* [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
@ 2018-10-29 22:22 ` Yong Zhi
2018-11-02 12:59 ` Mauro Carvalho Chehab
2018-11-29 19:16 ` Laurent Pinchart
2018-10-29 22:22 ` [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation Yong Zhi
` (16 subsequent siblings)
17 siblings, 2 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:22 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
Add IPU3-specific meta formats for parameter
processing and 3A, DVS statistics:
V4L2_META_FMT_IPU3_PARAMS
V4L2_META_FMT_IPU3_STAT_3A
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
include/uapi/linux/videodev2.h | 4 ++++
2 files changed, 6 insertions(+)
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 6489f25..abff64b 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1299,6 +1299,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
case V4L2_META_FMT_VSP1_HGO: descr = "R-Car VSP1 1-D Histogram"; break;
case V4L2_META_FMT_VSP1_HGT: descr = "R-Car VSP1 2-D Histogram"; break;
case V4L2_META_FMT_UVC: descr = "UVC payload header metadata"; break;
+ case V4L2_META_FMT_IPU3_PARAMS: descr = "IPU3 processing parameters"; break;
+ case V4L2_META_FMT_IPU3_STAT_3A: descr = "IPU3 3A statistics"; break;
default:
/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index f0a968a..bdccd7a 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -718,6 +718,10 @@ struct v4l2_pix_format {
#define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
#define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
+/* Vendor specific - used for IPU3 camera sub-system */
+#define V4L2_META_FMT_IPU3_PARAMS v4l2_fourcc('i', 'p', '3', 'p') /* IPU3 params */
+#define V4L2_META_FMT_IPU3_STAT_3A v4l2_fourcc('i', 'p', '3', 's') /* IPU3 3A statistics */
+
/* priv field value to indicates that subsequent fields are valid. */
#define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats
2018-10-29 22:22 ` [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats Yong Zhi
@ 2018-11-02 12:59 ` Mauro Carvalho Chehab
2018-11-02 13:05 ` Mauro Carvalho Chehab
2018-11-29 19:16 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Mauro Carvalho Chehab @ 2018-11-02 12:59 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Zhi-san,
Em Mon, 29 Oct 2018 15:22:55 -0700
Yong Zhi <yong.zhi@intel.com> escreveu:
> Add IPU3-specific meta formats for parameter
> processing and 3A, DVS statistics:
>
> V4L2_META_FMT_IPU3_PARAMS
> V4L2_META_FMT_IPU3_STAT_3A
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> ---
> drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
> include/uapi/linux/videodev2.h | 4 ++++
> 2 files changed, 6 insertions(+)
>
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 6489f25..abff64b 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1299,6 +1299,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> case V4L2_META_FMT_VSP1_HGO: descr = "R-Car VSP1 1-D Histogram"; break;
> case V4L2_META_FMT_VSP1_HGT: descr = "R-Car VSP1 2-D Histogram"; break;
> case V4L2_META_FMT_UVC: descr = "UVC payload header metadata"; break;
> + case V4L2_META_FMT_IPU3_PARAMS: descr = "IPU3 processing parameters"; break;
> + case V4L2_META_FMT_IPU3_STAT_3A: descr = "IPU3 3A statistics"; break;
>
> default:
> /* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index f0a968a..bdccd7a 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -718,6 +718,10 @@ struct v4l2_pix_format {
> #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
> #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
>
> +/* Vendor specific - used for IPU3 camera sub-system */
> +#define V4L2_META_FMT_IPU3_PARAMS v4l2_fourcc('i', 'p', '3', 'p') /* IPU3 params */
> +#define V4L2_META_FMT_IPU3_STAT_3A v4l2_fourcc('i', 'p', '3', 's') /* IPU3 3A statistics */
Where's the documentation for those two new formats? The best is to
always add the documentation bits for V4L2 uAPI stuff at the same
patch, as it makes easier for us to review.
> +
> /* priv field value to indicates that subsequent fields are valid. */
> #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
>
Thanks,
Mauro
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats
2018-11-02 12:59 ` Mauro Carvalho Chehab
@ 2018-11-02 13:05 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 123+ messages in thread
From: Mauro Carvalho Chehab @ 2018-11-02 13:05 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Em Fri, 2 Nov 2018 09:59:31 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> Hi Zhi-san,
>
> Em Mon, 29 Oct 2018 15:22:55 -0700
> Yong Zhi <yong.zhi@intel.com> escreveu:
>
> > Add IPU3-specific meta formats for parameter
> > processing and 3A, DVS statistics:
> >
> > V4L2_META_FMT_IPU3_PARAMS
> > V4L2_META_FMT_IPU3_STAT_3A
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > ---
> > drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
> > include/uapi/linux/videodev2.h | 4 ++++
> > 2 files changed, 6 insertions(+)
> >
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index 6489f25..abff64b 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1299,6 +1299,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > case V4L2_META_FMT_VSP1_HGO: descr = "R-Car VSP1 1-D Histogram"; break;
> > case V4L2_META_FMT_VSP1_HGT: descr = "R-Car VSP1 2-D Histogram"; break;
> > case V4L2_META_FMT_UVC: descr = "UVC payload header metadata"; break;
> > + case V4L2_META_FMT_IPU3_PARAMS: descr = "IPU3 processing parameters"; break;
> > + case V4L2_META_FMT_IPU3_STAT_3A: descr = "IPU3 3A statistics"; break;
> >
> > default:
> > /* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index f0a968a..bdccd7a 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -718,6 +718,10 @@ struct v4l2_pix_format {
> > #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
> > #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
> >
> > +/* Vendor specific - used for IPU3 camera sub-system */
> > +#define V4L2_META_FMT_IPU3_PARAMS v4l2_fourcc('i', 'p', '3', 'p') /* IPU3 params */
> > +#define V4L2_META_FMT_IPU3_STAT_3A v4l2_fourcc('i', 'p', '3', 's') /* IPU3 3A statistics */
>
> Where's the documentation for those two new formats? The best is to
> always add the documentation bits for V4L2 uAPI stuff at the same
> patch, as it makes easier for us to review.
Found it. It is at patch 3.
>
> > +
> > /* priv field value to indicates that subsequent fields are valid. */
> > #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
> >
>
>
>
> Thanks,
> Mauro
Thanks,
Mauro
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats
2018-10-29 22:22 ` [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats Yong Zhi
2018-11-02 12:59 ` Mauro Carvalho Chehab
@ 2018-11-29 19:16 ` Laurent Pinchart
2018-11-29 23:12 ` Zhi, Yong
1 sibling, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-11-29 19:16 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hello Yong,
Thank you for the patch.
On Tuesday, 30 October 2018 00:22:55 EET Yong Zhi wrote:
> Add IPU3-specific meta formats for parameter
> processing and 3A, DVS statistics:
Unless I'm mistaken DVS support has been removed. You can write this as
Add IPU3-specific meta formats for processing parameters and 3A
statistics.
>
> V4L2_META_FMT_IPU3_PARAMS
> V4L2_META_FMT_IPU3_STAT_3A
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> ---
> drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
> include/uapi/linux/videodev2.h | 4 ++++
> 2 files changed, 6 insertions(+)
I would squash this with patch 03/16.
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c
> b/drivers/media/v4l2-core/v4l2-ioctl.c index 6489f25..abff64b 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1299,6 +1299,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> case V4L2_META_FMT_VSP1_HGO: descr = "R-Car VSP1 1-D Histogram"; break;
> case V4L2_META_FMT_VSP1_HGT: descr = "R-Car VSP1 2-D Histogram"; break;
> case V4L2_META_FMT_UVC: descr = "UVC payload header metadata"; break;
> + case V4L2_META_FMT_IPU3_PARAMS: descr = "IPU3 processing parameters";
> break;
> + case V4L2_META_FMT_IPU3_STAT_3A: descr = "IPU3 3A statistics"; break;
>
> default:
> /* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index f0a968a..bdccd7a 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -718,6 +718,10 @@ struct v4l2_pix_format {
> #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC
> Payload Header metadata */ #define V4L2_META_FMT_D4XX
> v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
>
> +/* Vendor specific - used for IPU3 camera sub-system */
> +#define V4L2_META_FMT_IPU3_PARAMS v4l2_fourcc('i', 'p', '3', 'p') /* IPU3
> params */
Maybe "IPU3 processing parameters" in full ?
Apart from that the patch looks good to me.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> +#define V4L2_META_FMT_IPU3_STAT_3A v4l2_fourcc('i', 'p', '3', 's') /* IPU3
> 3A statistics */
> +
> /* priv field value to indicates that subsequent fields are valid. */
> #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats
2018-11-29 19:16 ` Laurent Pinchart
@ 2018-11-29 23:12 ` Zhi, Yong
0 siblings, 0 replies; 123+ messages in thread
From: Zhi, Yong @ 2018-11-29 23:12 UTC (permalink / raw)
To: Laurent Pinchart
Cc: linux-media, sakari.ailus, tfiga, mchehab, hans.verkuil, Mani,
Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu,
Tian Shu, Cao, Bingbu
Hi, Laurent,
Thanks for the review.
> -----Original Message-----
> From: Laurent Pinchart [mailto:laurent.pinchart@ideasonboard.com]
> Sent: Thursday, November 29, 2018 1:17 PM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> tfiga@chromium.org; mchehab@kernel.org; hans.verkuil@cisco.com; Mani,
> Rajmohan <rajmohan.mani@intel.com>; Zheng, Jian Xu
> <jian.xu.zheng@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>; Toivonen,
> Tuukka <tuukka.toivonen@intel.com>; Qiu, Tian Shu
> <tian.shu.qiu@intel.com>; Cao, Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats
>
> Hello Yong,
>
> Thank you for the patch.
>
> On Tuesday, 30 October 2018 00:22:55 EET Yong Zhi wrote:
> > Add IPU3-specific meta formats for parameter processing and 3A, DVS
> > statistics:
>
> Unless I'm mistaken DVS support has been removed. You can write this as
>
> Add IPU3-specific meta formats for processing parameters and 3A statistics.
>
Ack.
> >
> > V4L2_META_FMT_IPU3_PARAMS
> > V4L2_META_FMT_IPU3_STAT_3A
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > ---
> > drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
> > include/uapi/linux/videodev2.h | 4 ++++
> > 2 files changed, 6 insertions(+)
>
> I would squash this with patch 03/16.
>
OK, will squash patch 01 and 03 into single patch for v8.
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c
> > b/drivers/media/v4l2-core/v4l2-ioctl.c index 6489f25..abff64b 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1299,6 +1299,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc
> *fmt)
> > case V4L2_META_FMT_VSP1_HGO: descr = "R-Car VSP1 1-D Histogram";
> break;
> > case V4L2_META_FMT_VSP1_HGT: descr = "R-Car VSP1 2-D Histogram";
> break;
> > case V4L2_META_FMT_UVC: descr = "UVC payload header
> metadata"; break;
> > + case V4L2_META_FMT_IPU3_PARAMS: descr = "IPU3 processing
> parameters";
> > break;
> > + case V4L2_META_FMT_IPU3_STAT_3A: descr = "IPU3 3A statistics";
> break;
> >
> > default:
> > /* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h
> > b/include/uapi/linux/videodev2.h index f0a968a..bdccd7a 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -718,6 +718,10 @@ struct v4l2_pix_format {
> > #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC
> > Payload Header metadata */ #define V4L2_META_FMT_D4XX
> > v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
> >
> > +/* Vendor specific - used for IPU3 camera sub-system */
> > +#define V4L2_META_FMT_IPU3_PARAMS v4l2_fourcc('i', 'p', '3', 'p') /*
> IPU3
> > params */
>
> Maybe "IPU3 processing parameters" in full ?
>
Ack, thanks!!
> Apart from that the patch looks good to me.
>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> > +#define V4L2_META_FMT_IPU3_STAT_3A v4l2_fourcc('i', 'p', '3', 's') /*
> IPU3
> > 3A statistics */
> > +
> > /* priv field value to indicates that subsequent fields are valid. */
> > #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
>
>
> --
> Regards,
>
> Laurent Pinchart
>
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
2018-10-29 22:22 ` [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats Yong Zhi
@ 2018-10-29 22:22 ` Yong Zhi
2018-11-29 22:50 ` Laurent Pinchart
2018-10-29 22:22 ` [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI Yong Zhi
` (15 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:22 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao
From: Rajmohan Mani <rajmohan.mani@intel.com>
This patch adds the details about the IPU3 Imaging Unit driver.
Change-Id: I560cecf673df2dcc3ec72767cf8077708d649656
Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
---
Documentation/media/v4l-drivers/index.rst | 1 +
Documentation/media/v4l-drivers/ipu3.rst | 326 ++++++++++++++++++++++++++++++
2 files changed, 327 insertions(+)
create mode 100644 Documentation/media/v4l-drivers/ipu3.rst
diff --git a/Documentation/media/v4l-drivers/index.rst b/Documentation/media/v4l-drivers/index.rst
index 679238e..179a393 100644
--- a/Documentation/media/v4l-drivers/index.rst
+++ b/Documentation/media/v4l-drivers/index.rst
@@ -44,6 +44,7 @@ For more details see the file COPYING in the source distribution of Linux.
davinci-vpbe
fimc
imx
+ ipu3
ivtv
max2175
meye
diff --git a/Documentation/media/v4l-drivers/ipu3.rst b/Documentation/media/v4l-drivers/ipu3.rst
new file mode 100644
index 0000000..045bf42
--- /dev/null
+++ b/Documentation/media/v4l-drivers/ipu3.rst
@@ -0,0 +1,326 @@
+.. include:: <isonum.txt>
+
+===============================================================
+Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
+===============================================================
+
+Copyright |copy| 2018 Intel Corporation
+
+Introduction
+============
+
+This file documents Intel IPU3 (3rd generation Image Processing Unit) Imaging
+Unit driver located under drivers/media/pci/intel/ipu3.
+
+The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
+platforms (U/Y processor lines) is made up of two parts namely Imaging Unit
+(ImgU) and CIO2 device (MIPI CSI2 receiver).
+
+The CIO2 device receives the raw bayer data from the sensors and outputs the
+frames in a format that is specific to IPU3 (for consumption by IPU3 ImgU).
+CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2* and is
+enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
+
+The Imaging Unit (ImgU) is responsible for processing images captured
+through IPU3 CIO2 device. The ImgU driver sources can be found under
+drivers/media/pci/intel/ipu3 directory. The driver is enabled through the
+CONFIG_VIDEO_IPU3_IMGU config option.
+
+The two driver modules are named ipu3-csi2 and ipu3-imgu, respectively.
+
+The driver has been tested on Kaby Lake platforms (U/Y processor lines).
+
+The driver implements V4L2, Media controller and V4L2 sub-device interfaces.
+Camera sensors that have CSI-2 bus, which are connected to the IPU3 CIO2
+device are supported. Support for lens and flash drivers depends on the
+above sensors.
+
+ImgU device nodes
+=================
+
+The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
+subdev interface to the user space.
+
+Each V4L2 subdev represents a pipe, which can support a maximum of 2
+streams. A private ioctl can be used to configure the mode (video or still)
+of the pipe.
+
+This helps to support advanced camera features like Continuous View Finder
+(CVF) and Snapshot During Video(SDV).
+
+CIO2 device
+===========
+
+The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
+interface to the user space. There is a video node for each CSI-2 receiver,
+with a single media controller interface for the entire device.
+
+Media controller
+----------------
+
+The media device interface allows to configure the ImgU links, which defines
+the behavior of the IPU3 firmware.
+
+Device operation
+----------------
+
+With IPU3, once the input video node ("ipu3-imgu 0/1":0,
+in <entity>:<pad-number> format) is queued with buffer (in packed raw bayer
+format), IPU3 ISP starts processing the buffer and produces the video output
+in YUV format and statistics output on respective output nodes. The driver
+is expected to have buffers ready for all of parameter, output and
+statistics nodes, when input video node is queued with buffer.
+
+At a minimum, all of input, main output, 3A statistics and viewfinder
+video nodes should be enabled for IPU3 to start image processing.
+
+Each ImgU V4L2 subdev has the following set of video nodes.
+
+input, output and viewfinder video nodes
+----------------------------------------
+
+The frames (in packed raw bayer format specific to IPU3) received by the
+input video node is processed by the IPU3 Imaging Unit and is output to 2
+video nodes, with each targeting different purpose (main output and viewfinder
+output).
+
+Details on raw bayer format specific to IPU3 can be found as below.
+Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+
+The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
+
+Only the multi-planar API is supported. More details can be found at
+:ref:`planar-apis`.
+
+
+parameters video node
+---------------------
+
+The parameter video node receives the ISP algorithm parameters that are used
+to configure how the ISP algorithms process the image.
+
+Details on raw bayer format specific to IPU3 can be found as below.
+Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+
+3A statistics video node
+------------------------
+
+3A statistics video node is used by the ImgU driver to output the 3A (auto
+focus, auto exposure and auto white balance) statistics for the frames that
+are being processed by the ISP to user space applications. User space
+applications can use this statistics data to arrive at desired algorithm
+parameters for ISP.
+
+CIO2 device nodes
+=================
+
+CIO2 is represented as a single V4L2 sub-device with a video node for each
+CSI-2 receiver. The video node represents the DMA engine.
+
+Configuring the Intel IPU3
+==========================
+
+The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl calls,
+the ISP can be configured and enabled.
+
+The IPU3 ImgU pipelines can be configured using media controller APIs,
+defined at :ref:`media_controller`.
+
+Capturing frames in raw bayer format
+------------------------------------
+
+IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
+format) from the raw sensors connected to the CSI2 ports. The captured
+frames are used as input to the ImgU driver.
+
+Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
+raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique requirements
+and / or features specific to IPU3.
+
+-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in
+packed raw bayer format that is specific to IPU3
+
+-- Multiple video nodes have to be operated simultaneously
+
+Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
+2592x1944 image capture.
+
+Using the media contorller APIs, the ov5670 sensor is configured to send
+frames in packed raw bayer format to IPU3 CSI2 receiver.
+
+# This example assumes /dev/media0 as the ImgU media device
+
+export MDEV=/dev/media0
+
+# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
+
+export SDEV="ov5670 10-0036"
+
+# Establish the link for the media devices using media-ctl [#f3]_
+media-ctl -d $MDEV -l "ov5670 ":0 -> "ipu3-csi2 0":0[1]
+
+media-ctl -d $MDEV -l "ipu3-csi2 0":1 -> "ipu3-cio2 0":0[1]
+
+# Set the format for the media devices
+media-ctl -d $MDEV -V "ov5670 ":0 [fmt:SGRBG10/2592x1944]
+
+media-ctl -d $MDEV -V "ipu3-csi2 0":0 [fmt:SGRBG10/2592x1944]
+
+media-ctl -d $MDEV -V "ipu3-csi2 0":1 [fmt:SGRBG10/2592x1944]
+
+Once the media pipeline is configured, desired sensor specific settings
+(such as exposure and gain settings) can be set, using the yavta tool.
+
+e.g
+
+yavta -w 0x009e0903 444 $(media-ctl -d $MDEV -e "$SDEV")
+
+yavta -w 0x009e0913 1024 $(media-ctl -d $MDEV -e "$SDEV")
+
+yavta -w 0x009e0911 2046 $(media-ctl -d $MDEV -e "$SDEV")
+
+Once the desired sensor settings are set, frame captures can be done as below.
+
+e.g
+
+yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin
+-f IPU3_GRBG10 media-ctl -d $MDEV -e ipu3-cio2 0
+
+With the above command, 10 frames are captured at 2592x1944 resolution, with
+sGRBG10 format and output as IPU3_GRBG10 format.
+
+The captured frames are available as /tmp/frame-#.bin files.
+
+Processing the image in raw bayer format
+----------------------------------------
+
+Configuring ImgU V4L2 subdev for image processing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ImgU V4L2 subdevs have to be configured with media controller APIs to
+have all the video nodes setup correctly.
+
+Let us take "ipu3-imgu 0" subdev as an example.
+
+media-ctl -d $MDEV -r
+
+media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
+
+media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "output":0[1]
+
+media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "viewfinder":0[1]
+
+media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "3a stat":0[1]
+
+Also the pipe mode of the corresponding V4L2 subdev should be set as
+desired (e.g 0 for video mode or 1 for still mode) through the
+control id 0x009819a1 as below.
+
+e.g
+
+v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
+
+RAW bayer frames go through the following ISP pipeline HW blocks to
+have the processed image output to the DDR memory.
+
+RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
+Distortion Correction (GDC) -> DDR
+
+The ImgU V4L2 subdev has to be configured with the supported resolutions
+in all the above HW blocks, for a given input resolution.
+
+For a given supported resolution for an input frame, the Input Feeder,
+Bayer Down Scaling and GDC blocks should be configured with the supported
+resolutions. This information can be obtained by looking at the following
+IPU3 ISP configuration table.
+
+https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
+
+Under baseboard-poppy/media-libs/arc-camera3-hal-configs-poppy/files/gcss
+directory, graph_settings_ov5670.xml can be used as an example.
+
+The following steps prepare the ImgU ISP pipeline for the image processing.
+
+1. The ImgU V4L2 subdev data format should be set by using the
+VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
+
+2. The ImgU V4L2 subdev cropping should be set by using the
+VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
+using the input feeder height and width.
+
+3. The ImgU V4L2 subdev composing should be set by using the
+VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target,
+using the BDS height and width.
+
+For the ov5670 example, for an input frame with a resolution of 2592x1944
+(which is input to the ImgU subdev pad 0), the corresponding resolutions
+for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
+respectively.
+
+Once this is done, the received raw bayer frames can be input to the ImgU
+V4L2 subdev as below, using the open source application v4l2n.
+
+For an image captured with 2592x1944 [#f4]_ resolution, with desired output
+resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
+v4l2n command can be used. This helps process the raw bayer frames and
+produces the desired results for the main output image and the viewfinder
+output, in NV12 format.
+
+v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
+--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069
+--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 --output=/tmp/frames.out
+--open=/dev/video5
+--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
+--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 --output=/tmp/frames.vf
+--open=/dev/video6
+--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
+--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7
+--output=/tmp/frames.3A --fmt=type:META_CAPTURE,?
+--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
+
+where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
+input, output, viewfinder and 3A statistics video nodes respectively.
+
+Converting the raw bayer image into YUV domain
+----------------------------------------------
+
+The processed images after the above step, can be converted to YUV domain
+as below.
+
+Main output frames
+~~~~~~~~~~~~~~~~~~
+
+raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.pnm
+
+where 2560x1920 is output resolution, NV12 is the video format, followed
+by input frame and output PNM file.
+
+Viewfinder output frames
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.pnm
+
+where 2560x1920 is output resolution, NV12 is the video format, followed
+by input frame and output PNM file.
+
+Example user space code for IPU3
+================================
+
+User space code that configures and uses IPU3 is available here.
+
+https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
+
+The source can be located under hal/intel directory.
+
+References
+==========
+
+include/uapi/linux/intel-ipu3.h
+
+.. [#f1] https://github.com/intel/nvt
+
+.. [#f2] http://git.ideasonboard.org/yavta.git
+
+.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
+
+.. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation
2018-10-29 22:22 ` [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation Yong Zhi
@ 2018-11-29 22:50 ` Laurent Pinchart
2018-12-13 9:38 ` Sakari Ailus
2018-12-13 9:38 ` [PATCH 1/1] staging/ipu3-imgu: Address documentation comments Sakari Ailus
0 siblings, 2 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-11-29 22:50 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hello Yong,
Thank you for the patch.
On Tuesday, 30 October 2018 00:22:56 EET Yong Zhi wrote:
> From: Rajmohan Mani <rajmohan.mani@intel.com>
>
> This patch adds the details about the IPU3 Imaging Unit driver.
Strictly speaking this documents both the CIO2 and the IMGU. As they're
handled by two separate drivers, should they be split in two separate files ?
If you prefer keeping them together you should update the commit message
accordingly. I would in that case also split the documentation in a CIO2 and a
IMGU section in the file, instead of mixing them.
> Change-Id: I560cecf673df2dcc3ec72767cf8077708d649656
The Change-Id: tag isn't suitable for mainline, you can drop it.
> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> ---
> Documentation/media/v4l-drivers/index.rst | 1 +
> Documentation/media/v4l-drivers/ipu3.rst | 326 +++++++++++++++++++++++++++
> 2 files changed, 327 insertions(+)
> create mode 100644 Documentation/media/v4l-drivers/ipu3.rst
>
> diff --git a/Documentation/media/v4l-drivers/index.rst
> b/Documentation/media/v4l-drivers/index.rst index 679238e..179a393 100644
> --- a/Documentation/media/v4l-drivers/index.rst
> +++ b/Documentation/media/v4l-drivers/index.rst
> @@ -44,6 +44,7 @@ For more details see the file COPYING in the source
> distribution of Linux. davinci-vpbe
> fimc
> imx
> + ipu3
> ivtv
> max2175
> meye
> diff --git a/Documentation/media/v4l-drivers/ipu3.rst
> b/Documentation/media/v4l-drivers/ipu3.rst new file mode 100644
> index 0000000..045bf42
> --- /dev/null
> +++ b/Documentation/media/v4l-drivers/ipu3.rst
> @@ -0,0 +1,326 @@
> +.. include:: <isonum.txt>
> +
> +===============================================================
> +Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
> +===============================================================
> +
> +Copyright |copy| 2018 Intel Corporation
> +
> +Introduction
> +============
> +
> +This file documents Intel IPU3 (3rd generation Image Processing Unit)
> Imaging
s/documents Intel/documents the Intel/
> +Unit driver located under drivers/media/pci/intel/ipu3.
> +
> +The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
> +platforms (U/Y processor lines) is made up of two parts namely Imaging Unit
> +(ImgU) and CIO2 device (MIPI CSI2 receiver).
s/namely Imaging Unit/, namely the Imaging Unit/
s/and CIO2/and the CIO2/
> +
> +The CIO2 device receives the raw bayer data from the sensors and outputs
> the +frames in a format that is specific to IPU3 (for consumption by IPU3
> ImgU).
s/to IPU3/to the IPU3/
s/by IPU3/by the IPU3/
> +CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
> and is +enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
s/CIO2 driver/The CIO2 driver/
> +
> +The Imaging Unit (ImgU) is responsible for processing images captured
s/images/the images/
> +through IPU3 CIO2 device. The ImgU driver sources can be found under
s/IPU3 CIO2/the IPU3 CIO2/
> +drivers/media/pci/intel/ipu3 directory. The driver is enabled through the
> +CONFIG_VIDEO_IPU3_IMGU config option.
> +
> +The two driver modules are named ipu3-csi2 and ipu3-imgu, respectively.
> +
> +The driver has been tested on Kaby Lake platforms (U/Y processor lines).
I assume both drivers have been tested, so I would write
s/The driver has/The drivers have/
> +The driver implements V4L2, Media controller and V4L2 sub-device
> interfaces.
As this is true for both drivers,
s/The driver implements V4L2/Both drivers implement the V4L2/
s/Media controller/Media Controller/
> +Camera sensors that have CSI-2 bus, which are connected to the IPU3 CIO2
> +device are supported.
I would rephrase this slightly, as "The IPU3 CIO2 driver supports camera
sensors connected to the CIO2 MIPI CSI-2 interfaces through V4L2 sub-device
sensor drivers."
> Support for lens and flash drivers depends on the
> +above sensors.
That's a very good introduction !
I would follow with two sections, "IPU3 CIO2" followed by "IPU3 ImgU".
> +ImgU device nodes
> +=================
> +
> +The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
> +subdev interface to the user space.
Not just subdevs, but video nodes too. I would rephrase as follows (please
note that this might not be valid .rst content, I haven't tried compiling it,
it might need to be reworked slightly) :
"The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
exposed to userspace as a V4L2 sub-device node.
Each pipe has two input pads and three output pads for the following purpose:
Pad 0 (input): Input raw video stream
Pad 1 (input): Processing parameters
Pad 2 (output): Output processed video stream
Pad 3 (output): Output viewfinder video stream
Pad 4 (output): 3A statistics
Each pad is connected to a corresponding V4L2 video interface, exposed to
userspace as a V4L2 video device node."
> +Each V4L2 subdev represents a pipe, which can support a maximum of 2
> +streams. A private ioctl can be used to configure the mode (video or still)
> +of the pipe.
> +
> +This helps to support advanced camera features like Continuous View Finder
> +(CVF) and Snapshot During Video(SDV).
As far as I know there's no private ioctl anymore, at least I can't find it in
the source code. Could you thus rephrase that sentence and the next one to
explain the current method to implement CVF and SDV ?
> +CIO2 device
> +===========
> +
> +The CIO2 is represented as a single V4L2 subdev, which provides a V4L2
> subdev +interface to the user space. There is a video node for each CSI-2
> receiver, +with a single media controller interface for the entire device.
Similarly, I think we should explain here that there are four channels :
"The CIO2 contains four independent capture channel, each with its own MIPI
CSI-2 receiver and DMA engine. Each channel is modelled as a V4L2 sub-device
exposed to userspace as a V4L2 sub-device node and has two pads:
Pad 0 (input): MIPI CSI-2 input, connected to the sensor subdev
Pad 1 (output): Raw video capture, connected to the V4L2 video interface
The V4L2 video interfaces model the DMA engines. They are exposed to userspace
as V4L2 video device nodes."
The section should be moved above the IPU3 ImgU section.
> +Media controller
> +----------------
> +
> +The media device interface allows to configure the ImgU links, which
> defines +the behavior of the IPU3 firmware.
s/defines/define/ or possibly better s/defines/control/
> +
> +Device operation
> +----------------
> +
> +With IPU3, once the input video node ("ipu3-imgu 0/1":0,
> +in <entity>:<pad-number> format) is queued with buffer (in packed raw bayer
s/bayer/Bayer/
> +format), IPU3 ISP starts processing the buffer and produces the video
s/IPU3 ISP/the IPU3 ISP/
This is the first time you mention an ISP. Should the term ISP be replaced by
ImgU here and below ? I'm fine keeping it, but it should then be defined in
the introduction, in particular with an explanation of the difference between
ImgU and ISP.
> output +in YUV format and statistics output on respective output nodes. The
> driver +is expected to have buffers ready for all of parameter, output and
> +statistics nodes, when input video node is queued with buffer.
Why is that, shouldn't the driver wait for all necessary buffers to be ready
before processing ?
> +At a minimum, all of input, main output, 3A statistics and viewfinder
> +video nodes should be enabled for IPU3 to start image processing.
If they all need to be enabled, shouldn't the respective links be ENABLED and
IMMUTABLE ?
> +Each ImgU V4L2 subdev has the following set of video nodes.
> +
> +input, output and viewfinder video nodes
> +----------------------------------------
> +
> +The frames (in packed raw bayer format specific to IPU3) received by the
s/bayer/Bayer/
(same in a few other locations below)
> +input video node is processed by the IPU3 Imaging Unit and is output to 2
s/is output/are output/
> +video nodes, with each targeting different purpose (main output and
s/different purpose/a different purpose/
> viewfinder +output).
We will eventually need more information about this, but that could come as a
separate patch.
> +Details on raw bayer format specific to IPU3 can be found as below.
> +Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
How about linking to the document using :ref: instead of adding the file path
?
> +The driver supports V4L2 Video Capture Interface as defined at
> :ref:`devices`.
> +
> +Only the multi-planar API is supported. More details can be found at
> +:ref:`planar-apis`.
> +
> +
> +parameters video node
> +---------------------
> +
> +The parameter video node receives the ISP algorithm parameters that are
s/parameters/parameter/
> used +to configure how the ISP algorithms process the image.
> +
> +Details on raw bayer format specific to IPU3 can be found as below.
I assume you meant processing parameters, not raw Bayer format.
> +Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
:ref: here too.
> +3A statistics video node
> +------------------------
> +
> +3A statistics video node is used by the ImgU driver to output the 3A (auto
> +focus, auto exposure and auto white balance) statistics for the frames that
> +are being processed by the ISP to user space applications. User space
> +applications can use this statistics data to arrive at desired algorithm
> +parameters for ISP.
s/arrive at/compute the/
s/ISP/the ISP/
> +
> +CIO2 device nodes
> +=================
> +
> +CIO2 is represented as a single V4L2 sub-device with a video node for each
> +CSI-2 receiver. The video node represents the DMA engine.
I think you can remove this section, it's already explained above. If you want
to keep it, please move it with the rest of the CIO2 documentation above the
ImgU documentation.
> +Configuring the Intel IPU3
> +==========================
> +
> +The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl calls,
> +the ISP can be configured and enabled.
> +
> +The IPU3 ImgU pipelines can be configured using media controller APIs,
> +defined at :ref:`media_controller`.
> +
> +Capturing frames in raw bayer format
> +------------------------------------
> +
> +IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
> +format) from the raw sensors connected to the CSI2 ports. The captured
> +frames are used as input to the ImgU driver.
> +
> +Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
I would drop v4l2n from the documentation as it's not maintained and is not
functional (in particular it doesn't implement MPLANE support which the driver
requires).
> +raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique requirements
> +and / or features specific to IPU3.
> +
> +-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in
> +packed raw bayer format that is specific to IPU3
s/to IPU3/to the IPU3/.
> +
> +-- Multiple video nodes have to be operated simultaneously
> +
> +Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
> +2592x1944 image capture.
> +
> +Using the media contorller APIs, the ov5670 sensor is configured to send
> +frames in packed raw bayer format to IPU3 CSI2 receiver.
> +
> +# This example assumes /dev/media0 as the ImgU media device
Shouldn't that be CIO2 ?
> +
> +export MDEV=/dev/media0
> +
> +# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
> +
> +export SDEV="ov5670 10-0036"
> +
> +# Establish the link for the media devices using media-ctl [#f3]_
> +media-ctl -d $MDEV -l "ov5670 ":0 -> "ipu3-csi2 0":0[1]
You're missing quotes around the link description, and the entity name is
incorrect. Please test all the commands listed here with the upstream driver.
> +media-ctl -d $MDEV -l "ipu3-csi2 0":1 -> "ipu3-cio2 0":0[1]
Shouldn't this link be immutable ?
> +# Set the format for the media devices
> +media-ctl -d $MDEV -V "ov5670 ":0 [fmt:SGRBG10/2592x1944]
The entity name is incorrect.
> +media-ctl -d $MDEV -V "ipu3-csi2 0":0 [fmt:SGRBG10/2592x1944]
> +
> +media-ctl -d $MDEV -V "ipu3-csi2 0":1 [fmt:SGRBG10/2592x1944]
> +
> +Once the media pipeline is configured, desired sensor specific settings
> +(such as exposure and gain settings) can be set, using the yavta tool.
> +
> +e.g
> +
> +yavta -w 0x009e0903 444 $(media-ctl -d $MDEV -e "$SDEV")
> +
> +yavta -w 0x009e0913 1024 $(media-ctl -d $MDEV -e "$SDEV")
> +
> +yavta -w 0x009e0911 2046 $(media-ctl -d $MDEV -e "$SDEV")
> +
> +Once the desired sensor settings are set, frame captures can be done as
> below.
> +
> +e.g
> +
> +yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin
Do you need --data-prefix ?
> +-f IPU3_GRBG10 media-ctl -d $MDEV -e ipu3-cio2 0
The upstream yavta names the format IPU3_SGRBG10.
You're missing $(...) around the media-ctl command, and double quotes around
the entity name.
> +
> +With the above command, 10 frames are captured at 2592x1944 resolution,
> with +sGRBG10 format and output as IPU3_GRBG10 format.
IPU3_SGRBG10 here too.
> +
> +The captured frames are available as /tmp/frame-#.bin files.
All this should be moved to the CIO2 section.
> +Processing the image in raw bayer format
> +----------------------------------------
> +
> +Configuring ImgU V4L2 subdev for image processing
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The ImgU V4L2 subdevs have to be configured with media controller APIs to
> +have all the video nodes setup correctly.
> +
> +Let us take "ipu3-imgu 0" subdev as an example.
> +
> +media-ctl -d $MDEV -r
> +
> +media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
> +
> +media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "output":0[1]
> +
> +media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "viewfinder":0[1]
> +
> +media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "3a stat":0[1]
Entity names are incorrect here too.
> +Also the pipe mode of the corresponding V4L2 subdev should be set as
> +desired (e.g 0 for video mode or 1 for still mode) through the
> +control id 0x009819a1 as below.
> +
> +e.g
> +
> +v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
You can use yavta instead of v4l2n. Another option would be v4l2-ctl, but I'd
avoid adding a dependency to another tool if yavta is already used in other
places.
> +RAW bayer frames go through the following ISP pipeline HW blocks to
> +have the processed image output to the DDR memory.
> +
> +RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
> +Distortion Correction (GDC) -> DDR
A more detailed block diagram, with the other blocks included, should be added
to the ImgU description above. Each block should have a short description of
its purpose.
> +The ImgU V4L2 subdev has to be configured with the supported resolutions
> +in all the above HW blocks, for a given input resolution.
> +
> +For a given supported resolution for an input frame, the Input Feeder,
> +Bayer Down Scaling and GDC blocks should be configured with the supported
> +resolutions. This information can be obtained by looking at the following
> +IPU3 ISP configuration table.
Does this mean that the ImgU will not operate properly when exercised through
the MC and V4L2 only without configuration of the internal blocks through the
processing parameters device node ?
> +https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/mast
> er
> +
> +Under baseboard-poppy/media-libs/arc-camera3-hal-configs-poppy/files/gcss
> +directory, graph_settings_ov5670.xml can be used as an example.
The directory name is incorrect.
> +The following steps prepare the ImgU ISP pipeline for the image processing.
> +
> +1. The ImgU V4L2 subdev data format should be set by using the
> +VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> above.
> +
> +2. The ImgU V4L2 subdev cropping should be set by using the
> +VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
> +using the input feeder height and width.
> +
> +3. The ImgU V4L2 subdev composing should be set by using the
> +VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> target, +using the BDS height and width.
How the format and selection rectangles related to the internal processing
blocks should also be explained in more details.
> +For the ov5670 example, for an input frame with a resolution of 2592x1944
> +(which is input to the ImgU subdev pad 0), the corresponding resolutions
> +for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
> +respectively.
Why ? How is that computed ? If my input resolution was different, how would I
compute the other resolutions ?
> +Once this is done, the received raw bayer frames can be input to the ImgU
> +V4L2 subdev as below, using the open source application v4l2n.
> +
> +For an image captured with 2592x1944 [#f4]_ resolution, with desired output
> +resolution as 2560x1920 and viewfinder resolution as 2560x1920, the
> following +v4l2n command can be used. This helps process the raw bayer
> frames and +produces the desired results for the main output image and the
> viewfinder +output, in NV12 format.
> +
> +v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
> +--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X4733706
> 9 +--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1
> --output=/tmp/frames.out +--open=/dev/video5
> +--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
> +--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2
> --output=/tmp/frames.vf +--open=/dev/video6
> +--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
> +--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7
> +--output=/tmp/frames.3A --fmt=type:META_CAPTURE,?
> +--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
You can replace this with four yavta commands.
> +where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point
> to +input, output, viewfinder and 3A statistics video nodes respectively. +
> +Converting the raw bayer image into YUV domain
> +----------------------------------------------
> +
> +The processed images after the above step, can be converted to YUV domain
> +as below.
> +
> +Main output frames
> +~~~~~~~~~~~~~~~~~~
> +
> +raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.pnm
PNM is an umbrella term that refers to any of the PBM, PGM or PPM format. As
the raw2pnm tool outputs PPM files, let's name them .ppm here and below. This
helps with some image viewers that use file extensions to identify the format,
and have trouble handling .pnm files.
> +where 2560x1920 is output resolution, NV12 is the video format, followed
> +by input frame and output PNM file.
> +
> +Viewfinder output frames
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.pnm
> +
> +where 2560x1920 is output resolution, NV12 is the video format, followed
> +by input frame and output PNM file.
> +
> +Example user space code for IPU3
> +================================
> +
> +User space code that configures and uses IPU3 is available here.
> +
> +https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
> +
> +The source can be located under hal/intel directory.
> +
> +References
> +==========
> +
> +include/uapi/linux/intel-ipu3.h
> +
> +.. [#f1] https://github.com/intel/nvt
> +
> +.. [#f2] http://git.ideasonboard.org/yavta.git
> +
> +.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
> +
> +.. [#f4] ImgU limitation requires an additional 16x16 for all input
> resolutions
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation
2018-11-29 22:50 ` Laurent Pinchart
@ 2018-12-13 9:38 ` Sakari Ailus
2018-12-13 10:41 ` Laurent Pinchart
2018-12-13 9:38 ` [PATCH 1/1] staging/ipu3-imgu: Address documentation comments Sakari Ailus
1 sibling, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-12-13 9:38 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Laurent,
I'm sending a separate patch to address the comments.
On Fri, Nov 30, 2018 at 12:50:36AM +0200, Laurent Pinchart wrote:
> Hello Yong,
>
> Thank you for the patch.
>
> On Tuesday, 30 October 2018 00:22:56 EET Yong Zhi wrote:
> > From: Rajmohan Mani <rajmohan.mani@intel.com>
> >
> > This patch adds the details about the IPU3 Imaging Unit driver.
>
> Strictly speaking this documents both the CIO2 and the IMGU. As they're
> handled by two separate drivers, should they be split in two separate files ?
> If you prefer keeping them together you should update the commit message
> accordingly. I would in that case also split the documentation in a CIO2 and a
> IMGU section in the file, instead of mixing them.
I'm keeping it in a single document for now. In practice these devices
always come together as neither is really usable without the other.
>
> > Change-Id: I560cecf673df2dcc3ec72767cf8077708d649656
>
> The Change-Id: tag isn't suitable for mainline, you can drop it.
Fixed.
>
> > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> > ---
> > Documentation/media/v4l-drivers/index.rst | 1 +
> > Documentation/media/v4l-drivers/ipu3.rst | 326 +++++++++++++++++++++++++++
> > 2 files changed, 327 insertions(+)
> > create mode 100644 Documentation/media/v4l-drivers/ipu3.rst
> >
> > diff --git a/Documentation/media/v4l-drivers/index.rst
> > b/Documentation/media/v4l-drivers/index.rst index 679238e..179a393 100644
> > --- a/Documentation/media/v4l-drivers/index.rst
> > +++ b/Documentation/media/v4l-drivers/index.rst
> > @@ -44,6 +44,7 @@ For more details see the file COPYING in the source
> > distribution of Linux. davinci-vpbe
> > fimc
> > imx
> > + ipu3
> > ivtv
> > max2175
> > meye
> > diff --git a/Documentation/media/v4l-drivers/ipu3.rst
> > b/Documentation/media/v4l-drivers/ipu3.rst new file mode 100644
> > index 0000000..045bf42
> > --- /dev/null
> > +++ b/Documentation/media/v4l-drivers/ipu3.rst
> > @@ -0,0 +1,326 @@
> > +.. include:: <isonum.txt>
> > +
> > +===============================================================
> > +Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
> > +===============================================================
> > +
> > +Copyright |copy| 2018 Intel Corporation
> > +
> > +Introduction
> > +============
> > +
> > +This file documents Intel IPU3 (3rd generation Image Processing Unit)
> > Imaging
>
> s/documents Intel/documents the Intel/
Fixed.
>
> > +Unit driver located under drivers/media/pci/intel/ipu3.
Added the staging tree directory, too.
> > +
> > +The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
> > +platforms (U/Y processor lines) is made up of two parts namely Imaging Unit
> > +(ImgU) and CIO2 device (MIPI CSI2 receiver).
>
> s/namely Imaging Unit/, namely the Imaging Unit/
> s/and CIO2/and the CIO2/
Fixed.
>
> > +
> > +The CIO2 device receives the raw bayer data from the sensors and outputs
> > the +frames in a format that is specific to IPU3 (for consumption by IPU3
> > ImgU).
>
> s/to IPU3/to the IPU3/
> s/by IPU3/by the IPU3/
Fixed.
>
> > +CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
> > and is +enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
>
> s/CIO2 driver/The CIO2 driver/
Fixed.
>
> > +
> > +The Imaging Unit (ImgU) is responsible for processing images captured
>
> s/images/the images/
As these are not any particular images, I'd leave out "the".
>
> > +through IPU3 CIO2 device. The ImgU driver sources can be found under
>
> s/IPU3 CIO2/the IPU3 CIO2/
Yes, and through -> by.
>
> > +drivers/media/pci/intel/ipu3 directory. The driver is enabled through the
> > +CONFIG_VIDEO_IPU3_IMGU config option.
> > +
> > +The two driver modules are named ipu3-csi2 and ipu3-imgu, respectively.
s/-/_/g
> > +
> > +The driver has been tested on Kaby Lake platforms (U/Y processor lines).
>
> I assume both drivers have been tested, so I would write
>
> s/The driver has/The drivers have/
Yes.
>
> > +The driver implements V4L2, Media controller and V4L2 sub-device
> > interfaces.
>
> As this is true for both drivers,
>
> s/The driver implements V4L2/Both drivers implement the V4L2/
> s/Media controller/Media Controller/
Yes.
>
> > +Camera sensors that have CSI-2 bus, which are connected to the IPU3 CIO2
> > +device are supported.
>
> I would rephrase this slightly, as "The IPU3 CIO2 driver supports camera
> sensors connected to the CIO2 MIPI CSI-2 interfaces through V4L2 sub-device
> sensor drivers."
Yes.
>
> > Support for lens and flash drivers depends on the
> > +above sensors.
>
> That's a very good introduction !
I dropped the sentence as it's not relevant for IPU3.
>
> I would follow with two sections, "IPU3 CIO2" followed by "IPU3 ImgU".
Ack.
>
> > +ImgU device nodes
> > +=================
> > +
> > +The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
> > +subdev interface to the user space.
>
> Not just subdevs, but video nodes too. I would rephrase as follows (please
> note that this might not be valid .rst content, I haven't tried compiling it,
> it might need to be reworked slightly) :
>
> "The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
> exposed to userspace as a V4L2 sub-device node.
>
> Each pipe has two input pads and three output pads for the following purpose:
>
> Pad 0 (input): Input raw video stream
> Pad 1 (input): Processing parameters
> Pad 2 (output): Output processed video stream
> Pad 3 (output): Output viewfinder video stream
> Pad 4 (output): 3A statistics
>
> Each pad is connected to a corresponding V4L2 video interface, exposed to
> userspace as a V4L2 video device node."
Added that, with s/input/sink/; s/output/source/.
>
> > +Each V4L2 subdev represents a pipe, which can support a maximum of 2
> > +streams. A private ioctl can be used to configure the mode (video or still)
> > +of the pipe.
> > +
> > +This helps to support advanced camera features like Continuous View Finder
> > +(CVF) and Snapshot During Video(SDV).
>
> As far as I know there's no private ioctl anymore, at least I can't find it in
> the source code. Could you thus rephrase that sentence and the next one to
> explain the current method to implement CVF and SDV ?
I removed the sentence; the parameter video node is explained below.
>
> > +CIO2 device
> > +===========
> > +
> > +The CIO2 is represented as a single V4L2 subdev, which provides a V4L2
> > subdev +interface to the user space. There is a video node for each CSI-2
> > receiver, +with a single media controller interface for the entire device.
>
> Similarly, I think we should explain here that there are four channels :
>
> "The CIO2 contains four independent capture channel, each with its own MIPI
> CSI-2 receiver and DMA engine. Each channel is modelled as a V4L2 sub-device
> exposed to userspace as a V4L2 sub-device node and has two pads:
>
> Pad 0 (input): MIPI CSI-2 input, connected to the sensor subdev
> Pad 1 (output): Raw video capture, connected to the V4L2 video interface
>
> The V4L2 video interfaces model the DMA engines. They are exposed to userspace
> as V4L2 video device nodes."
>
> The section should be moved above the IPU3 ImgU section.
Fixed. I added a table for the pads, too.
>
> > +Media controller
> > +----------------
> > +
> > +The media device interface allows to configure the ImgU links, which
> > defines +the behavior of the IPU3 firmware.
>
> s/defines/define/ or possibly better s/defines/control/
I'm removing the section as the binary is selected using a control in the
current version. I'm adding this instead:
Firmware binary selection
-------------------------
The firmware binary is selected using the V4L2_CID_INTEL_IPU3_MODE, currently
defined in drivers/staging/media/ipu3/include/intel-ipu3.h . "VIDEO" and "STILL"
modes are available.
>
> > +
> > +Device operation
> > +----------------
> > +
> > +With IPU3, once the input video node ("ipu3-imgu 0/1":0,
> > +in <entity>:<pad-number> format) is queued with buffer (in packed raw bayer
>
> s/bayer/Bayer/
>
Fixed in the entire file.
> > +format), IPU3 ISP starts processing the buffer and produces the video
>
> s/IPU3 ISP/the IPU3 ISP/
>
> This is the first time you mention an ISP. Should the term ISP be replaced by
> ImgU here and below ? I'm fine keeping it, but it should then be defined in
> the introduction, in particular with an explanation of the difference between
> ImgU and ISP.
I replaced IPU3 with ImgU in this section.
>
> > output +in YUV format and statistics output on respective output nodes. The
> > driver +is expected to have buffers ready for all of parameter, output and
> > +statistics nodes, when input video node is queued with buffer.
>
> Why is that, shouldn't the driver wait for all necessary buffers to be ready
> before processing ?
Not all are mandatory.
>
> > +At a minimum, all of input, main output, 3A statistics and viewfinder
> > +video nodes should be enabled for IPU3 to start image processing.
>
> If they all need to be enabled, shouldn't the respective links be ENABLED and
> IMMUTABLE ?
Yes.
>
> > +Each ImgU V4L2 subdev has the following set of video nodes.
> > +
> > +input, output and viewfinder video nodes
> > +----------------------------------------
> > +
> > +The frames (in packed raw bayer format specific to IPU3) received by the
>
> s/bayer/Bayer/
>
> (same in a few other locations below)
>
> > +input video node is processed by the IPU3 Imaging Unit and is output to 2
>
> s/is output/are output/
Fixed.
>
> > +video nodes, with each targeting different purpose (main output and
>
> s/different purpose/a different purpose/
ditto.
>
> > viewfinder +output).
>
> We will eventually need more information about this, but that could come as a
> separate patch.
>
> > +Details on raw bayer format specific to IPU3 can be found as below.
> > +Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
>
> How about linking to the document using :ref: instead of adding the file path
> ?
Fixed.
>
> > +The driver supports V4L2 Video Capture Interface as defined at
> > :ref:`devices`.
> > +
> > +Only the multi-planar API is supported. More details can be found at
> > +:ref:`planar-apis`.
> > +
> > +
> > +parameters video node
> > +---------------------
> > +
> > +The parameter video node receives the ISP algorithm parameters that are
>
> s/parameters/parameter/
Fixed.
>
> > used +to configure how the ISP algorithms process the image.
> > +
> > +Details on raw bayer format specific to IPU3 can be found as below.
>
> I assume you meant processing parameters, not raw Bayer format.
Fixed.
>
> > +Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
>
> :ref: here too.
Yes.
>
> > +3A statistics video node
> > +------------------------
> > +
> > +3A statistics video node is used by the ImgU driver to output the 3A (auto
> > +focus, auto exposure and auto white balance) statistics for the frames that
> > +are being processed by the ISP to user space applications. User space
> > +applications can use this statistics data to arrive at desired algorithm
> > +parameters for ISP.
>
> s/arrive at/compute the/
> s/ISP/the ISP/
Fixed; ISP -> ImgU.
>
> > +
> > +CIO2 device nodes
> > +=================
> > +
> > +CIO2 is represented as a single V4L2 sub-device with a video node for each
> > +CSI-2 receiver. The video node represents the DMA engine.
>
> I think you can remove this section, it's already explained above. If you want
> to keep it, please move it with the rest of the CIO2 documentation above the
> ImgU documentation.
Fixed.
>
> > +Configuring the Intel IPU3
> > +==========================
> > +
> > +The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl calls,
> > +the ISP can be configured and enabled.
> > +
> > +The IPU3 ImgU pipelines can be configured using media controller APIs,
> > +defined at :ref:`media_controller`.
> > +
> > +Capturing frames in raw bayer format
> > +------------------------------------
> > +
> > +IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
> > +format) from the raw sensors connected to the CSI2 ports. The captured
> > +frames are used as input to the ImgU driver.
> > +
> > +Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
>
> I would drop v4l2n from the documentation as it's not maintained and is not
> functional (in particular it doesn't implement MPLANE support which the driver
> requires).
Ack. I'm leaving the command plus the reference to v4l2n, this might
contain information useful still.
>
> > +raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique requirements
> > +and / or features specific to IPU3.
> > +
> > +-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in
> > +packed raw bayer format that is specific to IPU3
>
> s/to IPU3/to the IPU3/.
Fixed.
>
> > +
> > +-- Multiple video nodes have to be operated simultaneously
> > +
> > +Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
> > +2592x1944 image capture.
> > +
> > +Using the media contorller APIs, the ov5670 sensor is configured to send
> > +frames in packed raw bayer format to IPU3 CSI2 receiver.
> > +
> > +# This example assumes /dev/media0 as the ImgU media device
>
> Shouldn't that be CIO2 ?
Fixed.
>
> > +
> > +export MDEV=/dev/media0
> > +
> > +# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
> > +
> > +export SDEV="ov5670 10-0036"
> > +
> > +# Establish the link for the media devices using media-ctl [#f3]_
> > +media-ctl -d $MDEV -l "ov5670 ":0 -> "ipu3-csi2 0":0[1]
>
> You're missing quotes around the link description, and the entity name is
> incorrect. Please test all the commands listed here with the upstream driver.
>
> > +media-ctl -d $MDEV -l "ipu3-csi2 0":1 -> "ipu3-cio2 0":0[1]
>
> Shouldn't this link be immutable ?
To be changed in the driver I presume. I'm removing this line for now.
>
> > +# Set the format for the media devices
> > +media-ctl -d $MDEV -V "ov5670 ":0 [fmt:SGRBG10/2592x1944]
>
> The entity name is incorrect.
I fixed the quotes below, too...
>
> > +media-ctl -d $MDEV -V "ipu3-csi2 0":0 [fmt:SGRBG10/2592x1944]
> > +
> > +media-ctl -d $MDEV -V "ipu3-csi2 0":1 [fmt:SGRBG10/2592x1944]
> > +
> > +Once the media pipeline is configured, desired sensor specific settings
> > +(such as exposure and gain settings) can be set, using the yavta tool.
> > +
> > +e.g
> > +
> > +yavta -w 0x009e0903 444 $(media-ctl -d $MDEV -e "$SDEV")
> > +
> > +yavta -w 0x009e0913 1024 $(media-ctl -d $MDEV -e "$SDEV")
> > +
> > +yavta -w 0x009e0911 2046 $(media-ctl -d $MDEV -e "$SDEV")
I simplified this a little...
> > +
> > +Once the desired sensor settings are set, frame captures can be done as
> > below.
> > +
> > +e.g
> > +
> > +yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin
>
> Do you need --data-prefix ?
>
> > +-f IPU3_GRBG10 media-ctl -d $MDEV -e ipu3-cio2 0
>
> The upstream yavta names the format IPU3_SGRBG10.
Fixed.
>
> You're missing $(...) around the media-ctl command, and double quotes around
> the entity name.
That, too.
>
> > +
> > +With the above command, 10 frames are captured at 2592x1944 resolution,
> > with +sGRBG10 format and output as IPU3_GRBG10 format.
>
> IPU3_SGRBG10 here too.
Yes.
>
> > +
> > +The captured frames are available as /tmp/frame-#.bin files.
>
> All this should be moved to the CIO2 section.
Done.
>
> > +Processing the image in raw bayer format
> > +----------------------------------------
> > +
> > +Configuring ImgU V4L2 subdev for image processing
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +The ImgU V4L2 subdevs have to be configured with media controller APIs to
> > +have all the video nodes setup correctly.
> > +
> > +Let us take "ipu3-imgu 0" subdev as an example.
> > +
> > +media-ctl -d $MDEV -r
> > +
> > +media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
> > +
> > +media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "output":0[1]
> > +
> > +media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "viewfinder":0[1]
> > +
> > +media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "3a stat":0[1]
>
> Entity names are incorrect here too.
Fixed.
>
> > +Also the pipe mode of the corresponding V4L2 subdev should be set as
> > +desired (e.g 0 for video mode or 1 for still mode) through the
> > +control id 0x009819a1 as below.
> > +
> > +e.g
> > +
> > +v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
>
> You can use yavta instead of v4l2n. Another option would be v4l2-ctl, but I'd
> avoid adding a dependency to another tool if yavta is already used in other
> places.
Switched to yavta.
>
> > +RAW bayer frames go through the following ISP pipeline HW blocks to
> > +have the processed image output to the DDR memory.
> > +
> > +RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
> > +Distortion Correction (GDC) -> DDR
>
> A more detailed block diagram, with the other blocks included, should be added
> to the ImgU description above. Each block should have a short description of
> its purpose.
I think we have one in conjunction with the parameter format description.
>
> > +The ImgU V4L2 subdev has to be configured with the supported resolutions
> > +in all the above HW blocks, for a given input resolution.
> > +
> > +For a given supported resolution for an input frame, the Input Feeder,
> > +Bayer Down Scaling and GDC blocks should be configured with the supported
> > +resolutions. This information can be obtained by looking at the following
> > +IPU3 ISP configuration table.
>
> Does this mean that the ImgU will not operate properly when exercised through
> the MC and V4L2 only without configuration of the internal blocks through the
> processing parameters device node ?
I need to let Raj and Yong to answer that. My understanding is the default
parameters should be usable. If they're not, that needs to be addressed.
>
> > +https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/mast
> > er
> > +
> > +Under baseboard-poppy/media-libs/arc-camera3-hal-configs-poppy/files/gcss
> > +directory, graph_settings_ov5670.xml can be used as an example.
>
> The directory name is incorrect.
Fixed.
>
> > +The following steps prepare the ImgU ISP pipeline for the image processing.
> > +
> > +1. The ImgU V4L2 subdev data format should be set by using the
> > +VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > above.
> > +
> > +2. The ImgU V4L2 subdev cropping should be set by using the
> > +VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
> > +using the input feeder height and width.
> > +
> > +3. The ImgU V4L2 subdev composing should be set by using the
> > +VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > target, +using the BDS height and width.
>
> How the format and selection rectangles related to the internal processing
> blocks should also be explained in more details.
To be added later.
>
> > +For the ov5670 example, for an input frame with a resolution of 2592x1944
> > +(which is input to the ImgU subdev pad 0), the corresponding resolutions
> > +for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
> > +respectively.
>
> Why ? How is that computed ? If my input resolution was different, how would I
> compute the other resolutions ?
Ditto.
>
> > +Once this is done, the received raw bayer frames can be input to the ImgU
> > +V4L2 subdev as below, using the open source application v4l2n.
> > +
> > +For an image captured with 2592x1944 [#f4]_ resolution, with desired output
> > +resolution as 2560x1920 and viewfinder resolution as 2560x1920, the
> > following +v4l2n command can be used. This helps process the raw bayer
> > frames and +produces the desired results for the main output image and the
> > viewfinder +output, in NV12 format.
> > +
> > +v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
> > +--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X4733706
> > 9 +--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1
> > --output=/tmp/frames.out +--open=/dev/video5
> > +--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
> > +--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2
> > --output=/tmp/frames.vf +--open=/dev/video6
> > +--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
> > +--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7
> > +--output=/tmp/frames.3A --fmt=type:META_CAPTURE,?
> > +--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
>
> You can replace this with four yavta commands.
Ditto.
>
> > +where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point
> > to +input, output, viewfinder and 3A statistics video nodes respectively. +
> > +Converting the raw bayer image into YUV domain
> > +----------------------------------------------
> > +
> > +The processed images after the above step, can be converted to YUV domain
> > +as below.
> > +
> > +Main output frames
> > +~~~~~~~~~~~~~~~~~~
> > +
> > +raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.pnm
>
> PNM is an umbrella term that refers to any of the PBM, PGM or PPM format. As
> the raw2pnm tool outputs PPM files, let's name them .ppm here and below. This
> helps with some image viewers that use file extensions to identify the format,
> and have trouble handling .pnm files.
Fixed.
>
> > +where 2560x1920 is output resolution, NV12 is the video format, followed
> > +by input frame and output PNM file.
> > +
> > +Viewfinder output frames
> > +~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.pnm
> > +
> > +where 2560x1920 is output resolution, NV12 is the video format, followed
> > +by input frame and output PNM file.
> > +
> > +Example user space code for IPU3
> > +================================
> > +
> > +User space code that configures and uses IPU3 is available here.
> > +
> > +https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
> > +
> > +The source can be located under hal/intel directory.
> > +
> > +References
> > +==========
> > +
> > +include/uapi/linux/intel-ipu3.h
> > +
> > +.. [#f1] https://github.com/intel/nvt
> > +
> > +.. [#f2] http://git.ideasonboard.org/yavta.git
> > +
> > +.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
> > +
> > +.. [#f4] ImgU limitation requires an additional 16x16 for all input
> > resolutions
>
> --
> Regards,
>
> Laurent Pinchart
>
>
>
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation
2018-12-13 9:38 ` Sakari Ailus
@ 2018-12-13 10:41 ` Laurent Pinchart
2018-12-13 10:50 ` Sakari Ailus
0 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-13 10:41 UTC (permalink / raw)
To: Sakari Ailus
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Sakari,
On Thursday, 13 December 2018 11:38:26 EET Sakari Ailus wrote:
> Hi Laurent,
>
> I'm sending a separate patch to address the comments.
>
> On Fri, Nov 30, 2018 at 12:50:36AM +0200, Laurent Pinchart wrote:
> > On Tuesday, 30 October 2018 00:22:56 EET Yong Zhi wrote:
> >> From: Rajmohan Mani <rajmohan.mani@intel.com>
> >>
> >> This patch adds the details about the IPU3 Imaging Unit driver.
> >
> > Strictly speaking this documents both the CIO2 and the IMGU. As they're
> > handled by two separate drivers, should they be split in two separate
> > files ? If you prefer keeping them together you should update the commit
> > message accordingly. I would in that case also split the documentation in
> > a CIO2 and a IMGU section in the file, instead of mixing them.
>
> I'm keeping it in a single document for now. In practice these devices
> always come together as neither is really usable without the other.
But they're still two separate devices. Splitting the documentation would
clarify which part is associated with each device. CIO2 and ImgU instructions
are currently interleaved and that's very confusing. If you really want to
keep everything in one file, the CIO2 and ImgU parts should be deinterleaved,
and the CIO2 should come first.
> >> Change-Id: I560cecf673df2dcc3ec72767cf8077708d649656
> >
> > The Change-Id: tag isn't suitable for mainline, you can drop it.
>
> Fixed.
>
> >> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> >> ---
> >>
> >> Documentation/media/v4l-drivers/index.rst | 1 +
> >> Documentation/media/v4l-drivers/ipu3.rst | 326 ++++++++++++++++++++++++
> >> 2 files changed, 327 insertions(+)
> >> create mode 100644 Documentation/media/v4l-drivers/ipu3.rst
[snip]
> >> diff --git a/Documentation/media/v4l-drivers/ipu3.rst
> >> b/Documentation/media/v4l-drivers/ipu3.rst new file mode 100644
> >> index 0000000..045bf42
> >> --- /dev/null
> >> +++ b/Documentation/media/v4l-drivers/ipu3.rst
[snip]
> >> +Media controller
> >> +----------------
> >> +
> >> +The media device interface allows to configure the ImgU links, which
> >> defines +the behavior of the IPU3 firmware.
> >
> > s/defines/define/ or possibly better s/defines/control/
>
> I'm removing the section as the binary is selected using a control in the
> current version. I'm adding this instead:
>
> Firmware binary selection
> -------------------------
>
> The firmware binary is selected using the V4L2_CID_INTEL_IPU3_MODE,
> currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h .
> "VIDEO" and "STILL" modes are available.
Do we need to expose the fact that they use different firmwares ? Or should we
instead document the two modes of operation, and explain that they need to be
selected before anything else ? In any case modes need to be documented.
Interestingly enough, the driver code includes
enum ipu3_css_pipe_id {
IPU3_CSS_PIPE_ID_PREVIEW,
IPU3_CSS_PIPE_ID_COPY,
IPU3_CSS_PIPE_ID_VIDEO,
IPU3_CSS_PIPE_ID_CAPTURE,
IPU3_CSS_PIPE_ID_YUVPP,
IPU3_CSS_PIPE_ID_ACC,
IPU3_CSS_PIPE_ID_NUM
};
but seems to only support the video and capture modes.
> >> +
> >> +Device operation
> >> +----------------
> >> +
> >> +With IPU3, once the input video node ("ipu3-imgu 0/1":0,
> >> +in <entity>:<pad-number> format) is queued with buffer (in packed raw
> >> bayer
> >
> > s/bayer/Bayer/
>
> Fixed in the entire file.
>
> >> +format), IPU3 ISP starts processing the buffer and produces the video
> >
> > s/IPU3 ISP/the IPU3 ISP/
> >
> > This is the first time you mention an ISP. Should the term ISP be replaced
> > by ImgU here and below ? I'm fine keeping it, but it should then be
> > defined in the introduction, in particular with an explanation of the
> > difference between ImgU and ISP.
>
> I replaced IPU3 with ImgU in this section.
>
> >> output +in YUV format and statistics output on respective output nodes.
> >> The driver +is expected to have buffers ready for all of parameter,
> >> output and +statistics nodes, when input video node is queued with
> >> buffer.
> >
> > Why is that, shouldn't the driver wait for all necessary buffers to be
> > ready before processing ?
>
> Not all are mandatory.
Which ones are mandatory, and which ones are not ? V4L2 doesn't enforce buffer
queuing order for M2M devices, it's a very bad idea to do so here. It would
make usage of the driver impossible with separate unsynchronized processes for
different queues (which is typically the case when using command line test
tools).
> >> +At a minimum, all of input, main output, 3A statistics and viewfinder
> >> +video nodes should be enabled for IPU3 to start image processing.
> >
> > If they all need to be enabled, shouldn't the respective links be ENABLED
> > and IMMUTABLE ?
>
> Yes.
Could you please capture this in the TODO file ?
[snip]
> >> +Configuring the Intel IPU3
> >> +==========================
> >> +
> >> +The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl
> >> calls, +the ISP can be configured and enabled.
> >> +
> >> +The IPU3 ImgU pipelines can be configured using media controller APIs,
> >> +defined at :ref:`media_controller`.
> >> +
> >> +Capturing frames in raw bayer format
> >> +------------------------------------
> >> +
> >> +IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
> >> +format) from the raw sensors connected to the CSI2 ports. The captured
> >> +frames are used as input to the ImgU driver.
> >> +
> >> +Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
> >
> > I would drop v4l2n from the documentation as it's not maintained and is
> > not functional (in particular it doesn't implement MPLANE support which
> > the driver requires).
>
> Ack. I'm leaving the command plus the reference to v4l2n, this might
> contain information useful still.
Who will address this TODO item ? :-)
> >> +raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique
> >> requirements
> >> +and / or features specific to IPU3.
[snip]
> >> +Processing the image in raw bayer format
> >> +----------------------------------------
> >> +
> >> +Configuring ImgU V4L2 subdev for image processing
> >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[snip]
> >> +RAW bayer frames go through the following ISP pipeline HW blocks to
> >> +have the processed image output to the DDR memory.
> >> +
> >> +RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >> Geometric +Distortion Correction (GDC) -> DDR
> >
> > A more detailed block diagram, with the other blocks included, should be
> > added to the ImgU description above. Each block should have a short
> > description of its purpose.
>
> I think we have one in conjunction with the parameter format description.
You're right. As mentioned in my review for that patch, I think it should be
moved to this document.
> >> +The ImgU V4L2 subdev has to be configured with the supported
> >> resolutions +in all the above HW blocks, for a given input resolution.
> >> +
> >> +For a given supported resolution for an input frame, the Input Feeder,
> >> +Bayer Down Scaling and GDC blocks should be configured with the
> >> supported +resolutions. This information can be obtained by looking at
> >> the following +IPU3 ISP configuration table.
> >
> > Does this mean that the ImgU will not operate properly when exercised
> > through the MC and V4L2 only without configuration of the internal blocks
> > through the processing parameters device node ?
>
> I need to let Raj and Yong to answer that. My understanding is the default
> parameters should be usable. If they're not, that needs to be addressed.
That's my assumption too :-)
> >> +https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/
> >> master
> >> +
> >> +Under
> >> baseboard-poppy/media-libs/arc-camera3-hal-configs-poppy/files/gcss
> >> +directory, graph_settings_ov5670.xml can be used as an example.
> >
> > The directory name is incorrect.
>
> Fixed.
>
> >> +The following steps prepare the ImgU ISP pipeline for the image
> >> processing.
> >> +
> >> +1. The ImgU V4L2 subdev data format should be set by using the
> >> +VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> >> above.
> >> +
> >> +2. The ImgU V4L2 subdev cropping should be set by using the
> >> +VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> >> target,
> >> +using the input feeder height and width.
> >> +
> >> +3. The ImgU V4L2 subdev composing should be set by using the
> >> +VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >> target, +using the BDS height and width.
> >
> > How the format and selection rectangles related to the internal processing
> > blocks should also be explained in more details.
>
> To be added later.
As it will likely require changes in the driver, I'm OK with that.
> >> +For the ov5670 example, for an input frame with a resolution of
> >> 2592x1944
> >> +(which is input to the ImgU subdev pad 0), the corresponding
> >> resolutions
> >> +for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
> >> +respectively.
> >
> > Why ? How is that computed ? If my input resolution was different, how
> > would I compute the other resolutions ?
>
> Ditto.
>
> >> +Once this is done, the received raw bayer frames can be input to the
> >> ImgU
> >> +V4L2 subdev as below, using the open source application v4l2n.
> >> +
> >> +For an image captured with 2592x1944 [#f4]_ resolution, with desired
> >> output +resolution as 2560x1920 and viewfinder resolution as 2560x1920,
> >> the following +v4l2n command can be used. This helps process the raw
> >> bayer frames and +produces the desired results for the main output
> >> image and the viewfinder +output, in NV12 format.
> >> +
> >> +v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
> >> +--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X473
> >> 3706 9 +--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1
> >> --output=/tmp/frames.out +--open=/dev/video5
> >> +--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
> >> +--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2
> >> --output=/tmp/frames.vf +--open=/dev/video6
> >> +--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
> >> +--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7
> >> +--output=/tmp/frames.3A --fmt=type:META_CAPTURE,?
> >> +--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
> >
> > You can replace this with four yavta commands.
>
> Ditto.
[snip]
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation
2018-12-13 10:41 ` Laurent Pinchart
@ 2018-12-13 10:50 ` Sakari Ailus
0 siblings, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-12-13 10:50 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Laurent,
Thanks for the review.
On Thu, Dec 13, 2018 at 12:41:36PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
>
> On Thursday, 13 December 2018 11:38:26 EET Sakari Ailus wrote:
> > Hi Laurent,
> >
> > I'm sending a separate patch to address the comments.
> >
> > On Fri, Nov 30, 2018 at 12:50:36AM +0200, Laurent Pinchart wrote:
> > > On Tuesday, 30 October 2018 00:22:56 EET Yong Zhi wrote:
> > >> From: Rajmohan Mani <rajmohan.mani@intel.com>
> > >>
> > >> This patch adds the details about the IPU3 Imaging Unit driver.
> > >
> > > Strictly speaking this documents both the CIO2 and the IMGU. As they're
> > > handled by two separate drivers, should they be split in two separate
> > > files ? If you prefer keeping them together you should update the commit
> > > message accordingly. I would in that case also split the documentation in
> > > a CIO2 and a IMGU section in the file, instead of mixing them.
> >
> > I'm keeping it in a single document for now. In practice these devices
> > always come together as neither is really usable without the other.
>
> But they're still two separate devices. Splitting the documentation would
> clarify which part is associated with each device. CIO2 and ImgU instructions
> are currently interleaved and that's very confusing. If you really want to
> keep everything in one file, the CIO2 and ImgU parts should be deinterleaved,
> and the CIO2 should come first.
That's implemented in the patch I submitted.
>
> > >> Change-Id: I560cecf673df2dcc3ec72767cf8077708d649656
> > >
> > > The Change-Id: tag isn't suitable for mainline, you can drop it.
> >
> > Fixed.
> >
> > >> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> > >> ---
> > >>
> > >> Documentation/media/v4l-drivers/index.rst | 1 +
> > >> Documentation/media/v4l-drivers/ipu3.rst | 326 ++++++++++++++++++++++++
> > >> 2 files changed, 327 insertions(+)
> > >> create mode 100644 Documentation/media/v4l-drivers/ipu3.rst
>
> [snip]
>
> > >> diff --git a/Documentation/media/v4l-drivers/ipu3.rst
> > >> b/Documentation/media/v4l-drivers/ipu3.rst new file mode 100644
> > >> index 0000000..045bf42
> > >> --- /dev/null
> > >> +++ b/Documentation/media/v4l-drivers/ipu3.rst
>
> [snip]
>
> > >> +Media controller
> > >> +----------------
> > >> +
> > >> +The media device interface allows to configure the ImgU links, which
> > >> defines +the behavior of the IPU3 firmware.
> > >
> > > s/defines/define/ or possibly better s/defines/control/
> >
> > I'm removing the section as the binary is selected using a control in the
> > current version. I'm adding this instead:
> >
> > Firmware binary selection
> > -------------------------
> >
> > The firmware binary is selected using the V4L2_CID_INTEL_IPU3_MODE,
> > currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h .
> > "VIDEO" and "STILL" modes are available.
>
> Do we need to expose the fact that they use different firmwares ? Or should we
> instead document the two modes of operation, and explain that they need to be
> selected before anything else ? In any case modes need to be documented.
I agree. I'll add this to the TODO file.
> Interestingly enough, the driver code includes
>
> enum ipu3_css_pipe_id {
> IPU3_CSS_PIPE_ID_PREVIEW,
> IPU3_CSS_PIPE_ID_COPY,
> IPU3_CSS_PIPE_ID_VIDEO,
> IPU3_CSS_PIPE_ID_CAPTURE,
> IPU3_CSS_PIPE_ID_YUVPP,
> IPU3_CSS_PIPE_ID_ACC,
> IPU3_CSS_PIPE_ID_NUM
> };
>
> but seems to only support the video and capture modes.
I wonder if the others exist in the firmware binary. Other devices with
e.g. integrated CSI-2 receivers have been supported using the same source
code base I presume.
>
> > >> +
> > >> +Device operation
> > >> +----------------
> > >> +
> > >> +With IPU3, once the input video node ("ipu3-imgu 0/1":0,
> > >> +in <entity>:<pad-number> format) is queued with buffer (in packed raw
> > >> bayer
> > >
> > > s/bayer/Bayer/
> >
> > Fixed in the entire file.
> >
> > >> +format), IPU3 ISP starts processing the buffer and produces the video
> > >
> > > s/IPU3 ISP/the IPU3 ISP/
> > >
> > > This is the first time you mention an ISP. Should the term ISP be replaced
> > > by ImgU here and below ? I'm fine keeping it, but it should then be
> > > defined in the introduction, in particular with an explanation of the
> > > difference between ImgU and ISP.
> >
> > I replaced IPU3 with ImgU in this section.
> >
> > >> output +in YUV format and statistics output on respective output nodes.
> > >> The driver +is expected to have buffers ready for all of parameter,
> > >> output and +statistics nodes, when input video node is queued with
> > >> buffer.
> > >
> > > Why is that, shouldn't the driver wait for all necessary buffers to be
> > > ready before processing ?
> >
> > Not all are mandatory.
>
> Which ones are mandatory, and which ones are not ? V4L2 doesn't enforce buffer
> queuing order for M2M devices, it's a very bad idea to do so here. It would
> make usage of the driver impossible with separate unsynchronized processes for
> different queues (which is typically the case when using command line test
> tools).
How about this:
- Document different operation modes, and which buffer queues are relevant
in each mode. To process an image, which queues require a buffer an in
which ones is it optional?
>
> > >> +At a minimum, all of input, main output, 3A statistics and viewfinder
> > >> +video nodes should be enabled for IPU3 to start image processing.
> > >
> > > If they all need to be enabled, shouldn't the respective links be ENABLED
> > > and IMMUTABLE ?
> >
> > Yes.
>
> Could you please capture this in the TODO file ?
>
> [snip]
>
> > >> +Configuring the Intel IPU3
> > >> +==========================
> > >> +
> > >> +The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl
> > >> calls, +the ISP can be configured and enabled.
> > >> +
> > >> +The IPU3 ImgU pipelines can be configured using media controller APIs,
> > >> +defined at :ref:`media_controller`.
> > >> +
> > >> +Capturing frames in raw bayer format
> > >> +------------------------------------
> > >> +
> > >> +IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
> > >> +format) from the raw sensors connected to the CSI2 ports. The captured
> > >> +frames are used as input to the ImgU driver.
> > >> +
> > >> +Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
> > >
> > > I would drop v4l2n from the documentation as it's not maintained and is
> > > not functional (in particular it doesn't implement MPLANE support which
> > > the driver requires).
> >
> > Ack. I'm leaving the command plus the reference to v4l2n, this might
> > contain information useful still.
>
> Who will address this TODO item ? :-)
I don't think we need to name anyone at this point. Presumably someone from
Intel.
>
> > >> +raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique
> > >> requirements
> > >> +and / or features specific to IPU3.
>
> [snip]
>
> > >> +Processing the image in raw bayer format
> > >> +----------------------------------------
> > >> +
> > >> +Configuring ImgU V4L2 subdev for image processing
> > >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> [snip]
>
> > >> +RAW bayer frames go through the following ISP pipeline HW blocks to
> > >> +have the processed image output to the DDR memory.
> > >> +
> > >> +RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > >> Geometric +Distortion Correction (GDC) -> DDR
> > >
> > > A more detailed block diagram, with the other blocks included, should be
> > > added to the ImgU description above. Each block should have a short
> > > description of its purpose.
> >
> > I think we have one in conjunction with the parameter format description.
>
> You're right. As mentioned in my review for that patch, I think it should be
> moved to this document.
Fair enough. Let's polish this more later.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH 1/1] staging/ipu3-imgu: Address documentation comments
2018-11-29 22:50 ` Laurent Pinchart
2018-12-13 9:38 ` Sakari Ailus
@ 2018-12-13 9:38 ` Sakari Ailus
1 sibling, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-12-13 9:38 UTC (permalink / raw)
To: linux-media; +Cc: laurent.pinchart, yong.zhi, rajmohan.mani
Address comments on the documentation after Yong's original patch.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
.../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 2 +-
Documentation/media/v4l-drivers/ipu3.rst | 343 ++++++++++++---------
drivers/staging/media/ipu3/TODO | 7 +
3 files changed, 201 insertions(+), 151 deletions(-)
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
index 8cd30ffbf8b8b..dc871006b41a5 100644
--- a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
@@ -175,4 +175,4 @@ video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
Intel IPU3 ImgU uAPI data types
===============================
-.. kernel-doc:: include/uapi/linux/intel-ipu3.h
+.. kernel-doc:: drivers/staging/media/ipu3/include/intel-ipu3.h
diff --git a/Documentation/media/v4l-drivers/ipu3.rst b/Documentation/media/v4l-drivers/ipu3.rst
index 045bf4222b1a0..9f95a8432f140 100644
--- a/Documentation/media/v4l-drivers/ipu3.rst
+++ b/Documentation/media/v4l-drivers/ipu3.rst
@@ -9,196 +9,241 @@ Copyright |copy| 2018 Intel Corporation
Introduction
============
-This file documents Intel IPU3 (3rd generation Image Processing Unit) Imaging
-Unit driver located under drivers/media/pci/intel/ipu3.
+This file documents the Intel IPU3 (3rd generation Image Processing Unit)
+Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
+as under drivers/staging/media/ipu3 (ImgU).
The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
-platforms (U/Y processor lines) is made up of two parts namely Imaging Unit
-(ImgU) and CIO2 device (MIPI CSI2 receiver).
+platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
+(ImgU) and the CIO2 device (MIPI CSI2 receiver).
-The CIO2 device receives the raw bayer data from the sensors and outputs the
-frames in a format that is specific to IPU3 (for consumption by IPU3 ImgU).
-CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2* and is
-enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
+The CIO2 device receives the raw Bayer data from the sensors and outputs the
+frames in a format that is specific to the IPU3 (for consumption by the IPU3
+ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
+and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
The Imaging Unit (ImgU) is responsible for processing images captured
-through IPU3 CIO2 device. The ImgU driver sources can be found under
-drivers/media/pci/intel/ipu3 directory. The driver is enabled through the
+by the IPU3 CIO2 device. The ImgU driver sources can be found under
+drivers/staging/media/ipu3 directory. The driver is enabled through the
CONFIG_VIDEO_IPU3_IMGU config option.
-The two driver modules are named ipu3-csi2 and ipu3-imgu, respectively.
+The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
-The driver has been tested on Kaby Lake platforms (U/Y processor lines).
+The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
-The driver implements V4L2, Media controller and V4L2 sub-device interfaces.
-Camera sensors that have CSI-2 bus, which are connected to the IPU3 CIO2
-device are supported. Support for lens and flash drivers depends on the
-above sensors.
+Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
+interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
+MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
-ImgU device nodes
-=================
+CIO2
+====
-The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
-subdev interface to the user space.
+The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
+interface to the user space. There is a video node for each CSI-2 receiver,
+with a single media controller interface for the entire device.
-Each V4L2 subdev represents a pipe, which can support a maximum of 2
-streams. A private ioctl can be used to configure the mode (video or still)
-of the pipe.
+The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
+receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
+to userspace as a V4L2 sub-device node and has two pads:
-This helps to support advanced camera features like Continuous View Finder
-(CVF) and Snapshot During Video(SDV).
+.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
-CIO2 device
-===========
+.. flat-table::
-The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
-interface to the user space. There is a video node for each CSI-2 receiver,
-with a single media controller interface for the entire device.
+ * - pad
+ - direction
+ - purpose
-Media controller
-----------------
+ * - 0
+ - sink
+ - MIPI CSI-2 input, connected to the sensor subdev
-The media device interface allows to configure the ImgU links, which defines
-the behavior of the IPU3 firmware.
+ * - 1
+ - source
+ - Raw video capture, connected to the V4L2 video interface
-Device operation
-----------------
+The V4L2 video interfaces model the DMA engines. They are exposed to userspace
+as V4L2 video device nodes.
-With IPU3, once the input video node ("ipu3-imgu 0/1":0,
-in <entity>:<pad-number> format) is queued with buffer (in packed raw bayer
-format), IPU3 ISP starts processing the buffer and produces the video output
-in YUV format and statistics output on respective output nodes. The driver
-is expected to have buffers ready for all of parameter, output and
-statistics nodes, when input video node is queued with buffer.
+Capturing frames in raw Bayer format
+------------------------------------
-At a minimum, all of input, main output, 3A statistics and viewfinder
-video nodes should be enabled for IPU3 to start image processing.
+CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
+from the raw sensors connected to the CSI2 ports. The captured frames are used
+as input to the ImgU driver.
-Each ImgU V4L2 subdev has the following set of video nodes.
+Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
+yavta [#f2]_ due to the following unique requirements and / or features specific
+to IPU3.
-input, output and viewfinder video nodes
-----------------------------------------
+-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
+raw Bayer format that is specific to IPU3.
-The frames (in packed raw bayer format specific to IPU3) received by the
-input video node is processed by the IPU3 Imaging Unit and is output to 2
-video nodes, with each targeting different purpose (main output and viewfinder
-output).
+-- Multiple video nodes have to be operated simultaneously.
-Details on raw bayer format specific to IPU3 can be found as below.
-Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
+2592x1944 image capture.
-The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
+Using the media contorller APIs, the ov5670 sensor is configured to send
+frames in packed raw Bayer format to IPU3 CSI2 receiver.
-Only the multi-planar API is supported. More details can be found at
-:ref:`planar-apis`.
+# This example assumes /dev/media0 as the CIO2 media device
+export MDEV=/dev/media0
-parameters video node
----------------------
+# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
-The parameter video node receives the ISP algorithm parameters that are used
-to configure how the ISP algorithms process the image.
+export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
-Details on raw bayer format specific to IPU3 can be found as below.
-Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+# Establish the link for the media devices using media-ctl [#f3]_
+media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
-3A statistics video node
-------------------------
+# Set the format for the media devices
+media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
-3A statistics video node is used by the ImgU driver to output the 3A (auto
-focus, auto exposure and auto white balance) statistics for the frames that
-are being processed by the ISP to user space applications. User space
-applications can use this statistics data to arrive at desired algorithm
-parameters for ISP.
+media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
-CIO2 device nodes
-=================
+media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
-CIO2 is represented as a single V4L2 sub-device with a video node for each
-CSI-2 receiver. The video node represents the DMA engine.
+Once the media pipeline is configured, desired sensor specific settings
+(such as exposure and gain settings) can be set, using the yavta tool.
-Configuring the Intel IPU3
-==========================
+e.g
-The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl calls,
-the ISP can be configured and enabled.
+yavta -w 0x009e0903 444 $SDEV
-The IPU3 ImgU pipelines can be configured using media controller APIs,
-defined at :ref:`media_controller`.
+yavta -w 0x009e0913 1024 $SDEV
-Capturing frames in raw bayer format
-------------------------------------
+yavta -w 0x009e0911 2046 $SDEV
-IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
-format) from the raw sensors connected to the CSI2 ports. The captured
-frames are used as input to the ImgU driver.
+Once the desired sensor settings are set, frame captures can be done as below.
-Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
-raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique requirements
-and / or features specific to IPU3.
+e.g
--- The IPU3 CSI2 receiver outputs the captured frames from the sensor in
-packed raw bayer format that is specific to IPU3
+yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
+ -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
--- Multiple video nodes have to be operated simultaneously
+With the above command, 10 frames are captured at 2592x1944 resolution, with
+sGRBG10 format and output as IPU3_SGRBG10 format.
-Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
-2592x1944 image capture.
+The captured frames are available as /tmp/frame-#.bin files.
-Using the media contorller APIs, the ov5670 sensor is configured to send
-frames in packed raw bayer format to IPU3 CSI2 receiver.
+ImgU
+====
-# This example assumes /dev/media0 as the ImgU media device
+The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
+subdev interface to the user space.
-export MDEV=/dev/media0
+Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
+This helps to support advanced camera features like Continuous View Finder (CVF)
+and Snapshot During Video(SDV).
-# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
+The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
+exposed to userspace as a V4L2 sub-device node.
-export SDEV="ov5670 10-0036"
+Each pipe has two sink pads and three source pads for the following purpose:
-# Establish the link for the media devices using media-ctl [#f3]_
-media-ctl -d $MDEV -l "ov5670 ":0 -> "ipu3-csi2 0":0[1]
+.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
-media-ctl -d $MDEV -l "ipu3-csi2 0":1 -> "ipu3-cio2 0":0[1]
+.. flat-table::
-# Set the format for the media devices
-media-ctl -d $MDEV -V "ov5670 ":0 [fmt:SGRBG10/2592x1944]
+ * - pad
+ - direction
+ - purpose
-media-ctl -d $MDEV -V "ipu3-csi2 0":0 [fmt:SGRBG10/2592x1944]
+ * - 0
+ - sink
+ - Input raw video stream
-media-ctl -d $MDEV -V "ipu3-csi2 0":1 [fmt:SGRBG10/2592x1944]
+ * - 1
+ - sink
+ - Processing parameters
-Once the media pipeline is configured, desired sensor specific settings
-(such as exposure and gain settings) can be set, using the yavta tool.
+ * - 2
+ - source
+ - Output processed video stream
-e.g
+ * - 3
+ - source
+ - Output viewfinder video stream
-yavta -w 0x009e0903 444 $(media-ctl -d $MDEV -e "$SDEV")
+ * - 4
+ - source
+ - 3A statistics
-yavta -w 0x009e0913 1024 $(media-ctl -d $MDEV -e "$SDEV")
+Each pad is connected to a corresponding V4L2 video interface, exposed to
+userspace as a V4L2 video device node.
+
+Device operation
+----------------
-yavta -w 0x009e0911 2046 $(media-ctl -d $MDEV -e "$SDEV")
+With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
+<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
+format), ImgU starts processing the buffer and produces the video output in YUV
+format and statistics output on respective output nodes. The driver is expected
+to have buffers ready for all of parameter, output and statistics nodes, when
+input video node is queued with buffer.
-Once the desired sensor settings are set, frame captures can be done as below.
+At a minimum, all of input, main output, 3A statistics and viewfinder
+video nodes should be enabled for IPU3 to start image processing.
-e.g
+Each ImgU V4L2 subdev has the following set of video nodes.
-yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin
--f IPU3_GRBG10 media-ctl -d $MDEV -e ipu3-cio2 0
+input, output and viewfinder video nodes
+----------------------------------------
-With the above command, 10 frames are captured at 2592x1944 resolution, with
-sGRBG10 format and output as IPU3_GRBG10 format.
+The frames (in packed raw Bayer format specific to the IPU3) received by the
+input video node is processed by the IPU3 Imaging Unit and are output to 2 video
+nodes, with each targeting a different purpose (main output and viewfinder
+output).
-The captured frames are available as /tmp/frame-#.bin files.
+Details onand the Bayer format specific to the IPU3 can be found in
+:ref:`v4l2-pix-fmt-ipu3-sbggr10`.
-Processing the image in raw bayer format
+The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
+
+Only the multi-planar API is supported. More details can be found at
+:ref:`planar-apis`.
+
+Parameters video node
+---------------------
+
+The parameters video node receives the ImgU algorithm parameters that are used
+to configure how the ImgU algorithms process the image.
+
+Details on processing parameters specific to the IPU3 can be found in
+:ref:`v4l2-meta-fmt-params`.
+
+3A statistics video node
+------------------------
+
+3A statistics video node is used by the ImgU driver to output the 3A (auto
+focus, auto exposure and auto white balance) statistics for the frames that are
+being processed by the ImgU to user space applications. User space applications
+can use this statistics data to compute the desired algorithm parameters for
+the ImgU.
+
+Configuring the Intel IPU3
+==========================
+
+The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
+:ref:`media_controller`.
+
+Firmware binary selection
+-------------------------
+
+The firmware binary is selected using the V4L2_CID_INTEL_IPU3_MODE, currently
+defined in drivers/staging/media/ipu3/include/intel-ipu3.h . "VIDEO" and "STILL"
+modes are available.
+
+Processing the image in raw Bayer format
----------------------------------------
Configuring ImgU V4L2 subdev for image processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The ImgU V4L2 subdevs have to be configured with media controller APIs to
-have all the video nodes setup correctly.
+The ImgU V4L2 subdevs have to be configured with media controller APIs to have
+all the video nodes setup correctly.
Let us take "ipu3-imgu 0" subdev as an example.
@@ -206,40 +251,38 @@ media-ctl -d $MDEV -r
media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
-media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "output":0[1]
+media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
-media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "viewfinder":0[1]
+media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
-media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "3a stat":0[1]
+media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
-Also the pipe mode of the corresponding V4L2 subdev should be set as
-desired (e.g 0 for video mode or 1 for still mode) through the
-control id 0x009819a1 as below.
-
-e.g
+Also the pipe mode of the corresponding V4L2 subdev should be set as desired
+(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
+below.
-v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
+yavta -w "0x009819A1 1" /dev/v4l-subdev7
-RAW bayer frames go through the following ISP pipeline HW blocks to
-have the processed image output to the DDR memory.
+RAW Bayer frames go through the following ImgU pipeline HW blocks to have the
+processed image output to the DDR memory.
-RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
+RAW Bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
Distortion Correction (GDC) -> DDR
-The ImgU V4L2 subdev has to be configured with the supported resolutions
-in all the above HW blocks, for a given input resolution.
+The ImgU V4L2 subdev has to be configured with the supported resolutions in all
+the above HW blocks, for a given input resolution.
-For a given supported resolution for an input frame, the Input Feeder,
-Bayer Down Scaling and GDC blocks should be configured with the supported
-resolutions. This information can be obtained by looking at the following
-IPU3 ISP configuration table.
+For a given supported resolution for an input frame, the Input Feeder, Bayer
+Down Scaling and GDC blocks should be configured with the supported resolutions.
+This information can be obtained by looking at the following IPU3 ImgU
+configuration table.
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
-Under baseboard-poppy/media-libs/arc-camera3-hal-configs-poppy/files/gcss
+Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
directory, graph_settings_ov5670.xml can be used as an example.
-The following steps prepare the ImgU ISP pipeline for the image processing.
+The following steps prepare the ImgU pipeline for the image processing.
1. The ImgU V4L2 subdev data format should be set by using the
VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
@@ -257,14 +300,14 @@ For the ov5670 example, for an input frame with a resolution of 2592x1944
for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
respectively.
-Once this is done, the received raw bayer frames can be input to the ImgU
-V4L2 subdev as below, using the open source application v4l2n.
+Once this is done, the received raw Bayer frames can be input to the ImgU
+V4L2 subdev as below, using the open source application v4l2n [#f1]_.
For an image captured with 2592x1944 [#f4]_ resolution, with desired output
resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
-v4l2n command can be used. This helps process the raw bayer frames and
-produces the desired results for the main output image and the viewfinder
-output, in NV12 format.
+v4l2n command can be used. This helps process the raw Bayer frames and produces
+the desired results for the main output image and the viewfinder output, in NV12
+format.
v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069
@@ -281,7 +324,7 @@ v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
input, output, viewfinder and 3A statistics video nodes respectively.
-Converting the raw bayer image into YUV domain
+Converting the raw Bayer image into YUV domain
----------------------------------------------
The processed images after the above step, can be converted to YUV domain
@@ -290,7 +333,7 @@ as below.
Main output frames
~~~~~~~~~~~~~~~~~~
-raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.pnm
+raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
where 2560x1920 is output resolution, NV12 is the video format, followed
by input frame and output PNM file.
@@ -298,7 +341,7 @@ by input frame and output PNM file.
Viewfinder output frames
~~~~~~~~~~~~~~~~~~~~~~~~
-raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.pnm
+raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
where 2560x1920 is output resolution, NV12 is the video format, followed
by input frame and output PNM file.
@@ -315,7 +358,7 @@ The source can be located under hal/intel directory.
References
==========
-include/uapi/linux/intel-ipu3.h
+.. [#f5] include/uapi/linux/intel-ipu3.h
.. [#f1] https://github.com/intel/nvt
diff --git a/drivers/staging/media/ipu3/TODO b/drivers/staging/media/ipu3/TODO
index 922b885f10a70..d3076f5ebec1a 100644
--- a/drivers/staging/media/ipu3/TODO
+++ b/drivers/staging/media/ipu3/TODO
@@ -21,3 +21,10 @@ staging directory.
Further clarification on some ambiguities such as data type conversion of
IEFD CU inputs. (Sakari)
Move acronyms to doc-rst file. (Mauro)
+
+- Switch to yavta from v4l2n in driver docs.
+
+- Elaborate the functionality of different selection rectangles in driver
+ documentation. This may require driver changes as well.
+
+- More detailed documentation on calculating BDS, GCD etc. sizes needed.
--
2.11.0
^ permalink raw reply related [flat|nested] 123+ messages in thread
* [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
2018-10-29 22:22 ` [PATCH v7 01/16] v4l: Add Intel IPU3 meta buffer formats Yong Zhi
2018-10-29 22:22 ` [PATCH v7 02/16] doc-rst: Add Intel IPU3 documentation Yong Zhi
@ 2018-10-29 22:22 ` Yong Zhi
2018-11-02 13:02 ` Sakari Ailus
` (2 more replies)
2018-10-29 22:22 ` [PATCH v7 04/16] intel-ipu3: abi: Add register definitions and enum Yong Zhi
` (14 subsequent siblings)
17 siblings, 3 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:22 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi, Chao C Li
These meta formats are used on Intel IPU3 ImgU video queues
to carry 3A statistics and ISP pipeline parameters.
V4L2_META_FMT_IPU3_3A
V4L2_META_FMT_IPU3_PARAMS
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
Signed-off-by: Chao C Li <chao.c.li@intel.com>
Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
---
Documentation/media/uapi/v4l/meta-formats.rst | 1 +
.../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
3 files changed, 3001 insertions(+)
create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
create mode 100644 include/uapi/linux/intel-ipu3.h
diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
index cf971d5..eafc534 100644
--- a/Documentation/media/uapi/v4l/meta-formats.rst
+++ b/Documentation/media/uapi/v4l/meta-formats.rst
@@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
.. toctree::
:maxdepth: 1
+ pixfmt-meta-intel-ipu3
pixfmt-meta-d4xx
pixfmt-meta-uvc
pixfmt-meta-vsp1-hgo
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
new file mode 100644
index 0000000..23b945b
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
@@ -0,0 +1,181 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _intel-ipu3:
+
+******************************************************************
+V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
+******************************************************************
+
+.. c:type:: ipu3_uapi_stats_3a
+
+3A statistics
+=============
+
+For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
+an input bayer frame. Those statistics, defined in data struct
+:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu 3a stat"
+video node, which are then passed to user space for statistics analysis
+using :c:type:`v4l2_meta_format` interface.
+
+The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
+Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
+and AE (Auto-exposure) histogram.
+
+struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.
+
+
+.. code-block:: c
+
+
+ struct ipu3_uapi_stats_3a {
+ struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
+ __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_raw_buffer_aligned
+ ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
+ struct ipu3_uapi_af_raw_buffer af_raw_buffer;
+ struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
+ struct ipu3_uapi_4a_config stats_4a_config;
+ __u32 ae_join_buffers;
+ __u8 padding[28];
+ struct ipu3_uapi_stats_3a_bubble_info_per_stripe
+ stats_3a_bubble_per_stripe;
+ struct ipu3_uapi_ff_status stats_3a_status;
+ } __packed;
+
+
+.. c:type:: ipu3_uapi_params
+
+Pipeline parameters
+===================
+
+IPU3 pipeline has a number of image processing stages, each of which takes a
+set of parameters as input. The major stages of pipelines are shown here:
+
+Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
+
+Linearization -> Lens Shading Correction -> White Balance / Exposure /
+
+Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
+
+Correction Matrix -> Gamma correction -> Color Space Conversion ->
+
+Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
+
+Correction -> XNR3 -> TNR -> DDR
+
+The table below presents a description of the above algorithms.
+
+======================== =======================================================
+Name Description
+======================== =======================================================
+Optical Black Correction Optical Black Correction block subtracts a pre-defined
+ value from the respective pixel values to obtain better
+ image quality.
+ Defined in :c:type:`ipu3_uapi_obgrid_param`.
+Linearization This algo block uses linearization parameters to
+ address non-linearity sensor effects. The Lookup table
+ table is defined in
+ :c:type:`ipu3_uapi_isp_lin_vmem_params`.
+SHD Lens shading correction is used to correct spatial
+ non-uniformity of the pixel response due to optical
+ lens shading. This is done by applying a different gain
+ for each pixel. The gain, black level etc are
+ configured in :c:type:`ipu3_uapi_shd_config_static`.
+BNR Bayer noise reduction block removes image noise by
+ applying a bilateral filter.
+ See :c:type:`ipu3_uapi_bnr_static_config` for details.
+ANR Advanced Noise Reduction is a block based algorithm
+ that performs noise reduction in the Bayer domain. The
+ convolution matrix etc can be found in
+ :c:type:`ipu3_uapi_anr_config`.
+Demosaicing Demosaicing converts raw sensor data in Bayer format
+ into RGB (Red, Green, Blue) presentation. Then add
+ outputs of estimation of Y channel for following stream
+ processing by Firmware. The struct is defined as
+ :c:type:`ipu3_uapi_dm_config`.
+Color Correction Color Correction algo transforms sensor specific color
+ space to the standard "sRGB" color space. This is done
+ by applying 3x3 matrix defined in
+ :c:type:`ipu3_uapi_ccm_mat_config`.
+Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
+ basic non-linear tone mapping correction that is
+ applied per pixel for each pixel component.
+CSC Color space conversion transforms each pixel from the
+ RGB primary presentation to YUV (Y - brightness,
+ UV - Luminance) presentation. This is done by applying
+ a 3x3 matrix defined in
+ :c:type:`ipu3_uapi_csc_mat_config`
+CDS Chroma down sampling
+ After the CSC is performed, the Chroma Down Sampling
+ is applied for a UV plane down sampling by a factor
+ of 2 in each direction for YUV 4:2:0 using a 4x2
+ configurable filter :c:type:`ipu3_uapi_cds_params`.
+CHNR Chroma noise reduction
+ This block processes only the chrominance pixels and
+ performs noise reduction by cleaning the high
+ frequency noise.
+ See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
+TCC Total color correction as defined in struct
+ :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
+XNR3 eXtreme Noise Reduction V3 is the third revision of
+ noise reduction algorithm used to improve image
+ quality. This removes the low frequency noise in the
+ captured image. Two related structs are being defined,
+ :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
+ and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
+ memory.
+TNR Temporal Noise Reduction block compares successive
+ frames in time to remove anomalies / noise in pixel
+ values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
+ :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
+ vector and data memory respectively.
+======================== =======================================================
+
+A few stages of the pipeline will be executed by firmware running on the ISP
+processor, while many others will use a set of fixed hardware blocks also
+called accelerator cluster (ACC) to crunch pixel data and produce statistics.
+
+ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be selectively
+enabled / disabled by the user space through struct :c:type:`ipu3_uapi_flags`
+embedded in :c:type:`ipu3_uapi_params` structure. For parameters that are not
+enabled by the user space, corresponding structs are ignored by the ISP.
+
+Both 3A statistics and pipeline parameters described here are closely tied to
+the underlying camera sub-system (CSS) APIs. They are usually consumed and
+produced by dedicated user space libraries that comprise the important tuning
+tools, thus freeing the developers from being bothered with the low level
+hardware and algorithm details.
+
+It should be noted that IPU3 DMA operations require the addresses of all data
+structures (that includes both input and output) to be aligned on 32 byte
+boundaries.
+
+The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
+video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
+
+.. code-block:: c
+
+ struct ipu3_uapi_params {
+ /* Flags which of the settings below are to be applied */
+ struct ipu3_uapi_flags use __attribute__((aligned(32)));
+
+ /* Accelerator cluster parameters */
+ struct ipu3_uapi_acc_param acc_param;
+
+ /* ISP vector address space parameters */
+ struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
+ struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
+ struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
+
+ /* ISP data memory (DMEM) parameters */
+ struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
+ struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
+
+ /* Optical black level compensation */
+ struct ipu3_uapi_obgrid_param obgrid_param;
+ } __packed;
+
+Intel IPU3 ImgU uAPI data types
+===============================
+
+.. kernel-doc:: include/uapi/linux/intel-ipu3.h
diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
new file mode 100644
index 0000000..c2608b6
--- /dev/null
+++ b/include/uapi/linux/intel-ipu3.h
@@ -0,0 +1,2819 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2017 - 2018 Intel Corporation */
+
+#ifndef __IPU3_UAPI_H
+#define __IPU3_UAPI_H
+
+#include <linux/types.h>
+
+/********************* Key Acronyms *************************/
+/*
+ * ACC - Accelerator cluster
+ * ANR - Adaptive noise reduction
+ * AWB_FR- Auto white balance filter response statistics
+ * BNR - Bayer noise reduction parameters
+ * BDS - Bayer downscaler parameters
+ * CCM - Color correction matrix coefficients
+ * CDS - Chroma down sample
+ * CHNR - Chroma noise reduction
+ * CSC - Color space conversion
+ * DM - De-mosaic
+ * IEFd - Image enhancement filter directed
+ * Obgrid - Optical black level compensation
+ * OSYS - Output system configuration
+ * ROI - Region of interest
+ * SHD - Lens shading correction table
+ * TCC - Total color correction
+ * YDS - Y down sampling
+ * YTM - Y-tone mapping
+ */
+
+/*
+ * IPU3 DMA operations require buffers to be aligned at
+ * 32 byte boundaries
+ */
+
+/******************* ipu3_uapi_stats_3a *******************/
+
+#define IPU3_UAPI_MAX_STRIPES 2
+#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
+
+#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
+#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
+
+/* controls generation of meta_data (like FF enable/disable) */
+#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
+#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
+
+/**
+ * struct ipu3_uapi_grid_config - Grid plane config
+ *
+ * @width: Grid horizontal dimensions, in number of grid blocks(cells).
+ * @height: Grid vertical dimensions, in number of grid cells.
+ * @block_width_log2: Log2 of the width of each cell in pixels.
+ * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
+ * @block_height_log2: Log2 of the height of each cell in pixels.
+ * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
+ * @height_per_slice: The number of blocks in vertical axis per slice.
+ * Default 2.
+ * @x_start: X value of top left corner of Region of Interest(ROI).
+ * @y_start: Y value of top left corner of ROI
+ * @x_end: X value of bottom right corner of ROI
+ * @y_end: Y value of bottom right corner of ROI
+ *
+ * Due to the size of total amount of collected data, most statistics
+ * create a grid-based output, and the data is then divided into "slices".
+ */
+struct ipu3_uapi_grid_config {
+ __u8 width;
+ __u8 height;
+ __u16 block_width_log2:3;
+ __u16 block_height_log2:3;
+ __u16 height_per_slice:8;
+ __u16 x_start;
+ __u16 y_start;
+ __u16 x_end;
+ __u16 y_end;
+} __packed;
+
+/*
+ * The grid based data is divided into "slices" called set, each slice of setX
+ * refers to ipu3_uapi_grid_config width * height_per_slice.
+ */
+#define IPU3_UAPI_AWB_MAX_SETS 60
+/* Based on grid size 80 * 60 and cell size 16 x 16 */
+#define IPU3_UAPI_AWB_SET_SIZE 1280
+#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
+#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
+ (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
+ IPU3_UAPI_AWB_MD_ITEM_SIZE)
+#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
+ (IPU3_UAPI_AWB_MAX_SETS * \
+ (IPU3_UAPI_AWB_SET_SIZE + IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
+/**
+ * struct ipu3_uapi_awb_meta_data - AWB meta data
+ *
+ * @meta_data_buffer: Average values for each color channel
+ */
+struct ipu3_uapi_awb_meta_data {
+ __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
+} __packed;
+
+/**
+ * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
+ *
+ * @meta_data: buffer to hold auto white balance meta data.
+ */
+struct ipu3_uapi_awb_raw_buffer {
+ struct ipu3_uapi_awb_meta_data meta_data;
+} __packed;
+
+/**
+ * struct ipu3_uapi_awb_config_s - AWB config
+ *
+ * @rgbs_thr_gr: gr threshold value.
+ * @rgbs_thr_r: Red threshold value.
+ * @rgbs_thr_gb: gb threshold value.
+ * @rgbs_thr_b: Blue threshold value.
+ * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
+ *
+ * The threshold is a saturation measure range [0, 8191], 8191 is default.
+ * Values over threshold may be optionally rejected for averaging.
+ */
+struct ipu3_uapi_awb_config_s {
+ __u16 rgbs_thr_gr;
+ __u16 rgbs_thr_r;
+ __u16 rgbs_thr_gb;
+ __u16 rgbs_thr_b;
+ struct ipu3_uapi_grid_config grid;
+} __attribute__((aligned(32))) __packed;
+
+/**
+ * struct ipu3_uapi_awb_config - AWB config wrapper
+ *
+ * @config: config for auto white balance as defined by &ipu3_uapi_awb_config_s
+ */
+struct ipu3_uapi_awb_config {
+ struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
+} __packed;
+
+#define IPU3_UAPI_AE_COLORS 4 /* R, G, B, Y */
+#define IPU3_UAPI_AE_BINS 256
+#define IPU3_UAPI_AE_WEIGHTS 96
+
+/**
+ * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
+ *
+ * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
+ *
+ * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit unsigned
+ * for counting the number of the pixel.
+ */
+struct ipu3_uapi_ae_raw_buffer {
+ __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
+ *
+ * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
+ */
+struct ipu3_uapi_ae_raw_buffer_aligned {
+ struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_ae_grid_config - AE weight grid
+ *
+ * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
+ * @height: Grid vertical dimensions. Value: [16, 24], default 16.
+ * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
+ * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
+ * @__reserved0: reserved
+ * @ae_en: 0: does not write to meta-data array, 1: write normally.
+ * @rst_hist_array: write 1 to trigger histogram array reset.
+ * @done_rst_hist_array: flag for histogram array reset done.
+ * @x_start: X value of top left corner of ROI, default 0.
+ * @y_start: Y value of top left corner of ROI, default 0.
+ * @x_end: X value of bottom right corner of ROI
+ * @y_end: Y value of bottom right corner of ROI
+ *
+ * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
+ * a defined ROI within the frame. The contribution of each pixel into the
+ * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a grid.
+ */
+struct ipu3_uapi_ae_grid_config {
+ __u8 width;
+ __u8 height;
+ __u8 block_width_log2:4;
+ __u8 block_height_log2:4;
+ __u8 __reserved0:5;
+ __u8 ae_en:1;
+ __u8 rst_hist_array:1;
+ __u8 done_rst_hist_array:1;
+ __u16 x_start;
+ __u16 y_start;
+ __u16 x_end;
+ __u16 y_end;
+} __packed;
+
+/**
+ * struct ipu3_uapi_ae_weight_elem - AE weights LUT
+ *
+ * @cell0: weighted histogram grid value.
+ * @cell1: weighted histogram grid value.
+ * @cell2: weighted histogram grid value.
+ * @cell3: weighted histogram grid value.
+ * @cell4: weighted histogram grid value.
+ * @cell5: weighted histogram grid value.
+ * @cell6: weighted histogram grid value.
+ * @cell7: weighted histogram grid value.
+ *
+ * Use weighted grid value to give a different contribution factor to each cell.
+ * Precision u4, range [0, 15].
+ */
+struct ipu3_uapi_ae_weight_elem {
+ __u32 cell0:4;
+ __u32 cell1:4;
+ __u32 cell2:4;
+ __u32 cell3:4;
+ __u32 cell4:4;
+ __u32 cell5:4;
+ __u32 cell6:4;
+ __u32 cell7:4;
+} __packed;
+
+/**
+ * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
+ *
+ * @gain_gr: WB gain factor for the gr channels. Default 256.
+ * @gain_r: WB gain factor for the r channel. Default 256.
+ * @gain_b: WB gain factor for the b channel. Default 256.
+ * @gain_gb: WB gain factor for the gb channels. Default 256.
+ * @mat: 4x4 matrix that transforms Bayer quad output from WB to RGB+Y.
+ *
+ * Default:
+ * 128, 0, 0, 0,
+ * 0, 128, 0, 0,
+ * 0, 0, 128, 0,
+ * 0, 0, 0, 128,
+ *
+ * As part of the raw frame pre-process stage, the WB and color conversion need
+ * to be applied to expose the impact of these gain operations.
+ */
+struct ipu3_uapi_ae_ccm {
+ __u16 gain_gr;
+ __u16 gain_r;
+ __u16 gain_b;
+ __u16 gain_gb;
+ __s16 mat[16];
+} __packed;
+
+/**
+ * struct ipu3_uapi_ae_config - AE config
+ *
+ * @grid_cfg: config for auto exposure statistics grid. See struct
+ * &ipu3_uapi_ae_grid_config
+ * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks in the grid.
+ * Each grid cell has a corresponding value in weights LUT called
+ * grid value, global histogram is updated based on grid value and
+ * pixel value.
+ * @ae_ccm: Color convert matrix pre-processing block.
+ *
+ * Calculate AE grid from image resolution, resample ae weights.
+ */
+struct ipu3_uapi_ae_config {
+ struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_weight_elem weights[
+ IPU3_UAPI_AE_WEIGHTS] __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast measurements
+ *
+ * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
+ * anti-symmetry type. A12 is center, A1-A11 are neighbours.
+ * for analyzing low frequency content, used to calculate sum
+ * of gradients in x direction.
+ * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
+ * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
+ * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
+ * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
+ * @y1_coeff_1: Struct
+ * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
+ * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
+ * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
+ * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
+ * @y1_coeff_2: Struct
+ * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
+ * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
+ * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
+ * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
+ * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
+ * 0: positive, 1: negative, default 0.
+ * @y2_coeff_0: Y2, same structure as Y1. For analyzing high frequency content.
+ * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
+ * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
+ * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
+ * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
+ * @y2_coeff_1: Struct
+ * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
+ * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
+ * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
+ * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
+ * @y2_coeff_2: Struct
+ * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
+ * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
+ * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
+ * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
+ * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
+ * 0: positive, 1: negative, default 0.
+ * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to be
+ * used for building histogram. Range [0, 32], default 8.
+ * Rule:
+ * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b + y_gen_rate_gb = 32
+ * A single Y is calculated based on sum of Gr/R/B/Gb based on
+ * their contribution ratio.
+ * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
+ * @y_calc.y_gen_rate_r: Contribution ratio R for Y
+ * @y_calc.y_gen_rate_b: Contribution ratio B for Y
+ * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
+ * @nf: The shift right value that should be applied during the Y1/Y2 filter to
+ * make sure the total memory needed is 2 bytes per grid cell.
+ * @nf.__reserved0: reserved
+ * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
+ * should be log2 of the sum of the abs values of the filter
+ * coeffs, default 7 (2^7 = 128).
+ * @nf.__reserved1: reserved
+ * @nf.y2_nf: Normalization factor for y2, should be log2 of the sum of the
+ * abs values of the filter coeffs.
+ * @nf.__reserved2: reserved
+ */
+struct ipu3_uapi_af_filter_config {
+ struct {
+ __u8 a1;
+ __u8 a2;
+ __u8 a3;
+ __u8 a4;
+ } y1_coeff_0;
+ struct {
+ __u8 a5;
+ __u8 a6;
+ __u8 a7;
+ __u8 a8;
+ } y1_coeff_1;
+ struct {
+ __u8 a9;
+ __u8 a10;
+ __u8 a11;
+ __u8 a12;
+ } y1_coeff_2;
+
+ __u32 y1_sign_vec;
+
+ struct {
+ __u8 a1;
+ __u8 a2;
+ __u8 a3;
+ __u8 a4;
+ } y2_coeff_0;
+ struct {
+ __u8 a5;
+ __u8 a6;
+ __u8 a7;
+ __u8 a8;
+ } y2_coeff_1;
+ struct {
+ __u8 a9;
+ __u8 a10;
+ __u8 a11;
+ __u8 a12;
+ } y2_coeff_2;
+
+ __u32 y2_sign_vec;
+
+ struct {
+ __u8 y_gen_rate_gr;
+ __u8 y_gen_rate_r;
+ __u8 y_gen_rate_b;
+ __u8 y_gen_rate_gb;
+ } y_calc;
+
+ struct {
+ __u32 __reserved0:8;
+ __u32 y1_nf:4;
+ __u32 __reserved1:4;
+ __u32 y2_nf:4;
+ __u32 __reserved2:12;
+ } nf;
+} __packed;
+
+#define IPU3_UAPI_AF_MAX_SETS 24
+#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
+#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
+ (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
+ IPU3_UAPI_AF_MD_ITEM_SIZE)
+#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
+#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
+ (IPU3_UAPI_AF_MAX_SETS * \
+ (IPU3_UAPI_AF_Y_TABLE_SET_SIZE + IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
+ IPU3_UAPI_MAX_STRIPES)
+
+/**
+ * struct ipu3_uapi_af_meta_data - AF meta data
+ *
+ * @y_table: Each color component will be convolved separately with filter1
+ * and filter2 and the result will be summed out and averaged for
+ * each cell.
+ */
+struct ipu3_uapi_af_meta_data {
+ __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE] __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_af_raw_buffer - AF raw buffer
+ *
+ * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus meta data.
+ */
+struct ipu3_uapi_af_raw_buffer {
+ struct ipu3_uapi_af_meta_data meta_data __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_af_config_s - AF config
+ *
+ * @filter_config: AF uses Y1 and Y2 filters as configured in
+ * &ipu3_uapi_af_filter_config
+ * @padding: paddings
+ * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use large
+ * grid size for large image and vice versa.
+ */
+struct ipu3_uapi_af_config_s {
+ struct ipu3_uapi_af_filter_config filter_config __attribute__((aligned(32)));
+ __u8 padding[4];
+ struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_af_config - AF config wrapper
+ *
+ * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
+ */
+struct ipu3_uapi_af_config {
+ struct ipu3_uapi_af_config_s config;
+} __packed;
+
+#define IPU3_UAPI_AWB_FR_MAX_SETS 24
+#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
+#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
+#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
+ (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
+ IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
+#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
+ (IPU3_UAPI_AWB_FR_MAX_SETS * \
+ (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
+ IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) * IPU3_UAPI_MAX_STRIPES)
+
+/**
+ * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
+ *
+ * @bayer_table: Statistics output on the grid after convolving with 1D filter.
+ */
+struct ipu3_uapi_awb_fr_meta_data {
+ __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE] __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
+ *
+ * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
+ */
+struct ipu3_uapi_awb_fr_raw_buffer {
+ struct ipu3_uapi_awb_fr_meta_data meta_data;
+} __packed;
+
+/**
+ * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
+ *
+ * @grid_cfg: grid config, default 16x16.
+ * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
+ * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
+ * Applied on whole image for each Bayer channel separately
+ * by a weighted sum of its 11x1 neighbors.
+ * @__reserved1: reserved
+ * @bayer_sign: sign of filter coeffcients, default 0.
+ * @bayer_nf: normalization factor for the convolution coeffs, to make sure
+ * total memory needed is within pre-determined range.
+ * NF should be the log2 of the sum of the abs values of the
+ * filter coeffs, range [7, 14], default 7.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_awb_fr_config_s {
+ struct ipu3_uapi_grid_config grid_cfg;
+ __u8 bayer_coeff[6];
+ __u16 __reserved1;
+ __u32 bayer_sign;
+ __u8 bayer_nf;
+ __u8 __reserved2[3];
+} __attribute__((aligned(32))) __packed;
+
+/**
+ * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
+ *
+ * @config: See &ipu3_uapi_awb_fr_config_s.
+ */
+struct ipu3_uapi_awb_fr_config {
+ struct ipu3_uapi_awb_fr_config_s config;
+} __packed;
+
+/**
+ * struct ipu3_uapi_4a_config - 4A config
+ *
+ * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
+ * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
+ * @padding: paddings
+ * @af_config: auto focus config &ipu3_uapi_af_config_s
+ * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution 16x16
+ */
+struct ipu3_uapi_4a_config {
+ struct ipu3_uapi_awb_config_s awb_config __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_grid_config ae_grd_config;
+ __u8 padding[20];
+ struct ipu3_uapi_af_config_s af_config;
+ struct ipu3_uapi_awb_fr_config_s awb_fr_config;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
+ *
+ * @num_of_stripes: A single frame is divided into several parts called stripes
+ * due to limitation on line buffer memory.
+ * The separation between the stripes is vertical. Each such
+ * stripe is processed as a single frame by the ISP pipe.
+ * @padding: padding bytes.
+ * @num_sets: number of sets.
+ * @padding1: padding bytes.
+ * @size_of_set: set size.
+ * @padding2: padding bytes.
+ * @bubble_size: is the amount of padding in the bubble expressed in "sets".
+ * @padding3: padding bytes.
+ */
+struct ipu3_uapi_bubble_info {
+ __u32 num_of_stripes __attribute__((aligned(32)));
+ __u8 padding[28];
+ __u32 num_sets;
+ __u8 padding1[28];
+ __u32 size_of_set;
+ __u8 padding2[28];
+ __u32 bubble_size;
+ __u8 padding3[28];
+} __packed;
+
+/*
+ * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
+ */
+struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
+ struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
+ struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
+ struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+/**
+ * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
+ *
+ * @awb_en: auto white balance enable
+ * @padding: padding config
+ * @ae_en: auto exposure enable
+ * @padding1: padding config
+ * @af_en: auto focus enable
+ * @padding2: padding config
+ * @awb_fr_en: awb filter response enable bit
+ * @padding3: padding config
+ */
+struct ipu3_uapi_ff_status {
+ __u32 awb_en __attribute__((aligned(32)));
+ __u8 padding[28];
+ __u32 ae_en;
+ __u8 padding1[28];
+ __u32 af_en;
+ __u8 padding2[28];
+ __u32 awb_fr_en;
+ __u8 padding3[28];
+} __packed;
+
+/**
+ * struct ipu3_uapi_stats_3a - 3A statistics
+ *
+ * @awb_raw_buffer: auto white balance meta data &ipu3_uapi_awb_raw_buffer
+ * @ae_raw_buffer: auto exposure raw data &ipu3_uapi_ae_raw_buffer_aligned
+ * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
+ * @awb_fr_raw_buffer: value as specified by &ipu3_uapi_awb_fr_raw_buffer
+ * @stats_4a_config: 4a statistics config as defined by &ipu3_uapi_4a_config.
+ * @ae_join_buffers: 1 to use ae_raw_buffer.
+ * @padding: padding config
+ * @stats_3a_bubble_per_stripe: a &ipu3_uapi_stats_3a_bubble_info_per_stripe
+ * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
+ */
+struct ipu3_uapi_stats_3a {
+ struct ipu3_uapi_awb_raw_buffer awb_raw_buffer __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_raw_buffer_aligned
+ ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
+ struct ipu3_uapi_af_raw_buffer af_raw_buffer;
+ struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
+ struct ipu3_uapi_4a_config stats_4a_config;
+ __u32 ae_join_buffers;
+ __u8 padding[28];
+ struct ipu3_uapi_stats_3a_bubble_info_per_stripe
+ stats_3a_bubble_per_stripe;
+ struct ipu3_uapi_ff_status stats_3a_status;
+} __packed;
+
+/******************* ipu3_uapi_acc_param *******************/
+
+#define IPU3_UAPI_ISP_VEC_ELEMS 64
+#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
+
+#define IPU3_UAPI_BNR_LUT_SIZE 32
+
+/* number of elements in gamma correction LUT */
+#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
+
+/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
+#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
+#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
+/* Normalization shift aka nf */
+#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
+#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
+
+#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
+#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
+#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
+#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
+
+#define IPU3_UAPI_ANR_LUT_SIZE 26
+#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
+
+#define IPU3_UAPI_LIN_LUT_SIZE 64
+
+/* Bayer Noise Reduction related structs */
+
+/**
+ * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance gains
+ *
+ * @gr: white balance gain for Gr channel.
+ * @r: white balance gain for R channel.
+ * @b: white balance gain for B channel.
+ * @gb: white balance gain for Gb channel.
+ *
+ * Precision u3.13, range [0, 8]. White balance correction is done by applying
+ * a multiplicative gain to each color channels prior to BNR.
+ */
+struct ipu3_uapi_bnr_static_config_wb_gains_config {
+ __u16 gr;
+ __u16 r;
+ __u16 b;
+ __u16 gb;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold config
+ *
+ * @gr: white balance threshold gain for Gr channel.
+ * @r: white balance threshold gain for R channel.
+ * @b: white balance threshold gain for B channel.
+ * @gb: white balance threshold gain for Gb channel.
+ *
+ * Defines the threshold that specifies how different a defect pixel can be from
+ * its neighbors.(used by dynamic defect pixel correction sub block)
+ * Precision u4.4 range [0, 8].
+ */
+struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
+ __u8 gr;
+ __u8 r;
+ __u8 b;
+ __u8 gb;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
+ * coefficients that controls noise threshold
+ *
+ * @cf: Free coefficient for threshold calculation, range [0, 8191], default 0.
+ * @__reserved0: reserved
+ * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
+ * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
+ * default 6.
+ * format: u3.2 (3 most significant bits represent whole number,
+ * 2 least significant bits represent the fractional part
+ * with each count representing 0.25)
+ * e.g 6 in binary format is 00110, that translates to 1.5
+ * @__reserved1: reserved
+ * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
+ * where r is a radius of pixel [row, col] from centor of sensor.
+ * default 14.
+ *
+ * Threshold used to distinguish between noise and details.
+ */
+struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
+ __u32 cf:13;
+ __u32 __reserved0:3;
+ __u32 cg:5;
+ __u32 ci:5;
+ __u32 __reserved1:1;
+ __u32 r_nf:5;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
+ *
+ * @gr: Coefficient defines lens shading gain approximation for gr channel
+ * @r: Coefficient defines lens shading gain approximation for r channel
+ * @b: Coefficient defines lens shading gain approximation for b channel
+ * @gb: Coefficient defines lens shading gain approximation for gb channel
+ *
+ * Parameters for noise model (NM) adaptation of BNR due to shading correction.
+ * All above have precision of u3.3, default to 0.
+ */
+struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
+ __u8 gr;
+ __u8 r;
+ __u8 b;
+ __u8 gb;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center config
+ *
+ * @x_reset: Reset value of X (col start - X center). Precision s12.0.
+ * @__reserved0: reserved
+ * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
+ * @__reserved2: reserved
+ *
+ * Distance from corner to optical center for NM adaptation due to shading
+ * correction (should be calculated based on shading tables)
+ */
+struct ipu3_uapi_bnr_static_config_opt_center_config {
+ __s32 x_reset:13;
+ __u32 __reserved0:3;
+ __s32 y_reset:13;
+ __u32 __reserved2:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup table
+ *
+ * @values: pre-calculated values of square root function.
+ *
+ * LUT implementation of square root operation.
+ */
+struct ipu3_uapi_bnr_static_config_lut_config {
+ __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels (bp)
+ *
+ * @bp_thr_gain: Defines the threshold that specifies how different a
+ * defect pixel can be from its neighbors. Threshold is
+ * dependent on de-noise threshold calculated by algorithm.
+ * Range [4, 31], default 4.
+ * @__reserved0: reserved
+ * @defect_mode: Mode of addressed defect pixels,
+ * 0 - single defect pixel is expected,
+ * 1 - 2 adjacent defect pixels are expected, default 1.
+ * @bp_gain: Defines how 2nd derivation that passes through a defect pixel
+ * is different from 2nd derivations that pass through
+ * neighbor pixels. u4.2, range [0, 256], default 8.
+ * @__reserved1: reserved
+ * @w0_coeff: Blending coefficient of defect pixel correction.
+ * Precision u4, range [0, 8], default 8.
+ * @__reserved2: reserved
+ * @w1_coeff: Enable influence of incorrect defect pixel correction to be
+ * avoided. Precision u4, range [1, 8], default 8.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
+ __u32 bp_thr_gain:5;
+ __u32 __reserved0:2;
+ __u32 defect_mode:1;
+ __u32 bp_gain:6;
+ __u32 __reserved1:18;
+ __u32 w0_coeff:4;
+ __u32 __reserved2:4;
+ __u32 w1_coeff:4;
+ __u32 __reserved3:20;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising config
+ *
+ * @alpha: Weight of central element of smoothing filter.
+ * @beta: Weight of peripheral elements of smoothing filter, default 4.
+ * @gamma: Weight of diagonal elements of smoothing filter, default 4.
+ *
+ * beta and gamma parameter define the strength of the noise removal filter.
+ * All above has precision u0.4, range [0, 0xf]
+ * format: u0.4 (no / zero bits represent whole number,
+ * 4 bits represent the fractional part
+ * with each count representing 0.0625)
+ * e.g 0xf translates to 0.0625x15 = 0.9375
+ *
+ * @__reserved0: reserved
+ * @max_inf: Maximum increase of peripheral or diagonal element influence
+ * relative to the pre-defined value range: [0x5, 0xa]
+ * @__reserved1: reserved
+ * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
+ * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 - enable.
+ * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 - enable.
+ * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
+ __u32 alpha:4;
+ __u32 beta:4;
+ __u32 gamma:4;
+ __u32 __reserved0:4;
+ __u32 max_inf:4;
+ __u32 __reserved1:7;
+ __u32 gd_enable:1;
+ __u32 bpc_enable:1;
+ __u32 bnr_enable:1;
+ __u32 ff_enable:1;
+ __u32 __reserved2:1;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical square
+ *
+ * @x_sqr_reset: Reset value of X^2.
+ * @y_sqr_reset: Reset value of Y^2.
+ *
+ * Please note:
+ *
+ * #. X and Y ref to
+ * &ipu3_uapi_bnr_static_config_opt_center_config
+ * #. Both structs are used in threshold formula to calculate r^2, where r
+ * is a radius of pixel [row, col] from centor of sensor.
+ */
+struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
+ __u32 x_sqr_reset;
+ __u32 y_sqr_reset;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config - BNR static config
+ *
+ * @wb_gains: white balance gains &ipu3_uapi_bnr_static_config_wb_gains_config
+ * @wb_gains_thr: white balance gains threshold as defined by
+ * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
+ * @thr_coeffs: coefficients of threshold
+ * &ipu3_uapi_bnr_static_config_thr_coeffs_config
+ * @thr_ctrl_shd: control of shading threshold
+ * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
+ * @opt_center: optical center &ipu3_uapi_bnr_static_config_opt_center_config
+ *
+ * Above parameters and opt_center_sqr are used for white balance and shading.
+ *
+ * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
+ * @bp_ctrl: detect and remove bad pixels as defined in struct
+ * &ipu3_uapi_bnr_static_config_bp_ctrl_config
+ * @dn_detect_ctrl: detect and remove noise.
+ * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
+ * @column_size: The number of pixels in column.
+ * @opt_center_sqr: Reset value of r^2 to optical center, see
+ * &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
+ */
+struct ipu3_uapi_bnr_static_config {
+ struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
+ struct ipu3_uapi_bnr_static_config_wb_gains_thr_config wb_gains_thr;
+ struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
+ struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
+ struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
+ struct ipu3_uapi_bnr_static_config_lut_config lut;
+ struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
+ struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config dn_detect_ctrl;
+ __u32 column_size;
+ struct ipu3_uapi_bnr_static_config_opt_center_sqr_config opt_center_sqr;
+} __packed;
+
+/**
+ * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green disparity
+ *
+ * @gd_red: Shading gain coeff for gr disparity level in bright red region.
+ * Precision u0.6, default 4(0.0625).
+ * @__reserved0: reserved
+ * @gd_green: Shading gain coeff for gr disparity level in bright green
+ * region. Precision u0.6, default 4(0.0625).
+ * @__reserved1: reserved
+ * @gd_blue: Shading gain coeff for gr disparity level in bright blue region.
+ * Precision u0.6, default 4(0.0625).
+ * @__reserved2: reserved
+ * @gd_black: Maximal green disparity level in dark region (stronger disparity
+ * assumed to be image detail). Precision u14, default 80.
+ * @__reserved3: reserved
+ * @gd_shading: Change maximal green disparity level according to square
+ * distance from image center.
+ * @__reserved4: reserved
+ * @gd_support: Lower bound for the number of second green color pixels in
+ * current pixel neighborhood with less than threshold difference
+ * from it.
+ *
+ * The shading gain coeff of red, green, blue and black are used to calculate
+ * threshold given a pixel's color value and its coordinates in the image.
+ *
+ * @__reserved5: reserved
+ * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
+ * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
+ */
+struct ipu3_uapi_bnr_static_config_green_disparity {
+ __u32 gd_red:6;
+ __u32 __reserved0:2;
+ __u32 gd_green:6;
+ __u32 __reserved1:2;
+ __u32 gd_blue:6;
+ __u32 __reserved2:10;
+ __u32 gd_black:14;
+ __u32 __reserved3:2;
+ __u32 gd_shading:7;
+ __u32 __reserved4:1;
+ __u32 gd_support:2;
+ __u32 __reserved5:1;
+ __u32 gd_clip:1;
+ __u32 gd_central_weight:4;
+} __packed;
+
+/**
+ * struct ipu3_uapi_dm_config - De-mosaic parameters
+ *
+ * @dm_en: de-mosaic enable.
+ * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
+ * @fcc_en: False color correction (FCC) enable flag. Default 0.
+ * @__reserved0: reserved
+ * @frame_width: do not care
+ * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation of
+ * complementary color in Hamilton-Adams interpolation).
+ * u5, range [0, 31], default 8.
+ * @__reserved1: reserved
+ * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity metric
+ * in calculation of final homogeneity metric.
+ * u5, range [0, 31], default 7.
+ * @__reserved2: reserved
+ * @cr_param1: First parameter that defines Checker artifact removal
+ * feature gain.Precision u5, range [0, 31], default 8.
+ * @__reserved3: reserved
+ * @cr_param2: Second parameter that defines Checker artifact removal
+ * feature gain. Precision u5, range [0, 31], default 8.
+ * @__reserved4: reserved
+ * @coring_param: Defines power of false color correction operation.
+ * low for preserving edge colors, high for preserving gray
+ * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
+ * @__reserved5: reserved
+ *
+ * The demosaic fixed function block is responsible to covert Bayer(mosaiced)
+ * images into color images based on demosaicing algorithm.
+ */
+struct ipu3_uapi_dm_config {
+ __u32 dm_en:1;
+ __u32 ch_ar_en:1;
+ __u32 fcc_en:1;
+ __u32 __reserved0:13;
+ __u32 frame_width:16;
+
+ __u32 gamma_sc:5;
+ __u32 __reserved1:3;
+ __u32 lc_ctrl:5;
+ __u32 __reserved2:3;
+ __u32 cr_param1:5;
+ __u32 __reserved3:3;
+ __u32 cr_param2:5;
+ __u32 __reserved4:3;
+
+ __u32 coring_param:5;
+ __u32 __reserved5:27;
+} __packed;
+
+/**
+ * struct ipu3_uapi_ccm_mat_config - Color correction matrix
+ *
+ * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
+ * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
+ * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
+ * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
+ * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
+ * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
+ * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
+ * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
+ * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
+ * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
+ * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
+ * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
+ *
+ * Transform sensor specific color space to standard sRGB by applying 3x3 matrix
+ * and adding a bias vector O. The transformation is basically a rotation and
+ * translation in the 3-dimensional color spaces. Here are the defaults:
+ *
+ * 9775, -2671, 1087, 0
+ * -1071, 8303, 815, 0
+ * -23, -7887, 16103, 0
+ */
+struct ipu3_uapi_ccm_mat_config {
+ __s16 coeff_m11;
+ __s16 coeff_m12;
+ __s16 coeff_m13;
+ __s16 coeff_o_r;
+ __s16 coeff_m21;
+ __s16 coeff_m22;
+ __s16 coeff_m23;
+ __s16 coeff_o_g;
+ __s16 coeff_m31;
+ __s16 coeff_m32;
+ __s16 coeff_m33;
+ __s16 coeff_o_b;
+} __packed;
+
+/**
+ * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
+ *
+ * @enable: gamma correction enable.
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_gamma_corr_ctrl {
+ __u32 enable:1;
+ __u32 __reserved:31;
+} __packed;
+
+/**
+ * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping implemented as LUT.
+ *
+ * @lut: 256 tabulated values of the gamma function. LUT[1].. LUT[256]
+ * format u13.0, range [0, 8191].
+ *
+ * The tone mapping operation is done by a Piece wise linear graph
+ * that is implemented as a lookup table(LUT). The pixel component input
+ * intensity is the X-axis of the graph which is the table entry.
+ */
+struct ipu3_uapi_gamma_corr_lut {
+ __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
+} __packed;
+
+/**
+ * struct ipu3_uapi_gamma_config - Gamma config
+ *
+ * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
+ * @gc_lut: lookup table of gamma correction &ipu3_uapi_gamma_corr_lut
+ */
+struct ipu3_uapi_gamma_config {
+ struct ipu3_uapi_gamma_corr_ctrl gc_ctrl __attribute__((aligned(32)));
+ struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
+ *
+ * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1], default 1.
+ * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1], default 0.
+ * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1], default 0.
+ * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
+ * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1], default 0.
+ * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1], default 1.
+ * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1], default 0.
+ * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
+ * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1], default 0.
+ * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1], default 0.
+ * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1], default 1.
+ * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
+ *
+ * To transform each pixel from RGB to YUV (Y - brightness/luminance,
+ * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding an
+ * optional bias 3x1 vector.
+ */
+struct ipu3_uapi_csc_mat_config {
+ __s16 coeff_c11;
+ __s16 coeff_c12;
+ __s16 coeff_c13;
+ __s16 coeff_b1;
+ __s16 coeff_c21;
+ __s16 coeff_c22;
+ __s16 coeff_c23;
+ __s16 coeff_b2;
+ __s16 coeff_c31;
+ __s16 coeff_c32;
+ __s16 coeff_c33;
+ __s16 coeff_b3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_cds_params - Chroma down-scaling
+ *
+ * @ds_c00: range [0, 3]
+ * @ds_c01: range [0, 3]
+ * @ds_c02: range [0, 3]
+ * @ds_c03: range [0, 3]
+ * @ds_c10: range [0, 3]
+ * @ds_c11: range [0, 3]
+ * @ds_c12: range [0, 3]
+ * @ds_c13: range [0, 3]
+ *
+ * In case user does not provide, above 4x2 filter will use following defaults:
+ * 1, 3, 3, 1,
+ * 1, 3, 3, 1,
+ *
+ * @ds_nf: Normalization factor for Chroma output downscaling filter,
+ * range 0,4, default 2.
+ * @__reserved0: reserved
+ * @csc_en: Color space conversion enable
+ * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_cds_params {
+ __u32 ds_c00:2;
+ __u32 ds_c01:2;
+ __u32 ds_c02:2;
+ __u32 ds_c03:2;
+ __u32 ds_c10:2;
+ __u32 ds_c11:2;
+ __u32 ds_c12:2;
+ __u32 ds_c13:2;
+ __u32 ds_nf:5;
+ __u32 __reserved0:3;
+ __u32 csc_en:1;
+ __u32 uv_bin_output:1;
+ __u32 __reserved1:6;
+} __packed;
+
+/**
+ * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
+ *
+ * @width: Grid horizontal dimensions, u8, [8, 128], default 73
+ * @height: Grid vertical dimensions, u8, [8, 128], default 56
+ * @block_width_log2: Log2 of the width of the grid cell in pixel count
+ * u4, [0, 15], default value 5.
+ * @__reserved0: reserved
+ * @block_height_log2: Log2 of the height of the grid cell in pixel count
+ * u4, [0, 15], default value 6.
+ * @__reserved1: reserved
+ * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
+ * (with SHD_MAX_CELLS_PER_SET = 146).
+ * @x_start: X value of top left corner of sensor relative to ROI
+ * u12, [-4096, 0]. default 0, only negative values.
+ * @y_start: Y value of top left corner of sensor relative to ROI
+ * u12, [-4096, 0]. default 0, only negative values.
+ */
+struct ipu3_uapi_shd_grid_config {
+ /* reg 0 */
+ __u8 width;
+ __u8 height;
+ __u8 block_width_log2:3;
+ __u8 __reserved0:1;
+ __u8 block_height_log2:3;
+ __u8 __reserved1:1;
+ __u8 grid_height_per_slice;
+ /* reg 1 */
+ __s16 x_start;
+ __s16 y_start;
+} __packed;
+
+/**
+ * struct ipu3_uapi_shd_general_config - Shading general config
+ *
+ * @init_set_vrt_offst_ul: set vertical offset,
+ * y_start >> block_height_log2 % grid_height_per_slice.
+ * @shd_enable: shading enable.
+ * @gain_factor: Gain factor. Shift calculated anti shading value. Precision u2.
+ * 0x0 - gain factor [1, 5], means no shift interpolated value.
+ * 0x1 - gain factor [1, 9], means shift interpolated by 1.
+ * 0x2 - gain factor [1, 17], means shift interpolated by 2.
+ * @__reserved: reserved
+ *
+ * Correction is performed by multiplying a gain factor for each of the 4 Bayer
+ * channels as a function of the pixel location in the sensor.
+ */
+struct ipu3_uapi_shd_general_config {
+ __u32 init_set_vrt_offst_ul:8;
+ __u32 shd_enable:1;
+ __u32 gain_factor:2;
+ __u32 __reserved:21;
+} __packed;
+
+/**
+ * struct ipu3_uapi_shd_black_level_config - Black level correction
+ *
+ * @bl_r: Bios values for green red. s11 range [-2048, 2047].
+ * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
+ * @bl_gb: Bios values for red. s11 range [-2048, 2047].
+ * @bl_b: Bios values for blue. s11 range [-2048, 2047].
+ */
+struct ipu3_uapi_shd_black_level_config {
+ __s16 bl_r;
+ __s16 bl_gr;
+ __s16 bl_gb;
+ __s16 bl_b;
+} __packed;
+
+/**
+ * struct ipu3_uapi_shd_config_static - Shading config static
+ *
+ * @grid: shading grid config &ipu3_uapi_shd_grid_config
+ * @general: shading general config &ipu3_uapi_shd_general_config
+ * @black_level: black level config for shading correction as defined by
+ * &ipu3_uapi_shd_black_level_config
+ */
+struct ipu3_uapi_shd_config_static {
+ struct ipu3_uapi_shd_grid_config grid;
+ struct ipu3_uapi_shd_general_config general;
+ struct ipu3_uapi_shd_black_level_config black_level;
+} __packed;
+
+/**
+ * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
+ *
+ * @sets: array
+ * @sets.r_and_gr: Red and GreenR Lookup table.
+ * @sets.r_and_gr.r: Red shading factor.
+ * @sets.r_and_gr.gr: GreenR shading factor.
+ * @sets.__reserved1: reserved
+ * @sets.gb_and_b: GreenB and Blue Lookup table.
+ * @sets.gb_and_b.gb: GreenB shading factor.
+ * @sets.gb_and_b.b: Blue shading factor.
+ * @sets.__reserved2: reserved
+ *
+ * Map to shading correction LUT register set.
+ */
+struct ipu3_uapi_shd_lut {
+ struct {
+ struct {
+ __u16 r;
+ __u16 gr;
+ } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
+ __u8 __reserved1[24];
+ struct {
+ __u16 gb;
+ __u16 b;
+ } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
+ __u8 __reserved2[24];
+ } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_shd_config - Shading config
+ *
+ * @shd: shading static config, see &ipu3_uapi_shd_config_static
+ * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
+ */
+struct ipu3_uapi_shd_config {
+ struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
+ struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
+} __packed;
+
+/* Image Enhancement Filter directed */
+
+/**
+ * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
+ *
+ * @x0: X0 point of Config Unit, u9.0, default 0.
+ * @x1: X1 point of Config Unit, u9.0, default 0.
+ * @a01: Slope A of Config Unit, s4.4, default 0.
+ * @b01: Always 0.
+ *
+ * Calculate weight for blending directed and non-directed denoise elements
+ *
+ * Note:
+ * Each instance of Config Unit needs X coordinate of n points and
+ * slope A factor between points calculated by driver based on calibration
+ * parameters.
+ */
+struct ipu3_uapi_iefd_cux2 {
+ __u32 x0:9;
+ __u32 x1:9;
+ __u32 a01:9;
+ __u32 b01:5; /* NOTE: hardcoded to zero */
+} __packed;
+
+/**
+ * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed sharpening
+ * element, Config Unit 6 for edge detail (ED).
+ *
+ * @x0: X coordinate of point 0, u9.0, default 0.
+ * @x1: X coordinate of point 1, u9.0, default 0.
+ * @x2: X coordinate of point 2, u9.0, default 0.
+ * @__reserved0: reserved
+ * @x3: X coordinate of point 3, u9.0, default 0.
+ * @x4: X coordinate of point 4, u9.0, default 0.
+ * @x5: X coordinate of point 5, u9.0, default 0.
+ * @__reserved1: reserved
+ * @a01: slope A points 01, s4.4, default 0.
+ * @a12: slope A points 12, s4.4, default 0.
+ * @a23: slope A points 23, s4.4, default 0.
+ * @__reserved2: reserved
+ * @a34: slope A points 34, s4.4, default 0.
+ * @a45: slope A points 45, s4.4, default 0.
+ * @__reserved3: reserved
+ * @b01: slope B points 01, s4.4, default 0.
+ * @b12: slope B points 12, s4.4, default 0.
+ * @b23: slope B points 23, s4.4, default 0.
+ * @__reserved4: reserved
+ * @b34: slope B points 34, s4.4, default 0.
+ * @b45: slope B points 45, s4.4, default 0.
+ * @__reserved5: reserved
+ */
+struct ipu3_uapi_iefd_cux6_ed {
+ __u32 x0:9;
+ __u32 x1:9;
+ __u32 x2:9;
+ __u32 __reserved0:5;
+
+ __u32 x3:9;
+ __u32 x4:9;
+ __u32 x5:9;
+ __u32 __reserved1:5;
+
+ __u32 a01:9;
+ __u32 a12:9;
+ __u32 a23:9;
+ __u32 __reserved2:5;
+
+ __u32 a34:9;
+ __u32 a45:9;
+ __u32 __reserved3:14;
+
+ __u32 b01:9;
+ __u32 b12:9;
+ __u32 b23:9;
+ __u32 __reserved4:5;
+
+ __u32 b34:9;
+ __u32 b45:9;
+ __u32 __reserved5:14;
+} __packed;
+
+/**
+ * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
+ * element apply.
+ * @x0: X0 point of Config Unit, u9.0, default 0.
+ * @x1: X1 point of Config Unit, u9.0, default 0.
+ * @a01: Slope A of Config Unit, s4.4, default 0.
+ * @__reserved1: reserved
+ * @b01: offset B0 of Config Unit, u7.0, default 0.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_iefd_cux2_1 {
+ __u32 x0:9;
+ __u32 x1:9;
+ __u32 a01:9;
+ __u32 __reserved1:5;
+
+ __u32 b01:8;
+ __u32 __reserved2:24;
+} __packed;
+
+/**
+ * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed sharpening
+ * element.
+ *
+ * @x0: X0 point of Config Unit, u9.0, default 0.
+ * @x1: X1 point of Config Unit, u9.0, default 0.
+ * @x2: X2 point of Config Unit, u9.0, default 0.
+ * @__reserved0: reserved
+ * @x3: X3 point of Config Unit, u9.0, default 0.
+ * @a01: Slope A0 of Config Unit, s4.4, default 0.
+ * @a12: Slope A1 of Config Unit, s4.4, default 0.
+ * @__reserved1: reserved
+ * @a23: Slope A2 of Config Unit, s4.4, default 0.
+ * @b01: Offset B0 of Config Unit, s7.0, default 0.
+ * @b12: Offset B1 of Config Unit, s7.0, default 0.
+ * @__reserved2: reserved
+ * @b23: Offset B2 of Config Unit, s7.0, default 0.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_iefd_cux4 {
+ __u32 x0:9;
+ __u32 x1:9;
+ __u32 x2:9;
+ __u32 __reserved0:5;
+
+ __u32 x3:9;
+ __u32 a01:9;
+ __u32 a12:9;
+ __u32 __reserved1:5;
+
+ __u32 a23:9;
+ __u32 b01:8;
+ __u32 b12:8;
+ __u32 __reserved2:7;
+
+ __u32 b23:8;
+ __u32 __reserved3:24;
+} __packed;
+
+/**
+ * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
+ *
+ * @x0: x0 points of Config Unit radial, u8.0
+ * @x1: x1 points of Config Unit radial, u8.0
+ * @x2: x2 points of Config Unit radial, u8.0
+ * @x3: x3 points of Config Unit radial, u8.0
+ * @x4: x4 points of Config Unit radial, u8.0
+ * @x5: x5 points of Config Unit radial, u8.0
+ * @__reserved1: reserved
+ * @a01: Slope A of Config Unit radial, s7.8
+ * @a12: Slope A of Config Unit radial, s7.8
+ * @a23: Slope A of Config Unit radial, s7.8
+ * @a34: Slope A of Config Unit radial, s7.8
+ * @a45: Slope A of Config Unit radial, s7.8
+ * @__reserved2: reserved
+ * @b01: Slope B of Config Unit radial, s9.0
+ * @b12: Slope B of Config Unit radial, s9.0
+ * @b23: Slope B of Config Unit radial, s9.0
+ * @__reserved4: reserved
+ * @b34: Slope B of Config Unit radial, s9.0
+ * @b45: Slope B of Config Unit radial, s9.0
+ * @__reserved5: reserved
+ */
+struct ipu3_uapi_iefd_cux6_rad {
+ __u32 x0:8;
+ __u32 x1:8;
+ __u32 x2:8;
+ __u32 x3:8;
+
+ __u32 x4:8;
+ __u32 x5:8;
+ __u32 __reserved1:16;
+
+ __u32 a01:16;
+ __u32 a12:16;
+
+ __u32 a23:16;
+ __u32 a34:16;
+
+ __u32 a45:16;
+ __u32 __reserved2:16;
+
+ __u32 b01:10;
+ __u32 b12:10;
+ __u32 b23:10;
+ __u32 __reserved4:2;
+
+ __u32 b34:10;
+ __u32 b45:10;
+ __u32 __reserved5:12;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
+ *
+ * @cu_1: calculate weight for blending directed and
+ * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
+ * @cu_ed: calculate power of non-directed sharpening element, see
+ * &ipu3_uapi_iefd_cux6_ed
+ * @cu_3: calculate weight for blending directed and
+ * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
+ * @cu_5: calculate power of non-directed denoise element apply, use
+ * &ipu3_uapi_iefd_cux2_1
+ * @cu_6: calculate power of non-directed sharpening element. See
+ * &ipu3_uapi_iefd_cux4
+ * @cu_7: calculate weight for blending directed and
+ * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
+ * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
+ * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
+ * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
+ */
+struct ipu3_uapi_yuvp1_iefd_cfg_units {
+ struct ipu3_uapi_iefd_cux2 cu_1;
+ struct ipu3_uapi_iefd_cux6_ed cu_ed;
+ struct ipu3_uapi_iefd_cux2 cu_3;
+ struct ipu3_uapi_iefd_cux2_1 cu_5;
+ struct ipu3_uapi_iefd_cux4 cu_6;
+ struct ipu3_uapi_iefd_cux2 cu_7;
+ struct ipu3_uapi_iefd_cux4 cu_unsharp;
+ struct ipu3_uapi_iefd_cux6_rad cu_radial;
+ struct ipu3_uapi_iefd_cux2 cu_vssnlm;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
+ *
+ * @horver_diag_coeff: Gradiant compensation, coefficient that compensates for
+ * different distance for vertical / horizontal and diagonal
+ * * gradient calculation (~1/sqrt(2)).
+ * @__reserved0: reserved
+ * @clamp_stitch: Slope to stitch between clamped and unclamped edge values
+ * @__reserved1: reserved
+ * @direct_metric_update: Update coeff for direction metric
+ * @__reserved2: reserved
+ * @ed_horver_diag_coeff: Radial Coefficient that compensates for
+ * different distance for vertical/horizontal and
+ * diagonal gradient calculation (~1/sqrt(2))
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_yuvp1_iefd_config_s {
+ __u32 horver_diag_coeff:7;
+ __u32 __reserved0:1;
+ __u32 clamp_stitch:6;
+ __u32 __reserved1:2;
+ __u32 direct_metric_update:5;
+ __u32 __reserved2:3;
+ __u32 ed_horver_diag_coeff:7;
+ __u32 __reserved3:1;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
+ *
+ * @iefd_en: Enable IEFd
+ * @denoise_en: Enable denoise
+ * @direct_smooth_en: Enable directional smooth
+ * @rad_en: Enable radial update
+ * @vssnlm_en: Enable VSSNLM output filter
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_yuvp1_iefd_control {
+ __u32 iefd_en:1;
+ __u32 denoise_en:1;
+ __u32 direct_smooth_en:1;
+ __u32 rad_en:1;
+ __u32 vssnlm_en:1;
+ __u32 __reserved:27;
+} __packed;
+
+/**
+ * struct ipu3_uapi_sharp_cfg - Sharpening config
+ *
+ * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
+ * @__reserved0: reserved
+ * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
+ * @__reserved1: reserved
+ * @nega_lmt_dir: Sharpening limit for negative overshoots for direction (edge).
+ * @__reserved2: reserved
+ * @posi_lmt_dir: Sharpening limit for positive overshoots for direction (edge).
+ * @__reserved3: reserved
+ *
+ * Fixed point type u13.0, range [0, 8191].
+ */
+struct ipu3_uapi_sharp_cfg {
+ __u32 nega_lmt_txt:13;
+ __u32 __reserved0:19;
+ __u32 posi_lmt_txt:13;
+ __u32 __reserved1:19;
+ __u32 nega_lmt_dir:13;
+ __u32 __reserved2:19;
+ __u32 posi_lmt_dir:13;
+ __u32 __reserved3:19;
+} __packed;
+
+/**
+ * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
+ *
+ * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64], default 64.
+ * @__reserved0: reserved
+ * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64], default 0.
+ * @__reserved1: reserved
+ * @ndir_dns_powr: Power of non-direct denoising,
+ * Precision u1.6, range [0, 64], default 64.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_far_w {
+ __u32 dir_shrp:7;
+ __u32 __reserved0:1;
+ __u32 dir_dns:7;
+ __u32 __reserved1:1;
+ __u32 ndir_dns_powr:7;
+ __u32 __reserved2:9;
+} __packed;
+
+/**
+ * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
+ *
+ * @unsharp_weight: Unsharp mask blending weight.
+ * u1.6, range [0, 64], default 16.
+ * 0 - disabled, 64 - use only unsharp.
+ * @__reserved0: reserved
+ * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511], default 0.
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_unsharp_cfg {
+ __u32 unsharp_weight:7;
+ __u32 __reserved0:1;
+ __u32 unsharp_amount:9;
+ __u32 __reserved1:15;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
+ *
+ * @cfg: sharpness config &ipu3_uapi_sharp_cfg
+ * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
+ * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
+ * neighbors (8 pixels called Near), and the second order neighborhood
+ * around them (16 pixels called Far).
+ * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
+ */
+struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
+ struct ipu3_uapi_sharp_cfg cfg;
+ struct ipu3_uapi_far_w far_w;
+ struct ipu3_uapi_unsharp_cfg unshrp_cfg;
+} __packed;
+
+/**
+ * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
+ *
+ * @c00: Coeff11, s0.8, range [-255, 255], default 1.
+ * @c01: Coeff12, s0.8, range [-255, 255], default 5.
+ * @c02: Coeff13, s0.8, range [-255, 255], default 9.
+ * @__reserved: reserved
+ *
+ * Configurable registers for common sharpening support.
+ */
+struct ipu3_uapi_unsharp_coef0 {
+ __u32 c00:9;
+ __u32 c01:9;
+ __u32 c02:9;
+ __u32 __reserved:5;
+} __packed;
+
+/**
+ * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
+ *
+ * @c11: Coeff22, s0.8, range [-255, 255], default 29.
+ * @c12: Coeff23, s0.8, range [-255, 255], default 55.
+ * @c22: Coeff33, s0.8, range [-255, 255], default 96.
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_unsharp_coef1 {
+ __u32 c11:9;
+ __u32 c12:9;
+ __u32 c22:9;
+ __u32 __reserved:5;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
+ *
+ * @unsharp_coef0: unsharp coefficient 0 config. See &ipu3_uapi_unsharp_coef0
+ * @unsharp_coef1: unsharp coefficient 1 config. See &ipu3_uapi_unsharp_coef1
+ */
+struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
+ struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
+ struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
+} __packed;
+
+/**
+ * struct ipu3_uapi_radial_reset_xy - Radial coordinate reset
+ *
+ * @x: Radial reset of x coordinate. Precision s12, [-4095, 4095], default 0.
+ * @__reserved0: reserved
+ * @y: Radial center y coordinate. Precision s12, [-4095, 4095], default 0.
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_radial_reset_xy {
+ __s32 x:13;
+ __u32 __reserved0:3;
+ __s32 y:13;
+ __u32 __reserved1:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_radial_reset_x2 - Radial X^2 reset
+ *
+ * @x2: Radial reset of x^2 coordinate. Precision u24, default 0.
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_radial_reset_x2 {
+ __u32 x2:24;
+ __u32 __reserved:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_radial_reset_y2 - Radial Y^2 reset
+ *
+ * @y2: Radial reset of y^2 coordinate. Precision u24, default 0.
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_radial_reset_y2 {
+ __u32 y2:24;
+ __u32 __reserved:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_radial_cfg - Radial config
+ *
+ * @rad_nf: Radial. R^2 normalization factor is scale down by 2^ - (15 + scale)
+ * @__reserved0: reserved
+ * @rad_inv_r2: Radial R^-2 normelized to (0.5..1), Prec' u7, range [0, 127].
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_radial_cfg {
+ __u32 rad_nf:4;
+ __u32 __reserved0:4;
+ __u32 rad_inv_r2:7;
+ __u32 __reserved1:17;
+} __packed;
+
+/**
+ * struct ipu3_uapi_rad_far_w - Radial FAR sub-group
+ *
+ * @rad_dir_far_sharp_w: Weight of wide direct sharpening, u1.6, range [0, 64],
+ * default 64.
+ * @rad_dir_far_dns_w: Weight of wide direct denoising, u1.6, range [0, 64],
+ * default 0.
+ * @rad_ndir_far_dns_power: power of non-direct sharpening, u1.6, range [0, 64],
+ * default 0.
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_rad_far_w {
+ __u32 rad_dir_far_sharp_w:8;
+ __u32 rad_dir_far_dns_w:8;
+ __u32 rad_ndir_far_dns_power:8;
+ __u32 __reserved:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_cu_cfg0 - Radius Config Unit cfg0 register
+ *
+ * @cu6_pow: Power of CU6. Power of non-direct sharpening, u3.4.
+ * @__reserved0: reserved
+ * @cu_unsharp_pow: Power of unsharp mask, u2.4.
+ * @__reserved1: reserved
+ * @rad_cu6_pow: Radial/corner CU6. Directed sharpening power, u3.4.
+ * @__reserved2: reserved
+ * @rad_cu_unsharp_pow: Radial power of unsharp mask, u2.4.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_cu_cfg0 {
+ __u32 cu6_pow:7;
+ __u32 __reserved0:1;
+ __u32 cu_unsharp_pow:7;
+ __u32 __reserved1:1;
+ __u32 rad_cu6_pow:7;
+ __u32 __reserved2:1;
+ __u32 rad_cu_unsharp_pow:6;
+ __u32 __reserved3:2;
+} __packed;
+
+/**
+ * struct ipu3_uapi_cu_cfg1 - Radius Config Unit cfg1 register
+ *
+ * @rad_cu6_x1: X1 point of Config Unit 6, precision u9.0.
+ * @__reserved0: reserved
+ * @rad_cu_unsharp_x1: X1 point for Config Unit unsharp for radial/corner point
+ * precision u9.0.
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_cu_cfg1 {
+ __u32 rad_cu6_x1:9;
+ __u32 __reserved0:1;
+ __u32 rad_cu_unsharp_x1:9;
+ __u32 __reserved1:13;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_rad_cfg - IEFd parameters changed radially over
+ * the picture plain.
+ *
+ * @reset_xy: reset xy value in radial calculation. &ipu3_uapi_radial_reset_xy
+ * @reset_x2: reset x square value in radial calculation. See struct
+ * &ipu3_uapi_radial_reset_x2
+ * @reset_y2: reset y square value in radial calculation. See struct
+ * &ipu3_uapi_radial_reset_y2
+ * @cfg: radial config defined in &ipu3_uapi_radial_cfg
+ * @rad_far_w: weight for wide range radial. &ipu3_uapi_rad_far_w
+ * @cu_cfg0: configuration unit 0. See &ipu3_uapi_cu_cfg0
+ * @cu_cfg1: configuration unit 1. See &ipu3_uapi_cu_cfg1
+ */
+struct ipu3_uapi_yuvp1_iefd_rad_cfg {
+ struct ipu3_uapi_radial_reset_xy reset_xy;
+ struct ipu3_uapi_radial_reset_x2 reset_x2;
+ struct ipu3_uapi_radial_reset_y2 reset_y2;
+ struct ipu3_uapi_radial_cfg cfg;
+ struct ipu3_uapi_rad_far_w rad_far_w;
+ struct ipu3_uapi_cu_cfg0 cu_cfg0;
+ struct ipu3_uapi_cu_cfg1 cu_cfg1;
+} __packed;
+
+/* Vssnlm - Very small scale non-local mean algorithm */
+
+/**
+ * struct ipu3_uapi_vss_lut_x - Vssnlm LUT x0/x1/x2
+ *
+ * @vs_x0: Vssnlm LUT x0, precision u8, range [0, 255], default 16.
+ * @vs_x1: Vssnlm LUT x1, precision u8, range [0, 255], default 32.
+ * @vs_x2: Vssnlm LUT x2, precision u8, range [0, 255], default 64.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_vss_lut_x {
+ __u32 vs_x0:8;
+ __u32 vs_x1:8;
+ __u32 vs_x2:8;
+ __u32 __reserved2:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_vss_lut_y - Vssnlm LUT y0/y1/y2
+ *
+ * @vs_y1: Vssnlm LUT y1, precision u4, range [0, 8], default 1.
+ * @__reserved0: reserved
+ * @vs_y2: Vssnlm LUT y2, precision u4, range [0, 8], default 3.
+ * @__reserved1: reserved
+ * @vs_y3: Vssnlm LUT y3, precision u4, range [0, 8], default 8.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_vss_lut_y {
+ __u32 vs_y1:4;
+ __u32 __reserved0:4;
+ __u32 vs_y2:4;
+ __u32 __reserved1:4;
+ __u32 vs_y3:4;
+ __u32 __reserved2:12;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_vssnlm_cf - IEFd Vssnlm Lookup table
+ *
+ * @vss_lut_x: vss lookup table. See &ipu3_uapi_vss_lut_x description
+ * @vss_lut_y: vss lookup table. See &ipu3_uapi_vss_lut_y description
+ */
+struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg {
+ struct ipu3_uapi_vss_lut_x vss_lut_x;
+ struct ipu3_uapi_vss_lut_y vss_lut_y;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_iefd_config - IEFd config
+ *
+ * @units: configuration unit setting, &ipu3_uapi_yuvp1_iefd_cfg_units
+ * @config: configuration, as defined by &ipu3_uapi_yuvp1_iefd_config_s
+ * @control: control setting, as defined by &ipu3_uapi_yuvp1_iefd_control
+ * @sharp: sharpness setting, as defined by &ipu3_uapi_yuvp1_iefd_shrp_cfg
+ * @unsharp: unsharpness setting, as defined by &ipu3_uapi_yuvp1_iefd_unshrp_cfg
+ * @rad: radial setting, as defined by &ipu3_uapi_yuvp1_iefd_rad_cfg
+ * @vsslnm: vsslnm setting, as defined by &ipu3_uapi_yuvp1_iefd_vssnlm_cfg
+ */
+struct ipu3_uapi_yuvp1_iefd_config {
+ struct ipu3_uapi_yuvp1_iefd_cfg_units units;
+ struct ipu3_uapi_yuvp1_iefd_config_s config;
+ struct ipu3_uapi_yuvp1_iefd_control control;
+ struct ipu3_uapi_yuvp1_iefd_shrp_cfg sharp;
+ struct ipu3_uapi_yuvp1_iefd_unshrp_cfg unsharp;
+ struct ipu3_uapi_yuvp1_iefd_rad_cfg rad;
+ struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg vsslnm;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_yds_config - Y Down-Sampling config
+ *
+ * @c00: range [0, 3], default 0x0
+ * @c01: range [0, 3], default 0x1
+ * @c02: range [0, 3], default 0x1
+ * @c03: range [0, 3], default 0x0
+ * @c10: range [0, 3], default 0x0
+ * @c11: range [0, 3], default 0x1
+ * @c12: range [0, 3], default 0x1
+ * @c13: range [0, 3], default 0x0
+ *
+ * Above are 4x2 filter coefficients for chroma output downscaling.
+ *
+ * @norm_factor: Normalization factor, range [0, 4], default 2
+ * 0 - divide by 1
+ * 1 - divide by 2
+ * 2 - divide by 4
+ * 3 - divide by 8
+ * 4 - divide by 16
+ * @__reserved0: reserved
+ * @bin_output: Down sampling on Luma channel in two optional modes
+ * 0 - Bin output 4.2.0 (default), 1 output 4.2.2.
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_yuvp1_yds_config {
+ __u32 c00:2;
+ __u32 c01:2;
+ __u32 c02:2;
+ __u32 c03:2;
+ __u32 c10:2;
+ __u32 c11:2;
+ __u32 c12:2;
+ __u32 c13:2;
+ __u32 norm_factor:5;
+ __u32 __reserved0:4;
+ __u32 bin_output:1;
+ __u32 __reserved1:6;
+} __packed;
+
+/* Chroma Noise Reduction */
+
+/**
+ * struct ipu3_uapi_yuvp1_chnr_enable_config - Chroma noise reduction enable
+ *
+ * @enable: enable/disable chroma noise reduction
+ * @yuv_mode: 0 - YUV420, 1 - YUV422
+ * @__reserved0: reserved
+ * @col_size: number of columns in the frame, max width is 2560
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_yuvp1_chnr_enable_config {
+ __u32 enable:1;
+ __u32 yuv_mode:1;
+ __u32 __reserved0:14;
+ __u32 col_size:12;
+ __u32 __reserved1:4;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_chnr_coring_config - Coring thresholds for UV
+ *
+ * @u: U coring level, u0.13, range [0.0, 1.0], default 0.0
+ * @__reserved0: reserved
+ * @v: V coring level, u0.13, range [0.0, 1.0], default 0.0
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_yuvp1_chnr_coring_config {
+ __u32 u:13;
+ __u32 __reserved0:3;
+ __u32 v:13;
+ __u32 __reserved1:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_chnr_sense_gain_config - Chroma noise reduction gains
+ *
+ * All sensitivity gain parameters have precision u13.0, range [0, 8191].
+ *
+ * @vy: Sensitivity of horizontal edge of Y, default 100
+ * @vu: Sensitivity of horizontal edge of U, default 100
+ * @vv: Sensitivity of horizontal edge of V, default 100
+ * @__reserved0: reserved
+ * @hy: Sensitivity of vertical edge of Y, default 50
+ * @hu: Sensitivity of vertical edge of U, default 50
+ * @hv: Sensitivity of vertical edge of V, default 50
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_yuvp1_chnr_sense_gain_config {
+ __u32 vy:8;
+ __u32 vu:8;
+ __u32 vv:8;
+ __u32 __reserved0:8;
+
+ __u32 hy:8;
+ __u32 hu:8;
+ __u32 hv:8;
+ __u32 __reserved1:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_chnr_iir_fir_config - Chroma IIR/FIR filter config
+ *
+ * @fir_0h: Value of center tap in horizontal FIR, range [0, 32], default 8.
+ * @__reserved0: reserved
+ * @fir_1h: Value of distance 1 in horizontal FIR, range [0, 32], default 12.
+ * @__reserved1: reserved
+ * @fir_2h: Value of distance 2 tap in horizontal FIR, range [0, 32], default 0.
+ * @dalpha_clip_val: weight for previous row in IIR, range [1, 256], default 0.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_yuvp1_chnr_iir_fir_config {
+ __u32 fir_0h:6;
+ __u32 __reserved0:2;
+ __u32 fir_1h:6;
+ __u32 __reserved1:2;
+ __u32 fir_2h:6;
+ __u32 dalpha_clip_val:9;
+ __u32 __reserved2:1;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_chnr_config - Chroma noise reduction config
+ *
+ * @enable: chroma noise reduction enable, see
+ * &ipu3_uapi_yuvp1_chnr_enable_config
+ * @coring: coring config for chroma noise reduction, see
+ * &ipu3_uapi_yuvp1_chnr_coring_config
+ * @sense_gain: sensitivity config for chroma noise reduction, see
+ * ipu3_uapi_yuvp1_chnr_sense_gain_config
+ * @iir_fir: iir and fir config for chroma noise reduction, see
+ * ipu3_uapi_yuvp1_chnr_iir_fir_config
+ */
+struct ipu3_uapi_yuvp1_chnr_config {
+ struct ipu3_uapi_yuvp1_chnr_enable_config enable;
+ struct ipu3_uapi_yuvp1_chnr_coring_config coring;
+ struct ipu3_uapi_yuvp1_chnr_sense_gain_config sense_gain;
+ struct ipu3_uapi_yuvp1_chnr_iir_fir_config iir_fir;
+} __packed;
+
+/* Edge Enhancement and Noise Reduction */
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config - Luma(Y) edge enhancement low-pass
+ * filter coefficients
+ *
+ * @a_diag: Smoothing diagonal coefficient, u5.0.
+ * @__reserved0: reserved
+ * @a_periph: Image smoothing perpherial, u5.0.
+ * @__reserved1: reserved
+ * @a_cent: Image Smoothing center coefficient, u5.0.
+ * @__reserved2: reserved
+ * @enable: 0: Y_EE_NR disabled, output = input; 1: Y_EE_NR enabled.
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config {
+ __u32 a_diag:5;
+ __u32 __reserved0:3;
+ __u32 a_periph:5;
+ __u32 __reserved1:3;
+ __u32 a_cent:5;
+ __u32 __reserved2:9;
+ __u32 enable:1;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_sense_config - Luma(Y) edge enhancement
+ * noise reduction sensitivity gains
+ *
+ * @edge_sense_0: Sensitivity of edge in dark area. u13.0, default 8191.
+ * @__reserved0: reserved
+ * @delta_edge_sense: Difference in the sensitivity of edges between
+ * the bright and dark areas. u13.0, default 0.
+ * @__reserved1: reserved
+ * @corner_sense_0: Sensitivity of corner in dark area. u13.0, default 0.
+ * @__reserved2: reserved
+ * @delta_corner_sense: Difference in the sensitivity of corners between
+ * the bright and dark areas. u13.0, default 8191.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_sense_config {
+ __u32 edge_sense_0:13;
+ __u32 __reserved0:3;
+ __u32 delta_edge_sense:13;
+ __u32 __reserved1:3;
+ __u32 corner_sense_0:13;
+ __u32 __reserved2:3;
+ __u32 delta_corner_sense:13;
+ __u32 __reserved3:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_gain_config - Luma(Y) edge enhancement
+ * noise reduction gain config
+ *
+ * @gain_pos_0: Gain for positive edge in dark area. u5.0, [0, 16], default 2.
+ * @__reserved0: reserved
+ * @delta_gain_posi: Difference in the gain of edges between the bright and
+ * dark areas for positive edges. u5.0, [0, 16], default 0.
+ * @__reserved1: reserved
+ * @gain_neg_0: Gain for negative edge in dark area. u5.0, [0, 16], default 8.
+ * @__reserved2: reserved
+ * @delta_gain_neg: Difference in the gain of edges between the bright and
+ * dark areas for negative edges. u5.0, [0, 16], default 0.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_gain_config {
+ __u32 gain_pos_0:5;
+ __u32 __reserved0:3;
+ __u32 delta_gain_posi:5;
+ __u32 __reserved1:3;
+ __u32 gain_neg_0:5;
+ __u32 __reserved2:3;
+ __u32 delta_gain_neg:5;
+ __u32 __reserved3:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_clip_config - Luma(Y) edge enhancement
+ * noise reduction clipping config
+ *
+ * @clip_pos_0: Limit of positive edge in dark area
+ * u5, value [0, 16], default 8.
+ * @__reserved0: reserved
+ * @delta_clip_posi: Difference in the limit of edges between the bright
+ * and dark areas for positive edges.
+ * u5, value [0, 16], default 8.
+ * @__reserved1: reserved
+ * @clip_neg_0: Limit of negative edge in dark area
+ * u5, value [0, 16], default 8.
+ * @__reserved2: reserved
+ * @delta_clip_neg: Difference in the limit of edges between the bright
+ * and dark areas for negative edges.
+ * u5, value [0, 16], default 8.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_clip_config {
+ __u32 clip_pos_0:5;
+ __u32 __reserved0:3;
+ __u32 delta_clip_posi:5;
+ __u32 __reserved1:3;
+ __u32 clip_neg_0:5;
+ __u32 __reserved2:3;
+ __u32 delta_clip_neg:5;
+ __u32 __reserved3:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_frng_config - Luma(Y) edge enhancement
+ * noise reduction fringe config
+ *
+ * @gain_exp: Common exponent of gains, u4, [0, 8], default 2.
+ * @__reserved0: reserved
+ * @min_edge: Threshold for edge and smooth stitching, u13.
+ * @__reserved1: reserved
+ * @lin_seg_param: Power of LinSeg, u4.
+ * @__reserved2: reserved
+ * @t1: Parameter for enabling/disabling the edge enhancement, u1.0, [0, 1],
+ * default 1.
+ * @t2: Parameter for enabling/disabling the smoothing, u1.0, [0, 1],
+ * default 1.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_frng_config {
+ __u32 gain_exp:4;
+ __u32 __reserved0:28;
+ __u32 min_edge:13;
+ __u32 __reserved1:3;
+ __u32 lin_seg_param:4;
+ __u32 __reserved2:4;
+ __u32 t1:1;
+ __u32 t2:1;
+ __u32 __reserved3:6;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_diag_config - Luma(Y) edge enhancement
+ * noise reduction diagonal config
+ *
+ * @diag_disc_g: Coefficient that prioritize diagonal edge direction on
+ * horizontal or vertical for final enhancement.
+ * u4.0, [1, 15], default 1.
+ * @__reserved0: reserved
+ * @hvw_hor: Weight of horizontal/vertical edge enhancement for hv edge.
+ * u2.2, [1, 15], default 4.
+ * @dw_hor: Weight of diagonal edge enhancement for hv edge.
+ * u2.2, [1, 15], default 1.
+ * @hvw_diag: Weight of horizontal/vertical edge enhancement for diagonal edge.
+ * u2.2, [1, 15], default 1.
+ * @dw_diag: Weight of diagonal edge enhancement for diagonal edge.
+ * u2.2, [1, 15], default 4.
+ * @__reserved1: reserved
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_diag_config {
+ __u32 diag_disc_g:4;
+ __u32 __reserved0:4;
+ __u32 hvw_hor:4;
+ __u32 dw_hor:4;
+ __u32 hvw_diag:4;
+ __u32 dw_diag:4;
+ __u32 __reserved1:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config - Luma(Y) edge enhancement
+ * noise reduction false color correction (FCC) coring config
+ *
+ * @pos_0: Gain for positive edge in dark, u13.0, [0, 16], default 0.
+ * @__reserved0: reserved
+ * @pos_delta: Gain for positive edge in bright, value: pos_0 + pos_delta <=16
+ * u13.0, default 0.
+ * @__reserved1: reserved
+ * @neg_0: Gain for negative edge in dark area, u13.0, range [0, 16], default 0.
+ * @__reserved2: reserved
+ * @neg_delta: Gain for negative edge in bright area. neg_0 + neg_delta <=16
+ * u13.0, default 0.
+ * @__reserved3: reserved
+ *
+ * Coring is a simple soft thresholding technique.
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config {
+ __u32 pos_0:13;
+ __u32 __reserved0:3;
+ __u32 pos_delta:13;
+ __u32 __reserved1:3;
+ __u32 neg_0:13;
+ __u32 __reserved2:3;
+ __u32 neg_delta:13;
+ __u32 __reserved3:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp1_y_ee_nr_config - Edge enhancement and noise reduction
+ *
+ * @lpf: low-pass filter config. See &ipu3_uapi_yuvp1_y_ee_nr_lpf_config
+ * @sense: sensitivity config. See &ipu3_uapi_yuvp1_y_ee_nr_sense_config
+ * @gain: gain config as defined in &ipu3_uapi_yuvp1_y_ee_nr_gain_config
+ * @clip: clip config as defined in &ipu3_uapi_yuvp1_y_ee_nr_clip_config
+ * @frng: fringe config as defined in &ipu3_uapi_yuvp1_y_ee_nr_frng_config
+ * @diag: diagonal edge config. See &ipu3_uapi_yuvp1_y_ee_nr_diag_config
+ * @fc_coring: coring config for fringe control. See
+ * &ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config
+ */
+struct ipu3_uapi_yuvp1_y_ee_nr_config {
+ struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config lpf;
+ struct ipu3_uapi_yuvp1_y_ee_nr_sense_config sense;
+ struct ipu3_uapi_yuvp1_y_ee_nr_gain_config gain;
+ struct ipu3_uapi_yuvp1_y_ee_nr_clip_config clip;
+ struct ipu3_uapi_yuvp1_y_ee_nr_frng_config frng;
+ struct ipu3_uapi_yuvp1_y_ee_nr_diag_config diag;
+ struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config fc_coring;
+} __packed;
+
+/* Total Color Correction */
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_gen_control_static_config - Total color correction
+ * general control config
+ *
+ * @en: 0 - TCC disabled. Output = input 1 - TCC enabled.
+ * @blend_shift: blend shift, Range[3, 4], default NA.
+ * @gain_according_to_y_only: 0: Gain is calculated according to YUV,
+ * 1: Gain is calculated according to Y only
+ * @__reserved0: reserved
+ * @gamma: Final blending coefficients. Values[-16, 16], default NA.
+ * @__reserved1: reserved
+ * @delta: Final blending coefficients. Values[-16, 16], default NA.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_yuvp2_tcc_gen_control_static_config {
+ __u32 en:1;
+ __u32 blend_shift:3;
+ __u32 gain_according_to_y_only:1;
+ __u32 __reserved0:11;
+ __s32 gamma:5;
+ __u32 __reserved1:3;
+ __s32 delta:5;
+ __u32 __reserved2:3;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config - Total color correction
+ * multi-axis color control (MACC) config
+ *
+ * @a: a coefficient for 2x2 MACC conversion matrix.
+ * @__reserved0: reserved
+ * @b: b coefficient 2x2 MACC conversion matrix.
+ * @__reserved1: reserved
+ * @c: c coefficient for 2x2 MACC conversion matrix.
+ * @__reserved2: reserved
+ * @d: d coefficient for 2x2 MACC conversion matrix.
+ * @__reserved3: reserved
+ */
+struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config {
+ __s32 a:12;
+ __u32 __reserved0:4;
+ __s32 b:12;
+ __u32 __reserved1:4;
+ __s32 c:12;
+ __u32 __reserved2:4;
+ __s32 d:12;
+ __u32 __reserved3:4;
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_macc_table_static_config - Total color correction
+ * multi-axis color control (MACC) table array
+ *
+ * @entries: config for multi axis color correction, as specified by
+ * &ipu3_uapi_yuvp2_tcc_macc_elem_static_config
+ */
+struct ipu3_uapi_yuvp2_tcc_macc_table_static_config {
+ struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config
+ entries[IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config - Total color correction
+ * inverse y lookup table
+ *
+ * @entries: lookup table for inverse y estimation, and use it to estimate the
+ * ratio between luma and chroma. Chroma by approximate the absolute
+ * value of the radius on the chroma plane (R = sqrt(u^2+v^2) ) and
+ * luma by approximate by 1/Y.
+ */
+struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config {
+ __u16 entries[IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config - Total color
+ * correction lookup table for PCWL
+ *
+ * @entries: lookup table for gain piece wise linear transformation (PCWL)
+ */
+struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config {
+ __u16 entries[IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config - Total color correction
+ * lookup table for r square root
+ *
+ * @entries: lookup table for r square root estimation
+ */
+struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config {
+ __s16 entries[IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_yuvp2_tcc_static_config- Total color correction static
+ *
+ * @gen_control: general config for Total Color Correction
+ * @macc_table: config for multi axis color correction
+ * @inv_y_lut: lookup table for inverse y estimation
+ * @gain_pcwl: lookup table for gain PCWL
+ * @r_sqr_lut: lookup table for r square root estimation.
+ */
+struct ipu3_uapi_yuvp2_tcc_static_config {
+ struct ipu3_uapi_yuvp2_tcc_gen_control_static_config gen_control;
+ struct ipu3_uapi_yuvp2_tcc_macc_table_static_config macc_table;
+ struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config inv_y_lut;
+ struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config gain_pcwl;
+ struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config r_sqr_lut;
+} __packed;
+
+/* Advanced Noise Reduction related structs */
+
+/*
+ * struct ipu3_uapi_anr_alpha - Advanced noise reduction alpha
+ *
+ * Tunable parameters that are subject to modification according to the
+ * total gain used.
+ */
+struct ipu3_uapi_anr_alpha {
+ __u16 gr;
+ __u16 r;
+ __u16 b;
+ __u16 gb;
+ __u16 dc_gr;
+ __u16 dc_r;
+ __u16 dc_b;
+ __u16 dc_gb;
+} __packed;
+
+/*
+ * struct ipu3_uapi_anr_beta - Advanced noise reduction beta
+ *
+ * Tunable parameters that are subject to modification according to the
+ * total gain used.
+ */
+struct ipu3_uapi_anr_beta {
+ __u16 beta_gr;
+ __u16 beta_r;
+ __u16 beta_b;
+ __u16 beta_gb;
+} __packed;
+
+/*
+ * struct ipu3_uapi_anr_plain_color - Advanced noise reduction plain color with
+ * 4x4 matrix
+ *
+ * Tunable parameters that are subject to modification according to the
+ * total gain used.
+ */
+struct ipu3_uapi_anr_plain_color {
+ __u16 reg_w_gr[16];
+ __u16 reg_w_r[16];
+ __u16 reg_w_b[16];
+ __u16 reg_w_gb[16];
+} __packed;
+
+/**
+ * struct ipu3_uapi_anr_transform_config - Advanced noise reduction transform
+ *
+ * @enable: advanced noise reduction enabled.
+ * @adaptive_treshhold_en: On IPU3, adaptive threshold is always enabled.
+ * @__reserved1: reserved
+ * @__reserved2: reserved
+ * @alpha: using following defaults:
+ * 13, 13, 13, 13, 0, 0, 0, 0
+ * 11, 11, 11, 11, 0, 0, 0, 0
+ * 14, 14, 14, 14, 0, 0, 0, 0
+ * @beta: use following defaults:
+ * 24, 24, 24, 24
+ * 21, 20, 20, 21
+ * 25, 25, 25, 25
+ * @color: use defaults defined in driver/media/pci/intel/ipu3-tables.c
+ * @sqrt_lut: 11 bits per element, values =
+ * [724 768 810 849 887
+ * 923 958 991 1024 1056
+ * 1116 1145 1173 1201 1086
+ * 1228 1254 1280 1305 1330
+ * 1355 1379 1402 1425 1448]
+ * @xreset: Reset value of X for r^2 calculation Value: col_start-X_center
+ * Constraint: Xreset + FrameWdith=4095 Xreset= -4095, default -1632.
+ * @__reserved3: reserved
+ * @yreset: Reset value of Y for r^2 calculation Value: row_start-Y_center
+ * Constraint: Yreset + FrameHeight=4095 Yreset= -4095, default -1224.
+ * @__reserved4: reserved
+ * @x_sqr_reset: Reset value of X^2 for r^2 calculation Value = (Xreset)^2
+ * @r_normfactor: Normalization factor for R. Default 14.
+ * @__reserved5: reserved
+ * @y_sqr_reset: Reset value of Y^2 for r^2 calculation Value = (Yreset)^2
+ * @gain_scale: Parameter describing shading gain as a function of distance
+ * from the image center.
+ * A single value per frame, loaded by the driver. Default 115.
+ */
+struct ipu3_uapi_anr_transform_config {
+ __u32 enable:1; /* 0 or 1, disabled or enabled */
+ __u32 adaptive_treshhold_en:1; /* On IPU3, always enabled */
+
+ __u32 __reserved1:30;
+ __u8 __reserved2[44];
+
+ struct ipu3_uapi_anr_alpha alpha[3];
+ struct ipu3_uapi_anr_beta beta[3];
+ struct ipu3_uapi_anr_plain_color color[3];
+
+ __u16 sqrt_lut[IPU3_UAPI_ANR_LUT_SIZE]; /* 11 bits per element */
+
+ __s16 xreset:13;
+ __u16 __reserved3:3;
+ __s16 yreset:13;
+ __u16 __reserved4:3;
+
+ __u32 x_sqr_reset:24;
+ __u32 r_normfactor:5;
+ __u32 __reserved5:3;
+
+ __u32 y_sqr_reset:24;
+ __u32 gain_scale:8;
+} __packed;
+
+/**
+ * struct ipu3_uapi_anr_stitch_pyramid - ANR stitch pyramid
+ *
+ * @entry0: pyramid LUT entry0, range [0x0, 0x3f]
+ * @entry1: pyramid LUT entry1, range [0x0, 0x3f]
+ * @entry2: pyramid LUT entry2, range [0x0, 0x3f]
+ * @__reserved: reserved
+ */
+struct ipu3_uapi_anr_stitch_pyramid {
+ __u32 entry0:6;
+ __u32 entry1:6;
+ __u32 entry2:6;
+ __u32 __reserved:14;
+} __packed;
+
+/**
+ * struct ipu3_uapi_anr_stitch_config - ANR stitch config
+ *
+ * @anr_stitch_en: enable stitch. Enabled with 1.
+ * @__reserved: reserved
+ * @pyramid: pyramid table as defined by &ipu3_uapi_anr_stitch_pyramid
+ * default values:
+ * { 1, 3, 5 }, { 7, 7, 5 }, { 3, 1, 3 },
+ * { 9, 15, 21 }, { 21, 15, 9 }, { 3, 5, 15 },
+ * { 25, 35, 35 }, { 25, 15, 5 }, { 7, 21, 35 },
+ * { 49, 49, 35 }, { 21, 7, 7 }, { 21, 35, 49 },
+ * { 49, 35, 21 }, { 7, 5, 15 }, { 25, 35, 35 },
+ * { 25, 15, 5 }, { 3, 9, 15 }, { 21, 21, 15 },
+ * { 9, 3, 1 }, { 3, 5, 7 }, { 7, 5, 3}, { 1 }
+ */
+struct ipu3_uapi_anr_stitch_config {
+ __u32 anr_stitch_en;
+ __u8 __reserved[44];
+ struct ipu3_uapi_anr_stitch_pyramid pyramid[IPU3_UAPI_ANR_PYRAMID_SIZE];
+} __packed;
+
+/**
+ * struct ipu3_uapi_anr_config - ANR config
+ *
+ * @transform: advanced noise reduction transform config as specified by
+ * &ipu3_uapi_anr_transform_config
+ * @stitch: create 4x4 patch from 4 surrounding 8x8 patches.
+ */
+struct ipu3_uapi_anr_config {
+ struct ipu3_uapi_anr_transform_config transform __attribute__((aligned(32)));
+ struct ipu3_uapi_anr_stitch_config stitch __attribute__((aligned(32)));
+} __packed;
+
+/**
+ * struct ipu3_uapi_acc_param - Accelerator cluster parameters
+ *
+ * ACC refers to the HW cluster containing all Fixed Functions(FFs). Each FF
+ * implements a specific algorithm.
+ *
+ * @bnr: parameters for bayer noise reduction static config. See
+ * &ipu3_uapi_bnr_static_config
+ * @green_disparity: disparity static config between gr and gb channel.
+ * See &ipu3_uapi_bnr_static_config_green_disparity
+ * @dm: de-mosaic config. See &ipu3_uapi_dm_config
+ * @ccm: color correction matrix. See &ipu3_uapi_ccm_mat_config
+ * @gamma: gamma correction config. See &ipu3_uapi_gamma_config
+ * @csc: color space conversion matrix. See &ipu3_uapi_csc_mat_config
+ * @cds: color down sample config. See &ipu3_uapi_cds_params
+ * @shd: lens shading correction config. See &ipu3_uapi_shd_config
+ * @iefd: Image enhancement filter and denoise config.
+ * &ipu3_uapi_yuvp1_iefd_config
+ * @yds_c0: y down scaler config. &ipu3_uapi_yuvp1_yds_config
+ * @chnr_c0: chroma noise reduction config. &ipu3_uapi_yuvp1_chnr_config
+ * @y_ee_nr: y edge enhancement and noise reduction config.
+ * &ipu3_uapi_yuvp1_y_ee_nr_config
+ * @yds: y down scaler config. See &ipu3_uapi_yuvp1_yds_config
+ * @chnr: chroma noise reduction config. See &ipu3_uapi_yuvp1_chnr_config
+ * @__reserved1: reserved
+ * @yds2: y channel down scaler config. See &ipu3_uapi_yuvp1_yds_config
+ * @tcc: total color correction config as defined in struct
+ * &ipu3_uapi_yuvp2_tcc_static_config
+ * @__reserved2: reserved
+ * @anr: advanced noise reduction config.See &ipu3_uapi_anr_config
+ * @awb_fr: AWB filter response config. See ipu3_uapi_awb_fr_config
+ * @ae: auto exposure config As specified by &ipu3_uapi_ae_config
+ * @af: auto focus config. As specified by &ipu3_uapi_af_config
+ * @awb: auto white balance config. As specified by &ipu3_uapi_awb_config
+ */
+struct ipu3_uapi_acc_param {
+ struct ipu3_uapi_bnr_static_config bnr;
+ struct ipu3_uapi_bnr_static_config_green_disparity
+ green_disparity __attribute__((aligned(32)));
+ struct ipu3_uapi_dm_config dm __attribute__((aligned(32)));
+ struct ipu3_uapi_ccm_mat_config ccm __attribute__((aligned(32)));
+ struct ipu3_uapi_gamma_config gamma __attribute__((aligned(32)));
+ struct ipu3_uapi_csc_mat_config csc __attribute__((aligned(32)));
+ struct ipu3_uapi_cds_params cds __attribute__((aligned(32)));
+ struct ipu3_uapi_shd_config shd __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_iefd_config iefd __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_yds_config yds_c0 __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_chnr_config chnr_c0 __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_y_ee_nr_config y_ee_nr __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_yds_config yds __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_chnr_config chnr __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_yds_config yds2 __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp2_tcc_static_config tcc __attribute__((aligned(32)));
+ struct ipu3_uapi_anr_config anr;
+ struct ipu3_uapi_awb_fr_config awb_fr;
+ struct ipu3_uapi_ae_config ae;
+ struct ipu3_uapi_af_config af;
+ struct ipu3_uapi_awb_config awb;
+} __packed;
+
+/**
+ * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
+ *
+ * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
+ * @lin_lutlow_r: linearization look-up table for R channel interpolation.
+ * @lin_lutlow_b: linearization look-up table for B channel interpolation.
+ * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
+ * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
+ * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
+ * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
+ * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
+ * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
+ * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
+ */
+struct ipu3_uapi_isp_lin_vmem_params {
+ __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
+ __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
+} __packed;
+
+/* Temporal Noise Reduction */
+
+/**
+ * struct ipu3_uapi_isp_tnr3_vmem_params - Temporal noise reduction vector
+ * memory parameters
+ *
+ * @slope: slope setting in interpolation curve for temporal noise reduction.
+ * @__reserved1: reserved
+ * @sigma: knee point setting in interpolation curve for temporal
+ * noise reduction.
+ * @__reserved2: reserved
+ */
+struct ipu3_uapi_isp_tnr3_vmem_params {
+ __u16 slope[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
+ __u16 __reserved1[IPU3_UAPI_ISP_VEC_ELEMS
+ - IPU3_UAPI_ISP_TNR3_VMEM_LEN];
+ __u16 sigma[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
+ __u16 __reserved2[IPU3_UAPI_ISP_VEC_ELEMS
+ - IPU3_UAPI_ISP_TNR3_VMEM_LEN];
+} __packed;
+
+/**
+ * struct ipu3_uapi_isp_tnr3_params - Temporal noise reduction v3 parameters
+ *
+ * @knee_y1: Knee point TNR3 assumes standard deviation of Y,U and
+ * V at Y1 are TnrY1_Sigma_Y, U and V.
+ * @knee_y2: Knee point TNR3 assumes standard deviation of Y,U and
+ * V at Y2 are TnrY2_Sigma_Y, U and V.
+ * @maxfb_y: Max feedback gain for Y
+ * @maxfb_u: Max feedback gain for U
+ * @maxfb_v: Max feedback gain for V
+ * @round_adj_y: rounding Adjust for Y
+ * @round_adj_u: rounding Adjust for U
+ * @round_adj_v: rounding Adjust for V
+ * @ref_buf_select: selection of the reference frame buffer to be used.
+ */
+struct ipu3_uapi_isp_tnr3_params {
+ __u32 knee_y1;
+ __u32 knee_y2;
+ __u32 maxfb_y;
+ __u32 maxfb_u;
+ __u32 maxfb_v;
+ __u32 round_adj_y;
+ __u32 round_adj_u;
+ __u32 round_adj_v;
+ __u32 ref_buf_select;
+} __packed;
+
+/* Extreme Noise Reduction version 3 */
+
+/**
+ * struct ipu3_uapi_isp_xnr3_vmem_params - Extreme noise reduction v3
+ * vector memory parameters
+ *
+ * @x: xnr3 parameters.
+ * @a: xnr3 parameters.
+ * @b: xnr3 parameters.
+ * @c: xnr3 parameters.
+ */
+struct ipu3_uapi_isp_xnr3_vmem_params {
+ __u16 x[IPU3_UAPI_ISP_VEC_ELEMS];
+ __u16 a[IPU3_UAPI_ISP_VEC_ELEMS];
+ __u16 b[IPU3_UAPI_ISP_VEC_ELEMS];
+ __u16 c[IPU3_UAPI_ISP_VEC_ELEMS];
+} __packed;
+
+/**
+ * struct ipu3_uapi_xnr3_alpha_params - Extreme noise reduction v3
+ * alpha tuning parameters
+ *
+ * @y0: Sigma for Y range similarity in dark area.
+ * @u0: Sigma for U range similarity in dark area.
+ * @v0: Sigma for V range similarity in dark area.
+ * @ydiff: Sigma difference for Y between bright area and dark area.
+ * @udiff: Sigma difference for U between bright area and dark area.
+ * @vdiff: Sigma difference for V between bright area and dark area.
+ */
+struct ipu3_uapi_xnr3_alpha_params {
+ __u32 y0;
+ __u32 u0;
+ __u32 v0;
+ __u32 ydiff;
+ __u32 udiff;
+ __u32 vdiff;
+} __packed;
+
+/**
+ * struct ipu3_uapi_xnr3_coring_params - Extreme noise reduction v3
+ * coring parameters
+ *
+ * @u0: Coring Threshold of U channel in dark area.
+ * @v0: Coring Threshold of V channel in dark area.
+ * @udiff: Threshold difference of U channel between bright and dark area.
+ * @vdiff: Threshold difference of V channel between bright and dark area.
+ */
+struct ipu3_uapi_xnr3_coring_params {
+ __u32 u0;
+ __u32 v0;
+ __u32 udiff;
+ __u32 vdiff;
+} __packed;
+
+/**
+ * struct ipu3_uapi_xnr3_blending_params - Blending factor
+ *
+ * @strength: The factor for blending output with input. This is tuning
+ * parameterHigher values lead to more aggressive XNR operation.
+ */
+struct ipu3_uapi_xnr3_blending_params {
+ __u32 strength;
+} __packed;
+
+/**
+ * struct ipu3_uapi_isp_xnr3_params - Extreme noise reduction v3 parameters
+ *
+ * @alpha: parameters for xnr3 alpha. See &ipu3_uapi_xnr3_alpha_params
+ * @coring: parameters for xnr3 coring. See &ipu3_uapi_xnr3_coring_params
+ * @blending: parameters for xnr3 blending. See &ipu3_uapi_xnr3_blending_params
+ */
+struct ipu3_uapi_isp_xnr3_params {
+ struct ipu3_uapi_xnr3_alpha_params alpha;
+ struct ipu3_uapi_xnr3_coring_params coring;
+ struct ipu3_uapi_xnr3_blending_params blending;
+} __packed;
+
+/***** Obgrid (optical black level compensation) table entry *****/
+
+/**
+ * struct ipu3_uapi_obgrid_param - Optical black level compensation parameters
+ *
+ * @gr: Grid table values for color GR
+ * @r: Grid table values for color R
+ * @b: Grid table values for color B
+ * @gb: Grid table values for color GB
+ *
+ * Black level is different for red, green, and blue channels. So black level
+ * compensation is different per channel.
+ */
+struct ipu3_uapi_obgrid_param {
+ __u16 gr;
+ __u16 r;
+ __u16 b;
+ __u16 gb;
+} __packed;
+
+/******************* V4L2_META_FMT_IPU3_PARAMS *******************/
+
+/**
+ * struct ipu3_uapi_flags - bits to indicate which pipeline needs update
+ *
+ * @gdc: 0 = no update, 1 = update.
+ * @obgrid: 0 = no update, 1 = update.
+ * @__reserved1: Not used.
+ * @acc_bnr: 0 = no update, 1 = update.
+ * @acc_green_disparity: 0 = no update, 1 = update.
+ * @acc_dm: 0 = no update, 1 = update.
+ * @acc_ccm: 0 = no update, 1 = update.
+ * @acc_gamma: 0 = no update, 1 = update.
+ * @acc_csc: 0 = no update, 1 = update.
+ * @acc_cds: 0 = no update, 1 = update.
+ * @acc_shd: 0 = no update, 1 = update.
+ * @__reserved2: Not used.
+ * @acc_iefd: 0 = no update, 1 = update.
+ * @acc_yds_c0: 0 = no update, 1 = update.
+ * @acc_chnr_c0: 0 = no update, 1 = update.
+ * @acc_y_ee_nr: 0 = no update, 1 = update.
+ * @acc_yds: 0 = no update, 1 = update.
+ * @acc_chnr: 0 = no update, 1 = update.
+ * @acc_ytm: 0 = no update, 1 = update.
+ * @acc_yds2: 0 = no update, 1 = update.
+ * @acc_tcc: 0 = no update, 1 = update.
+ * @acc_dpc: 0 = no update, 1 = update.
+ * @acc_bds: 0 = no update, 1 = update.
+ * @acc_anr: 0 = no update, 1 = update.
+ * @acc_awb_fr: 0 = no update, 1 = update.
+ * @acc_ae: 0 = no update, 1 = update.
+ * @acc_af: 0 = no update, 1 = update.
+ * @acc_awb: 0 = no update, 1 = update.
+ * @__acc_osys: 0 = no update, 1 = update.
+ * @__reserved3: Not used.
+ * @lin_vmem_params: 0 = no update, 1 = update.
+ * @tnr3_vmem_params: 0 = no update, 1 = update.
+ * @xnr3_vmem_params: 0 = no update, 1 = update.
+ * @tnr3_dmem_params: 0 = no update, 1 = update.
+ * @xnr3_dmem_params: 0 = no update, 1 = update.
+ * @__reserved4: Not used.
+ * @obgrid_param: 0 = no update, 1 = update.
+ * @__reserved5: Not used.
+ */
+struct ipu3_uapi_flags {
+ __u32 gdc:1;
+ __u32 obgrid:1;
+ __u32 __reserved1:30;
+
+ __u32 acc_bnr:1;
+ __u32 acc_green_disparity:1;
+ __u32 acc_dm:1;
+ __u32 acc_ccm:1;
+ __u32 acc_gamma:1;
+ __u32 acc_csc:1;
+ __u32 acc_cds:1;
+ __u32 acc_shd:1;
+ __u32 __reserved2:2;
+ __u32 acc_iefd:1;
+ __u32 acc_yds_c0:1;
+ __u32 acc_chnr_c0:1;
+ __u32 acc_y_ee_nr:1;
+ __u32 acc_yds:1;
+ __u32 acc_chnr:1;
+ __u32 acc_ytm:1;
+ __u32 acc_yds2:1;
+ __u32 acc_tcc:1;
+ __u32 acc_dpc:1;
+ __u32 acc_bds:1;
+ __u32 acc_anr:1;
+ __u32 acc_awb_fr:1;
+ __u32 acc_ae:1;
+ __u32 acc_af:1;
+ __u32 acc_awb:1;
+ __u32 __reserved3:4;
+
+ __u32 lin_vmem_params:1;
+ __u32 tnr3_vmem_params:1;
+ __u32 xnr3_vmem_params:1;
+ __u32 tnr3_dmem_params:1;
+ __u32 xnr3_dmem_params:1;
+ __u32 __reserved4:1;
+ __u32 obgrid_param:1;
+ __u32 __reserved5:25;
+} __packed;
+
+/**
+ * struct ipu3_uapi_params - V4L2_META_FMT_IPU3_PARAMS
+ *
+ * @use: select which parameters to apply, see &ipu3_uapi_flags
+ * @acc_param: ACC parameters, as specified by &ipu3_uapi_acc_param
+ * @lin_vmem_params: linearization VMEM, as specified by
+ * &ipu3_uapi_isp_lin_vmem_params
+ * @tnr3_vmem_params: tnr3 VMEM as specified by
+ * &ipu3_uapi_isp_tnr3_vmem_params
+ * @xnr3_vmem_params: xnr3 VMEM as specified by
+ * &ipu3_uapi_isp_xnr3_vmem_params
+ * @tnr3_dmem_params: tnr3 DMEM as specified by &ipu3_uapi_isp_tnr3_params
+ * @xnr3_dmem_params: xnr3 DMEM as specified by &ipu3_uapi_isp_xnr3_params
+ * @obgrid_param: obgrid parameters as specified by
+ * &ipu3_uapi_obgrid_param
+ *
+ * The video queue "parameters" is of format V4L2_META_FMT_IPU3_PARAMS.
+ * This is a "single plane" v4l2_meta_format using V4L2_BUF_TYPE_META_OUTPUT.
+ *
+ * struct ipu3_uapi_params as defined below contains a lot of parameters and
+ * ipu3_uapi_flags selects which parameters to apply.
+ */
+struct ipu3_uapi_params {
+ /* Flags which of the settings below are to be applied */
+ struct ipu3_uapi_flags use __attribute__((aligned(32)));
+
+ /* Accelerator cluster parameters */
+ struct ipu3_uapi_acc_param acc_param;
+
+ /* ISP vector address space parameters */
+ struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
+ struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
+ struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
+
+ /* ISP data memory (DMEM) parameters */
+ struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
+ struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
+
+ /* Optical black level compensation */
+ struct ipu3_uapi_obgrid_param obgrid_param;
+} __packed;
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-10-29 22:22 ` [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI Yong Zhi
@ 2018-11-02 13:02 ` Sakari Ailus
2018-11-16 22:37 ` Zhi, Yong
2018-11-02 13:49 ` Mauro Carvalho Chehab
2018-11-15 12:51 ` Hans Verkuil
2 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-02 13:02 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao, Chao C Li
Hi Yong,
Thanks for the update! I went through this again... a few comments below
but I'd say they're mostly pretty minor issues.
On Mon, Oct 29, 2018 at 03:22:57PM -0700, Yong Zhi wrote:
> These meta formats are used on Intel IPU3 ImgU video queues
> to carry 3A statistics and ISP pipeline parameters.
>
> V4L2_META_FMT_IPU3_3A
> V4L2_META_FMT_IPU3_PARAMS
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> Signed-off-by: Chao C Li <chao.c.li@intel.com>
> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> ---
> Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
> include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
> 3 files changed, 3001 insertions(+)
> create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> create mode 100644 include/uapi/linux/intel-ipu3.h
>
> diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
> index cf971d5..eafc534 100644
> --- a/Documentation/media/uapi/v4l/meta-formats.rst
> +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
> .. toctree::
> :maxdepth: 1
>
> + pixfmt-meta-intel-ipu3
> pixfmt-meta-d4xx
> pixfmt-meta-uvc
> pixfmt-meta-vsp1-hgo
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> new file mode 100644
> index 0000000..23b945b
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> @@ -0,0 +1,181 @@
> +.. -*- coding: utf-8; mode: rst -*-
> +
> +.. _intel-ipu3:
Instead, to avoid a warning from Sphinx, replace the line with these:
.. _v4l2-meta-fmt-ipu3-params:
.. _v4l2-meta-fmt-ipu3-stat-3a:
> +
> +******************************************************************
> +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
> +******************************************************************
> +
> +.. c:type:: ipu3_uapi_stats_3a
> +
> +3A statistics
> +=============
> +
> +For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
> +an input bayer frame. Those statistics, defined in data struct
> +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu 3a stat"
> +video node, which are then passed to user space for statistics analysis
> +using :c:type:`v4l2_meta_format` interface.
> +
> +The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
Extra whitespace at the end of the line.
> +Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
> +and AE (Auto-exposure) histogram.
> +
> +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.
> +
> +
> +.. code-block:: c
> +
> +
> + struct ipu3_uapi_stats_3a {
> + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> + __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_raw_buffer_aligned
> + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> + struct ipu3_uapi_4a_config stats_4a_config;
> + __u32 ae_join_buffers;
> + __u8 padding[28];
> + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + stats_3a_bubble_per_stripe;
I think you could just unwrap these, even if it causes them to be over 80
characters per line. They display better in a web browser that way. Or
alternatively align the wrapped lines to the same column.
> + struct ipu3_uapi_ff_status stats_3a_status;
> + } __packed;
> +
> +
> +.. c:type:: ipu3_uapi_params
> +
> +Pipeline parameters
> +===================
> +
> +IPU3 pipeline has a number of image processing stages, each of which takes a
> +set of parameters as input. The major stages of pipelines are shown here:
> +
> +Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
> +
> +Linearization -> Lens Shading Correction -> White Balance / Exposure /
> +
> +Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
> +
> +Correction Matrix -> Gamma correction -> Color Space Conversion ->
> +
> +Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
> +
> +Correction -> XNR3 -> TNR -> DDR
> +
> +The table below presents a description of the above algorithms.
> +
> +======================== =======================================================
> +Name Description
> +======================== =======================================================
> +Optical Black Correction Optical Black Correction block subtracts a pre-defined
> + value from the respective pixel values to obtain better
> + image quality.
> + Defined in :c:type:`ipu3_uapi_obgrid_param`.
> +Linearization This algo block uses linearization parameters to
> + address non-linearity sensor effects. The Lookup table
> + table is defined in
> + :c:type:`ipu3_uapi_isp_lin_vmem_params`.
> +SHD Lens shading correction is used to correct spatial
> + non-uniformity of the pixel response due to optical
> + lens shading. This is done by applying a different gain
> + for each pixel. The gain, black level etc are
> + configured in :c:type:`ipu3_uapi_shd_config_static`.
> +BNR Bayer noise reduction block removes image noise by
> + applying a bilateral filter.
> + See :c:type:`ipu3_uapi_bnr_static_config` for details.
> +ANR Advanced Noise Reduction is a block based algorithm
> + that performs noise reduction in the Bayer domain. The
> + convolution matrix etc can be found in
> + :c:type:`ipu3_uapi_anr_config`.
> +Demosaicing Demosaicing converts raw sensor data in Bayer format
> + into RGB (Red, Green, Blue) presentation. Then add
> + outputs of estimation of Y channel for following stream
> + processing by Firmware. The struct is defined as
> + :c:type:`ipu3_uapi_dm_config`.
> +Color Correction Color Correction algo transforms sensor specific color
> + space to the standard "sRGB" color space. This is done
> + by applying 3x3 matrix defined in
> + :c:type:`ipu3_uapi_ccm_mat_config`.
> +Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
> + basic non-linear tone mapping correction that is
> + applied per pixel for each pixel component.
> +CSC Color space conversion transforms each pixel from the
> + RGB primary presentation to YUV (Y - brightness,
> + UV - Luminance) presentation. This is done by applying
> + a 3x3 matrix defined in
> + :c:type:`ipu3_uapi_csc_mat_config`
> +CDS Chroma down sampling
> + After the CSC is performed, the Chroma Down Sampling
> + is applied for a UV plane down sampling by a factor
> + of 2 in each direction for YUV 4:2:0 using a 4x2
> + configurable filter :c:type:`ipu3_uapi_cds_params`.
> +CHNR Chroma noise reduction
> + This block processes only the chrominance pixels and
> + performs noise reduction by cleaning the high
> + frequency noise.
> + See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
> +TCC Total color correction as defined in struct
> + :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
> +XNR3 eXtreme Noise Reduction V3 is the third revision of
> + noise reduction algorithm used to improve image
> + quality. This removes the low frequency noise in the
> + captured image. Two related structs are being defined,
> + :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
> + and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
> + memory.
> +TNR Temporal Noise Reduction block compares successive
> + frames in time to remove anomalies / noise in pixel
> + values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
> + :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
> + vector and data memory respectively.
> +======================== =======================================================
> +
> +A few stages of the pipeline will be executed by firmware running on the ISP
> +processor, while many others will use a set of fixed hardware blocks also
> +called accelerator cluster (ACC) to crunch pixel data and produce statistics.
> +
> +ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be selectively
> +enabled / disabled by the user space through struct :c:type:`ipu3_uapi_flags`
> +embedded in :c:type:`ipu3_uapi_params` structure. For parameters that are not
> +enabled by the user space, corresponding structs are ignored by the ISP.
I presume the "enabled" here means enabling the use of the new parameter
values. How about this:
ACC parameters of individual algorithms, as defined by
:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
space through struct :c:type:`ipu3_uapi_flags` embedded in
:c:type:`ipu3_uapi_params` structure. For parameters that are configured as
not enabled by the user space, the corresponding structs are ignored by the
driver, in which case the existing configuration of the algorithm will be
preserved.
> +
> +Both 3A statistics and pipeline parameters described here are closely tied to
> +the underlying camera sub-system (CSS) APIs. They are usually consumed and
> +produced by dedicated user space libraries that comprise the important tuning
> +tools, thus freeing the developers from being bothered with the low level
> +hardware and algorithm details.
> +
> +It should be noted that IPU3 DMA operations require the addresses of all data
> +structures (that includes both input and output) to be aligned on 32 byte
> +boundaries.
> +
> +The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
> +video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
> +
> +.. code-block:: c
> +
> + struct ipu3_uapi_params {
> + /* Flags which of the settings below are to be applied */
> + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> +
> + /* Accelerator cluster parameters */
> + struct ipu3_uapi_acc_param acc_param;
> +
> + /* ISP vector address space parameters */
> + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> +
> + /* ISP data memory (DMEM) parameters */
> + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> +
> + /* Optical black level compensation */
> + struct ipu3_uapi_obgrid_param obgrid_param;
> + } __packed;
> +
> +Intel IPU3 ImgU uAPI data types
> +===============================
> +
> +.. kernel-doc:: include/uapi/linux/intel-ipu3.h
> diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
> new file mode 100644
> index 0000000..c2608b6
> --- /dev/null
> +++ b/include/uapi/linux/intel-ipu3.h
> @@ -0,0 +1,2819 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2017 - 2018 Intel Corporation */
> +
> +#ifndef __IPU3_UAPI_H
> +#define __IPU3_UAPI_H
> +
> +#include <linux/types.h>
> +
> +/********************* Key Acronyms *************************/
> +/*
> + * ACC - Accelerator cluster
> + * ANR - Adaptive noise reduction
> + * AWB_FR- Auto white balance filter response statistics
> + * BNR - Bayer noise reduction parameters
> + * BDS - Bayer downscaler parameters
> + * CCM - Color correction matrix coefficients
> + * CDS - Chroma down sample
> + * CHNR - Chroma noise reduction
> + * CSC - Color space conversion
> + * DM - De-mosaic
> + * IEFd - Image enhancement filter directed
> + * Obgrid - Optical black level compensation
> + * OSYS - Output system configuration
> + * ROI - Region of interest
> + * SHD - Lens shading correction table
> + * TCC - Total color correction
> + * YDS - Y down sampling
> + * YTM - Y-tone mapping
> + */
> +
> +/*
> + * IPU3 DMA operations require buffers to be aligned at
> + * 32 byte boundaries
> + */
> +
> +/******************* ipu3_uapi_stats_3a *******************/
> +
> +#define IPU3_UAPI_MAX_STRIPES 2
> +#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
> +
> +#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
> +#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
> +
> +/* controls generation of meta_data (like FF enable/disable) */
> +#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
> +#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
> +
> +/**
> + * struct ipu3_uapi_grid_config - Grid plane config
> + *
> + * @width: Grid horizontal dimensions, in number of grid blocks(cells).
> + * @height: Grid vertical dimensions, in number of grid cells.
> + * @block_width_log2: Log2 of the width of each cell in pixels.
> + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> + * @block_height_log2: Log2 of the height of each cell in pixels.
> + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> + * @height_per_slice: The number of blocks in vertical axis per slice.
> + * Default 2.
> + * @x_start: X value of top left corner of Region of Interest(ROI).
> + * @y_start: Y value of top left corner of ROI
> + * @x_end: X value of bottom right corner of ROI
> + * @y_end: Y value of bottom right corner of ROI
> + *
> + * Due to the size of total amount of collected data, most statistics
> + * create a grid-based output, and the data is then divided into "slices".
> + */
> +struct ipu3_uapi_grid_config {
> + __u8 width;
> + __u8 height;
> + __u16 block_width_log2:3;
> + __u16 block_height_log2:3;
> + __u16 height_per_slice:8;
> + __u16 x_start;
> + __u16 y_start;
> + __u16 x_end;
> + __u16 y_end;
> +} __packed;
> +
> +/*
> + * The grid based data is divided into "slices" called set, each slice of setX
> + * refers to ipu3_uapi_grid_config width * height_per_slice.
> + */
> +#define IPU3_UAPI_AWB_MAX_SETS 60
> +/* Based on grid size 80 * 60 and cell size 16 x 16 */
> +#define IPU3_UAPI_AWB_SET_SIZE 1280
> +#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
> +#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AWB_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
> + (IPU3_UAPI_AWB_MAX_SETS * \
> + (IPU3_UAPI_AWB_SET_SIZE + IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
> +/**
> + * struct ipu3_uapi_awb_meta_data - AWB meta data
> + *
> + * @meta_data_buffer: Average values for each color channel
> + */
> +struct ipu3_uapi_awb_meta_data {
> + __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
> + *
> + * @meta_data: buffer to hold auto white balance meta data.
> + */
> +struct ipu3_uapi_awb_raw_buffer {
> + struct ipu3_uapi_awb_meta_data meta_data;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_config_s - AWB config
> + *
> + * @rgbs_thr_gr: gr threshold value.
> + * @rgbs_thr_r: Red threshold value.
> + * @rgbs_thr_gb: gb threshold value.
> + * @rgbs_thr_b: Blue threshold value.
> + * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
> + *
> + * The threshold is a saturation measure range [0, 8191], 8191 is default.
> + * Values over threshold may be optionally rejected for averaging.
> + */
> +struct ipu3_uapi_awb_config_s {
> + __u16 rgbs_thr_gr;
> + __u16 rgbs_thr_r;
> + __u16 rgbs_thr_gb;
> + __u16 rgbs_thr_b;
> + struct ipu3_uapi_grid_config grid;
> +} __attribute__((aligned(32))) __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_config - AWB config wrapper
> + *
> + * @config: config for auto white balance as defined by &ipu3_uapi_awb_config_s
> + */
> +struct ipu3_uapi_awb_config {
> + struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
> +} __packed;
> +
> +#define IPU3_UAPI_AE_COLORS 4 /* R, G, B, Y */
> +#define IPU3_UAPI_AE_BINS 256
> +#define IPU3_UAPI_AE_WEIGHTS 96
> +
> +/**
> + * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
> + *
> + * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
> + *
> + * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit unsigned
> + * for counting the number of the pixel.
> + */
> +struct ipu3_uapi_ae_raw_buffer {
> + __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
What's the order of the colour components? Do the components for the same
bin come together, or all bins for a given component?
Could this be written instead as:
struct {
__u32 gr;
__u32 g;
__u32 b;
__u32 gb;
} vals[IPU3_UAPI_AE_BINS] __packed __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
> + *
> + * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
> + */
> +struct ipu3_uapi_ae_raw_buffer_aligned {
> + struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
Please use struct ipu3_uapi_ae_raw_buffer directly.
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_grid_config - AE weight grid
> + *
> + * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
> + * @height: Grid vertical dimensions. Value: [16, 24], default 16.
> + * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
> + * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
2^3 == 16? Is the example wrong or the description of the field?
> + * @__reserved0: reserved
> + * @ae_en: 0: does not write to meta-data array, 1: write normally.
Is the meta-data array here the AE raw buffer, as defined above? If so,
please align the terms used.
> + * @rst_hist_array: write 1 to trigger histogram array reset.
> + * @done_rst_hist_array: flag for histogram array reset done.
> + * @x_start: X value of top left corner of ROI, default 0.
> + * @y_start: Y value of top left corner of ROI, default 0.
> + * @x_end: X value of bottom right corner of ROI
> + * @y_end: Y value of bottom right corner of ROI
> + *
> + * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
> + * a defined ROI within the frame. The contribution of each pixel into the
> + * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a grid.
> + */
> +struct ipu3_uapi_ae_grid_config {
> + __u8 width;
> + __u8 height;
> + __u8 block_width_log2:4;
> + __u8 block_height_log2:4;
> + __u8 __reserved0:5;
> + __u8 ae_en:1;
> + __u8 rst_hist_array:1;
> + __u8 done_rst_hist_array:1;
> + __u16 x_start;
> + __u16 y_start;
> + __u16 x_end;
> + __u16 y_end;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_weight_elem - AE weights LUT
> + *
> + * @cell0: weighted histogram grid value.
> + * @cell1: weighted histogram grid value.
> + * @cell2: weighted histogram grid value.
> + * @cell3: weighted histogram grid value.
> + * @cell4: weighted histogram grid value.
> + * @cell5: weighted histogram grid value.
> + * @cell6: weighted histogram grid value.
> + * @cell7: weighted histogram grid value.
> + *
> + * Use weighted grid value to give a different contribution factor to each cell.
> + * Precision u4, range [0, 15].
> + */
> +struct ipu3_uapi_ae_weight_elem {
> + __u32 cell0:4;
> + __u32 cell1:4;
> + __u32 cell2:4;
> + __u32 cell3:4;
> + __u32 cell4:4;
> + __u32 cell5:4;
> + __u32 cell6:4;
> + __u32 cell7:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
> + *
> + * @gain_gr: WB gain factor for the gr channels. Default 256.
> + * @gain_r: WB gain factor for the r channel. Default 256.
> + * @gain_b: WB gain factor for the b channel. Default 256.
> + * @gain_gb: WB gain factor for the gb channels. Default 256.
> + * @mat: 4x4 matrix that transforms Bayer quad output from WB to RGB+Y.
> + *
> + * Default:
> + * 128, 0, 0, 0,
> + * 0, 128, 0, 0,
> + * 0, 0, 128, 0,
> + * 0, 0, 0, 128,
> + *
> + * As part of the raw frame pre-process stage, the WB and color conversion need
> + * to be applied to expose the impact of these gain operations.
> + */
> +struct ipu3_uapi_ae_ccm {
> + __u16 gain_gr;
> + __u16 gain_r;
> + __u16 gain_b;
> + __u16 gain_gb;
> + __s16 mat[16];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_config - AE config
> + *
> + * @grid_cfg: config for auto exposure statistics grid. See struct
> + * &ipu3_uapi_ae_grid_config
> + * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks in the grid.
> + * Each grid cell has a corresponding value in weights LUT called
> + * grid value, global histogram is updated based on grid value and
> + * pixel value.
> + * @ae_ccm: Color convert matrix pre-processing block.
> + *
> + * Calculate AE grid from image resolution, resample ae weights.
> + */
> +struct ipu3_uapi_ae_config {
> + struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_weight_elem weights[
> + IPU3_UAPI_AE_WEIGHTS] __attribute__((aligned(32)));
Over 80 characters per line.
> + struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast measurements
> + *
> + * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
> + * anti-symmetry type. A12 is center, A1-A11 are neighbours.
> + * for analyzing low frequency content, used to calculate sum
> + * of gradients in x direction.
> + * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
> + * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
> + * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
> + * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
> + * @y1_coeff_1: Struct
> + * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
> + * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
> + * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
> + * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
> + * @y1_coeff_2: Struct
> + * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> + * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> + * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> + * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> + * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
> + * 0: positive, 1: negative, default 0.
> + * @y2_coeff_0: Y2, same structure as Y1. For analyzing high frequency content.
> + * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
> + * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
> + * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
> + * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
> + * @y2_coeff_1: Struct
> + * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
> + * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
> + * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
> + * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
> + * @y2_coeff_2: Struct
> + * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> + * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> + * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> + * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> + * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
> + * 0: positive, 1: negative, default 0.
> + * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to be
> + * used for building histogram. Range [0, 32], default 8.
> + * Rule:
> + * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b + y_gen_rate_gb = 32
> + * A single Y is calculated based on sum of Gr/R/B/Gb based on
> + * their contribution ratio.
> + * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
> + * @y_calc.y_gen_rate_r: Contribution ratio R for Y
> + * @y_calc.y_gen_rate_b: Contribution ratio B for Y
> + * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
> + * @nf: The shift right value that should be applied during the Y1/Y2 filter to
> + * make sure the total memory needed is 2 bytes per grid cell.
> + * @nf.__reserved0: reserved
> + * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
> + * should be log2 of the sum of the abs values of the filter
> + * coeffs, default 7 (2^7 = 128).
> + * @nf.__reserved1: reserved
> + * @nf.y2_nf: Normalization factor for y2, should be log2 of the sum of the
> + * abs values of the filter coeffs.
> + * @nf.__reserved2: reserved
> + */
> +struct ipu3_uapi_af_filter_config {
> + struct {
> + __u8 a1;
> + __u8 a2;
> + __u8 a3;
> + __u8 a4;
> + } y1_coeff_0;
> + struct {
> + __u8 a5;
> + __u8 a6;
> + __u8 a7;
> + __u8 a8;
> + } y1_coeff_1;
> + struct {
> + __u8 a9;
> + __u8 a10;
> + __u8 a11;
> + __u8 a12;
> + } y1_coeff_2;
> +
> + __u32 y1_sign_vec;
> +
> + struct {
> + __u8 a1;
> + __u8 a2;
> + __u8 a3;
> + __u8 a4;
> + } y2_coeff_0;
> + struct {
> + __u8 a5;
> + __u8 a6;
> + __u8 a7;
> + __u8 a8;
> + } y2_coeff_1;
> + struct {
> + __u8 a9;
> + __u8 a10;
> + __u8 a11;
> + __u8 a12;
> + } y2_coeff_2;
> +
> + __u32 y2_sign_vec;
> +
> + struct {
> + __u8 y_gen_rate_gr;
> + __u8 y_gen_rate_r;
> + __u8 y_gen_rate_b;
> + __u8 y_gen_rate_gb;
> + } y_calc;
> +
> + struct {
> + __u32 __reserved0:8;
> + __u32 y1_nf:4;
> + __u32 __reserved1:4;
> + __u32 y2_nf:4;
> + __u32 __reserved2:12;
> + } nf;
> +} __packed;
> +
> +#define IPU3_UAPI_AF_MAX_SETS 24
> +#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
> +#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AF_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
> +#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
> + (IPU3_UAPI_AF_MAX_SETS * \
> + (IPU3_UAPI_AF_Y_TABLE_SET_SIZE + IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
> + IPU3_UAPI_MAX_STRIPES)
> +
> +/**
> + * struct ipu3_uapi_af_meta_data - AF meta data
> + *
> + * @y_table: Each color component will be convolved separately with filter1
> + * and filter2 and the result will be summed out and averaged for
> + * each cell.
> + */
> +struct ipu3_uapi_af_meta_data {
> + __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE] __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_raw_buffer - AF raw buffer
> + *
> + * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus meta data.
> + */
> +struct ipu3_uapi_af_raw_buffer {
> + struct ipu3_uapi_af_meta_data meta_data __attribute__((aligned(32)));
How about:
__u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE]
__attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_config_s - AF config
> + *
> + * @filter_config: AF uses Y1 and Y2 filters as configured in
> + * &ipu3_uapi_af_filter_config
> + * @padding: paddings
> + * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use large
> + * grid size for large image and vice versa.
> + */
> +struct ipu3_uapi_af_config_s {
> + struct ipu3_uapi_af_filter_config filter_config __attribute__((aligned(32)));
> + __u8 padding[4];
> + struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_config - AF config wrapper
> + *
> + * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
> + */
> +struct ipu3_uapi_af_config {
Could you drop struct ipu3_uapi_af_config and use ipu3_uapi_af_config_s
instead?
> + struct ipu3_uapi_af_config_s config;
> +} __packed;
> +
> +#define IPU3_UAPI_AWB_FR_MAX_SETS 24
> +#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
> +#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
> +#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
> + (IPU3_UAPI_AWB_FR_MAX_SETS * \
> + (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
> + IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) * IPU3_UAPI_MAX_STRIPES)
> +
> +/**
> + * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
> + *
> + * @bayer_table: Statistics output on the grid after convolving with 1D filter.
> + */
> +struct ipu3_uapi_awb_fr_meta_data {
> + __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE] __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
> + *
> + * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
> + */
> +struct ipu3_uapi_awb_fr_raw_buffer {
Same here, please use ipu3_uapi_awb_fr_meta_data instead.
> + struct ipu3_uapi_awb_fr_meta_data meta_data;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
> + *
> + * @grid_cfg: grid config, default 16x16.
> + * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
> + * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
> + * Applied on whole image for each Bayer channel separately
> + * by a weighted sum of its 11x1 neighbors.
> + * @__reserved1: reserved
> + * @bayer_sign: sign of filter coeffcients, default 0.
> + * @bayer_nf: normalization factor for the convolution coeffs, to make sure
> + * total memory needed is within pre-determined range.
> + * NF should be the log2 of the sum of the abs values of the
> + * filter coeffs, range [7, 14], default 7.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_awb_fr_config_s {
> + struct ipu3_uapi_grid_config grid_cfg;
> + __u8 bayer_coeff[6];
> + __u16 __reserved1;
> + __u32 bayer_sign;
> + __u8 bayer_nf;
> + __u8 __reserved2[3];
> +} __attribute__((aligned(32))) __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
> + *
> + * @config: See &ipu3_uapi_awb_fr_config_s.
> + */
> +struct ipu3_uapi_awb_fr_config {
Ditto.
> + struct ipu3_uapi_awb_fr_config_s config;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_4a_config - 4A config
> + *
> + * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
> + * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
> + * @padding: paddings
> + * @af_config: auto focus config &ipu3_uapi_af_config_s
> + * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution 16x16
> + */
> +struct ipu3_uapi_4a_config {
> + struct ipu3_uapi_awb_config_s awb_config __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_grid_config ae_grd_config;
> + __u8 padding[20];
> + struct ipu3_uapi_af_config_s af_config;
> + struct ipu3_uapi_awb_fr_config_s awb_fr_config;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
> + *
> + * @num_of_stripes: A single frame is divided into several parts called stripes
> + * due to limitation on line buffer memory.
> + * The separation between the stripes is vertical. Each such
> + * stripe is processed as a single frame by the ISP pipe.
> + * @padding: padding bytes.
> + * @num_sets: number of sets.
> + * @padding1: padding bytes.
> + * @size_of_set: set size.
> + * @padding2: padding bytes.
> + * @bubble_size: is the amount of padding in the bubble expressed in "sets".
> + * @padding3: padding bytes.
> + */
> +struct ipu3_uapi_bubble_info {
> + __u32 num_of_stripes __attribute__((aligned(32)));
> + __u8 padding[28];
> + __u32 num_sets;
> + __u8 padding1[28];
> + __u32 size_of_set;
> + __u8 padding2[28];
> + __u32 bubble_size;
> + __u8 padding3[28];
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + */
> +struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
> + struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
> + *
> + * @awb_en: auto white balance enable
> + * @padding: padding config
> + * @ae_en: auto exposure enable
> + * @padding1: padding config
> + * @af_en: auto focus enable
> + * @padding2: padding config
> + * @awb_fr_en: awb filter response enable bit
> + * @padding3: padding config
> + */
> +struct ipu3_uapi_ff_status {
> + __u32 awb_en __attribute__((aligned(32)));
> + __u8 padding[28];
> + __u32 ae_en;
> + __u8 padding1[28];
> + __u32 af_en;
> + __u8 padding2[28];
> + __u32 awb_fr_en;
> + __u8 padding3[28];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_stats_3a - 3A statistics
> + *
> + * @awb_raw_buffer: auto white balance meta data &ipu3_uapi_awb_raw_buffer
> + * @ae_raw_buffer: auto exposure raw data &ipu3_uapi_ae_raw_buffer_aligned
> + * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
> + * @awb_fr_raw_buffer: value as specified by &ipu3_uapi_awb_fr_raw_buffer
> + * @stats_4a_config: 4a statistics config as defined by &ipu3_uapi_4a_config.
> + * @ae_join_buffers: 1 to use ae_raw_buffer.
> + * @padding: padding config
> + * @stats_3a_bubble_per_stripe: a &ipu3_uapi_stats_3a_bubble_info_per_stripe
> + * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
> + */
> +struct ipu3_uapi_stats_3a {
> + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_raw_buffer_aligned
> + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> + struct ipu3_uapi_4a_config stats_4a_config;
> + __u32 ae_join_buffers;
> + __u8 padding[28];
> + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + stats_3a_bubble_per_stripe;
> + struct ipu3_uapi_ff_status stats_3a_status;
> +} __packed;
> +
> +/******************* ipu3_uapi_acc_param *******************/
> +
> +#define IPU3_UAPI_ISP_VEC_ELEMS 64
> +#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
> +
> +#define IPU3_UAPI_BNR_LUT_SIZE 32
> +
> +/* number of elements in gamma correction LUT */
> +#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
> +
> +/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
> +#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
> +#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
> +/* Normalization shift aka nf */
> +#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
> +#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
> +
> +#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
> +#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
> +#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
> +#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
> +
> +#define IPU3_UAPI_ANR_LUT_SIZE 26
> +#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
> +
> +#define IPU3_UAPI_LIN_LUT_SIZE 64
> +
> +/* Bayer Noise Reduction related structs */
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance gains
> + *
> + * @gr: white balance gain for Gr channel.
> + * @r: white balance gain for R channel.
> + * @b: white balance gain for B channel.
> + * @gb: white balance gain for Gb channel.
> + *
> + * Precision u3.13, range [0, 8]. White balance correction is done by applying
[0, 8[
(or [0, 8), but the same notation needs to be used everywhere).
> + * a multiplicative gain to each color channels prior to BNR.
> + */
> +struct ipu3_uapi_bnr_static_config_wb_gains_config {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold config
> + *
> + * @gr: white balance threshold gain for Gr channel.
> + * @r: white balance threshold gain for R channel.
> + * @b: white balance threshold gain for B channel.
> + * @gb: white balance threshold gain for Gb channel.
> + *
> + * Defines the threshold that specifies how different a defect pixel can be from
> + * its neighbors.(used by dynamic defect pixel correction sub block)
> + * Precision u4.4 range [0, 8].
> + */
> +struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
> + __u8 gr;
> + __u8 r;
> + __u8 b;
> + __u8 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
> + * coefficients that controls noise threshold
> + *
> + * @cf: Free coefficient for threshold calculation, range [0, 8191], default 0.
> + * @__reserved0: reserved
> + * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
> + * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
> + * default 6.
> + * format: u3.2 (3 most significant bits represent whole number,
> + * 2 least significant bits represent the fractional part
> + * with each count representing 0.25)
> + * e.g 6 in binary format is 00110, that translates to 1.5
> + * @__reserved1: reserved
> + * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
> + * where r is a radius of pixel [row, col] from centor of sensor.
> + * default 14.
> + *
> + * Threshold used to distinguish between noise and details.
> + */
> +struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
> + __u32 cf:13;
> + __u32 __reserved0:3;
> + __u32 cg:5;
> + __u32 ci:5;
> + __u32 __reserved1:1;
> + __u32 r_nf:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
> + *
> + * @gr: Coefficient defines lens shading gain approximation for gr channel
> + * @r: Coefficient defines lens shading gain approximation for r channel
> + * @b: Coefficient defines lens shading gain approximation for b channel
> + * @gb: Coefficient defines lens shading gain approximation for gb channel
> + *
> + * Parameters for noise model (NM) adaptation of BNR due to shading correction.
> + * All above have precision of u3.3, default to 0.
> + */
> +struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
> + __u8 gr;
> + __u8 r;
> + __u8 b;
> + __u8 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center config
> + *
> + * @x_reset: Reset value of X (col start - X center). Precision s12.0.
> + * @__reserved0: reserved
> + * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
> + * @__reserved2: reserved
> + *
> + * Distance from corner to optical center for NM adaptation due to shading
> + * correction (should be calculated based on shading tables)
> + */
> +struct ipu3_uapi_bnr_static_config_opt_center_config {
> + __s32 x_reset:13;
> + __u32 __reserved0:3;
> + __s32 y_reset:13;
> + __u32 __reserved2:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup table
> + *
> + * @values: pre-calculated values of square root function.
> + *
> + * LUT implementation of square root operation.
> + */
> +struct ipu3_uapi_bnr_static_config_lut_config {
> + __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels (bp)
> + *
> + * @bp_thr_gain: Defines the threshold that specifies how different a
> + * defect pixel can be from its neighbors. Threshold is
> + * dependent on de-noise threshold calculated by algorithm.
> + * Range [4, 31], default 4.
> + * @__reserved0: reserved
> + * @defect_mode: Mode of addressed defect pixels,
> + * 0 - single defect pixel is expected,
> + * 1 - 2 adjacent defect pixels are expected, default 1.
> + * @bp_gain: Defines how 2nd derivation that passes through a defect pixel
> + * is different from 2nd derivations that pass through
> + * neighbor pixels. u4.2, range [0, 256], default 8.
> + * @__reserved1: reserved
> + * @w0_coeff: Blending coefficient of defect pixel correction.
> + * Precision u4, range [0, 8], default 8.
> + * @__reserved2: reserved
> + * @w1_coeff: Enable influence of incorrect defect pixel correction to be
> + * avoided. Precision u4, range [1, 8], default 8.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
> + __u32 bp_thr_gain:5;
> + __u32 __reserved0:2;
> + __u32 defect_mode:1;
> + __u32 bp_gain:6;
> + __u32 __reserved1:18;
> + __u32 w0_coeff:4;
> + __u32 __reserved2:4;
> + __u32 w1_coeff:4;
> + __u32 __reserved3:20;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising config
> + *
> + * @alpha: Weight of central element of smoothing filter.
> + * @beta: Weight of peripheral elements of smoothing filter, default 4.
> + * @gamma: Weight of diagonal elements of smoothing filter, default 4.
> + *
> + * beta and gamma parameter define the strength of the noise removal filter.
> + * All above has precision u0.4, range [0, 0xf]
> + * format: u0.4 (no / zero bits represent whole number,
> + * 4 bits represent the fractional part
> + * with each count representing 0.0625)
> + * e.g 0xf translates to 0.0625x15 = 0.9375
> + *
> + * @__reserved0: reserved
> + * @max_inf: Maximum increase of peripheral or diagonal element influence
> + * relative to the pre-defined value range: [0x5, 0xa]
> + * @__reserved1: reserved
> + * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
> + * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 - enable.
> + * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 - enable.
> + * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
> + __u32 alpha:4;
> + __u32 beta:4;
> + __u32 gamma:4;
> + __u32 __reserved0:4;
> + __u32 max_inf:4;
> + __u32 __reserved1:7;
> + __u32 gd_enable:1;
> + __u32 bpc_enable:1;
> + __u32 bnr_enable:1;
> + __u32 ff_enable:1;
> + __u32 __reserved2:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical square
> + *
> + * @x_sqr_reset: Reset value of X^2.
> + * @y_sqr_reset: Reset value of Y^2.
> + *
> + * Please note:
> + *
> + * #. X and Y ref to
> + * &ipu3_uapi_bnr_static_config_opt_center_config
> + * #. Both structs are used in threshold formula to calculate r^2, where r
> + * is a radius of pixel [row, col] from centor of sensor.
> + */
> +struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
> + __u32 x_sqr_reset;
> + __u32 y_sqr_reset;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config - BNR static config
> + *
> + * @wb_gains: white balance gains &ipu3_uapi_bnr_static_config_wb_gains_config
> + * @wb_gains_thr: white balance gains threshold as defined by
> + * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
> + * @thr_coeffs: coefficients of threshold
> + * &ipu3_uapi_bnr_static_config_thr_coeffs_config
> + * @thr_ctrl_shd: control of shading threshold
> + * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
> + * @opt_center: optical center &ipu3_uapi_bnr_static_config_opt_center_config
> + *
> + * Above parameters and opt_center_sqr are used for white balance and shading.
> + *
> + * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
> + * @bp_ctrl: detect and remove bad pixels as defined in struct
> + * &ipu3_uapi_bnr_static_config_bp_ctrl_config
> + * @dn_detect_ctrl: detect and remove noise.
> + * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> + * @column_size: The number of pixels in column.
> + * @opt_center_sqr: Reset value of r^2 to optical center, see
> + * &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
> + */
> +struct ipu3_uapi_bnr_static_config {
> + struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
> + struct ipu3_uapi_bnr_static_config_wb_gains_thr_config wb_gains_thr;
> + struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
> + struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
> + struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
> + struct ipu3_uapi_bnr_static_config_lut_config lut;
> + struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
> + struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config dn_detect_ctrl;
> + __u32 column_size;
> + struct ipu3_uapi_bnr_static_config_opt_center_sqr_config opt_center_sqr;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green disparity
> + *
> + * @gd_red: Shading gain coeff for gr disparity level in bright red region.
> + * Precision u0.6, default 4(0.0625).
> + * @__reserved0: reserved
> + * @gd_green: Shading gain coeff for gr disparity level in bright green
> + * region. Precision u0.6, default 4(0.0625).
> + * @__reserved1: reserved
> + * @gd_blue: Shading gain coeff for gr disparity level in bright blue region.
> + * Precision u0.6, default 4(0.0625).
> + * @__reserved2: reserved
> + * @gd_black: Maximal green disparity level in dark region (stronger disparity
> + * assumed to be image detail). Precision u14, default 80.
> + * @__reserved3: reserved
> + * @gd_shading: Change maximal green disparity level according to square
> + * distance from image center.
> + * @__reserved4: reserved
> + * @gd_support: Lower bound for the number of second green color pixels in
> + * current pixel neighborhood with less than threshold difference
> + * from it.
> + *
> + * The shading gain coeff of red, green, blue and black are used to calculate
> + * threshold given a pixel's color value and its coordinates in the image.
> + *
> + * @__reserved5: reserved
> + * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
> + * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
> + */
> +struct ipu3_uapi_bnr_static_config_green_disparity {
> + __u32 gd_red:6;
> + __u32 __reserved0:2;
> + __u32 gd_green:6;
> + __u32 __reserved1:2;
> + __u32 gd_blue:6;
> + __u32 __reserved2:10;
> + __u32 gd_black:14;
> + __u32 __reserved3:2;
> + __u32 gd_shading:7;
> + __u32 __reserved4:1;
> + __u32 gd_support:2;
> + __u32 __reserved5:1;
> + __u32 gd_clip:1;
> + __u32 gd_central_weight:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_dm_config - De-mosaic parameters
> + *
> + * @dm_en: de-mosaic enable.
> + * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
> + * @fcc_en: False color correction (FCC) enable flag. Default 0.
> + * @__reserved0: reserved
> + * @frame_width: do not care
> + * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation of
> + * complementary color in Hamilton-Adams interpolation).
> + * u5, range [0, 31], default 8.
> + * @__reserved1: reserved
> + * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity metric
> + * in calculation of final homogeneity metric.
> + * u5, range [0, 31], default 7.
> + * @__reserved2: reserved
> + * @cr_param1: First parameter that defines Checker artifact removal
> + * feature gain.Precision u5, range [0, 31], default 8.
> + * @__reserved3: reserved
> + * @cr_param2: Second parameter that defines Checker artifact removal
> + * feature gain. Precision u5, range [0, 31], default 8.
> + * @__reserved4: reserved
> + * @coring_param: Defines power of false color correction operation.
> + * low for preserving edge colors, high for preserving gray
> + * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
> + * @__reserved5: reserved
> + *
> + * The demosaic fixed function block is responsible to covert Bayer(mosaiced)
> + * images into color images based on demosaicing algorithm.
> + */
> +struct ipu3_uapi_dm_config {
> + __u32 dm_en:1;
> + __u32 ch_ar_en:1;
> + __u32 fcc_en:1;
> + __u32 __reserved0:13;
> + __u32 frame_width:16;
> +
> + __u32 gamma_sc:5;
> + __u32 __reserved1:3;
> + __u32 lc_ctrl:5;
> + __u32 __reserved2:3;
> + __u32 cr_param1:5;
> + __u32 __reserved3:3;
> + __u32 cr_param2:5;
> + __u32 __reserved4:3;
> +
> + __u32 coring_param:5;
> + __u32 __reserved5:27;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ccm_mat_config - Color correction matrix
> + *
> + * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
> + * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
> + * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
> + * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
> + * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
> + *
> + * Transform sensor specific color space to standard sRGB by applying 3x3 matrix
> + * and adding a bias vector O. The transformation is basically a rotation and
> + * translation in the 3-dimensional color spaces. Here are the defaults:
> + *
> + * 9775, -2671, 1087, 0
> + * -1071, 8303, 815, 0
> + * -23, -7887, 16103, 0
> + */
> +struct ipu3_uapi_ccm_mat_config {
> + __s16 coeff_m11;
> + __s16 coeff_m12;
> + __s16 coeff_m13;
> + __s16 coeff_o_r;
> + __s16 coeff_m21;
> + __s16 coeff_m22;
> + __s16 coeff_m23;
> + __s16 coeff_o_g;
> + __s16 coeff_m31;
> + __s16 coeff_m32;
> + __s16 coeff_m33;
> + __s16 coeff_o_b;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
> + *
> + * @enable: gamma correction enable.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_gamma_corr_ctrl {
> + __u32 enable:1;
> + __u32 __reserved:31;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping implemented as LUT.
> + *
> + * @lut: 256 tabulated values of the gamma function. LUT[1].. LUT[256]
> + * format u13.0, range [0, 8191].
> + *
> + * The tone mapping operation is done by a Piece wise linear graph
> + * that is implemented as a lookup table(LUT). The pixel component input
> + * intensity is the X-axis of the graph which is the table entry.
> + */
> +struct ipu3_uapi_gamma_corr_lut {
> + __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_config - Gamma config
> + *
> + * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
> + * @gc_lut: lookup table of gamma correction &ipu3_uapi_gamma_corr_lut
> + */
> +struct ipu3_uapi_gamma_config {
> + struct ipu3_uapi_gamma_corr_ctrl gc_ctrl __attribute__((aligned(32)));
> + struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
> + *
> + * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1], default 1.
You can't represent 1; it's 8191/8192. How about [-1, 1[ ?
Is the default 1 or 8191? The numerical value is used elsewhere but here
it seems that this might not be the case. The same for other cases below.
> + * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + *
> + * To transform each pixel from RGB to YUV (Y - brightness/luminance,
> + * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding an
> + * optional bias 3x1 vector.
> + */
> +struct ipu3_uapi_csc_mat_config {
> + __s16 coeff_c11;
> + __s16 coeff_c12;
> + __s16 coeff_c13;
> + __s16 coeff_b1;
> + __s16 coeff_c21;
> + __s16 coeff_c22;
> + __s16 coeff_c23;
> + __s16 coeff_b2;
> + __s16 coeff_c31;
> + __s16 coeff_c32;
> + __s16 coeff_c33;
> + __s16 coeff_b3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cds_params - Chroma down-scaling
> + *
> + * @ds_c00: range [0, 3]
> + * @ds_c01: range [0, 3]
> + * @ds_c02: range [0, 3]
> + * @ds_c03: range [0, 3]
> + * @ds_c10: range [0, 3]
> + * @ds_c11: range [0, 3]
> + * @ds_c12: range [0, 3]
> + * @ds_c13: range [0, 3]
> + *
> + * In case user does not provide, above 4x2 filter will use following defaults:
> + * 1, 3, 3, 1,
> + * 1, 3, 3, 1,
> + *
> + * @ds_nf: Normalization factor for Chroma output downscaling filter,
> + * range 0,4, default 2.
> + * @__reserved0: reserved
> + * @csc_en: Color space conversion enable
> + * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_cds_params {
> + __u32 ds_c00:2;
> + __u32 ds_c01:2;
> + __u32 ds_c02:2;
> + __u32 ds_c03:2;
> + __u32 ds_c10:2;
> + __u32 ds_c11:2;
> + __u32 ds_c12:2;
> + __u32 ds_c13:2;
> + __u32 ds_nf:5;
> + __u32 __reserved0:3;
> + __u32 csc_en:1;
> + __u32 uv_bin_output:1;
> + __u32 __reserved1:6;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
> + *
> + * @width: Grid horizontal dimensions, u8, [8, 128], default 73
> + * @height: Grid vertical dimensions, u8, [8, 128], default 56
> + * @block_width_log2: Log2 of the width of the grid cell in pixel count
> + * u4, [0, 15], default value 5.
> + * @__reserved0: reserved
> + * @block_height_log2: Log2 of the height of the grid cell in pixel count
> + * u4, [0, 15], default value 6.
> + * @__reserved1: reserved
> + * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
> + * (with SHD_MAX_CELLS_PER_SET = 146).
> + * @x_start: X value of top left corner of sensor relative to ROI
> + * u12, [-4096, 0]. default 0, only negative values.
> + * @y_start: Y value of top left corner of sensor relative to ROI
> + * u12, [-4096, 0]. default 0, only negative values.
I suppose u12 is incorrect here, if the value is signed --- and negative
(sign bit) if not 0?
> + */
> +struct ipu3_uapi_shd_grid_config {
> + /* reg 0 */
> + __u8 width;
> + __u8 height;
> + __u8 block_width_log2:3;
> + __u8 __reserved0:1;
> + __u8 block_height_log2:3;
> + __u8 __reserved1:1;
> + __u8 grid_height_per_slice;
> + /* reg 1 */
> + __s16 x_start;
> + __s16 y_start;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_general_config - Shading general config
> + *
> + * @init_set_vrt_offst_ul: set vertical offset,
> + * y_start >> block_height_log2 % grid_height_per_slice.
> + * @shd_enable: shading enable.
> + * @gain_factor: Gain factor. Shift calculated anti shading value. Precision u2.
> + * 0x0 - gain factor [1, 5], means no shift interpolated value.
> + * 0x1 - gain factor [1, 9], means shift interpolated by 1.
> + * 0x2 - gain factor [1, 17], means shift interpolated by 2.
> + * @__reserved: reserved
> + *
> + * Correction is performed by multiplying a gain factor for each of the 4 Bayer
> + * channels as a function of the pixel location in the sensor.
> + */
> +struct ipu3_uapi_shd_general_config {
> + __u32 init_set_vrt_offst_ul:8;
> + __u32 shd_enable:1;
> + __u32 gain_factor:2;
> + __u32 __reserved:21;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_black_level_config - Black level correction
> + *
> + * @bl_r: Bios values for green red. s11 range [-2048, 2047].
> + * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
> + * @bl_gb: Bios values for red. s11 range [-2048, 2047].
> + * @bl_b: Bios values for blue. s11 range [-2048, 2047].
> + */
> +struct ipu3_uapi_shd_black_level_config {
> + __s16 bl_r;
> + __s16 bl_gr;
> + __s16 bl_gb;
> + __s16 bl_b;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_config_static - Shading config static
> + *
> + * @grid: shading grid config &ipu3_uapi_shd_grid_config
> + * @general: shading general config &ipu3_uapi_shd_general_config
> + * @black_level: black level config for shading correction as defined by
> + * &ipu3_uapi_shd_black_level_config
> + */
> +struct ipu3_uapi_shd_config_static {
> + struct ipu3_uapi_shd_grid_config grid;
> + struct ipu3_uapi_shd_general_config general;
> + struct ipu3_uapi_shd_black_level_config black_level;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
> + *
> + * @sets: array
> + * @sets.r_and_gr: Red and GreenR Lookup table.
> + * @sets.r_and_gr.r: Red shading factor.
> + * @sets.r_and_gr.gr: GreenR shading factor.
> + * @sets.__reserved1: reserved
> + * @sets.gb_and_b: GreenB and Blue Lookup table.
> + * @sets.gb_and_b.gb: GreenB shading factor.
> + * @sets.gb_and_b.b: Blue shading factor.
> + * @sets.__reserved2: reserved
> + *
> + * Map to shading correction LUT register set.
> + */
> +struct ipu3_uapi_shd_lut {
> + struct {
> + struct {
> + __u16 r;
> + __u16 gr;
> + } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> + __u8 __reserved1[24];
> + struct {
> + __u16 gb;
> + __u16 b;
> + } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> + __u8 __reserved2[24];
> + } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_config - Shading config
> + *
> + * @shd: shading static config, see &ipu3_uapi_shd_config_static
> + * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
> + */
> +struct ipu3_uapi_shd_config {
> + struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
> + struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
> +} __packed;
> +
> +/* Image Enhancement Filter directed */
> +
> +/**
> + * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
> + *
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A of Config Unit, s4.4, default 0.
> + * @b01: Always 0.
> + *
> + * Calculate weight for blending directed and non-directed denoise elements
> + *
> + * Note:
> + * Each instance of Config Unit needs X coordinate of n points and
> + * slope A factor between points calculated by driver based on calibration
> + * parameters.
> + */
> +struct ipu3_uapi_iefd_cux2 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 a01:9;
> + __u32 b01:5; /* NOTE: hardcoded to zero */
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed sharpening
> + * element, Config Unit 6 for edge detail (ED).
> + *
> + * @x0: X coordinate of point 0, u9.0, default 0.
> + * @x1: X coordinate of point 1, u9.0, default 0.
> + * @x2: X coordinate of point 2, u9.0, default 0.
> + * @__reserved0: reserved
> + * @x3: X coordinate of point 3, u9.0, default 0.
> + * @x4: X coordinate of point 4, u9.0, default 0.
> + * @x5: X coordinate of point 5, u9.0, default 0.
> + * @__reserved1: reserved
> + * @a01: slope A points 01, s4.4, default 0.
> + * @a12: slope A points 12, s4.4, default 0.
> + * @a23: slope A points 23, s4.4, default 0.
> + * @__reserved2: reserved
> + * @a34: slope A points 34, s4.4, default 0.
> + * @a45: slope A points 45, s4.4, default 0.
> + * @__reserved3: reserved
> + * @b01: slope B points 01, s4.4, default 0.
> + * @b12: slope B points 12, s4.4, default 0.
> + * @b23: slope B points 23, s4.4, default 0.
> + * @__reserved4: reserved
> + * @b34: slope B points 34, s4.4, default 0.
> + * @b45: slope B points 45, s4.4, default 0.
> + * @__reserved5: reserved
> + */
> +struct ipu3_uapi_iefd_cux6_ed {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 x2:9;
> + __u32 __reserved0:5;
> +
> + __u32 x3:9;
> + __u32 x4:9;
> + __u32 x5:9;
> + __u32 __reserved1:5;
> +
> + __u32 a01:9;
> + __u32 a12:9;
> + __u32 a23:9;
> + __u32 __reserved2:5;
> +
> + __u32 a34:9;
> + __u32 a45:9;
> + __u32 __reserved3:14;
> +
> + __u32 b01:9;
> + __u32 b12:9;
> + __u32 b23:9;
> + __u32 __reserved4:5;
> +
> + __u32 b34:9;
> + __u32 b45:9;
> + __u32 __reserved5:14;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
> + * element apply.
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A of Config Unit, s4.4, default 0.
The field is marked unsigned below. Which one is correct?
> + * @__reserved1: reserved
> + * @b01: offset B0 of Config Unit, u7.0, default 0.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_iefd_cux2_1 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 a01:9;
> + __u32 __reserved1:5;
> +
> + __u32 b01:8;
> + __u32 __reserved2:24;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed sharpening
> + * element.
> + *
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @x2: X2 point of Config Unit, u9.0, default 0.
> + * @__reserved0: reserved
> + * @x3: X3 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A0 of Config Unit, s4.4, default 0.
> + * @a12: Slope A1 of Config Unit, s4.4, default 0.
Same here, suggest __s32 below if this is signed.
> + * @__reserved1: reserved
> + * @a23: Slope A2 of Config Unit, s4.4, default 0.
> + * @b01: Offset B0 of Config Unit, s7.0, default 0.
> + * @b12: Offset B1 of Config Unit, s7.0, default 0.
> + * @__reserved2: reserved
> + * @b23: Offset B2 of Config Unit, s7.0, default 0.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_iefd_cux4 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 x2:9;
> + __u32 __reserved0:5;
> +
> + __u32 x3:9;
> + __u32 a01:9;
> + __u32 a12:9;
> + __u32 __reserved1:5;
> +
> + __u32 a23:9;
> + __u32 b01:8;
> + __u32 b12:8;
> + __u32 __reserved2:7;
> +
> + __u32 b23:8;
> + __u32 __reserved3:24;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
> + *
> + * @x0: x0 points of Config Unit radial, u8.0
> + * @x1: x1 points of Config Unit radial, u8.0
> + * @x2: x2 points of Config Unit radial, u8.0
> + * @x3: x3 points of Config Unit radial, u8.0
> + * @x4: x4 points of Config Unit radial, u8.0
> + * @x5: x5 points of Config Unit radial, u8.0
> + * @__reserved1: reserved
> + * @a01: Slope A of Config Unit radial, s7.8
> + * @a12: Slope A of Config Unit radial, s7.8
> + * @a23: Slope A of Config Unit radial, s7.8
> + * @a34: Slope A of Config Unit radial, s7.8
> + * @a45: Slope A of Config Unit radial, s7.8
> + * @__reserved2: reserved
> + * @b01: Slope B of Config Unit radial, s9.0
> + * @b12: Slope B of Config Unit radial, s9.0
> + * @b23: Slope B of Config Unit radial, s9.0
> + * @__reserved4: reserved
> + * @b34: Slope B of Config Unit radial, s9.0
> + * @b45: Slope B of Config Unit radial, s9.0
> + * @__reserved5: reserved
> + */
> +struct ipu3_uapi_iefd_cux6_rad {
> + __u32 x0:8;
> + __u32 x1:8;
> + __u32 x2:8;
> + __u32 x3:8;
> +
> + __u32 x4:8;
> + __u32 x5:8;
> + __u32 __reserved1:16;
> +
> + __u32 a01:16;
> + __u32 a12:16;
> +
> + __u32 a23:16;
> + __u32 a34:16;
> +
> + __u32 a45:16;
> + __u32 __reserved2:16;
> +
> + __u32 b01:10;
> + __u32 b12:10;
> + __u32 b23:10;
> + __u32 __reserved4:2;
> +
> + __u32 b34:10;
> + __u32 b45:10;
> + __u32 __reserved5:12;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
> + *
> + * @cu_1: calculate weight for blending directed and
> + * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
> + * @cu_ed: calculate power of non-directed sharpening element, see
> + * &ipu3_uapi_iefd_cux6_ed
> + * @cu_3: calculate weight for blending directed and
> + * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
> + * @cu_5: calculate power of non-directed denoise element apply, use
> + * &ipu3_uapi_iefd_cux2_1
> + * @cu_6: calculate power of non-directed sharpening element. See
> + * &ipu3_uapi_iefd_cux4
> + * @cu_7: calculate weight for blending directed and
> + * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
> + * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
> + * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
> + * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
> + */
> +struct ipu3_uapi_yuvp1_iefd_cfg_units {
> + struct ipu3_uapi_iefd_cux2 cu_1;
> + struct ipu3_uapi_iefd_cux6_ed cu_ed;
> + struct ipu3_uapi_iefd_cux2 cu_3;
> + struct ipu3_uapi_iefd_cux2_1 cu_5;
> + struct ipu3_uapi_iefd_cux4 cu_6;
> + struct ipu3_uapi_iefd_cux2 cu_7;
> + struct ipu3_uapi_iefd_cux4 cu_unsharp;
> + struct ipu3_uapi_iefd_cux6_rad cu_radial;
> + struct ipu3_uapi_iefd_cux2 cu_vssnlm;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
> + *
> + * @horver_diag_coeff: Gradiant compensation, coefficient that compensates for
> + * different distance for vertical / horizontal and diagonal
> + * * gradient calculation (~1/sqrt(2)).
> + * @__reserved0: reserved
> + * @clamp_stitch: Slope to stitch between clamped and unclamped edge values
> + * @__reserved1: reserved
> + * @direct_metric_update: Update coeff for direction metric
> + * @__reserved2: reserved
> + * @ed_horver_diag_coeff: Radial Coefficient that compensates for
> + * different distance for vertical/horizontal and
> + * diagonal gradient calculation (~1/sqrt(2))
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_iefd_config_s {
> + __u32 horver_diag_coeff:7;
> + __u32 __reserved0:1;
> + __u32 clamp_stitch:6;
> + __u32 __reserved1:2;
> + __u32 direct_metric_update:5;
> + __u32 __reserved2:3;
> + __u32 ed_horver_diag_coeff:7;
> + __u32 __reserved3:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
> + *
> + * @iefd_en: Enable IEFd
> + * @denoise_en: Enable denoise
> + * @direct_smooth_en: Enable directional smooth
> + * @rad_en: Enable radial update
> + * @vssnlm_en: Enable VSSNLM output filter
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_yuvp1_iefd_control {
> + __u32 iefd_en:1;
> + __u32 denoise_en:1;
> + __u32 direct_smooth_en:1;
> + __u32 rad_en:1;
> + __u32 vssnlm_en:1;
> + __u32 __reserved:27;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_sharp_cfg - Sharpening config
> + *
> + * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
> + * @__reserved0: reserved
> + * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
> + * @__reserved1: reserved
> + * @nega_lmt_dir: Sharpening limit for negative overshoots for direction (edge).
> + * @__reserved2: reserved
> + * @posi_lmt_dir: Sharpening limit for positive overshoots for direction (edge).
> + * @__reserved3: reserved
> + *
> + * Fixed point type u13.0, range [0, 8191].
> + */
> +struct ipu3_uapi_sharp_cfg {
> + __u32 nega_lmt_txt:13;
> + __u32 __reserved0:19;
> + __u32 posi_lmt_txt:13;
> + __u32 __reserved1:19;
> + __u32 nega_lmt_dir:13;
> + __u32 __reserved2:19;
> + __u32 posi_lmt_dir:13;
> + __u32 __reserved3:19;
> +} __packed;
> +
> +/**
> + * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
> + *
> + * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64], default 64.
> + * @__reserved0: reserved
> + * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64], default 0.
> + * @__reserved1: reserved
> + * @ndir_dns_powr: Power of non-direct denoising,
> + * Precision u1.6, range [0, 64], default 64.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_far_w {
> + __u32 dir_shrp:7;
> + __u32 __reserved0:1;
> + __u32 dir_dns:7;
> + __u32 __reserved1:1;
> + __u32 ndir_dns_powr:7;
> + __u32 __reserved2:9;
> +} __packed;
> +
> +/**
> + * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
> + *
> + * @unsharp_weight: Unsharp mask blending weight.
> + * u1.6, range [0, 64], default 16.
> + * 0 - disabled, 64 - use only unsharp.
> + * @__reserved0: reserved
> + * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511], default 0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_unsharp_cfg {
> + __u32 unsharp_weight:7;
> + __u32 __reserved0:1;
> + __u32 unsharp_amount:9;
> + __u32 __reserved1:15;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
> + *
> + * @cfg: sharpness config &ipu3_uapi_sharp_cfg
> + * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
> + * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
> + * neighbors (8 pixels called Near), and the second order neighborhood
> + * around them (16 pixels called Far).
> + * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
> + */
> +struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
> + struct ipu3_uapi_sharp_cfg cfg;
> + struct ipu3_uapi_far_w far_w;
> + struct ipu3_uapi_unsharp_cfg unshrp_cfg;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
> + *
> + * @c00: Coeff11, s0.8, range [-255, 255], default 1.
> + * @c01: Coeff12, s0.8, range [-255, 255], default 5.
> + * @c02: Coeff13, s0.8, range [-255, 255], default 9.
> + * @__reserved: reserved
> + *
> + * Configurable registers for common sharpening support.
> + */
> +struct ipu3_uapi_unsharp_coef0 {
> + __u32 c00:9;
> + __u32 c01:9;
> + __u32 c02:9;
> + __u32 __reserved:5;
__s32?
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
> + *
> + * @c11: Coeff22, s0.8, range [-255, 255], default 29.
> + * @c12: Coeff23, s0.8, range [-255, 255], default 55.
> + * @c22: Coeff33, s0.8, range [-255, 255], default 96.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_unsharp_coef1 {
> + __u32 c11:9;
> + __u32 c12:9;
> + __u32 c22:9;
__s32?
> + __u32 __reserved:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
> + *
> + * @unsharp_coef0: unsharp coefficient 0 config. See &ipu3_uapi_unsharp_coef0
> + * @unsharp_coef1: unsharp coefficient 1 config. See &ipu3_uapi_unsharp_coef1
> + */
> +struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
> + struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
> + struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
> +} __packed;
> +
...
> +/**
> + * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
> + *
> + * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
> + * @lin_lutlow_r: linearization look-up table for R channel interpolation.
> + * @lin_lutlow_b: linearization look-up table for B channel interpolation.
> + * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
> + * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
Copy & paste issue here? Should the postfixes be gr, r, b and gb instead?
> + * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
> + * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
> + * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
> + * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
> + * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
> + */
> +struct ipu3_uapi_isp_lin_vmem_params {
> + __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
> +} __packed;
--
Kind regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-02 13:02 ` Sakari Ailus
@ 2018-11-16 22:37 ` Zhi, Yong
[not found] ` <20181129224548.qwbkau6suipt2veq@kekkonen.localdomain>
0 siblings, 1 reply; 123+ messages in thread
From: Zhi, Yong @ 2018-11-16 22:37 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu, Li, Chao C
Hi, Sakari,
Thanks for the thorough review.
> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> Sent: Friday, November 2, 2018 8:03 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>; Li, Chao C <chao.c.li@intel.com>
> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
>
> Hi Yong,
>
> Thanks for the update! I went through this again... a few comments below
> but I'd say they're mostly pretty minor issues.
>
> On Mon, Oct 29, 2018 at 03:22:57PM -0700, Yong Zhi wrote:
> > These meta formats are used on Intel IPU3 ImgU video queues
> > to carry 3A statistics and ISP pipeline parameters.
> >
> > V4L2_META_FMT_IPU3_3A
> > V4L2_META_FMT_IPU3_PARAMS
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > Signed-off-by: Chao C Li <chao.c.li@intel.com>
> > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> > ---
> > Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> > .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
> > include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
> > 3 files changed, 3001 insertions(+)
> > create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-
> ipu3.rst
> > create mode 100644 include/uapi/linux/intel-ipu3.h
> >
> > diff --git a/Documentation/media/uapi/v4l/meta-formats.rst
> b/Documentation/media/uapi/v4l/meta-formats.rst
> > index cf971d5..eafc534 100644
> > --- a/Documentation/media/uapi/v4l/meta-formats.rst
> > +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> > @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata`
> interface only.
> > .. toctree::
> > :maxdepth: 1
> >
> > + pixfmt-meta-intel-ipu3
> > pixfmt-meta-d4xx
> > pixfmt-meta-uvc
> > pixfmt-meta-vsp1-hgo
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > new file mode 100644
> > index 0000000..23b945b
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > @@ -0,0 +1,181 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _intel-ipu3:
>
> Instead, to avoid a warning from Sphinx, replace the line with these:
>
> .. _v4l2-meta-fmt-ipu3-params:
> .. _v4l2-meta-fmt-ipu3-stat-3a:
>
Ack.
> > +
> >
> +***************************************************************
> ***
> > +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A
> ('ip3s')
> >
> +***************************************************************
> ***
> > +
> > +.. c:type:: ipu3_uapi_stats_3a
> > +
> > +3A statistics
> > +=============
> > +
> > +For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
> > +an input bayer frame. Those statistics, defined in data struct
> > +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu
> 3a stat"
> > +video node, which are then passed to user space for statistics analysis
> > +using :c:type:`v4l2_meta_format` interface.
> > +
> > +The statistics collected are AWB (Auto-white balance) RGBS (Red, Green,
> Blue and
>
> Extra whitespace at the end of the line.
>
Ack.
> > +Saturation measure) cells, AWB filter response, AF (Auto-focus) filter
> response,
> > +and AE (Auto-exposure) histogram.
> > +
> > +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all
> above.
> > +
> > +
> > +.. code-block:: c
> > +
> > +
> > + struct ipu3_uapi_stats_3a {
> > + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> > + __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_raw_buffer_aligned
> > + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> > + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> > + struct ipu3_uapi_4a_config stats_4a_config;
> > + __u32 ae_join_buffers;
> > + __u8 padding[28];
> > + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + stats_3a_bubble_per_stripe;
>
> I think you could just unwrap these, even if it causes them to be over 80
> characters per line. They display better in a web browser that way. Or
> alternatively align the wrapped lines to the same column.
>
Ack.
> > + struct ipu3_uapi_ff_status stats_3a_status;
> > + } __packed;
> > +
> > +
> > +.. c:type:: ipu3_uapi_params
> > +
> > +Pipeline parameters
> > +===================
> > +
> > +IPU3 pipeline has a number of image processing stages, each of which
> takes a
> > +set of parameters as input. The major stages of pipelines are shown here:
> > +
> > +Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
> > +
> > +Linearization -> Lens Shading Correction -> White Balance / Exposure /
> > +
> > +Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
> > +
> > +Correction Matrix -> Gamma correction -> Color Space Conversion ->
> > +
> > +Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
> > +
> > +Correction -> XNR3 -> TNR -> DDR
> > +
> > +The table below presents a description of the above algorithms.
> > +
> > +========================
> =======================================================
> > +Name Description
> > +========================
> =======================================================
> > +Optical Black Correction Optical Black Correction block subtracts a pre-
> defined
> > + value from the respective pixel values to obtain
> better
> > + image quality.
> > + Defined in :c:type:`ipu3_uapi_obgrid_param`.
> > +Linearization This algo block uses linearization parameters
> to
> > + address non-linearity sensor effects. The Lookup
> table
> > + table is defined in
> > + :c:type:`ipu3_uapi_isp_lin_vmem_params`.
> > +SHD Lens shading correction is used to correct spatial
> > + non-uniformity of the pixel response due to optical
> > + lens shading. This is done by applying a different
> gain
> > + for each pixel. The gain, black level etc are
> > + configured in :c:type:`ipu3_uapi_shd_config_static`.
> > +BNR Bayer noise reduction block removes image noise by
> > + applying a bilateral filter.
> > + See :c:type:`ipu3_uapi_bnr_static_config` for details.
> > +ANR Advanced Noise Reduction is a block based algorithm
> > + that performs noise reduction in the Bayer domain.
> The
> > + convolution matrix etc can be found in
> > + :c:type:`ipu3_uapi_anr_config`.
> > +Demosaicing Demosaicing converts raw sensor data in
> Bayer format
> > + into RGB (Red, Green, Blue) presentation. Then add
> > + outputs of estimation of Y channel for following
> stream
> > + processing by Firmware. The struct is defined as
> > + :c:type:`ipu3_uapi_dm_config`.
> > +Color Correction Color Correction algo transforms sensor specific
> color
> > + space to the standard "sRGB" color space. This is
> done
> > + by applying 3x3 matrix defined in
> > + :c:type:`ipu3_uapi_ccm_mat_config`.
> > +Gamma correction Gamma
> correction :c:type:`ipu3_uapi_gamma_config` is a
> > + basic non-linear tone mapping correction that is
> > + applied per pixel for each pixel component.
> > +CSC Color space conversion transforms each pixel from
> the
> > + RGB primary presentation to YUV (Y - brightness,
> > + UV - Luminance) presentation. This is done by
> applying
> > + a 3x3 matrix defined in
> > + :c:type:`ipu3_uapi_csc_mat_config`
> > +CDS Chroma down sampling
> > + After the CSC is performed, the Chroma Down
> Sampling
> > + is applied for a UV plane down sampling by a factor
> > + of 2 in each direction for YUV 4:2:0 using a 4x2
> > + configurable filter :c:type:`ipu3_uapi_cds_params`.
> > +CHNR Chroma noise reduction
> > + This block processes only the chrominance pixels
> and
> > + performs noise reduction by cleaning the high
> > + frequency noise.
> > + See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
> > +TCC Total color correction as defined in struct
> > + :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
> > +XNR3 eXtreme Noise Reduction V3 is the third
> revision of
> > + noise reduction algorithm used to improve image
> > + quality. This removes the low frequency noise in the
> > + captured image. Two related structs are being
> defined,
> > + :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data
> memory
> > + and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for
> vector
> > + memory.
> > +TNR Temporal Noise Reduction block compares
> successive
> > + frames in time to remove anomalies / noise in pixel
> > + values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params`
> and
> > + :c:type:`ipu3_uapi_isp_tnr3_params` are defined for
> ISP
> > + vector and data memory respectively.
> > +========================
> =======================================================
> > +
> > +A few stages of the pipeline will be executed by firmware running on the
> ISP
> > +processor, while many others will use a set of fixed hardware blocks also
> > +called accelerator cluster (ACC) to crunch pixel data and produce statistics.
> > +
> > +ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be
> selectively
> > +enabled / disabled by the user space through
> struct :c:type:`ipu3_uapi_flags`
> > +embedded in :c:type:`ipu3_uapi_params` structure. For parameters that
> are not
> > +enabled by the user space, corresponding structs are ignored by the ISP.
>
> I presume the "enabled" here means enabling the use of the new parameter
> values. How about this:
>
> ACC parameters of individual algorithms, as defined by
> :c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
> space through struct :c:type:`ipu3_uapi_flags` embedded in
> :c:type:`ipu3_uapi_params` structure. For parameters that are configured as
> not enabled by the user space, the corresponding structs are ignored by the
> driver, in which case the existing configuration of the algorithm will be
> preserved.
>
Sure, thanks for the fine tuning the sentences.
> > +
> > +Both 3A statistics and pipeline parameters described here are closely tied
> to
> > +the underlying camera sub-system (CSS) APIs. They are usually consumed
> and
> > +produced by dedicated user space libraries that comprise the important
> tuning
> > +tools, thus freeing the developers from being bothered with the low level
> > +hardware and algorithm details.
> > +
> > +It should be noted that IPU3 DMA operations require the addresses of all
> data
> > +structures (that includes both input and output) to be aligned on 32 byte
> > +boundaries.
> > +
> > +The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu
> parameters"
> > +video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
> > +
> > +.. code-block:: c
> > +
> > + struct ipu3_uapi_params {
> > + /* Flags which of the settings below are to be applied */
> > + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> > +
> > + /* Accelerator cluster parameters */
> > + struct ipu3_uapi_acc_param acc_param;
> > +
> > + /* ISP vector address space parameters */
> > + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> > + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> > + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> > +
> > + /* ISP data memory (DMEM) parameters */
> > + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> > + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> > +
> > + /* Optical black level compensation */
> > + struct ipu3_uapi_obgrid_param obgrid_param;
> > + } __packed;
> > +
> > +Intel IPU3 ImgU uAPI data types
> > +===============================
> > +
> > +.. kernel-doc:: include/uapi/linux/intel-ipu3.h
> > diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
> > new file mode 100644
> > index 0000000..c2608b6
> > --- /dev/null
> > +++ b/include/uapi/linux/intel-ipu3.h
> > @@ -0,0 +1,2819 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2017 - 2018 Intel Corporation */
> > +
> > +#ifndef __IPU3_UAPI_H
> > +#define __IPU3_UAPI_H
> > +
> > +#include <linux/types.h>
> > +
> > +/********************* Key Acronyms *************************/
> > +/*
> > + * ACC - Accelerator cluster
> > + * ANR - Adaptive noise reduction
> > + * AWB_FR- Auto white balance filter response statistics
> > + * BNR - Bayer noise reduction parameters
> > + * BDS - Bayer downscaler parameters
> > + * CCM - Color correction matrix coefficients
> > + * CDS - Chroma down sample
> > + * CHNR - Chroma noise reduction
> > + * CSC - Color space conversion
> > + * DM - De-mosaic
> > + * IEFd - Image enhancement filter directed
> > + * Obgrid - Optical black level compensation
> > + * OSYS - Output system configuration
> > + * ROI - Region of interest
> > + * SHD - Lens shading correction table
> > + * TCC - Total color correction
> > + * YDS - Y down sampling
> > + * YTM - Y-tone mapping
> > + */
> > +
> > +/*
> > + * IPU3 DMA operations require buffers to be aligned at
> > + * 32 byte boundaries
> > + */
> > +
> > +/******************* ipu3_uapi_stats_3a *******************/
> > +
> > +#define IPU3_UAPI_MAX_STRIPES 2
> > +#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
> > +
> > +#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
> > +#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
> > +
> > +/* controls generation of meta_data (like FF enable/disable) */
> > +#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
> > +#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
> > +
> > +/**
> > + * struct ipu3_uapi_grid_config - Grid plane config
> > + *
> > + * @width: Grid horizontal dimensions, in number of grid blocks(cells).
> > + * @height: Grid vertical dimensions, in number of grid cells.
> > + * @block_width_log2: Log2 of the width of each cell in pixels.
> > + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> > + * @block_height_log2: Log2 of the height of each cell in pixels.
> > + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> > + * @height_per_slice: The number of blocks in vertical axis per slice.
> > + * Default 2.
> > + * @x_start: X value of top left corner of Region of Interest(ROI).
> > + * @y_start: Y value of top left corner of ROI
> > + * @x_end: X value of bottom right corner of ROI
> > + * @y_end: Y value of bottom right corner of ROI
> > + *
> > + * Due to the size of total amount of collected data, most statistics
> > + * create a grid-based output, and the data is then divided into "slices".
> > + */
> > +struct ipu3_uapi_grid_config {
> > + __u8 width;
> > + __u8 height;
> > + __u16 block_width_log2:3;
> > + __u16 block_height_log2:3;
> > + __u16 height_per_slice:8;
> > + __u16 x_start;
> > + __u16 y_start;
> > + __u16 x_end;
> > + __u16 y_end;
> > +} __packed;
> > +
> > +/*
> > + * The grid based data is divided into "slices" called set, each slice of setX
> > + * refers to ipu3_uapi_grid_config width * height_per_slice.
> > + */
> > +#define IPU3_UAPI_AWB_MAX_SETS 60
> > +/* Based on grid size 80 * 60 and cell size 16 x 16 */
> > +#define IPU3_UAPI_AWB_SET_SIZE 1280
> > +#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
> > +#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AWB_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
> > + (IPU3_UAPI_AWB_MAX_SETS * \
> > + (IPU3_UAPI_AWB_SET_SIZE +
> IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
> > +/**
> > + * struct ipu3_uapi_awb_meta_data - AWB meta data
> > + *
> > + * @meta_data_buffer: Average values for each color channel
> > + */
> > +struct ipu3_uapi_awb_meta_data {
> > + __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
> > + *
> > + * @meta_data: buffer to hold auto white balance meta data.
> > + */
> > +struct ipu3_uapi_awb_raw_buffer {
> > + struct ipu3_uapi_awb_meta_data meta_data;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_config_s - AWB config
> > + *
> > + * @rgbs_thr_gr: gr threshold value.
> > + * @rgbs_thr_r: Red threshold value.
> > + * @rgbs_thr_gb: gb threshold value.
> > + * @rgbs_thr_b: Blue threshold value.
> > + * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
> > + *
> > + * The threshold is a saturation measure range [0, 8191], 8191 is default.
> > + * Values over threshold may be optionally rejected for averaging.
> > + */
> > +struct ipu3_uapi_awb_config_s {
> > + __u16 rgbs_thr_gr;
> > + __u16 rgbs_thr_r;
> > + __u16 rgbs_thr_gb;
> > + __u16 rgbs_thr_b;
> > + struct ipu3_uapi_grid_config grid;
> > +} __attribute__((aligned(32))) __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_config - AWB config wrapper
> > + *
> > + * @config: config for auto white balance as defined by
> &ipu3_uapi_awb_config_s
> > + */
> > +struct ipu3_uapi_awb_config {
> > + struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AE_COLORS 4 /* R,
> G, B, Y */
> > +#define IPU3_UAPI_AE_BINS 256
> > +#define IPU3_UAPI_AE_WEIGHTS 96
> > +
> > +/**
> > + * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
> > + *
> > + * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
> > + *
> > + * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit
> unsigned
> > + * for counting the number of the pixel.
> > + */
> > +struct ipu3_uapi_ae_raw_buffer {
> > + __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
>
> What's the order of the colour components? Do the components for the
> same
> bin come together, or all bins for a given component?
>
> Could this be written instead as:
>
> struct {
> __u32 gr;
> __u32 g;
> __u32 b;
> __u32 gb;
> } vals[IPU3_UAPI_AE_BINS] __packed __attribute__((aligned(32)));
>
> > +} __packed;
> > +
My understanding is that there are 4 histograms, with 256 bins per histogram, so 256 R followed by 256 G et cetera.
> > +/**
> > + * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
> > + *
> > + * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
> > + */
> > +struct ipu3_uapi_ae_raw_buffer_aligned {
> > + struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
>
> Please use struct ipu3_uapi_ae_raw_buffer directly.
>
Not sure about this one, but we can use meta_data directly at other places as you suggested.
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_grid_config - AE weight grid
> > + *
> > + * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
> > + * @height: Grid vertical dimensions. Value: [16, 24], default 16.
> > + * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
> > + * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
>
> 2^3 == 16? Is the example wrong or the description of the field?
>
My bad, should be 2^3 = 8, cell size is 8x8, 4 cell per grid.
> > + * @__reserved0: reserved
> > + * @ae_en: 0: does not write to meta-data array, 1: write normally.
>
> Is the meta-data array here the AE raw buffer, as defined above? If so,
> please align the terms used.
>
Ack.
> > + * @rst_hist_array: write 1 to trigger histogram array reset.
> > + * @done_rst_hist_array: flag for histogram array reset done.
> > + * @x_start: X value of top left corner of ROI, default 0.
> > + * @y_start: Y value of top left corner of ROI, default 0.
> > + * @x_end: X value of bottom right corner of ROI
> > + * @y_end: Y value of bottom right corner of ROI
> > + *
> > + * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
> > + * a defined ROI within the frame. The contribution of each pixel into the
> > + * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a
> grid.
> > + */
> > +struct ipu3_uapi_ae_grid_config {
> > + __u8 width;
> > + __u8 height;
> > + __u8 block_width_log2:4;
> > + __u8 block_height_log2:4;
> > + __u8 __reserved0:5;
> > + __u8 ae_en:1;
> > + __u8 rst_hist_array:1;
> > + __u8 done_rst_hist_array:1;
> > + __u16 x_start;
> > + __u16 y_start;
> > + __u16 x_end;
> > + __u16 y_end;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_weight_elem - AE weights LUT
> > + *
> > + * @cell0: weighted histogram grid value.
> > + * @cell1: weighted histogram grid value.
> > + * @cell2: weighted histogram grid value.
> > + * @cell3: weighted histogram grid value.
> > + * @cell4: weighted histogram grid value.
> > + * @cell5: weighted histogram grid value.
> > + * @cell6: weighted histogram grid value.
> > + * @cell7: weighted histogram grid value.
> > + *
> > + * Use weighted grid value to give a different contribution factor to each
> cell.
> > + * Precision u4, range [0, 15].
> > + */
> > +struct ipu3_uapi_ae_weight_elem {
> > + __u32 cell0:4;
> > + __u32 cell1:4;
> > + __u32 cell2:4;
> > + __u32 cell3:4;
> > + __u32 cell4:4;
> > + __u32 cell5:4;
> > + __u32 cell6:4;
> > + __u32 cell7:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
> > + *
> > + * @gain_gr: WB gain factor for the gr channels. Default 256.
> > + * @gain_r: WB gain factor for the r channel. Default 256.
> > + * @gain_b: WB gain factor for the b channel. Default 256.
> > + * @gain_gb: WB gain factor for the gb channels. Default 256.
> > + * @mat: 4x4 matrix that transforms Bayer quad output from WB to
> RGB+Y.
> > + *
> > + * Default:
> > + * 128, 0, 0, 0,
> > + * 0, 128, 0, 0,
> > + * 0, 0, 128, 0,
> > + * 0, 0, 0, 128,
> > + *
> > + * As part of the raw frame pre-process stage, the WB and color
> conversion need
> > + * to be applied to expose the impact of these gain operations.
> > + */
> > +struct ipu3_uapi_ae_ccm {
> > + __u16 gain_gr;
> > + __u16 gain_r;
> > + __u16 gain_b;
> > + __u16 gain_gb;
> > + __s16 mat[16];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_config - AE config
> > + *
> > + * @grid_cfg: config for auto exposure statistics grid. See struct
> > + * &ipu3_uapi_ae_grid_config
> > + * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks
> in the grid.
> > + * Each grid cell has a corresponding value in weights LUT called
> > + * grid value, global histogram is updated based on grid value
> and
> > + * pixel value.
> > + * @ae_ccm: Color convert matrix pre-processing block.
> > + *
> > + * Calculate AE grid from image resolution, resample ae weights.
> > + */
> > +struct ipu3_uapi_ae_config {
> > + struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_weight_elem weights[
> > + IPU3_UAPI_AE_WEIGHTS]
> __attribute__((aligned(32)));
>
> Over 80 characters per line.
>
Ack.
> > + struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast
> measurements
> > + *
> > + * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
> > + * anti-symmetry type. A12 is center, A1-A11 are neighbours.
> > + * for analyzing low frequency content, used to calculate sum
> > + * of gradients in x direction.
> > + * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
> > + * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
> > + * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
> > + * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
> > + * @y1_coeff_1: Struct
> > + * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
> > + * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
> > + * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
> > + * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
> > + * @y1_coeff_2: Struct
> > + * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> > + * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> > + * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> > + * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> > + * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
> > + * 0: positive, 1: negative, default 0.
> > + * @y2_coeff_0: Y2, same structure as Y1. For analyzing high
> frequency content.
> > + * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
> > + * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
> > + * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
> > + * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
> > + * @y2_coeff_1: Struct
> > + * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
> > + * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
> > + * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
> > + * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
> > + * @y2_coeff_2: Struct
> > + * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> > + * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> > + * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> > + * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> > + * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
> > + * 0: positive, 1: negative, default 0.
> > + * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to
> be
> > + * used for building histogram. Range [0, 32], default 8.
> > + * Rule:
> > + * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b +
> y_gen_rate_gb = 32
> > + * A single Y is calculated based on sum of Gr/R/B/Gb based on
> > + * their contribution ratio.
> > + * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
> > + * @y_calc.y_gen_rate_r: Contribution ratio R for Y
> > + * @y_calc.y_gen_rate_b: Contribution ratio B for Y
> > + * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
> > + * @nf: The shift right value that should be applied during the Y1/Y2
> filter to
> > + * make sure the total memory needed is 2 bytes per grid cell.
> > + * @nf.__reserved0: reserved
> > + * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
> > + * should be log2 of the sum of the abs values of the filter
> > + * coeffs, default 7 (2^7 = 128).
> > + * @nf.__reserved1: reserved
> > + * @nf.y2_nf: Normalization factor for y2, should be log2 of the
> sum of the
> > + * abs values of the filter coeffs.
> > + * @nf.__reserved2: reserved
> > + */
> > +struct ipu3_uapi_af_filter_config {
> > + struct {
> > + __u8 a1;
> > + __u8 a2;
> > + __u8 a3;
> > + __u8 a4;
> > + } y1_coeff_0;
> > + struct {
> > + __u8 a5;
> > + __u8 a6;
> > + __u8 a7;
> > + __u8 a8;
> > + } y1_coeff_1;
> > + struct {
> > + __u8 a9;
> > + __u8 a10;
> > + __u8 a11;
> > + __u8 a12;
> > + } y1_coeff_2;
> > +
> > + __u32 y1_sign_vec;
> > +
> > + struct {
> > + __u8 a1;
> > + __u8 a2;
> > + __u8 a3;
> > + __u8 a4;
> > + } y2_coeff_0;
> > + struct {
> > + __u8 a5;
> > + __u8 a6;
> > + __u8 a7;
> > + __u8 a8;
> > + } y2_coeff_1;
> > + struct {
> > + __u8 a9;
> > + __u8 a10;
> > + __u8 a11;
> > + __u8 a12;
> > + } y2_coeff_2;
> > +
> > + __u32 y2_sign_vec;
> > +
> > + struct {
> > + __u8 y_gen_rate_gr;
> > + __u8 y_gen_rate_r;
> > + __u8 y_gen_rate_b;
> > + __u8 y_gen_rate_gb;
> > + } y_calc;
> > +
> > + struct {
> > + __u32 __reserved0:8;
> > + __u32 y1_nf:4;
> > + __u32 __reserved1:4;
> > + __u32 y2_nf:4;
> > + __u32 __reserved2:12;
> > + } nf;
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AF_MAX_SETS 24
> > +#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
> > +#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AF_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
> > +#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
> > + (IPU3_UAPI_AF_MAX_SETS * \
> > + (IPU3_UAPI_AF_Y_TABLE_SET_SIZE +
> IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
> > + IPU3_UAPI_MAX_STRIPES)
> > +
> > +/**
> > + * struct ipu3_uapi_af_meta_data - AF meta data
> > + *
> > + * @y_table: Each color component will be convolved separately with
> filter1
> > + * and filter2 and the result will be summed out and averaged
> for
> > + * each cell.
> > + */
> > +struct ipu3_uapi_af_meta_data {
> > + __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_raw_buffer - AF raw buffer
> > + *
> > + * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus
> meta data.
> > + */
> > +struct ipu3_uapi_af_raw_buffer {
> > + struct ipu3_uapi_af_meta_data meta_data
> __attribute__((aligned(32)));
>
> How about:
>
> __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
>
Ack.
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_config_s - AF config
> > + *
> > + * @filter_config: AF uses Y1 and Y2 filters as configured in
> > + * &ipu3_uapi_af_filter_config
> > + * @padding: paddings
> > + * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use
> large
> > + * grid size for large image and vice versa.
> > + */
> > +struct ipu3_uapi_af_config_s {
> > + struct ipu3_uapi_af_filter_config filter_config
> __attribute__((aligned(32)));
> > + __u8 padding[4];
> > + struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_config - AF config wrapper
> > + *
> > + * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
> > + */
> > +struct ipu3_uapi_af_config {
>
> Could you drop struct ipu3_uapi_af_config and use ipu3_uapi_af_config_s
> instead?
Ack.
>
> > + struct ipu3_uapi_af_config_s config;
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AWB_FR_MAX_SETS 24
> > +#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
> > +#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
> > +#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
> > + (IPU3_UAPI_AWB_FR_MAX_SETS * \
> > + (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
> > + IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) *
> IPU3_UAPI_MAX_STRIPES)
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
> > + *
> > + * @bayer_table: Statistics output on the grid after convolving with 1D
> filter.
> > + */
> > +struct ipu3_uapi_awb_fr_meta_data {
> > + __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
> > + *
> > + * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
> > + */
> > +struct ipu3_uapi_awb_fr_raw_buffer {
>
> Same here, please use ipu3_uapi_awb_fr_meta_data instead.
>
Ack.
> > + struct ipu3_uapi_awb_fr_meta_data meta_data;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
> > + *
> > + * @grid_cfg: grid config, default 16x16.
> > + * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
> > + * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
> > + * Applied on whole image for each Bayer channel
> separately
> > + * by a weighted sum of its 11x1 neighbors.
> > + * @__reserved1: reserved
> > + * @bayer_sign: sign of filter coeffcients, default 0.
> > + * @bayer_nf: normalization factor for the convolution coeffs, to
> make sure
> > + * total memory needed is within pre-determined range.
> > + * NF should be the log2 of the sum of the abs values of the
> > + * filter coeffs, range [7, 14], default 7.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_awb_fr_config_s {
> > + struct ipu3_uapi_grid_config grid_cfg;
> > + __u8 bayer_coeff[6];
> > + __u16 __reserved1;
> > + __u32 bayer_sign;
> > + __u8 bayer_nf;
> > + __u8 __reserved2[3];
> > +} __attribute__((aligned(32))) __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
> > + *
> > + * @config: See &ipu3_uapi_awb_fr_config_s.
> > + */
> > +struct ipu3_uapi_awb_fr_config {
>
> Ditto.
Ack, will implement and test with corresponding user space changes.
>
> > + struct ipu3_uapi_awb_fr_config_s config;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_4a_config - 4A config
> > + *
> > + * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
> > + * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
> > + * @padding: paddings
> > + * @af_config: auto focus config &ipu3_uapi_af_config_s
> > + * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution
> 16x16
> > + */
> > +struct ipu3_uapi_4a_config {
> > + struct ipu3_uapi_awb_config_s awb_config
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_grid_config ae_grd_config;
> > + __u8 padding[20];
> > + struct ipu3_uapi_af_config_s af_config;
> > + struct ipu3_uapi_awb_fr_config_s awb_fr_config;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
> > + *
> > + * @num_of_stripes: A single frame is divided into several parts called
> stripes
> > + * due to limitation on line buffer memory.
> > + * The separation between the stripes is vertical. Each such
> > + * stripe is processed as a single frame by the ISP pipe.
> > + * @padding: padding bytes.
> > + * @num_sets: number of sets.
> > + * @padding1: padding bytes.
> > + * @size_of_set: set size.
> > + * @padding2: padding bytes.
> > + * @bubble_size: is the amount of padding in the bubble expressed in
> "sets".
> > + * @padding3: padding bytes.
> > + */
> > +struct ipu3_uapi_bubble_info {
> > + __u32 num_of_stripes __attribute__((aligned(32)));
> > + __u8 padding[28];
> > + __u32 num_sets;
> > + __u8 padding1[28];
> > + __u32 size_of_set;
> > + __u8 padding2[28];
> > + __u32 bubble_size;
> > + __u8 padding3[28];
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + */
> > +struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
> > + struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
> > + *
> > + * @awb_en: auto white balance enable
> > + * @padding: padding config
> > + * @ae_en: auto exposure enable
> > + * @padding1: padding config
> > + * @af_en: auto focus enable
> > + * @padding2: padding config
> > + * @awb_fr_en: awb filter response enable bit
> > + * @padding3: padding config
> > + */
> > +struct ipu3_uapi_ff_status {
> > + __u32 awb_en __attribute__((aligned(32)));
> > + __u8 padding[28];
> > + __u32 ae_en;
> > + __u8 padding1[28];
> > + __u32 af_en;
> > + __u8 padding2[28];
> > + __u32 awb_fr_en;
> > + __u8 padding3[28];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_stats_3a - 3A statistics
> > + *
> > + * @awb_raw_buffer: auto white balance meta data
> &ipu3_uapi_awb_raw_buffer
> > + * @ae_raw_buffer: auto exposure raw data
> &ipu3_uapi_ae_raw_buffer_aligned
> > + * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
> > + * @awb_fr_raw_buffer: value as specified by
> &ipu3_uapi_awb_fr_raw_buffer
> > + * @stats_4a_config: 4a statistics config as defined by
> &ipu3_uapi_4a_config.
> > + * @ae_join_buffers: 1 to use ae_raw_buffer.
> > + * @padding: padding config
> > + * @stats_3a_bubble_per_stripe: a
> &ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
> > + */
> > +struct ipu3_uapi_stats_3a {
> > + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_raw_buffer_aligned
> > + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> > + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> > + struct ipu3_uapi_4a_config stats_4a_config;
> > + __u32 ae_join_buffers;
> > + __u8 padding[28];
> > + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + stats_3a_bubble_per_stripe;
> > + struct ipu3_uapi_ff_status stats_3a_status;
> > +} __packed;
> > +
> > +/******************* ipu3_uapi_acc_param *******************/
> > +
> > +#define IPU3_UAPI_ISP_VEC_ELEMS 64
> > +#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
> > +
> > +#define IPU3_UAPI_BNR_LUT_SIZE 32
> > +
> > +/* number of elements in gamma correction LUT */
> > +#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
> > +
> > +/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
> > +#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
> > +#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
> > +/* Normalization shift aka nf */
> > +#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
> > +#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
> > +
> > +#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
> > +#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
> > +#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
> > +#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
> > +
> > +#define IPU3_UAPI_ANR_LUT_SIZE 26
> > +#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
> > +
> > +#define IPU3_UAPI_LIN_LUT_SIZE 64
> > +
> > +/* Bayer Noise Reduction related structs */
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance
> gains
> > + *
> > + * @gr: white balance gain for Gr channel.
> > + * @r: white balance gain for R channel.
> > + * @b: white balance gain for B channel.
> > + * @gb: white balance gain for Gb channel.
> > + *
> > + * Precision u3.13, range [0, 8]. White balance correction is done by
> applying
>
> [0, 8[
>
> (or [0, 8), but the same notation needs to be used everywhere).
>
Should be [0, 8).
> > + * a multiplicative gain to each color channels prior to BNR.
> > + */
> > +struct ipu3_uapi_bnr_static_config_wb_gains_config {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold
> config
> > + *
> > + * @gr: white balance threshold gain for Gr channel.
> > + * @r: white balance threshold gain for R channel.
> > + * @b: white balance threshold gain for B channel.
> > + * @gb: white balance threshold gain for Gb channel.
> > + *
> > + * Defines the threshold that specifies how different a defect pixel can be
> from
> > + * its neighbors.(used by dynamic defect pixel correction sub block)
> > + * Precision u4.4 range [0, 8].
> > + */
> > +struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
> > + __u8 gr;
> > + __u8 r;
> > + __u8 b;
> > + __u8 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
> > + * coefficients that controls noise threshold
> > + *
> > + * @cf: Free coefficient for threshold calculation, range [0, 8191],
> default 0.
> > + * @__reserved0: reserved
> > + * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
> > + * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
> > + * default 6.
> > + * format: u3.2 (3 most significant bits represent whole number,
> > + * 2 least significant bits represent the fractional part
> > + * with each count representing 0.25)
> > + * e.g 6 in binary format is 00110, that translates to 1.5
> > + * @__reserved1: reserved
> > + * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
> > + * where r is a radius of pixel [row, col] from centor of sensor.
> > + * default 14.
> > + *
> > + * Threshold used to distinguish between noise and details.
> > + */
> > +struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
> > + __u32 cf:13;
> > + __u32 __reserved0:3;
> > + __u32 cg:5;
> > + __u32 ci:5;
> > + __u32 __reserved1:1;
> > + __u32 r_nf:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
> > + *
> > + * @gr: Coefficient defines lens shading gain approximation for gr
> channel
> > + * @r: Coefficient defines lens shading gain approximation for r
> channel
> > + * @b: Coefficient defines lens shading gain approximation for b
> channel
> > + * @gb: Coefficient defines lens shading gain approximation for gb
> channel
> > + *
> > + * Parameters for noise model (NM) adaptation of BNR due to shading
> correction.
> > + * All above have precision of u3.3, default to 0.
> > + */
> > +struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
> > + __u8 gr;
> > + __u8 r;
> > + __u8 b;
> > + __u8 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center
> config
> > + *
> > + * @x_reset: Reset value of X (col start - X center). Precision s12.0.
> > + * @__reserved0: reserved
> > + * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
> > + * @__reserved2: reserved
> > + *
> > + * Distance from corner to optical center for NM adaptation due to
> shading
> > + * correction (should be calculated based on shading tables)
> > + */
> > +struct ipu3_uapi_bnr_static_config_opt_center_config {
> > + __s32 x_reset:13;
> > + __u32 __reserved0:3;
> > + __s32 y_reset:13;
> > + __u32 __reserved2:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup
> table
> > + *
> > + * @values: pre-calculated values of square root function.
> > + *
> > + * LUT implementation of square root operation.
> > + */
> > +struct ipu3_uapi_bnr_static_config_lut_config {
> > + __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels
> (bp)
> > + *
> > + * @bp_thr_gain: Defines the threshold that specifies how different a
> > + * defect pixel can be from its neighbors. Threshold is
> > + * dependent on de-noise threshold calculated by
> algorithm.
> > + * Range [4, 31], default 4.
> > + * @__reserved0: reserved
> > + * @defect_mode: Mode of addressed defect pixels,
> > + * 0 - single defect pixel is expected,
> > + * 1 - 2 adjacent defect pixels are expected, default 1.
> > + * @bp_gain: Defines how 2nd derivation that passes through a
> defect pixel
> > + * is different from 2nd derivations that pass through
> > + * neighbor pixels. u4.2, range [0, 256], default 8.
> > + * @__reserved1: reserved
> > + * @w0_coeff: Blending coefficient of defect pixel correction.
> > + * Precision u4, range [0, 8], default 8.
> > + * @__reserved2: reserved
> > + * @w1_coeff: Enable influence of incorrect defect pixel correction
> to be
> > + * avoided. Precision u4, range [1, 8], default 8.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
> > + __u32 bp_thr_gain:5;
> > + __u32 __reserved0:2;
> > + __u32 defect_mode:1;
> > + __u32 bp_gain:6;
> > + __u32 __reserved1:18;
> > + __u32 w0_coeff:4;
> > + __u32 __reserved2:4;
> > + __u32 w1_coeff:4;
> > + __u32 __reserved3:20;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising
> config
> > + *
> > + * @alpha: Weight of central element of smoothing filter.
> > + * @beta: Weight of peripheral elements of smoothing filter, default 4.
> > + * @gamma: Weight of diagonal elements of smoothing filter, default 4.
> > + *
> > + * beta and gamma parameter define the strength of the noise removal
> filter.
> > + * All above has precision u0.4, range [0, 0xf]
> > + * format: u0.4 (no / zero bits represent whole number,
> > + * 4 bits represent the fractional part
> > + * with each count representing 0.0625)
> > + * e.g 0xf translates to 0.0625x15 = 0.9375
> > + *
> > + * @__reserved0: reserved
> > + * @max_inf: Maximum increase of peripheral or diagonal element
> influence
> > + * relative to the pre-defined value range: [0x5, 0xa]
> > + * @__reserved1: reserved
> > + * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
> > + * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 -
> enable.
> > + * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 -
> enable.
> > + * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
> > + __u32 alpha:4;
> > + __u32 beta:4;
> > + __u32 gamma:4;
> > + __u32 __reserved0:4;
> > + __u32 max_inf:4;
> > + __u32 __reserved1:7;
> > + __u32 gd_enable:1;
> > + __u32 bpc_enable:1;
> > + __u32 bnr_enable:1;
> > + __u32 ff_enable:1;
> > + __u32 __reserved2:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical
> square
> > + *
> > + * @x_sqr_reset: Reset value of X^2.
> > + * @y_sqr_reset: Reset value of Y^2.
> > + *
> > + * Please note:
> > + *
> > + * #. X and Y ref to
> > + * &ipu3_uapi_bnr_static_config_opt_center_config
> > + * #. Both structs are used in threshold formula to calculate r^2, where r
> > + * is a radius of pixel [row, col] from centor of sensor.
> > + */
> > +struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
> > + __u32 x_sqr_reset;
> > + __u32 y_sqr_reset;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config - BNR static config
> > + *
> > + * @wb_gains: white balance gains
> &ipu3_uapi_bnr_static_config_wb_gains_config
> > + * @wb_gains_thr: white balance gains threshold as defined by
> > + * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
> > + * @thr_coeffs: coefficients of threshold
> > + * &ipu3_uapi_bnr_static_config_thr_coeffs_config
> > + * @thr_ctrl_shd: control of shading threshold
> > + * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
> > + * @opt_center: optical center
> &ipu3_uapi_bnr_static_config_opt_center_config
> > + *
> > + * Above parameters and opt_center_sqr are used for white balance and
> shading.
> > + *
> > + * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
> > + * @bp_ctrl: detect and remove bad pixels as defined in struct
> > + * &ipu3_uapi_bnr_static_config_bp_ctrl_config
> > + * @dn_detect_ctrl: detect and remove noise.
> > + * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> > + * @column_size: The number of pixels in column.
> > + * @opt_center_sqr: Reset value of r^2 to optical center, see
> > + *
> &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
> > + */
> > +struct ipu3_uapi_bnr_static_config {
> > + struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
> > + struct ipu3_uapi_bnr_static_config_wb_gains_thr_config
> wb_gains_thr;
> > + struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
> > + struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
> > + struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
> > + struct ipu3_uapi_bnr_static_config_lut_config lut;
> > + struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
> > + struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> dn_detect_ctrl;
> > + __u32 column_size;
> > + struct ipu3_uapi_bnr_static_config_opt_center_sqr_config
> opt_center_sqr;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green
> disparity
> > + *
> > + * @gd_red: Shading gain coeff for gr disparity level in bright red region.
> > + * Precision u0.6, default 4(0.0625).
> > + * @__reserved0: reserved
> > + * @gd_green: Shading gain coeff for gr disparity level in bright
> green
> > + * region. Precision u0.6, default 4(0.0625).
> > + * @__reserved1: reserved
> > + * @gd_blue: Shading gain coeff for gr disparity level in bright blue
> region.
> > + * Precision u0.6, default 4(0.0625).
> > + * @__reserved2: reserved
> > + * @gd_black: Maximal green disparity level in dark region (stronger
> disparity
> > + * assumed to be image detail). Precision u14, default 80.
> > + * @__reserved3: reserved
> > + * @gd_shading: Change maximal green disparity level according to
> square
> > + * distance from image center.
> > + * @__reserved4: reserved
> > + * @gd_support: Lower bound for the number of second green color
> pixels in
> > + * current pixel neighborhood with less than threshold
> difference
> > + * from it.
> > + *
> > + * The shading gain coeff of red, green, blue and black are used to
> calculate
> > + * threshold given a pixel's color value and its coordinates in the image.
> > + *
> > + * @__reserved5: reserved
> > + * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
> > + * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
> > + */
> > +struct ipu3_uapi_bnr_static_config_green_disparity {
> > + __u32 gd_red:6;
> > + __u32 __reserved0:2;
> > + __u32 gd_green:6;
> > + __u32 __reserved1:2;
> > + __u32 gd_blue:6;
> > + __u32 __reserved2:10;
> > + __u32 gd_black:14;
> > + __u32 __reserved3:2;
> > + __u32 gd_shading:7;
> > + __u32 __reserved4:1;
> > + __u32 gd_support:2;
> > + __u32 __reserved5:1;
> > + __u32 gd_clip:1;
> > + __u32 gd_central_weight:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_dm_config - De-mosaic parameters
> > + *
> > + * @dm_en: de-mosaic enable.
> > + * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
> > + * @fcc_en: False color correction (FCC) enable flag. Default 0.
> > + * @__reserved0: reserved
> > + * @frame_width: do not care
> > + * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation
> of
> > + * complementary color in Hamilton-Adams interpolation).
> > + * u5, range [0, 31], default 8.
> > + * @__reserved1: reserved
> > + * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity
> metric
> > + * in calculation of final homogeneity metric.
> > + * u5, range [0, 31], default 7.
> > + * @__reserved2: reserved
> > + * @cr_param1: First parameter that defines Checker artifact removal
> > + * feature gain.Precision u5, range [0, 31], default 8.
> > + * @__reserved3: reserved
> > + * @cr_param2: Second parameter that defines Checker artifact
> removal
> > + * feature gain. Precision u5, range [0, 31], default 8.
> > + * @__reserved4: reserved
> > + * @coring_param: Defines power of false color correction operation.
> > + * low for preserving edge colors, high for preserving
> gray
> > + * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
> > + * @__reserved5: reserved
> > + *
> > + * The demosaic fixed function block is responsible to covert
> Bayer(mosaiced)
> > + * images into color images based on demosaicing algorithm.
> > + */
> > +struct ipu3_uapi_dm_config {
> > + __u32 dm_en:1;
> > + __u32 ch_ar_en:1;
> > + __u32 fcc_en:1;
> > + __u32 __reserved0:13;
> > + __u32 frame_width:16;
> > +
> > + __u32 gamma_sc:5;
> > + __u32 __reserved1:3;
> > + __u32 lc_ctrl:5;
> > + __u32 __reserved2:3;
> > + __u32 cr_param1:5;
> > + __u32 __reserved3:3;
> > + __u32 cr_param2:5;
> > + __u32 __reserved4:3;
> > +
> > + __u32 coring_param:5;
> > + __u32 __reserved5:27;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ccm_mat_config - Color correction matrix
> > + *
> > + * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
> > + * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
> > + * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
> > + * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
> > + * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
> > + *
> > + * Transform sensor specific color space to standard sRGB by applying 3x3
> matrix
> > + * and adding a bias vector O. The transformation is basically a rotation
> and
> > + * translation in the 3-dimensional color spaces. Here are the defaults:
> > + *
> > + * 9775, -2671, 1087, 0
> > + * -1071, 8303, 815, 0
> > + * -23, -7887, 16103, 0
> > + */
> > +struct ipu3_uapi_ccm_mat_config {
> > + __s16 coeff_m11;
> > + __s16 coeff_m12;
> > + __s16 coeff_m13;
> > + __s16 coeff_o_r;
> > + __s16 coeff_m21;
> > + __s16 coeff_m22;
> > + __s16 coeff_m23;
> > + __s16 coeff_o_g;
> > + __s16 coeff_m31;
> > + __s16 coeff_m32;
> > + __s16 coeff_m33;
> > + __s16 coeff_o_b;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
> > + *
> > + * @enable: gamma correction enable.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_gamma_corr_ctrl {
> > + __u32 enable:1;
> > + __u32 __reserved:31;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping
> implemented as LUT.
> > + *
> > + * @lut: 256 tabulated values of the gamma function. LUT[1]..
> LUT[256]
> > + * format u13.0, range [0, 8191].
> > + *
> > + * The tone mapping operation is done by a Piece wise linear graph
> > + * that is implemented as a lookup table(LUT). The pixel component input
> > + * intensity is the X-axis of the graph which is the table entry.
> > + */
> > +struct ipu3_uapi_gamma_corr_lut {
> > + __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_config - Gamma config
> > + *
> > + * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
> > + * @gc_lut: lookup table of gamma correction
> &ipu3_uapi_gamma_corr_lut
> > + */
> > +struct ipu3_uapi_gamma_config {
> > + struct ipu3_uapi_gamma_corr_ctrl gc_ctrl
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
> > + *
> > + * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
>
> You can't represent 1; it's 8191/8192. How about [-1, 1[ ?
>
> Is the default 1 or 8191? The numerical value is used elsewhere but here
> it seems that this might not be the case. The same for other cases below.
>
For C11, the default is 4898 (which 0.299 * 2^ 14), will update the defaults and range for this struct.
> > + * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + *
> > + * To transform each pixel from RGB to YUV (Y - brightness/luminance,
> > + * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding
> an
> > + * optional bias 3x1 vector.
> > + */
> > +struct ipu3_uapi_csc_mat_config {
> > + __s16 coeff_c11;
> > + __s16 coeff_c12;
> > + __s16 coeff_c13;
> > + __s16 coeff_b1;
> > + __s16 coeff_c21;
> > + __s16 coeff_c22;
> > + __s16 coeff_c23;
> > + __s16 coeff_b2;
> > + __s16 coeff_c31;
> > + __s16 coeff_c32;
> > + __s16 coeff_c33;
> > + __s16 coeff_b3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cds_params - Chroma down-scaling
> > + *
> > + * @ds_c00: range [0, 3]
> > + * @ds_c01: range [0, 3]
> > + * @ds_c02: range [0, 3]
> > + * @ds_c03: range [0, 3]
> > + * @ds_c10: range [0, 3]
> > + * @ds_c11: range [0, 3]
> > + * @ds_c12: range [0, 3]
> > + * @ds_c13: range [0, 3]
> > + *
> > + * In case user does not provide, above 4x2 filter will use following
> defaults:
> > + * 1, 3, 3, 1,
> > + * 1, 3, 3, 1,
> > + *
> > + * @ds_nf: Normalization factor for Chroma output downscaling filter,
> > + * range 0,4, default 2.
> > + * @__reserved0: reserved
> > + * @csc_en: Color space conversion enable
> > + * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_cds_params {
> > + __u32 ds_c00:2;
> > + __u32 ds_c01:2;
> > + __u32 ds_c02:2;
> > + __u32 ds_c03:2;
> > + __u32 ds_c10:2;
> > + __u32 ds_c11:2;
> > + __u32 ds_c12:2;
> > + __u32 ds_c13:2;
> > + __u32 ds_nf:5;
> > + __u32 __reserved0:3;
> > + __u32 csc_en:1;
> > + __u32 uv_bin_output:1;
> > + __u32 __reserved1:6;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
> > + *
> > + * @width: Grid horizontal dimensions, u8, [8, 128], default 73
> > + * @height: Grid vertical dimensions, u8, [8, 128], default 56
> > + * @block_width_log2: Log2 of the width of the grid cell in pixel
> count
> > + * u4, [0, 15], default value 5.
> > + * @__reserved0: reserved
> > + * @block_height_log2: Log2 of the height of the grid cell in pixel
> count
> > + * u4, [0, 15], default value 6.
> > + * @__reserved1: reserved
> > + * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
> > + * (with SHD_MAX_CELLS_PER_SET = 146).
> > + * @x_start: X value of top left corner of sensor relative to ROI
> > + * u12, [-4096, 0]. default 0, only negative values.
> > + * @y_start: Y value of top left corner of sensor relative to ROI
> > + * u12, [-4096, 0]. default 0, only negative values.
>
> I suppose u12 is incorrect here, if the value is signed --- and negative
> (sign bit) if not 0?
>
The value will be written to 13 bit register, should use s12.0.
> > + */
> > +struct ipu3_uapi_shd_grid_config {
> > + /* reg 0 */
> > + __u8 width;
> > + __u8 height;
> > + __u8 block_width_log2:3;
> > + __u8 __reserved0:1;
> > + __u8 block_height_log2:3;
> > + __u8 __reserved1:1;
> > + __u8 grid_height_per_slice;
> > + /* reg 1 */
> > + __s16 x_start;
> > + __s16 y_start;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_general_config - Shading general config
> > + *
> > + * @init_set_vrt_offst_ul: set vertical offset,
> > + * y_start >> block_height_log2 % grid_height_per_slice.
> > + * @shd_enable: shading enable.
> > + * @gain_factor: Gain factor. Shift calculated anti shading value. Precision
> u2.
> > + * 0x0 - gain factor [1, 5], means no shift interpolated value.
> > + * 0x1 - gain factor [1, 9], means shift interpolated by 1.
> > + * 0x2 - gain factor [1, 17], means shift interpolated by 2.
> > + * @__reserved: reserved
> > + *
> > + * Correction is performed by multiplying a gain factor for each of the 4
> Bayer
> > + * channels as a function of the pixel location in the sensor.
> > + */
> > +struct ipu3_uapi_shd_general_config {
> > + __u32 init_set_vrt_offst_ul:8;
> > + __u32 shd_enable:1;
> > + __u32 gain_factor:2;
> > + __u32 __reserved:21;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_black_level_config - Black level correction
> > + *
> > + * @bl_r: Bios values for green red. s11 range [-2048, 2047].
> > + * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
> > + * @bl_gb: Bios values for red. s11 range [-2048, 2047].
> > + * @bl_b: Bios values for blue. s11 range [-2048, 2047].
> > + */
> > +struct ipu3_uapi_shd_black_level_config {
> > + __s16 bl_r;
> > + __s16 bl_gr;
> > + __s16 bl_gb;
> > + __s16 bl_b;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_config_static - Shading config static
> > + *
> > + * @grid: shading grid config &ipu3_uapi_shd_grid_config
> > + * @general: shading general config &ipu3_uapi_shd_general_config
> > + * @black_level: black level config for shading correction as defined by
> > + * &ipu3_uapi_shd_black_level_config
> > + */
> > +struct ipu3_uapi_shd_config_static {
> > + struct ipu3_uapi_shd_grid_config grid;
> > + struct ipu3_uapi_shd_general_config general;
> > + struct ipu3_uapi_shd_black_level_config black_level;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
> > + *
> > + * @sets: array
> > + * @sets.r_and_gr: Red and GreenR Lookup table.
> > + * @sets.r_and_gr.r: Red shading factor.
> > + * @sets.r_and_gr.gr: GreenR shading factor.
> > + * @sets.__reserved1: reserved
> > + * @sets.gb_and_b: GreenB and Blue Lookup table.
> > + * @sets.gb_and_b.gb: GreenB shading factor.
> > + * @sets.gb_and_b.b: Blue shading factor.
> > + * @sets.__reserved2: reserved
> > + *
> > + * Map to shading correction LUT register set.
> > + */
> > +struct ipu3_uapi_shd_lut {
> > + struct {
> > + struct {
> > + __u16 r;
> > + __u16 gr;
> > + } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> > + __u8 __reserved1[24];
> > + struct {
> > + __u16 gb;
> > + __u16 b;
> > + } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> > + __u8 __reserved2[24];
> > + } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_config - Shading config
> > + *
> > + * @shd: shading static config, see &ipu3_uapi_shd_config_static
> > + * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
> > + */
> > +struct ipu3_uapi_shd_config {
> > + struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
> > + struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/* Image Enhancement Filter directed */
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
> > + *
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A of Config Unit, s4.4, default 0.
> > + * @b01: Always 0.
> > + *
> > + * Calculate weight for blending directed and non-directed denoise
> elements
> > + *
> > + * Note:
> > + * Each instance of Config Unit needs X coordinate of n points and
> > + * slope A factor between points calculated by driver based on calibration
> > + * parameters.
> > + */
> > +struct ipu3_uapi_iefd_cux2 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 a01:9;
> > + __u32 b01:5; /* NOTE: hardcoded to zero */
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed
> sharpening
> > + * element, Config Unit 6 for edge detail (ED).
> > + *
> > + * @x0: X coordinate of point 0, u9.0, default 0.
> > + * @x1: X coordinate of point 1, u9.0, default 0.
> > + * @x2: X coordinate of point 2, u9.0, default 0.
> > + * @__reserved0: reserved
> > + * @x3: X coordinate of point 3, u9.0, default 0.
> > + * @x4: X coordinate of point 4, u9.0, default 0.
> > + * @x5: X coordinate of point 5, u9.0, default 0.
> > + * @__reserved1: reserved
> > + * @a01: slope A points 01, s4.4, default 0.
> > + * @a12: slope A points 12, s4.4, default 0.
> > + * @a23: slope A points 23, s4.4, default 0.
> > + * @__reserved2: reserved
> > + * @a34: slope A points 34, s4.4, default 0.
> > + * @a45: slope A points 45, s4.4, default 0.
> > + * @__reserved3: reserved
> > + * @b01: slope B points 01, s4.4, default 0.
> > + * @b12: slope B points 12, s4.4, default 0.
> > + * @b23: slope B points 23, s4.4, default 0.
> > + * @__reserved4: reserved
> > + * @b34: slope B points 34, s4.4, default 0.
> > + * @b45: slope B points 45, s4.4, default 0.
> > + * @__reserved5: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux6_ed {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 x2:9;
> > + __u32 __reserved0:5;
> > +
> > + __u32 x3:9;
> > + __u32 x4:9;
> > + __u32 x5:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 a01:9;
> > + __u32 a12:9;
> > + __u32 a23:9;
> > + __u32 __reserved2:5;
> > +
> > + __u32 a34:9;
> > + __u32 a45:9;
> > + __u32 __reserved3:14;
> > +
> > + __u32 b01:9;
> > + __u32 b12:9;
> > + __u32 b23:9;
> > + __u32 __reserved4:5;
> > +
> > + __u32 b34:9;
> > + __u32 b45:9;
> > + __u32 __reserved5:14;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
> > + * element apply.
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A of Config Unit, s4.4, default 0.
>
> The field is marked unsigned below. Which one is correct?
>
They are both correct, however, s4.4 is the internal representation used by CU, the inputs are unsigned, I will add a note in v8, same applies to the few other places as you commented.
> > + * @__reserved1: reserved
> > + * @b01: offset B0 of Config Unit, u7.0, default 0.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux2_1 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 a01:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 b01:8;
> > + __u32 __reserved2:24;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed
> sharpening
> > + * element.
> > + *
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @x2: X2 point of Config Unit, u9.0, default 0.
> > + * @__reserved0: reserved
> > + * @x3: X3 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A0 of Config Unit, s4.4, default 0.
> > + * @a12: Slope A1 of Config Unit, s4.4, default 0.
>
> Same here, suggest __s32 below if this is signed.
>
Ack, same reason as ipu3_uapi_iefd_cux2_1, will add a comments.
> > + * @__reserved1: reserved
> > + * @a23: Slope A2 of Config Unit, s4.4, default 0.
> > + * @b01: Offset B0 of Config Unit, s7.0, default 0.
> > + * @b12: Offset B1 of Config Unit, s7.0, default 0.
> > + * @__reserved2: reserved
> > + * @b23: Offset B2 of Config Unit, s7.0, default 0.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux4 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 x2:9;
> > + __u32 __reserved0:5;
> > +
> > + __u32 x3:9;
> > + __u32 a01:9;
> > + __u32 a12:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 a23:9;
> > + __u32 b01:8;
> > + __u32 b12:8;
> > + __u32 __reserved2:7;
> > +
> > + __u32 b23:8;
> > + __u32 __reserved3:24;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
> > + *
> > + * @x0: x0 points of Config Unit radial, u8.0
> > + * @x1: x1 points of Config Unit radial, u8.0
> > + * @x2: x2 points of Config Unit radial, u8.0
> > + * @x3: x3 points of Config Unit radial, u8.0
> > + * @x4: x4 points of Config Unit radial, u8.0
> > + * @x5: x5 points of Config Unit radial, u8.0
> > + * @__reserved1: reserved
> > + * @a01: Slope A of Config Unit radial, s7.8
> > + * @a12: Slope A of Config Unit radial, s7.8
> > + * @a23: Slope A of Config Unit radial, s7.8
> > + * @a34: Slope A of Config Unit radial, s7.8
> > + * @a45: Slope A of Config Unit radial, s7.8
> > + * @__reserved2: reserved
> > + * @b01: Slope B of Config Unit radial, s9.0
> > + * @b12: Slope B of Config Unit radial, s9.0
> > + * @b23: Slope B of Config Unit radial, s9.0
> > + * @__reserved4: reserved
> > + * @b34: Slope B of Config Unit radial, s9.0
> > + * @b45: Slope B of Config Unit radial, s9.0
> > + * @__reserved5: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux6_rad {
> > + __u32 x0:8;
> > + __u32 x1:8;
> > + __u32 x2:8;
> > + __u32 x3:8;
> > +
> > + __u32 x4:8;
> > + __u32 x5:8;
> > + __u32 __reserved1:16;
> > +
> > + __u32 a01:16;
> > + __u32 a12:16;
> > +
> > + __u32 a23:16;
> > + __u32 a34:16;
> > +
> > + __u32 a45:16;
> > + __u32 __reserved2:16;
> > +
> > + __u32 b01:10;
> > + __u32 b12:10;
> > + __u32 b23:10;
> > + __u32 __reserved4:2;
> > +
> > + __u32 b34:10;
> > + __u32 b45:10;
> > + __u32 __reserved5:12;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
> > + *
> > + * @cu_1: calculate weight for blending directed and
> > + * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
> > + * @cu_ed: calculate power of non-directed sharpening element, see
> > + * &ipu3_uapi_iefd_cux6_ed
> > + * @cu_3: calculate weight for blending directed and
> > + * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
> > + * @cu_5: calculate power of non-directed denoise element apply, use
> > + * &ipu3_uapi_iefd_cux2_1
> > + * @cu_6: calculate power of non-directed sharpening element. See
> > + * &ipu3_uapi_iefd_cux4
> > + * @cu_7: calculate weight for blending directed and
> > + * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
> > + * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
> > + * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
> > + * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_cfg_units {
> > + struct ipu3_uapi_iefd_cux2 cu_1;
> > + struct ipu3_uapi_iefd_cux6_ed cu_ed;
> > + struct ipu3_uapi_iefd_cux2 cu_3;
> > + struct ipu3_uapi_iefd_cux2_1 cu_5;
> > + struct ipu3_uapi_iefd_cux4 cu_6;
> > + struct ipu3_uapi_iefd_cux2 cu_7;
> > + struct ipu3_uapi_iefd_cux4 cu_unsharp;
> > + struct ipu3_uapi_iefd_cux6_rad cu_radial;
> > + struct ipu3_uapi_iefd_cux2 cu_vssnlm;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
> > + *
> > + * @horver_diag_coeff: Gradiant compensation, coefficient that
> compensates for
> > + * different distance for vertical / horizontal and diagonal
> > + * * gradient calculation (~1/sqrt(2)).
> > + * @__reserved0: reserved
> > + * @clamp_stitch: Slope to stitch between clamped and unclamped edge
> values
> > + * @__reserved1: reserved
> > + * @direct_metric_update: Update coeff for direction metric
> > + * @__reserved2: reserved
> > + * @ed_horver_diag_coeff: Radial Coefficient that compensates for
> > + * different distance for vertical/horizontal and
> > + * diagonal gradient calculation (~1/sqrt(2))
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_config_s {
> > + __u32 horver_diag_coeff:7;
> > + __u32 __reserved0:1;
> > + __u32 clamp_stitch:6;
> > + __u32 __reserved1:2;
> > + __u32 direct_metric_update:5;
> > + __u32 __reserved2:3;
> > + __u32 ed_horver_diag_coeff:7;
> > + __u32 __reserved3:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
> > + *
> > + * @iefd_en: Enable IEFd
> > + * @denoise_en: Enable denoise
> > + * @direct_smooth_en: Enable directional smooth
> > + * @rad_en: Enable radial update
> > + * @vssnlm_en: Enable VSSNLM output filter
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_control {
> > + __u32 iefd_en:1;
> > + __u32 denoise_en:1;
> > + __u32 direct_smooth_en:1;
> > + __u32 rad_en:1;
> > + __u32 vssnlm_en:1;
> > + __u32 __reserved:27;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_sharp_cfg - Sharpening config
> > + *
> > + * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
> > + * @__reserved0: reserved
> > + * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
> > + * @__reserved1: reserved
> > + * @nega_lmt_dir: Sharpening limit for negative overshoots for direction
> (edge).
> > + * @__reserved2: reserved
> > + * @posi_lmt_dir: Sharpening limit for positive overshoots for direction
> (edge).
> > + * @__reserved3: reserved
> > + *
> > + * Fixed point type u13.0, range [0, 8191].
> > + */
> > +struct ipu3_uapi_sharp_cfg {
> > + __u32 nega_lmt_txt:13;
> > + __u32 __reserved0:19;
> > + __u32 posi_lmt_txt:13;
> > + __u32 __reserved1:19;
> > + __u32 nega_lmt_dir:13;
> > + __u32 __reserved2:19;
> > + __u32 posi_lmt_dir:13;
> > + __u32 __reserved3:19;
> > +} __packed;
> > +
> > +/**
> > + * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
> > + *
> > + * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64],
> default 64.
> > + * @__reserved0: reserved
> > + * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64],
> default 0.
> > + * @__reserved1: reserved
> > + * @ndir_dns_powr: Power of non-direct denoising,
> > + * Precision u1.6, range [0, 64], default 64.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_far_w {
> > + __u32 dir_shrp:7;
> > + __u32 __reserved0:1;
> > + __u32 dir_dns:7;
> > + __u32 __reserved1:1;
> > + __u32 ndir_dns_powr:7;
> > + __u32 __reserved2:9;
> > +} __packed;
> > +
> > +/**
> > + * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
> > + *
> > + * @unsharp_weight: Unsharp mask blending weight.
> > + * u1.6, range [0, 64], default 16.
> > + * 0 - disabled, 64 - use only unsharp.
> > + * @__reserved0: reserved
> > + * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511],
> default 0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_unsharp_cfg {
> > + __u32 unsharp_weight:7;
> > + __u32 __reserved0:1;
> > + __u32 unsharp_amount:9;
> > + __u32 __reserved1:15;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
> > + *
> > + * @cfg: sharpness config &ipu3_uapi_sharp_cfg
> > + * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
> > + * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
> > + * neighbors (8 pixels called Near), and the second order neighborhood
> > + * around them (16 pixels called Far).
> > + * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
> > + struct ipu3_uapi_sharp_cfg cfg;
> > + struct ipu3_uapi_far_w far_w;
> > + struct ipu3_uapi_unsharp_cfg unshrp_cfg;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
> > + *
> > + * @c00: Coeff11, s0.8, range [-255, 255], default 1.
> > + * @c01: Coeff12, s0.8, range [-255, 255], default 5.
> > + * @c02: Coeff13, s0.8, range [-255, 255], default 9.
> > + * @__reserved: reserved
> > + *
> > + * Configurable registers for common sharpening support.
> > + */
> > +struct ipu3_uapi_unsharp_coef0 {
> > + __u32 c00:9;
> > + __u32 c01:9;
> > + __u32 c02:9;
> > + __u32 __reserved:5;
>
> __s32?
>
Will add a note, same as ipu3_uapi_iefd_cux2_1.
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
> > + *
> > + * @c11: Coeff22, s0.8, range [-255, 255], default 29.
> > + * @c12: Coeff23, s0.8, range [-255, 255], default 55.
> > + * @c22: Coeff33, s0.8, range [-255, 255], default 96.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_unsharp_coef1 {
> > + __u32 c11:9;
> > + __u32 c12:9;
> > + __u32 c22:9;
>
> __s32?
>
Ack.
> > + __u32 __reserved:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
> > + *
> > + * @unsharp_coef0: unsharp coefficient 0 config. See
> &ipu3_uapi_unsharp_coef0
> > + * @unsharp_coef1: unsharp coefficient 1 config. See
> &ipu3_uapi_unsharp_coef1
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
> > + struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
> > + struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
> > +} __packed;
> > +
>
> ...
>
> > +/**
> > + * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
> > + *
> > + * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
> > + * @lin_lutlow_r: linearization look-up table for R channel interpolation.
> > + * @lin_lutlow_b: linearization look-up table for B channel interpolation.
> > + * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
> > + * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
>
> Copy & paste issue here? Should the postfixes be gr, r, b and gb instead?
>
Ack.
It's a long file, thanks a lot for your time.
Yong
> > + * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
> > + * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
> > + * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
> > + * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
> > + * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
> > + */
> > +struct ipu3_uapi_isp_lin_vmem_params {
> > + __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
> > +} __packed;
>
> --
> Kind regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-10-29 22:22 ` [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI Yong Zhi
2018-11-02 13:02 ` Sakari Ailus
@ 2018-11-02 13:49 ` Mauro Carvalho Chehab
2018-11-02 14:04 ` Tomasz Figa
2018-11-06 18:25 ` Zhi, Yong
2018-11-15 12:51 ` Hans Verkuil
2 siblings, 2 replies; 123+ messages in thread
From: Mauro Carvalho Chehab @ 2018-11-02 13:49 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao, Chao C Li
Em Mon, 29 Oct 2018 15:22:57 -0700
Yong Zhi <yong.zhi@intel.com> escreveu:
> These meta formats are used on Intel IPU3 ImgU video queues
> to carry 3A statistics and ISP pipeline parameters.
Just minor things. See below.
>
> V4L2_META_FMT_IPU3_3A
> V4L2_META_FMT_IPU3_PARAMS
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> Signed-off-by: Chao C Li <chao.c.li@intel.com>
> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> ---
> Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
I would actually prefer to have those two changes merged together with
patch 1, as it makes easier for review.
> include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
This one makes sense to have a separate patch.
> 3 files changed, 3001 insertions(+)
> create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> create mode 100644 include/uapi/linux/intel-ipu3.h
>
> diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
> index cf971d5..eafc534 100644
> --- a/Documentation/media/uapi/v4l/meta-formats.rst
> +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
> .. toctree::
> :maxdepth: 1
>
> + pixfmt-meta-intel-ipu3
> pixfmt-meta-d4xx
> pixfmt-meta-uvc
> pixfmt-meta-vsp1-hgo
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> new file mode 100644
> index 0000000..23b945b
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> @@ -0,0 +1,181 @@
> +.. -*- coding: utf-8; mode: rst -*-
> +
> +.. _intel-ipu3:
> +
> +******************************************************************
> +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
> +******************************************************************
> +
> +.. c:type:: ipu3_uapi_stats_3a
> +
> +3A statistics
> +=============
> +
> +For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
> +an input bayer frame. Those statistics, defined in data struct
> +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu 3a stat"
> +video node, which are then passed to user space for statistics analysis
> +using :c:type:`v4l2_meta_format` interface.
> +
> +The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
> +Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
> +and AE (Auto-exposure) histogram.
> +
> +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.
> +
> +
> +.. code-block:: c
> +
> +
> + struct ipu3_uapi_stats_3a {
> + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> + __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_raw_buffer_aligned
> + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> + struct ipu3_uapi_4a_config stats_4a_config;
> + __u32 ae_join_buffers;
> + __u8 padding[28];
> + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + stats_3a_bubble_per_stripe;
> + struct ipu3_uapi_ff_status stats_3a_status;
> + } __packed;
> +
> +
> +.. c:type:: ipu3_uapi_params
> +
> +Pipeline parameters
> +===================
> +
> +IPU3 pipeline has a number of image processing stages, each of which takes a
> +set of parameters as input. The major stages of pipelines are shown here:
> +
> +Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
> +
> +Linearization -> Lens Shading Correction -> White Balance / Exposure /
> +
> +Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
> +
> +Correction Matrix -> Gamma correction -> Color Space Conversion ->
> +
> +Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
> +
> +Correction -> XNR3 -> TNR -> DDR
> +
> +The table below presents a description of the above algorithms.
> +
> +======================== =======================================================
> +Name Description
> +======================== =======================================================
> +Optical Black Correction Optical Black Correction block subtracts a pre-defined
> + value from the respective pixel values to obtain better
> + image quality.
> + Defined in :c:type:`ipu3_uapi_obgrid_param`.
> +Linearization This algo block uses linearization parameters to
> + address non-linearity sensor effects. The Lookup table
> + table is defined in
> + :c:type:`ipu3_uapi_isp_lin_vmem_params`.
> +SHD Lens shading correction is used to correct spatial
> + non-uniformity of the pixel response due to optical
> + lens shading. This is done by applying a different gain
> + for each pixel. The gain, black level etc are
> + configured in :c:type:`ipu3_uapi_shd_config_static`.
> +BNR Bayer noise reduction block removes image noise by
> + applying a bilateral filter.
> + See :c:type:`ipu3_uapi_bnr_static_config` for details.
> +ANR Advanced Noise Reduction is a block based algorithm
> + that performs noise reduction in the Bayer domain. The
> + convolution matrix etc can be found in
> + :c:type:`ipu3_uapi_anr_config`.
> +Demosaicing Demosaicing converts raw sensor data in Bayer format
> + into RGB (Red, Green, Blue) presentation. Then add
> + outputs of estimation of Y channel for following stream
> + processing by Firmware. The struct is defined as
> + :c:type:`ipu3_uapi_dm_config`.
> +Color Correction Color Correction algo transforms sensor specific color
> + space to the standard "sRGB" color space. This is done
> + by applying 3x3 matrix defined in
> + :c:type:`ipu3_uapi_ccm_mat_config`.
> +Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
> + basic non-linear tone mapping correction that is
> + applied per pixel for each pixel component.
> +CSC Color space conversion transforms each pixel from the
> + RGB primary presentation to YUV (Y - brightness,
> + UV - Luminance) presentation. This is done by applying
> + a 3x3 matrix defined in
> + :c:type:`ipu3_uapi_csc_mat_config`
> +CDS Chroma down sampling
> + After the CSC is performed, the Chroma Down Sampling
> + is applied for a UV plane down sampling by a factor
> + of 2 in each direction for YUV 4:2:0 using a 4x2
> + configurable filter :c:type:`ipu3_uapi_cds_params`.
> +CHNR Chroma noise reduction
> + This block processes only the chrominance pixels and
> + performs noise reduction by cleaning the high
> + frequency noise.
> + See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
> +TCC Total color correction as defined in struct
> + :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
> +XNR3 eXtreme Noise Reduction V3 is the third revision of
> + noise reduction algorithm used to improve image
> + quality. This removes the low frequency noise in the
> + captured image. Two related structs are being defined,
> + :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
> + and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
> + memory.
> +TNR Temporal Noise Reduction block compares successive
> + frames in time to remove anomalies / noise in pixel
> + values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
> + :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
> + vector and data memory respectively.
> +======================== =======================================================
> +
> +A few stages of the pipeline will be executed by firmware running on the ISP
> +processor, while many others will use a set of fixed hardware blocks also
> +called accelerator cluster (ACC) to crunch pixel data and produce statistics.
> +
> +ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be selectively
> +enabled / disabled by the user space through struct :c:type:`ipu3_uapi_flags`
> +embedded in :c:type:`ipu3_uapi_params` structure. For parameters that are not
> +enabled by the user space, corresponding structs are ignored by the ISP.
> +
> +Both 3A statistics and pipeline parameters described here are closely tied to
> +the underlying camera sub-system (CSS) APIs. They are usually consumed and
> +produced by dedicated user space libraries that comprise the important tuning
> +tools, thus freeing the developers from being bothered with the low level
> +hardware and algorithm details.
> +
> +It should be noted that IPU3 DMA operations require the addresses of all data
> +structures (that includes both input and output) to be aligned on 32 byte
> +boundaries.
> +
> +The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
> +video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
> +
> +.. code-block:: c
> +
> + struct ipu3_uapi_params {
> + /* Flags which of the settings below are to be applied */
> + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> +
> + /* Accelerator cluster parameters */
> + struct ipu3_uapi_acc_param acc_param;
> +
> + /* ISP vector address space parameters */
> + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> +
> + /* ISP data memory (DMEM) parameters */
> + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> +
> + /* Optical black level compensation */
> + struct ipu3_uapi_obgrid_param obgrid_param;
> + } __packed;
> +
> +Intel IPU3 ImgU uAPI data types
> +===============================
> +
> +.. kernel-doc:: include/uapi/linux/intel-ipu3.h
> diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
> new file mode 100644
> index 0000000..c2608b6
> --- /dev/null
> +++ b/include/uapi/linux/intel-ipu3.h
> @@ -0,0 +1,2819 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2017 - 2018 Intel Corporation */
> +
> +#ifndef __IPU3_UAPI_H
> +#define __IPU3_UAPI_H
> +
> +#include <linux/types.h>
> +
> +/********************* Key Acronyms *************************/
> +/*
> + * ACC - Accelerator cluster
> + * ANR - Adaptive noise reduction
> + * AWB_FR- Auto white balance filter response statistics
> + * BNR - Bayer noise reduction parameters
> + * BDS - Bayer downscaler parameters
> + * CCM - Color correction matrix coefficients
> + * CDS - Chroma down sample
> + * CHNR - Chroma noise reduction
> + * CSC - Color space conversion
> + * DM - De-mosaic
> + * IEFd - Image enhancement filter directed
> + * Obgrid - Optical black level compensation
> + * OSYS - Output system configuration
> + * ROI - Region of interest
> + * SHD - Lens shading correction table
> + * TCC - Total color correction
> + * YDS - Y down sampling
> + * YTM - Y-tone mapping
> + */
Hmm... It probably makes sense to convert this into a documentation
block, e. g.:
/**
* DOC: Key Acronyms used by IPU3 ImgU driver
*
...
*/
And then include this header inside Documentation/media/v4l-drivers/ipu3.rst.
> +
> +/*
> + * IPU3 DMA operations require buffers to be aligned at
> + * 32 byte boundaries
> + */
> +
> +/******************* ipu3_uapi_stats_3a *******************/
> +
> +#define IPU3_UAPI_MAX_STRIPES 2
> +#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
> +
> +#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
> +#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
> +
> +/* controls generation of meta_data (like FF enable/disable) */
> +#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
> +#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
> +
> +/**
> + * struct ipu3_uapi_grid_config - Grid plane config
> + *
> + * @width: Grid horizontal dimensions, in number of grid blocks(cells).
> + * @height: Grid vertical dimensions, in number of grid cells.
> + * @block_width_log2: Log2 of the width of each cell in pixels.
> + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> + * @block_height_log2: Log2 of the height of each cell in pixels.
> + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> + * @height_per_slice: The number of blocks in vertical axis per slice.
> + * Default 2.
> + * @x_start: X value of top left corner of Region of Interest(ROI).
> + * @y_start: Y value of top left corner of ROI
> + * @x_end: X value of bottom right corner of ROI
> + * @y_end: Y value of bottom right corner of ROI
> + *
> + * Due to the size of total amount of collected data, most statistics
> + * create a grid-based output, and the data is then divided into "slices".
> + */
> +struct ipu3_uapi_grid_config {
> + __u8 width;
> + __u8 height;
> + __u16 block_width_log2:3;
> + __u16 block_height_log2:3;
> + __u16 height_per_slice:8;
> + __u16 x_start;
> + __u16 y_start;
> + __u16 x_end;
> + __u16 y_end;
> +} __packed;
> +
> +/*
> + * The grid based data is divided into "slices" called set, each slice of setX
> + * refers to ipu3_uapi_grid_config width * height_per_slice.
> + */
> +#define IPU3_UAPI_AWB_MAX_SETS 60
> +/* Based on grid size 80 * 60 and cell size 16 x 16 */
> +#define IPU3_UAPI_AWB_SET_SIZE 1280
> +#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
> +#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AWB_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
> + (IPU3_UAPI_AWB_MAX_SETS * \
> + (IPU3_UAPI_AWB_SET_SIZE + IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
> +/**
> + * struct ipu3_uapi_awb_meta_data - AWB meta data
> + *
> + * @meta_data_buffer: Average values for each color channel
> + */
> +struct ipu3_uapi_awb_meta_data {
> + __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
> + *
> + * @meta_data: buffer to hold auto white balance meta data.
> + */
> +struct ipu3_uapi_awb_raw_buffer {
> + struct ipu3_uapi_awb_meta_data meta_data;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_config_s - AWB config
> + *
> + * @rgbs_thr_gr: gr threshold value.
> + * @rgbs_thr_r: Red threshold value.
> + * @rgbs_thr_gb: gb threshold value.
> + * @rgbs_thr_b: Blue threshold value.
> + * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
> + *
> + * The threshold is a saturation measure range [0, 8191], 8191 is default.
> + * Values over threshold may be optionally rejected for averaging.
> + */
> +struct ipu3_uapi_awb_config_s {
> + __u16 rgbs_thr_gr;
> + __u16 rgbs_thr_r;
> + __u16 rgbs_thr_gb;
> + __u16 rgbs_thr_b;
> + struct ipu3_uapi_grid_config grid;
> +} __attribute__((aligned(32))) __packed;
Hmm... Kernel defines a macro for aligned attribute:
include/linux/compiler_types.h:#define __aligned(x) __attribute__((aligned(x)))
I'm not a gcc expert, but it sounds weird to first ask it to align
with 32 bits and then have __packed (with means that pads should be
removed).
In other words, I *guess* is it should either be __packed
or __aligned(32).
Not that it would do any difference, in practice, as this
specific struct has a size with is multiple of 32 bits, but
let's do the right annotation here, not mixing two incompatible
alignment requirements.
> +
> +/**
> + * struct ipu3_uapi_awb_config - AWB config wrapper
> + *
> + * @config: config for auto white balance as defined by &ipu3_uapi_awb_config_s
> + */
> +struct ipu3_uapi_awb_config {
> + struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
> +} __packed;
> +
> +#define IPU3_UAPI_AE_COLORS 4 /* R, G, B, Y */
> +#define IPU3_UAPI_AE_BINS 256
> +#define IPU3_UAPI_AE_WEIGHTS 96
> +
> +/**
> + * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
> + *
> + * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
> + *
> + * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit unsigned
> + * for counting the number of the pixel.
> + */
> +struct ipu3_uapi_ae_raw_buffer {
> + __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
> + *
> + * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
> + */
> +struct ipu3_uapi_ae_raw_buffer_aligned {
> + struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
Please use __aligned(32).
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_grid_config - AE weight grid
> + *
> + * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
> + * @height: Grid vertical dimensions. Value: [16, 24], default 16.
> + * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
> + * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
> + * @__reserved0: reserved
> + * @ae_en: 0: does not write to meta-data array, 1: write normally.
> + * @rst_hist_array: write 1 to trigger histogram array reset.
> + * @done_rst_hist_array: flag for histogram array reset done.
> + * @x_start: X value of top left corner of ROI, default 0.
> + * @y_start: Y value of top left corner of ROI, default 0.
> + * @x_end: X value of bottom right corner of ROI
> + * @y_end: Y value of bottom right corner of ROI
> + *
> + * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
> + * a defined ROI within the frame. The contribution of each pixel into the
> + * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a grid.
> + */
> +struct ipu3_uapi_ae_grid_config {
> + __u8 width;
> + __u8 height;
> + __u8 block_width_log2:4;
> + __u8 block_height_log2:4;
> + __u8 __reserved0:5;
> + __u8 ae_en:1;
> + __u8 rst_hist_array:1;
> + __u8 done_rst_hist_array:1;
> + __u16 x_start;
> + __u16 y_start;
> + __u16 x_end;
> + __u16 y_end;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_weight_elem - AE weights LUT
> + *
> + * @cell0: weighted histogram grid value.
> + * @cell1: weighted histogram grid value.
> + * @cell2: weighted histogram grid value.
> + * @cell3: weighted histogram grid value.
> + * @cell4: weighted histogram grid value.
> + * @cell5: weighted histogram grid value.
> + * @cell6: weighted histogram grid value.
> + * @cell7: weighted histogram grid value.
> + *
> + * Use weighted grid value to give a different contribution factor to each cell.
> + * Precision u4, range [0, 15].
> + */
> +struct ipu3_uapi_ae_weight_elem {
> + __u32 cell0:4;
> + __u32 cell1:4;
> + __u32 cell2:4;
> + __u32 cell3:4;
> + __u32 cell4:4;
> + __u32 cell5:4;
> + __u32 cell6:4;
> + __u32 cell7:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
> + *
> + * @gain_gr: WB gain factor for the gr channels. Default 256.
> + * @gain_r: WB gain factor for the r channel. Default 256.
> + * @gain_b: WB gain factor for the b channel. Default 256.
> + * @gain_gb: WB gain factor for the gb channels. Default 256.
> + * @mat: 4x4 matrix that transforms Bayer quad output from WB to RGB+Y.
> + *
> + * Default:
> + * 128, 0, 0, 0,
> + * 0, 128, 0, 0,
> + * 0, 0, 128, 0,
> + * 0, 0, 0, 128,
> + *
> + * As part of the raw frame pre-process stage, the WB and color conversion need
> + * to be applied to expose the impact of these gain operations.
> + */
> +struct ipu3_uapi_ae_ccm {
> + __u16 gain_gr;
> + __u16 gain_r;
> + __u16 gain_b;
> + __u16 gain_gb;
> + __s16 mat[16];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_config - AE config
> + *
> + * @grid_cfg: config for auto exposure statistics grid. See struct
> + * &ipu3_uapi_ae_grid_config
> + * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks in the grid.
> + * Each grid cell has a corresponding value in weights LUT called
> + * grid value, global histogram is updated based on grid value and
> + * pixel value.
> + * @ae_ccm: Color convert matrix pre-processing block.
> + *
> + * Calculate AE grid from image resolution, resample ae weights.
> + */
> +struct ipu3_uapi_ae_config {
> + struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_weight_elem weights[
> + IPU3_UAPI_AE_WEIGHTS] __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
Same above: __aligned(32). Please review the remaining of this header,
as there are other occurrences of this pattern.
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast measurements
> + *
> + * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
> + * anti-symmetry type. A12 is center, A1-A11 are neighbours.
> + * for analyzing low frequency content, used to calculate sum
> + * of gradients in x direction.
> + * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
> + * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
> + * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
> + * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
> + * @y1_coeff_1: Struct
> + * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
> + * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
> + * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
> + * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
> + * @y1_coeff_2: Struct
> + * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> + * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> + * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> + * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> + * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
> + * 0: positive, 1: negative, default 0.
> + * @y2_coeff_0: Y2, same structure as Y1. For analyzing high frequency content.
> + * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
> + * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
> + * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
> + * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
> + * @y2_coeff_1: Struct
> + * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
> + * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
> + * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
> + * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
> + * @y2_coeff_2: Struct
> + * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> + * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> + * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> + * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> + * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
> + * 0: positive, 1: negative, default 0.
> + * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to be
> + * used for building histogram. Range [0, 32], default 8.
> + * Rule:
> + * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b + y_gen_rate_gb = 32
> + * A single Y is calculated based on sum of Gr/R/B/Gb based on
> + * their contribution ratio.
> + * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
> + * @y_calc.y_gen_rate_r: Contribution ratio R for Y
> + * @y_calc.y_gen_rate_b: Contribution ratio B for Y
> + * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
> + * @nf: The shift right value that should be applied during the Y1/Y2 filter to
> + * make sure the total memory needed is 2 bytes per grid cell.
> + * @nf.__reserved0: reserved
> + * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
> + * should be log2 of the sum of the abs values of the filter
> + * coeffs, default 7 (2^7 = 128).
> + * @nf.__reserved1: reserved
> + * @nf.y2_nf: Normalization factor for y2, should be log2 of the sum of the
> + * abs values of the filter coeffs.
> + * @nf.__reserved2: reserved
> + */
> +struct ipu3_uapi_af_filter_config {
> + struct {
> + __u8 a1;
> + __u8 a2;
> + __u8 a3;
> + __u8 a4;
> + } y1_coeff_0;
> + struct {
> + __u8 a5;
> + __u8 a6;
> + __u8 a7;
> + __u8 a8;
> + } y1_coeff_1;
> + struct {
> + __u8 a9;
> + __u8 a10;
> + __u8 a11;
> + __u8 a12;
> + } y1_coeff_2;
> +
> + __u32 y1_sign_vec;
> +
> + struct {
> + __u8 a1;
> + __u8 a2;
> + __u8 a3;
> + __u8 a4;
> + } y2_coeff_0;
> + struct {
> + __u8 a5;
> + __u8 a6;
> + __u8 a7;
> + __u8 a8;
> + } y2_coeff_1;
> + struct {
> + __u8 a9;
> + __u8 a10;
> + __u8 a11;
> + __u8 a12;
> + } y2_coeff_2;
> +
> + __u32 y2_sign_vec;
> +
> + struct {
> + __u8 y_gen_rate_gr;
> + __u8 y_gen_rate_r;
> + __u8 y_gen_rate_b;
> + __u8 y_gen_rate_gb;
> + } y_calc;
> +
> + struct {
> + __u32 __reserved0:8;
> + __u32 y1_nf:4;
> + __u32 __reserved1:4;
> + __u32 y2_nf:4;
> + __u32 __reserved2:12;
> + } nf;
> +} __packed;
> +
> +#define IPU3_UAPI_AF_MAX_SETS 24
> +#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
> +#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AF_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
> +#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
> + (IPU3_UAPI_AF_MAX_SETS * \
> + (IPU3_UAPI_AF_Y_TABLE_SET_SIZE + IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
> + IPU3_UAPI_MAX_STRIPES)
> +
> +/**
> + * struct ipu3_uapi_af_meta_data - AF meta data
> + *
> + * @y_table: Each color component will be convolved separately with filter1
> + * and filter2 and the result will be summed out and averaged for
> + * each cell.
> + */
> +struct ipu3_uapi_af_meta_data {
> + __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE] __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_raw_buffer - AF raw buffer
> + *
> + * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus meta data.
> + */
> +struct ipu3_uapi_af_raw_buffer {
> + struct ipu3_uapi_af_meta_data meta_data __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_config_s - AF config
> + *
> + * @filter_config: AF uses Y1 and Y2 filters as configured in
> + * &ipu3_uapi_af_filter_config
> + * @padding: paddings
> + * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use large
> + * grid size for large image and vice versa.
> + */
> +struct ipu3_uapi_af_config_s {
> + struct ipu3_uapi_af_filter_config filter_config __attribute__((aligned(32)));
> + __u8 padding[4];
> + struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_config - AF config wrapper
> + *
> + * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
> + */
> +struct ipu3_uapi_af_config {
> + struct ipu3_uapi_af_config_s config;
> +} __packed;
> +
> +#define IPU3_UAPI_AWB_FR_MAX_SETS 24
> +#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
> +#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
> +#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
> + (IPU3_UAPI_AWB_FR_MAX_SETS * \
> + (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
> + IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) * IPU3_UAPI_MAX_STRIPES)
> +
> +/**
> + * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
> + *
> + * @bayer_table: Statistics output on the grid after convolving with 1D filter.
> + */
> +struct ipu3_uapi_awb_fr_meta_data {
> + __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE] __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
> + *
> + * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
> + */
> +struct ipu3_uapi_awb_fr_raw_buffer {
> + struct ipu3_uapi_awb_fr_meta_data meta_data;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
> + *
> + * @grid_cfg: grid config, default 16x16.
> + * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
> + * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
> + * Applied on whole image for each Bayer channel separately
> + * by a weighted sum of its 11x1 neighbors.
> + * @__reserved1: reserved
> + * @bayer_sign: sign of filter coeffcients, default 0.
> + * @bayer_nf: normalization factor for the convolution coeffs, to make sure
> + * total memory needed is within pre-determined range.
> + * NF should be the log2 of the sum of the abs values of the
> + * filter coeffs, range [7, 14], default 7.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_awb_fr_config_s {
> + struct ipu3_uapi_grid_config grid_cfg;
> + __u8 bayer_coeff[6];
> + __u16 __reserved1;
> + __u32 bayer_sign;
> + __u8 bayer_nf;
> + __u8 __reserved2[3];
> +} __attribute__((aligned(32))) __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
> + *
> + * @config: See &ipu3_uapi_awb_fr_config_s.
> + */
> +struct ipu3_uapi_awb_fr_config {
> + struct ipu3_uapi_awb_fr_config_s config;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_4a_config - 4A config
> + *
> + * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
> + * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
> + * @padding: paddings
> + * @af_config: auto focus config &ipu3_uapi_af_config_s
> + * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution 16x16
> + */
> +struct ipu3_uapi_4a_config {
> + struct ipu3_uapi_awb_config_s awb_config __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_grid_config ae_grd_config;
> + __u8 padding[20];
> + struct ipu3_uapi_af_config_s af_config;
> + struct ipu3_uapi_awb_fr_config_s awb_fr_config;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
> + *
> + * @num_of_stripes: A single frame is divided into several parts called stripes
> + * due to limitation on line buffer memory.
> + * The separation between the stripes is vertical. Each such
> + * stripe is processed as a single frame by the ISP pipe.
> + * @padding: padding bytes.
> + * @num_sets: number of sets.
> + * @padding1: padding bytes.
> + * @size_of_set: set size.
> + * @padding2: padding bytes.
> + * @bubble_size: is the amount of padding in the bubble expressed in "sets".
> + * @padding3: padding bytes.
> + */
> +struct ipu3_uapi_bubble_info {
> + __u32 num_of_stripes __attribute__((aligned(32)));
> + __u8 padding[28];
> + __u32 num_sets;
> + __u8 padding1[28];
> + __u32 size_of_set;
> + __u8 padding2[28];
> + __u32 bubble_size;
> + __u8 padding3[28];
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + */
> +struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
> + struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
> + *
> + * @awb_en: auto white balance enable
> + * @padding: padding config
> + * @ae_en: auto exposure enable
> + * @padding1: padding config
> + * @af_en: auto focus enable
> + * @padding2: padding config
> + * @awb_fr_en: awb filter response enable bit
> + * @padding3: padding config
> + */
> +struct ipu3_uapi_ff_status {
> + __u32 awb_en __attribute__((aligned(32)));
> + __u8 padding[28];
> + __u32 ae_en;
> + __u8 padding1[28];
> + __u32 af_en;
> + __u8 padding2[28];
> + __u32 awb_fr_en;
> + __u8 padding3[28];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_stats_3a - 3A statistics
> + *
> + * @awb_raw_buffer: auto white balance meta data &ipu3_uapi_awb_raw_buffer
> + * @ae_raw_buffer: auto exposure raw data &ipu3_uapi_ae_raw_buffer_aligned
> + * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
> + * @awb_fr_raw_buffer: value as specified by &ipu3_uapi_awb_fr_raw_buffer
> + * @stats_4a_config: 4a statistics config as defined by &ipu3_uapi_4a_config.
> + * @ae_join_buffers: 1 to use ae_raw_buffer.
> + * @padding: padding config
> + * @stats_3a_bubble_per_stripe: a &ipu3_uapi_stats_3a_bubble_info_per_stripe
> + * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
> + */
> +struct ipu3_uapi_stats_3a {
> + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_raw_buffer_aligned
> + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> + struct ipu3_uapi_4a_config stats_4a_config;
> + __u32 ae_join_buffers;
> + __u8 padding[28];
> + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + stats_3a_bubble_per_stripe;
> + struct ipu3_uapi_ff_status stats_3a_status;
> +} __packed;
> +
> +/******************* ipu3_uapi_acc_param *******************/
> +
> +#define IPU3_UAPI_ISP_VEC_ELEMS 64
> +#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
> +
> +#define IPU3_UAPI_BNR_LUT_SIZE 32
> +
> +/* number of elements in gamma correction LUT */
> +#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
> +
> +/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
> +#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
> +#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
> +/* Normalization shift aka nf */
> +#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
> +#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
> +
> +#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
> +#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
> +#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
> +#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
> +
> +#define IPU3_UAPI_ANR_LUT_SIZE 26
> +#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
> +
> +#define IPU3_UAPI_LIN_LUT_SIZE 64
> +
> +/* Bayer Noise Reduction related structs */
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance gains
> + *
> + * @gr: white balance gain for Gr channel.
> + * @r: white balance gain for R channel.
> + * @b: white balance gain for B channel.
> + * @gb: white balance gain for Gb channel.
> + *
> + * Precision u3.13, range [0, 8]. White balance correction is done by applying
> + * a multiplicative gain to each color channels prior to BNR.
> + */
> +struct ipu3_uapi_bnr_static_config_wb_gains_config {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold config
> + *
> + * @gr: white balance threshold gain for Gr channel.
> + * @r: white balance threshold gain for R channel.
> + * @b: white balance threshold gain for B channel.
> + * @gb: white balance threshold gain for Gb channel.
> + *
> + * Defines the threshold that specifies how different a defect pixel can be from
> + * its neighbors.(used by dynamic defect pixel correction sub block)
> + * Precision u4.4 range [0, 8].
> + */
> +struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
> + __u8 gr;
> + __u8 r;
> + __u8 b;
> + __u8 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
> + * coefficients that controls noise threshold
> + *
> + * @cf: Free coefficient for threshold calculation, range [0, 8191], default 0.
> + * @__reserved0: reserved
> + * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
> + * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
> + * default 6.
> + * format: u3.2 (3 most significant bits represent whole number,
> + * 2 least significant bits represent the fractional part
> + * with each count representing 0.25)
> + * e.g 6 in binary format is 00110, that translates to 1.5
> + * @__reserved1: reserved
> + * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
> + * where r is a radius of pixel [row, col] from centor of sensor.
> + * default 14.
> + *
> + * Threshold used to distinguish between noise and details.
> + */
> +struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
> + __u32 cf:13;
> + __u32 __reserved0:3;
> + __u32 cg:5;
> + __u32 ci:5;
> + __u32 __reserved1:1;
> + __u32 r_nf:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
> + *
> + * @gr: Coefficient defines lens shading gain approximation for gr channel
> + * @r: Coefficient defines lens shading gain approximation for r channel
> + * @b: Coefficient defines lens shading gain approximation for b channel
> + * @gb: Coefficient defines lens shading gain approximation for gb channel
> + *
> + * Parameters for noise model (NM) adaptation of BNR due to shading correction.
> + * All above have precision of u3.3, default to 0.
> + */
> +struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
> + __u8 gr;
> + __u8 r;
> + __u8 b;
> + __u8 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center config
> + *
> + * @x_reset: Reset value of X (col start - X center). Precision s12.0.
> + * @__reserved0: reserved
> + * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
> + * @__reserved2: reserved
> + *
> + * Distance from corner to optical center for NM adaptation due to shading
> + * correction (should be calculated based on shading tables)
> + */
> +struct ipu3_uapi_bnr_static_config_opt_center_config {
> + __s32 x_reset:13;
> + __u32 __reserved0:3;
> + __s32 y_reset:13;
> + __u32 __reserved2:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup table
> + *
> + * @values: pre-calculated values of square root function.
> + *
> + * LUT implementation of square root operation.
> + */
> +struct ipu3_uapi_bnr_static_config_lut_config {
> + __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels (bp)
> + *
> + * @bp_thr_gain: Defines the threshold that specifies how different a
> + * defect pixel can be from its neighbors. Threshold is
> + * dependent on de-noise threshold calculated by algorithm.
> + * Range [4, 31], default 4.
> + * @__reserved0: reserved
> + * @defect_mode: Mode of addressed defect pixels,
> + * 0 - single defect pixel is expected,
> + * 1 - 2 adjacent defect pixels are expected, default 1.
> + * @bp_gain: Defines how 2nd derivation that passes through a defect pixel
> + * is different from 2nd derivations that pass through
> + * neighbor pixels. u4.2, range [0, 256], default 8.
> + * @__reserved1: reserved
> + * @w0_coeff: Blending coefficient of defect pixel correction.
> + * Precision u4, range [0, 8], default 8.
> + * @__reserved2: reserved
> + * @w1_coeff: Enable influence of incorrect defect pixel correction to be
> + * avoided. Precision u4, range [1, 8], default 8.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
> + __u32 bp_thr_gain:5;
> + __u32 __reserved0:2;
> + __u32 defect_mode:1;
> + __u32 bp_gain:6;
> + __u32 __reserved1:18;
> + __u32 w0_coeff:4;
> + __u32 __reserved2:4;
> + __u32 w1_coeff:4;
> + __u32 __reserved3:20;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising config
> + *
> + * @alpha: Weight of central element of smoothing filter.
> + * @beta: Weight of peripheral elements of smoothing filter, default 4.
> + * @gamma: Weight of diagonal elements of smoothing filter, default 4.
> + *
> + * beta and gamma parameter define the strength of the noise removal filter.
> + * All above has precision u0.4, range [0, 0xf]
> + * format: u0.4 (no / zero bits represent whole number,
> + * 4 bits represent the fractional part
> + * with each count representing 0.0625)
> + * e.g 0xf translates to 0.0625x15 = 0.9375
> + *
> + * @__reserved0: reserved
> + * @max_inf: Maximum increase of peripheral or diagonal element influence
> + * relative to the pre-defined value range: [0x5, 0xa]
> + * @__reserved1: reserved
> + * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
> + * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 - enable.
> + * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 - enable.
> + * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
> + __u32 alpha:4;
> + __u32 beta:4;
> + __u32 gamma:4;
> + __u32 __reserved0:4;
> + __u32 max_inf:4;
> + __u32 __reserved1:7;
> + __u32 gd_enable:1;
> + __u32 bpc_enable:1;
> + __u32 bnr_enable:1;
> + __u32 ff_enable:1;
> + __u32 __reserved2:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical square
> + *
> + * @x_sqr_reset: Reset value of X^2.
> + * @y_sqr_reset: Reset value of Y^2.
> + *
> + * Please note:
> + *
> + * #. X and Y ref to
> + * &ipu3_uapi_bnr_static_config_opt_center_config
> + * #. Both structs are used in threshold formula to calculate r^2, where r
> + * is a radius of pixel [row, col] from centor of sensor.
> + */
> +struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
> + __u32 x_sqr_reset;
> + __u32 y_sqr_reset;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config - BNR static config
> + *
> + * @wb_gains: white balance gains &ipu3_uapi_bnr_static_config_wb_gains_config
> + * @wb_gains_thr: white balance gains threshold as defined by
> + * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
> + * @thr_coeffs: coefficients of threshold
> + * &ipu3_uapi_bnr_static_config_thr_coeffs_config
> + * @thr_ctrl_shd: control of shading threshold
> + * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
> + * @opt_center: optical center &ipu3_uapi_bnr_static_config_opt_center_config
> + *
> + * Above parameters and opt_center_sqr are used for white balance and shading.
> + *
> + * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
> + * @bp_ctrl: detect and remove bad pixels as defined in struct
> + * &ipu3_uapi_bnr_static_config_bp_ctrl_config
> + * @dn_detect_ctrl: detect and remove noise.
> + * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> + * @column_size: The number of pixels in column.
> + * @opt_center_sqr: Reset value of r^2 to optical center, see
> + * &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
> + */
> +struct ipu3_uapi_bnr_static_config {
> + struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
> + struct ipu3_uapi_bnr_static_config_wb_gains_thr_config wb_gains_thr;
> + struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
> + struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
> + struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
> + struct ipu3_uapi_bnr_static_config_lut_config lut;
> + struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
> + struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config dn_detect_ctrl;
> + __u32 column_size;
> + struct ipu3_uapi_bnr_static_config_opt_center_sqr_config opt_center_sqr;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green disparity
> + *
> + * @gd_red: Shading gain coeff for gr disparity level in bright red region.
> + * Precision u0.6, default 4(0.0625).
> + * @__reserved0: reserved
> + * @gd_green: Shading gain coeff for gr disparity level in bright green
> + * region. Precision u0.6, default 4(0.0625).
> + * @__reserved1: reserved
> + * @gd_blue: Shading gain coeff for gr disparity level in bright blue region.
> + * Precision u0.6, default 4(0.0625).
> + * @__reserved2: reserved
> + * @gd_black: Maximal green disparity level in dark region (stronger disparity
> + * assumed to be image detail). Precision u14, default 80.
> + * @__reserved3: reserved
> + * @gd_shading: Change maximal green disparity level according to square
> + * distance from image center.
> + * @__reserved4: reserved
> + * @gd_support: Lower bound for the number of second green color pixels in
> + * current pixel neighborhood with less than threshold difference
> + * from it.
> + *
> + * The shading gain coeff of red, green, blue and black are used to calculate
> + * threshold given a pixel's color value and its coordinates in the image.
> + *
> + * @__reserved5: reserved
> + * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
> + * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
> + */
> +struct ipu3_uapi_bnr_static_config_green_disparity {
> + __u32 gd_red:6;
> + __u32 __reserved0:2;
> + __u32 gd_green:6;
> + __u32 __reserved1:2;
> + __u32 gd_blue:6;
> + __u32 __reserved2:10;
> + __u32 gd_black:14;
> + __u32 __reserved3:2;
> + __u32 gd_shading:7;
> + __u32 __reserved4:1;
> + __u32 gd_support:2;
> + __u32 __reserved5:1;
> + __u32 gd_clip:1;
> + __u32 gd_central_weight:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_dm_config - De-mosaic parameters
> + *
> + * @dm_en: de-mosaic enable.
> + * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
> + * @fcc_en: False color correction (FCC) enable flag. Default 0.
> + * @__reserved0: reserved
> + * @frame_width: do not care
> + * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation of
> + * complementary color in Hamilton-Adams interpolation).
> + * u5, range [0, 31], default 8.
> + * @__reserved1: reserved
> + * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity metric
> + * in calculation of final homogeneity metric.
> + * u5, range [0, 31], default 7.
> + * @__reserved2: reserved
> + * @cr_param1: First parameter that defines Checker artifact removal
> + * feature gain.Precision u5, range [0, 31], default 8.
> + * @__reserved3: reserved
> + * @cr_param2: Second parameter that defines Checker artifact removal
> + * feature gain. Precision u5, range [0, 31], default 8.
> + * @__reserved4: reserved
> + * @coring_param: Defines power of false color correction operation.
> + * low for preserving edge colors, high for preserving gray
> + * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
> + * @__reserved5: reserved
> + *
> + * The demosaic fixed function block is responsible to covert Bayer(mosaiced)
> + * images into color images based on demosaicing algorithm.
> + */
> +struct ipu3_uapi_dm_config {
> + __u32 dm_en:1;
> + __u32 ch_ar_en:1;
> + __u32 fcc_en:1;
> + __u32 __reserved0:13;
> + __u32 frame_width:16;
> +
> + __u32 gamma_sc:5;
> + __u32 __reserved1:3;
> + __u32 lc_ctrl:5;
> + __u32 __reserved2:3;
> + __u32 cr_param1:5;
> + __u32 __reserved3:3;
> + __u32 cr_param2:5;
> + __u32 __reserved4:3;
> +
> + __u32 coring_param:5;
> + __u32 __reserved5:27;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ccm_mat_config - Color correction matrix
> + *
> + * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
> + * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
> + * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
> + * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
> + * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
> + *
> + * Transform sensor specific color space to standard sRGB by applying 3x3 matrix
> + * and adding a bias vector O. The transformation is basically a rotation and
> + * translation in the 3-dimensional color spaces. Here are the defaults:
> + *
> + * 9775, -2671, 1087, 0
> + * -1071, 8303, 815, 0
> + * -23, -7887, 16103, 0
> + */
> +struct ipu3_uapi_ccm_mat_config {
> + __s16 coeff_m11;
> + __s16 coeff_m12;
> + __s16 coeff_m13;
> + __s16 coeff_o_r;
> + __s16 coeff_m21;
> + __s16 coeff_m22;
> + __s16 coeff_m23;
> + __s16 coeff_o_g;
> + __s16 coeff_m31;
> + __s16 coeff_m32;
> + __s16 coeff_m33;
> + __s16 coeff_o_b;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
> + *
> + * @enable: gamma correction enable.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_gamma_corr_ctrl {
> + __u32 enable:1;
> + __u32 __reserved:31;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping implemented as LUT.
> + *
> + * @lut: 256 tabulated values of the gamma function. LUT[1].. LUT[256]
> + * format u13.0, range [0, 8191].
> + *
> + * The tone mapping operation is done by a Piece wise linear graph
> + * that is implemented as a lookup table(LUT). The pixel component input
> + * intensity is the X-axis of the graph which is the table entry.
> + */
> +struct ipu3_uapi_gamma_corr_lut {
> + __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_config - Gamma config
> + *
> + * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
> + * @gc_lut: lookup table of gamma correction &ipu3_uapi_gamma_corr_lut
> + */
> +struct ipu3_uapi_gamma_config {
> + struct ipu3_uapi_gamma_corr_ctrl gc_ctrl __attribute__((aligned(32)));
> + struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
> + *
> + * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + *
> + * To transform each pixel from RGB to YUV (Y - brightness/luminance,
> + * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding an
> + * optional bias 3x1 vector.
> + */
> +struct ipu3_uapi_csc_mat_config {
> + __s16 coeff_c11;
> + __s16 coeff_c12;
> + __s16 coeff_c13;
> + __s16 coeff_b1;
> + __s16 coeff_c21;
> + __s16 coeff_c22;
> + __s16 coeff_c23;
> + __s16 coeff_b2;
> + __s16 coeff_c31;
> + __s16 coeff_c32;
> + __s16 coeff_c33;
> + __s16 coeff_b3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cds_params - Chroma down-scaling
> + *
> + * @ds_c00: range [0, 3]
> + * @ds_c01: range [0, 3]
> + * @ds_c02: range [0, 3]
> + * @ds_c03: range [0, 3]
> + * @ds_c10: range [0, 3]
> + * @ds_c11: range [0, 3]
> + * @ds_c12: range [0, 3]
> + * @ds_c13: range [0, 3]
> + *
> + * In case user does not provide, above 4x2 filter will use following defaults:
> + * 1, 3, 3, 1,
> + * 1, 3, 3, 1,
> + *
> + * @ds_nf: Normalization factor for Chroma output downscaling filter,
> + * range 0,4, default 2.
> + * @__reserved0: reserved
> + * @csc_en: Color space conversion enable
> + * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_cds_params {
> + __u32 ds_c00:2;
> + __u32 ds_c01:2;
> + __u32 ds_c02:2;
> + __u32 ds_c03:2;
> + __u32 ds_c10:2;
> + __u32 ds_c11:2;
> + __u32 ds_c12:2;
> + __u32 ds_c13:2;
> + __u32 ds_nf:5;
> + __u32 __reserved0:3;
> + __u32 csc_en:1;
> + __u32 uv_bin_output:1;
> + __u32 __reserved1:6;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
> + *
> + * @width: Grid horizontal dimensions, u8, [8, 128], default 73
> + * @height: Grid vertical dimensions, u8, [8, 128], default 56
> + * @block_width_log2: Log2 of the width of the grid cell in pixel count
> + * u4, [0, 15], default value 5.
> + * @__reserved0: reserved
> + * @block_height_log2: Log2 of the height of the grid cell in pixel count
> + * u4, [0, 15], default value 6.
> + * @__reserved1: reserved
> + * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
> + * (with SHD_MAX_CELLS_PER_SET = 146).
> + * @x_start: X value of top left corner of sensor relative to ROI
> + * u12, [-4096, 0]. default 0, only negative values.
> + * @y_start: Y value of top left corner of sensor relative to ROI
> + * u12, [-4096, 0]. default 0, only negative values.
> + */
> +struct ipu3_uapi_shd_grid_config {
> + /* reg 0 */
> + __u8 width;
> + __u8 height;
> + __u8 block_width_log2:3;
> + __u8 __reserved0:1;
> + __u8 block_height_log2:3;
> + __u8 __reserved1:1;
> + __u8 grid_height_per_slice;
> + /* reg 1 */
> + __s16 x_start;
> + __s16 y_start;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_general_config - Shading general config
> + *
> + * @init_set_vrt_offst_ul: set vertical offset,
> + * y_start >> block_height_log2 % grid_height_per_slice.
> + * @shd_enable: shading enable.
> + * @gain_factor: Gain factor. Shift calculated anti shading value. Precision u2.
> + * 0x0 - gain factor [1, 5], means no shift interpolated value.
> + * 0x1 - gain factor [1, 9], means shift interpolated by 1.
> + * 0x2 - gain factor [1, 17], means shift interpolated by 2.
> + * @__reserved: reserved
> + *
> + * Correction is performed by multiplying a gain factor for each of the 4 Bayer
> + * channels as a function of the pixel location in the sensor.
> + */
> +struct ipu3_uapi_shd_general_config {
> + __u32 init_set_vrt_offst_ul:8;
> + __u32 shd_enable:1;
> + __u32 gain_factor:2;
> + __u32 __reserved:21;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_black_level_config - Black level correction
> + *
> + * @bl_r: Bios values for green red. s11 range [-2048, 2047].
> + * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
> + * @bl_gb: Bios values for red. s11 range [-2048, 2047].
> + * @bl_b: Bios values for blue. s11 range [-2048, 2047].
> + */
> +struct ipu3_uapi_shd_black_level_config {
> + __s16 bl_r;
> + __s16 bl_gr;
> + __s16 bl_gb;
> + __s16 bl_b;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_config_static - Shading config static
> + *
> + * @grid: shading grid config &ipu3_uapi_shd_grid_config
> + * @general: shading general config &ipu3_uapi_shd_general_config
> + * @black_level: black level config for shading correction as defined by
> + * &ipu3_uapi_shd_black_level_config
> + */
> +struct ipu3_uapi_shd_config_static {
> + struct ipu3_uapi_shd_grid_config grid;
> + struct ipu3_uapi_shd_general_config general;
> + struct ipu3_uapi_shd_black_level_config black_level;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
> + *
> + * @sets: array
> + * @sets.r_and_gr: Red and GreenR Lookup table.
> + * @sets.r_and_gr.r: Red shading factor.
> + * @sets.r_and_gr.gr: GreenR shading factor.
> + * @sets.__reserved1: reserved
> + * @sets.gb_and_b: GreenB and Blue Lookup table.
> + * @sets.gb_and_b.gb: GreenB shading factor.
> + * @sets.gb_and_b.b: Blue shading factor.
> + * @sets.__reserved2: reserved
> + *
> + * Map to shading correction LUT register set.
> + */
> +struct ipu3_uapi_shd_lut {
> + struct {
> + struct {
> + __u16 r;
> + __u16 gr;
> + } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> + __u8 __reserved1[24];
> + struct {
> + __u16 gb;
> + __u16 b;
> + } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> + __u8 __reserved2[24];
> + } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_config - Shading config
> + *
> + * @shd: shading static config, see &ipu3_uapi_shd_config_static
> + * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
> + */
> +struct ipu3_uapi_shd_config {
> + struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
> + struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
> +} __packed;
> +
> +/* Image Enhancement Filter directed */
> +
> +/**
> + * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
> + *
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A of Config Unit, s4.4, default 0.
> + * @b01: Always 0.
> + *
> + * Calculate weight for blending directed and non-directed denoise elements
> + *
> + * Note:
> + * Each instance of Config Unit needs X coordinate of n points and
> + * slope A factor between points calculated by driver based on calibration
> + * parameters.
> + */
> +struct ipu3_uapi_iefd_cux2 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 a01:9;
> + __u32 b01:5; /* NOTE: hardcoded to zero */
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed sharpening
> + * element, Config Unit 6 for edge detail (ED).
> + *
> + * @x0: X coordinate of point 0, u9.0, default 0.
> + * @x1: X coordinate of point 1, u9.0, default 0.
> + * @x2: X coordinate of point 2, u9.0, default 0.
> + * @__reserved0: reserved
> + * @x3: X coordinate of point 3, u9.0, default 0.
> + * @x4: X coordinate of point 4, u9.0, default 0.
> + * @x5: X coordinate of point 5, u9.0, default 0.
> + * @__reserved1: reserved
> + * @a01: slope A points 01, s4.4, default 0.
> + * @a12: slope A points 12, s4.4, default 0.
> + * @a23: slope A points 23, s4.4, default 0.
> + * @__reserved2: reserved
> + * @a34: slope A points 34, s4.4, default 0.
> + * @a45: slope A points 45, s4.4, default 0.
> + * @__reserved3: reserved
> + * @b01: slope B points 01, s4.4, default 0.
> + * @b12: slope B points 12, s4.4, default 0.
> + * @b23: slope B points 23, s4.4, default 0.
> + * @__reserved4: reserved
> + * @b34: slope B points 34, s4.4, default 0.
> + * @b45: slope B points 45, s4.4, default 0.
> + * @__reserved5: reserved
> + */
> +struct ipu3_uapi_iefd_cux6_ed {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 x2:9;
> + __u32 __reserved0:5;
> +
> + __u32 x3:9;
> + __u32 x4:9;
> + __u32 x5:9;
> + __u32 __reserved1:5;
> +
> + __u32 a01:9;
> + __u32 a12:9;
> + __u32 a23:9;
> + __u32 __reserved2:5;
> +
> + __u32 a34:9;
> + __u32 a45:9;
> + __u32 __reserved3:14;
> +
> + __u32 b01:9;
> + __u32 b12:9;
> + __u32 b23:9;
> + __u32 __reserved4:5;
> +
> + __u32 b34:9;
> + __u32 b45:9;
> + __u32 __reserved5:14;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
> + * element apply.
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A of Config Unit, s4.4, default 0.
> + * @__reserved1: reserved
> + * @b01: offset B0 of Config Unit, u7.0, default 0.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_iefd_cux2_1 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 a01:9;
> + __u32 __reserved1:5;
> +
> + __u32 b01:8;
> + __u32 __reserved2:24;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed sharpening
> + * element.
> + *
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @x2: X2 point of Config Unit, u9.0, default 0.
> + * @__reserved0: reserved
> + * @x3: X3 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A0 of Config Unit, s4.4, default 0.
> + * @a12: Slope A1 of Config Unit, s4.4, default 0.
> + * @__reserved1: reserved
> + * @a23: Slope A2 of Config Unit, s4.4, default 0.
> + * @b01: Offset B0 of Config Unit, s7.0, default 0.
> + * @b12: Offset B1 of Config Unit, s7.0, default 0.
> + * @__reserved2: reserved
> + * @b23: Offset B2 of Config Unit, s7.0, default 0.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_iefd_cux4 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 x2:9;
> + __u32 __reserved0:5;
> +
> + __u32 x3:9;
> + __u32 a01:9;
> + __u32 a12:9;
> + __u32 __reserved1:5;
> +
> + __u32 a23:9;
> + __u32 b01:8;
> + __u32 b12:8;
> + __u32 __reserved2:7;
> +
> + __u32 b23:8;
> + __u32 __reserved3:24;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
> + *
> + * @x0: x0 points of Config Unit radial, u8.0
> + * @x1: x1 points of Config Unit radial, u8.0
> + * @x2: x2 points of Config Unit radial, u8.0
> + * @x3: x3 points of Config Unit radial, u8.0
> + * @x4: x4 points of Config Unit radial, u8.0
> + * @x5: x5 points of Config Unit radial, u8.0
> + * @__reserved1: reserved
> + * @a01: Slope A of Config Unit radial, s7.8
> + * @a12: Slope A of Config Unit radial, s7.8
> + * @a23: Slope A of Config Unit radial, s7.8
> + * @a34: Slope A of Config Unit radial, s7.8
> + * @a45: Slope A of Config Unit radial, s7.8
> + * @__reserved2: reserved
> + * @b01: Slope B of Config Unit radial, s9.0
> + * @b12: Slope B of Config Unit radial, s9.0
> + * @b23: Slope B of Config Unit radial, s9.0
> + * @__reserved4: reserved
> + * @b34: Slope B of Config Unit radial, s9.0
> + * @b45: Slope B of Config Unit radial, s9.0
> + * @__reserved5: reserved
> + */
> +struct ipu3_uapi_iefd_cux6_rad {
> + __u32 x0:8;
> + __u32 x1:8;
> + __u32 x2:8;
> + __u32 x3:8;
> +
> + __u32 x4:8;
> + __u32 x5:8;
> + __u32 __reserved1:16;
> +
> + __u32 a01:16;
> + __u32 a12:16;
> +
> + __u32 a23:16;
> + __u32 a34:16;
> +
> + __u32 a45:16;
> + __u32 __reserved2:16;
> +
> + __u32 b01:10;
> + __u32 b12:10;
> + __u32 b23:10;
> + __u32 __reserved4:2;
> +
> + __u32 b34:10;
> + __u32 b45:10;
> + __u32 __reserved5:12;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
> + *
> + * @cu_1: calculate weight for blending directed and
> + * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
> + * @cu_ed: calculate power of non-directed sharpening element, see
> + * &ipu3_uapi_iefd_cux6_ed
> + * @cu_3: calculate weight for blending directed and
> + * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
> + * @cu_5: calculate power of non-directed denoise element apply, use
> + * &ipu3_uapi_iefd_cux2_1
> + * @cu_6: calculate power of non-directed sharpening element. See
> + * &ipu3_uapi_iefd_cux4
> + * @cu_7: calculate weight for blending directed and
> + * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
> + * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
> + * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
> + * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
> + */
> +struct ipu3_uapi_yuvp1_iefd_cfg_units {
> + struct ipu3_uapi_iefd_cux2 cu_1;
> + struct ipu3_uapi_iefd_cux6_ed cu_ed;
> + struct ipu3_uapi_iefd_cux2 cu_3;
> + struct ipu3_uapi_iefd_cux2_1 cu_5;
> + struct ipu3_uapi_iefd_cux4 cu_6;
> + struct ipu3_uapi_iefd_cux2 cu_7;
> + struct ipu3_uapi_iefd_cux4 cu_unsharp;
> + struct ipu3_uapi_iefd_cux6_rad cu_radial;
> + struct ipu3_uapi_iefd_cux2 cu_vssnlm;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
> + *
> + * @horver_diag_coeff: Gradiant compensation, coefficient that compensates for
> + * different distance for vertical / horizontal and diagonal
> + * * gradient calculation (~1/sqrt(2)).
> + * @__reserved0: reserved
> + * @clamp_stitch: Slope to stitch between clamped and unclamped edge values
> + * @__reserved1: reserved
> + * @direct_metric_update: Update coeff for direction metric
> + * @__reserved2: reserved
> + * @ed_horver_diag_coeff: Radial Coefficient that compensates for
> + * different distance for vertical/horizontal and
> + * diagonal gradient calculation (~1/sqrt(2))
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_iefd_config_s {
> + __u32 horver_diag_coeff:7;
> + __u32 __reserved0:1;
> + __u32 clamp_stitch:6;
> + __u32 __reserved1:2;
> + __u32 direct_metric_update:5;
> + __u32 __reserved2:3;
> + __u32 ed_horver_diag_coeff:7;
> + __u32 __reserved3:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
> + *
> + * @iefd_en: Enable IEFd
> + * @denoise_en: Enable denoise
> + * @direct_smooth_en: Enable directional smooth
> + * @rad_en: Enable radial update
> + * @vssnlm_en: Enable VSSNLM output filter
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_yuvp1_iefd_control {
> + __u32 iefd_en:1;
> + __u32 denoise_en:1;
> + __u32 direct_smooth_en:1;
> + __u32 rad_en:1;
> + __u32 vssnlm_en:1;
> + __u32 __reserved:27;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_sharp_cfg - Sharpening config
> + *
> + * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
> + * @__reserved0: reserved
> + * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
> + * @__reserved1: reserved
> + * @nega_lmt_dir: Sharpening limit for negative overshoots for direction (edge).
> + * @__reserved2: reserved
> + * @posi_lmt_dir: Sharpening limit for positive overshoots for direction (edge).
> + * @__reserved3: reserved
> + *
> + * Fixed point type u13.0, range [0, 8191].
> + */
> +struct ipu3_uapi_sharp_cfg {
> + __u32 nega_lmt_txt:13;
> + __u32 __reserved0:19;
> + __u32 posi_lmt_txt:13;
> + __u32 __reserved1:19;
> + __u32 nega_lmt_dir:13;
> + __u32 __reserved2:19;
> + __u32 posi_lmt_dir:13;
> + __u32 __reserved3:19;
> +} __packed;
> +
> +/**
> + * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
> + *
> + * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64], default 64.
> + * @__reserved0: reserved
> + * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64], default 0.
> + * @__reserved1: reserved
> + * @ndir_dns_powr: Power of non-direct denoising,
> + * Precision u1.6, range [0, 64], default 64.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_far_w {
> + __u32 dir_shrp:7;
> + __u32 __reserved0:1;
> + __u32 dir_dns:7;
> + __u32 __reserved1:1;
> + __u32 ndir_dns_powr:7;
> + __u32 __reserved2:9;
> +} __packed;
> +
> +/**
> + * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
> + *
> + * @unsharp_weight: Unsharp mask blending weight.
> + * u1.6, range [0, 64], default 16.
> + * 0 - disabled, 64 - use only unsharp.
> + * @__reserved0: reserved
> + * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511], default 0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_unsharp_cfg {
> + __u32 unsharp_weight:7;
> + __u32 __reserved0:1;
> + __u32 unsharp_amount:9;
> + __u32 __reserved1:15;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
> + *
> + * @cfg: sharpness config &ipu3_uapi_sharp_cfg
> + * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
> + * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
> + * neighbors (8 pixels called Near), and the second order neighborhood
> + * around them (16 pixels called Far).
> + * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
> + */
> +struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
> + struct ipu3_uapi_sharp_cfg cfg;
> + struct ipu3_uapi_far_w far_w;
> + struct ipu3_uapi_unsharp_cfg unshrp_cfg;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
> + *
> + * @c00: Coeff11, s0.8, range [-255, 255], default 1.
> + * @c01: Coeff12, s0.8, range [-255, 255], default 5.
> + * @c02: Coeff13, s0.8, range [-255, 255], default 9.
> + * @__reserved: reserved
> + *
> + * Configurable registers for common sharpening support.
> + */
> +struct ipu3_uapi_unsharp_coef0 {
> + __u32 c00:9;
> + __u32 c01:9;
> + __u32 c02:9;
> + __u32 __reserved:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
> + *
> + * @c11: Coeff22, s0.8, range [-255, 255], default 29.
> + * @c12: Coeff23, s0.8, range [-255, 255], default 55.
> + * @c22: Coeff33, s0.8, range [-255, 255], default 96.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_unsharp_coef1 {
> + __u32 c11:9;
> + __u32 c12:9;
> + __u32 c22:9;
> + __u32 __reserved:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
> + *
> + * @unsharp_coef0: unsharp coefficient 0 config. See &ipu3_uapi_unsharp_coef0
> + * @unsharp_coef1: unsharp coefficient 1 config. See &ipu3_uapi_unsharp_coef1
> + */
> +struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
> + struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
> + struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_reset_xy - Radial coordinate reset
> + *
> + * @x: Radial reset of x coordinate. Precision s12, [-4095, 4095], default 0.
> + * @__reserved0: reserved
> + * @y: Radial center y coordinate. Precision s12, [-4095, 4095], default 0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_radial_reset_xy {
> + __s32 x:13;
> + __u32 __reserved0:3;
> + __s32 y:13;
> + __u32 __reserved1:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_reset_x2 - Radial X^2 reset
> + *
> + * @x2: Radial reset of x^2 coordinate. Precision u24, default 0.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_radial_reset_x2 {
> + __u32 x2:24;
> + __u32 __reserved:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_reset_y2 - Radial Y^2 reset
> + *
> + * @y2: Radial reset of y^2 coordinate. Precision u24, default 0.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_radial_reset_y2 {
> + __u32 y2:24;
> + __u32 __reserved:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_cfg - Radial config
> + *
> + * @rad_nf: Radial. R^2 normalization factor is scale down by 2^ - (15 + scale)
> + * @__reserved0: reserved
> + * @rad_inv_r2: Radial R^-2 normelized to (0.5..1), Prec' u7, range [0, 127].
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_radial_cfg {
> + __u32 rad_nf:4;
> + __u32 __reserved0:4;
> + __u32 rad_inv_r2:7;
> + __u32 __reserved1:17;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_rad_far_w - Radial FAR sub-group
> + *
> + * @rad_dir_far_sharp_w: Weight of wide direct sharpening, u1.6, range [0, 64],
> + * default 64.
> + * @rad_dir_far_dns_w: Weight of wide direct denoising, u1.6, range [0, 64],
> + * default 0.
> + * @rad_ndir_far_dns_power: power of non-direct sharpening, u1.6, range [0, 64],
> + * default 0.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_rad_far_w {
> + __u32 rad_dir_far_sharp_w:8;
> + __u32 rad_dir_far_dns_w:8;
> + __u32 rad_ndir_far_dns_power:8;
> + __u32 __reserved:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cu_cfg0 - Radius Config Unit cfg0 register
> + *
> + * @cu6_pow: Power of CU6. Power of non-direct sharpening, u3.4.
> + * @__reserved0: reserved
> + * @cu_unsharp_pow: Power of unsharp mask, u2.4.
> + * @__reserved1: reserved
> + * @rad_cu6_pow: Radial/corner CU6. Directed sharpening power, u3.4.
> + * @__reserved2: reserved
> + * @rad_cu_unsharp_pow: Radial power of unsharp mask, u2.4.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_cu_cfg0 {
> + __u32 cu6_pow:7;
> + __u32 __reserved0:1;
> + __u32 cu_unsharp_pow:7;
> + __u32 __reserved1:1;
> + __u32 rad_cu6_pow:7;
> + __u32 __reserved2:1;
> + __u32 rad_cu_unsharp_pow:6;
> + __u32 __reserved3:2;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cu_cfg1 - Radius Config Unit cfg1 register
> + *
> + * @rad_cu6_x1: X1 point of Config Unit 6, precision u9.0.
> + * @__reserved0: reserved
> + * @rad_cu_unsharp_x1: X1 point for Config Unit unsharp for radial/corner point
> + * precision u9.0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_cu_cfg1 {
> + __u32 rad_cu6_x1:9;
> + __u32 __reserved0:1;
> + __u32 rad_cu_unsharp_x1:9;
> + __u32 __reserved1:13;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_rad_cfg - IEFd parameters changed radially over
> + * the picture plain.
> + *
> + * @reset_xy: reset xy value in radial calculation. &ipu3_uapi_radial_reset_xy
> + * @reset_x2: reset x square value in radial calculation. See struct
> + * &ipu3_uapi_radial_reset_x2
> + * @reset_y2: reset y square value in radial calculation. See struct
> + * &ipu3_uapi_radial_reset_y2
> + * @cfg: radial config defined in &ipu3_uapi_radial_cfg
> + * @rad_far_w: weight for wide range radial. &ipu3_uapi_rad_far_w
> + * @cu_cfg0: configuration unit 0. See &ipu3_uapi_cu_cfg0
> + * @cu_cfg1: configuration unit 1. See &ipu3_uapi_cu_cfg1
> + */
> +struct ipu3_uapi_yuvp1_iefd_rad_cfg {
> + struct ipu3_uapi_radial_reset_xy reset_xy;
> + struct ipu3_uapi_radial_reset_x2 reset_x2;
> + struct ipu3_uapi_radial_reset_y2 reset_y2;
> + struct ipu3_uapi_radial_cfg cfg;
> + struct ipu3_uapi_rad_far_w rad_far_w;
> + struct ipu3_uapi_cu_cfg0 cu_cfg0;
> + struct ipu3_uapi_cu_cfg1 cu_cfg1;
> +} __packed;
> +
> +/* Vssnlm - Very small scale non-local mean algorithm */
> +
> +/**
> + * struct ipu3_uapi_vss_lut_x - Vssnlm LUT x0/x1/x2
> + *
> + * @vs_x0: Vssnlm LUT x0, precision u8, range [0, 255], default 16.
> + * @vs_x1: Vssnlm LUT x1, precision u8, range [0, 255], default 32.
> + * @vs_x2: Vssnlm LUT x2, precision u8, range [0, 255], default 64.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_vss_lut_x {
> + __u32 vs_x0:8;
> + __u32 vs_x1:8;
> + __u32 vs_x2:8;
> + __u32 __reserved2:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_vss_lut_y - Vssnlm LUT y0/y1/y2
> + *
> + * @vs_y1: Vssnlm LUT y1, precision u4, range [0, 8], default 1.
> + * @__reserved0: reserved
> + * @vs_y2: Vssnlm LUT y2, precision u4, range [0, 8], default 3.
> + * @__reserved1: reserved
> + * @vs_y3: Vssnlm LUT y3, precision u4, range [0, 8], default 8.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_vss_lut_y {
> + __u32 vs_y1:4;
> + __u32 __reserved0:4;
> + __u32 vs_y2:4;
> + __u32 __reserved1:4;
> + __u32 vs_y3:4;
> + __u32 __reserved2:12;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_vssnlm_cf - IEFd Vssnlm Lookup table
> + *
> + * @vss_lut_x: vss lookup table. See &ipu3_uapi_vss_lut_x description
> + * @vss_lut_y: vss lookup table. See &ipu3_uapi_vss_lut_y description
> + */
> +struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg {
> + struct ipu3_uapi_vss_lut_x vss_lut_x;
> + struct ipu3_uapi_vss_lut_y vss_lut_y;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_config - IEFd config
> + *
> + * @units: configuration unit setting, &ipu3_uapi_yuvp1_iefd_cfg_units
> + * @config: configuration, as defined by &ipu3_uapi_yuvp1_iefd_config_s
> + * @control: control setting, as defined by &ipu3_uapi_yuvp1_iefd_control
> + * @sharp: sharpness setting, as defined by &ipu3_uapi_yuvp1_iefd_shrp_cfg
> + * @unsharp: unsharpness setting, as defined by &ipu3_uapi_yuvp1_iefd_unshrp_cfg
> + * @rad: radial setting, as defined by &ipu3_uapi_yuvp1_iefd_rad_cfg
> + * @vsslnm: vsslnm setting, as defined by &ipu3_uapi_yuvp1_iefd_vssnlm_cfg
> + */
> +struct ipu3_uapi_yuvp1_iefd_config {
> + struct ipu3_uapi_yuvp1_iefd_cfg_units units;
> + struct ipu3_uapi_yuvp1_iefd_config_s config;
> + struct ipu3_uapi_yuvp1_iefd_control control;
> + struct ipu3_uapi_yuvp1_iefd_shrp_cfg sharp;
> + struct ipu3_uapi_yuvp1_iefd_unshrp_cfg unsharp;
> + struct ipu3_uapi_yuvp1_iefd_rad_cfg rad;
> + struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg vsslnm;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_yds_config - Y Down-Sampling config
> + *
> + * @c00: range [0, 3], default 0x0
> + * @c01: range [0, 3], default 0x1
> + * @c02: range [0, 3], default 0x1
> + * @c03: range [0, 3], default 0x0
> + * @c10: range [0, 3], default 0x0
> + * @c11: range [0, 3], default 0x1
> + * @c12: range [0, 3], default 0x1
> + * @c13: range [0, 3], default 0x0
> + *
> + * Above are 4x2 filter coefficients for chroma output downscaling.
> + *
> + * @norm_factor: Normalization factor, range [0, 4], default 2
> + * 0 - divide by 1
> + * 1 - divide by 2
> + * 2 - divide by 4
> + * 3 - divide by 8
> + * 4 - divide by 16
> + * @__reserved0: reserved
> + * @bin_output: Down sampling on Luma channel in two optional modes
> + * 0 - Bin output 4.2.0 (default), 1 output 4.2.2.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_yds_config {
> + __u32 c00:2;
> + __u32 c01:2;
> + __u32 c02:2;
> + __u32 c03:2;
> + __u32 c10:2;
> + __u32 c11:2;
> + __u32 c12:2;
> + __u32 c13:2;
> + __u32 norm_factor:5;
> + __u32 __reserved0:4;
> + __u32 bin_output:1;
> + __u32 __reserved1:6;
> +} __packed;
> +
> +/* Chroma Noise Reduction */
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_enable_config - Chroma noise reduction enable
> + *
> + * @enable: enable/disable chroma noise reduction
> + * @yuv_mode: 0 - YUV420, 1 - YUV422
> + * @__reserved0: reserved
> + * @col_size: number of columns in the frame, max width is 2560
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_enable_config {
> + __u32 enable:1;
> + __u32 yuv_mode:1;
> + __u32 __reserved0:14;
> + __u32 col_size:12;
> + __u32 __reserved1:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_coring_config - Coring thresholds for UV
> + *
> + * @u: U coring level, u0.13, range [0.0, 1.0], default 0.0
> + * @__reserved0: reserved
> + * @v: V coring level, u0.13, range [0.0, 1.0], default 0.0
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_coring_config {
> + __u32 u:13;
> + __u32 __reserved0:3;
> + __u32 v:13;
> + __u32 __reserved1:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_sense_gain_config - Chroma noise reduction gains
> + *
> + * All sensitivity gain parameters have precision u13.0, range [0, 8191].
> + *
> + * @vy: Sensitivity of horizontal edge of Y, default 100
> + * @vu: Sensitivity of horizontal edge of U, default 100
> + * @vv: Sensitivity of horizontal edge of V, default 100
> + * @__reserved0: reserved
> + * @hy: Sensitivity of vertical edge of Y, default 50
> + * @hu: Sensitivity of vertical edge of U, default 50
> + * @hv: Sensitivity of vertical edge of V, default 50
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_sense_gain_config {
> + __u32 vy:8;
> + __u32 vu:8;
> + __u32 vv:8;
> + __u32 __reserved0:8;
> +
> + __u32 hy:8;
> + __u32 hu:8;
> + __u32 hv:8;
> + __u32 __reserved1:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_iir_fir_config - Chroma IIR/FIR filter config
> + *
> + * @fir_0h: Value of center tap in horizontal FIR, range [0, 32], default 8.
> + * @__reserved0: reserved
> + * @fir_1h: Value of distance 1 in horizontal FIR, range [0, 32], default 12.
> + * @__reserved1: reserved
> + * @fir_2h: Value of distance 2 tap in horizontal FIR, range [0, 32], default 0.
> + * @dalpha_clip_val: weight for previous row in IIR, range [1, 256], default 0.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_iir_fir_config {
> + __u32 fir_0h:6;
> + __u32 __reserved0:2;
> + __u32 fir_1h:6;
> + __u32 __reserved1:2;
> + __u32 fir_2h:6;
> + __u32 dalpha_clip_val:9;
> + __u32 __reserved2:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_config - Chroma noise reduction config
> + *
> + * @enable: chroma noise reduction enable, see
> + * &ipu3_uapi_yuvp1_chnr_enable_config
> + * @coring: coring config for chroma noise reduction, see
> + * &ipu3_uapi_yuvp1_chnr_coring_config
> + * @sense_gain: sensitivity config for chroma noise reduction, see
> + * ipu3_uapi_yuvp1_chnr_sense_gain_config
> + * @iir_fir: iir and fir config for chroma noise reduction, see
> + * ipu3_uapi_yuvp1_chnr_iir_fir_config
> + */
> +struct ipu3_uapi_yuvp1_chnr_config {
> + struct ipu3_uapi_yuvp1_chnr_enable_config enable;
> + struct ipu3_uapi_yuvp1_chnr_coring_config coring;
> + struct ipu3_uapi_yuvp1_chnr_sense_gain_config sense_gain;
> + struct ipu3_uapi_yuvp1_chnr_iir_fir_config iir_fir;
> +} __packed;
> +
> +/* Edge Enhancement and Noise Reduction */
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config - Luma(Y) edge enhancement low-pass
> + * filter coefficients
> + *
> + * @a_diag: Smoothing diagonal coefficient, u5.0.
> + * @__reserved0: reserved
> + * @a_periph: Image smoothing perpherial, u5.0.
> + * @__reserved1: reserved
> + * @a_cent: Image Smoothing center coefficient, u5.0.
> + * @__reserved2: reserved
> + * @enable: 0: Y_EE_NR disabled, output = input; 1: Y_EE_NR enabled.
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config {
> + __u32 a_diag:5;
> + __u32 __reserved0:3;
> + __u32 a_periph:5;
> + __u32 __reserved1:3;
> + __u32 a_cent:5;
> + __u32 __reserved2:9;
> + __u32 enable:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_sense_config - Luma(Y) edge enhancement
> + * noise reduction sensitivity gains
> + *
> + * @edge_sense_0: Sensitivity of edge in dark area. u13.0, default 8191.
> + * @__reserved0: reserved
> + * @delta_edge_sense: Difference in the sensitivity of edges between
> + * the bright and dark areas. u13.0, default 0.
> + * @__reserved1: reserved
> + * @corner_sense_0: Sensitivity of corner in dark area. u13.0, default 0.
> + * @__reserved2: reserved
> + * @delta_corner_sense: Difference in the sensitivity of corners between
> + * the bright and dark areas. u13.0, default 8191.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_sense_config {
> + __u32 edge_sense_0:13;
> + __u32 __reserved0:3;
> + __u32 delta_edge_sense:13;
> + __u32 __reserved1:3;
> + __u32 corner_sense_0:13;
> + __u32 __reserved2:3;
> + __u32 delta_corner_sense:13;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_gain_config - Luma(Y) edge enhancement
> + * noise reduction gain config
> + *
> + * @gain_pos_0: Gain for positive edge in dark area. u5.0, [0, 16], default 2.
> + * @__reserved0: reserved
> + * @delta_gain_posi: Difference in the gain of edges between the bright and
> + * dark areas for positive edges. u5.0, [0, 16], default 0.
> + * @__reserved1: reserved
> + * @gain_neg_0: Gain for negative edge in dark area. u5.0, [0, 16], default 8.
> + * @__reserved2: reserved
> + * @delta_gain_neg: Difference in the gain of edges between the bright and
> + * dark areas for negative edges. u5.0, [0, 16], default 0.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_gain_config {
> + __u32 gain_pos_0:5;
> + __u32 __reserved0:3;
> + __u32 delta_gain_posi:5;
> + __u32 __reserved1:3;
> + __u32 gain_neg_0:5;
> + __u32 __reserved2:3;
> + __u32 delta_gain_neg:5;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_clip_config - Luma(Y) edge enhancement
> + * noise reduction clipping config
> + *
> + * @clip_pos_0: Limit of positive edge in dark area
> + * u5, value [0, 16], default 8.
> + * @__reserved0: reserved
> + * @delta_clip_posi: Difference in the limit of edges between the bright
> + * and dark areas for positive edges.
> + * u5, value [0, 16], default 8.
> + * @__reserved1: reserved
> + * @clip_neg_0: Limit of negative edge in dark area
> + * u5, value [0, 16], default 8.
> + * @__reserved2: reserved
> + * @delta_clip_neg: Difference in the limit of edges between the bright
> + * and dark areas for negative edges.
> + * u5, value [0, 16], default 8.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_clip_config {
> + __u32 clip_pos_0:5;
> + __u32 __reserved0:3;
> + __u32 delta_clip_posi:5;
> + __u32 __reserved1:3;
> + __u32 clip_neg_0:5;
> + __u32 __reserved2:3;
> + __u32 delta_clip_neg:5;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_frng_config - Luma(Y) edge enhancement
> + * noise reduction fringe config
> + *
> + * @gain_exp: Common exponent of gains, u4, [0, 8], default 2.
> + * @__reserved0: reserved
> + * @min_edge: Threshold for edge and smooth stitching, u13.
> + * @__reserved1: reserved
> + * @lin_seg_param: Power of LinSeg, u4.
> + * @__reserved2: reserved
> + * @t1: Parameter for enabling/disabling the edge enhancement, u1.0, [0, 1],
> + * default 1.
> + * @t2: Parameter for enabling/disabling the smoothing, u1.0, [0, 1],
> + * default 1.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_frng_config {
> + __u32 gain_exp:4;
> + __u32 __reserved0:28;
> + __u32 min_edge:13;
> + __u32 __reserved1:3;
> + __u32 lin_seg_param:4;
> + __u32 __reserved2:4;
> + __u32 t1:1;
> + __u32 t2:1;
> + __u32 __reserved3:6;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_diag_config - Luma(Y) edge enhancement
> + * noise reduction diagonal config
> + *
> + * @diag_disc_g: Coefficient that prioritize diagonal edge direction on
> + * horizontal or vertical for final enhancement.
> + * u4.0, [1, 15], default 1.
> + * @__reserved0: reserved
> + * @hvw_hor: Weight of horizontal/vertical edge enhancement for hv edge.
> + * u2.2, [1, 15], default 4.
> + * @dw_hor: Weight of diagonal edge enhancement for hv edge.
> + * u2.2, [1, 15], default 1.
> + * @hvw_diag: Weight of horizontal/vertical edge enhancement for diagonal edge.
> + * u2.2, [1, 15], default 1.
> + * @dw_diag: Weight of diagonal edge enhancement for diagonal edge.
> + * u2.2, [1, 15], default 4.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_diag_config {
> + __u32 diag_disc_g:4;
> + __u32 __reserved0:4;
> + __u32 hvw_hor:4;
> + __u32 dw_hor:4;
> + __u32 hvw_diag:4;
> + __u32 dw_diag:4;
> + __u32 __reserved1:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config - Luma(Y) edge enhancement
> + * noise reduction false color correction (FCC) coring config
> + *
> + * @pos_0: Gain for positive edge in dark, u13.0, [0, 16], default 0.
> + * @__reserved0: reserved
> + * @pos_delta: Gain for positive edge in bright, value: pos_0 + pos_delta <=16
> + * u13.0, default 0.
> + * @__reserved1: reserved
> + * @neg_0: Gain for negative edge in dark area, u13.0, range [0, 16], default 0.
> + * @__reserved2: reserved
> + * @neg_delta: Gain for negative edge in bright area. neg_0 + neg_delta <=16
> + * u13.0, default 0.
> + * @__reserved3: reserved
> + *
> + * Coring is a simple soft thresholding technique.
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config {
> + __u32 pos_0:13;
> + __u32 __reserved0:3;
> + __u32 pos_delta:13;
> + __u32 __reserved1:3;
> + __u32 neg_0:13;
> + __u32 __reserved2:3;
> + __u32 neg_delta:13;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_config - Edge enhancement and noise reduction
> + *
> + * @lpf: low-pass filter config. See &ipu3_uapi_yuvp1_y_ee_nr_lpf_config
> + * @sense: sensitivity config. See &ipu3_uapi_yuvp1_y_ee_nr_sense_config
> + * @gain: gain config as defined in &ipu3_uapi_yuvp1_y_ee_nr_gain_config
> + * @clip: clip config as defined in &ipu3_uapi_yuvp1_y_ee_nr_clip_config
> + * @frng: fringe config as defined in &ipu3_uapi_yuvp1_y_ee_nr_frng_config
> + * @diag: diagonal edge config. See &ipu3_uapi_yuvp1_y_ee_nr_diag_config
> + * @fc_coring: coring config for fringe control. See
> + * &ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_config {
> + struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config lpf;
> + struct ipu3_uapi_yuvp1_y_ee_nr_sense_config sense;
> + struct ipu3_uapi_yuvp1_y_ee_nr_gain_config gain;
> + struct ipu3_uapi_yuvp1_y_ee_nr_clip_config clip;
> + struct ipu3_uapi_yuvp1_y_ee_nr_frng_config frng;
> + struct ipu3_uapi_yuvp1_y_ee_nr_diag_config diag;
> + struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config fc_coring;
> +} __packed;
> +
> +/* Total Color Correction */
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_gen_control_static_config - Total color correction
> + * general control config
> + *
> + * @en: 0 - TCC disabled. Output = input 1 - TCC enabled.
> + * @blend_shift: blend shift, Range[3, 4], default NA.
> + * @gain_according_to_y_only: 0: Gain is calculated according to YUV,
> + * 1: Gain is calculated according to Y only
> + * @__reserved0: reserved
> + * @gamma: Final blending coefficients. Values[-16, 16], default NA.
> + * @__reserved1: reserved
> + * @delta: Final blending coefficients. Values[-16, 16], default NA.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_yuvp2_tcc_gen_control_static_config {
> + __u32 en:1;
> + __u32 blend_shift:3;
> + __u32 gain_according_to_y_only:1;
> + __u32 __reserved0:11;
> + __s32 gamma:5;
> + __u32 __reserved1:3;
> + __s32 delta:5;
> + __u32 __reserved2:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config - Total color correction
> + * multi-axis color control (MACC) config
> + *
> + * @a: a coefficient for 2x2 MACC conversion matrix.
> + * @__reserved0: reserved
> + * @b: b coefficient 2x2 MACC conversion matrix.
> + * @__reserved1: reserved
> + * @c: c coefficient for 2x2 MACC conversion matrix.
> + * @__reserved2: reserved
> + * @d: d coefficient for 2x2 MACC conversion matrix.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config {
> + __s32 a:12;
> + __u32 __reserved0:4;
> + __s32 b:12;
> + __u32 __reserved1:4;
> + __s32 c:12;
> + __u32 __reserved2:4;
> + __s32 d:12;
> + __u32 __reserved3:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_macc_table_static_config - Total color correction
> + * multi-axis color control (MACC) table array
> + *
> + * @entries: config for multi axis color correction, as specified by
> + * &ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> + */
> +struct ipu3_uapi_yuvp2_tcc_macc_table_static_config {
> + struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> + entries[IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config - Total color correction
> + * inverse y lookup table
> + *
> + * @entries: lookup table for inverse y estimation, and use it to estimate the
> + * ratio between luma and chroma. Chroma by approximate the absolute
> + * value of the radius on the chroma plane (R = sqrt(u^2+v^2) ) and
> + * luma by approximate by 1/Y.
> + */
> +struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config {
> + __u16 entries[IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config - Total color
> + * correction lookup table for PCWL
> + *
> + * @entries: lookup table for gain piece wise linear transformation (PCWL)
> + */
> +struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config {
> + __u16 entries[IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config - Total color correction
> + * lookup table for r square root
> + *
> + * @entries: lookup table for r square root estimation
> + */
> +struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config {
> + __s16 entries[IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_static_config- Total color correction static
> + *
> + * @gen_control: general config for Total Color Correction
> + * @macc_table: config for multi axis color correction
> + * @inv_y_lut: lookup table for inverse y estimation
> + * @gain_pcwl: lookup table for gain PCWL
> + * @r_sqr_lut: lookup table for r square root estimation.
> + */
> +struct ipu3_uapi_yuvp2_tcc_static_config {
> + struct ipu3_uapi_yuvp2_tcc_gen_control_static_config gen_control;
> + struct ipu3_uapi_yuvp2_tcc_macc_table_static_config macc_table;
> + struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config inv_y_lut;
> + struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config gain_pcwl;
> + struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config r_sqr_lut;
> +} __packed;
> +
> +/* Advanced Noise Reduction related structs */
> +
> +/*
> + * struct ipu3_uapi_anr_alpha - Advanced noise reduction alpha
> + *
> + * Tunable parameters that are subject to modification according to the
> + * total gain used.
> + */
> +struct ipu3_uapi_anr_alpha {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> + __u16 dc_gr;
> + __u16 dc_r;
> + __u16 dc_b;
> + __u16 dc_gb;
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_anr_beta - Advanced noise reduction beta
> + *
> + * Tunable parameters that are subject to modification according to the
> + * total gain used.
> + */
> +struct ipu3_uapi_anr_beta {
> + __u16 beta_gr;
> + __u16 beta_r;
> + __u16 beta_b;
> + __u16 beta_gb;
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_anr_plain_color - Advanced noise reduction plain color with
> + * 4x4 matrix
> + *
> + * Tunable parameters that are subject to modification according to the
> + * total gain used.
> + */
> +struct ipu3_uapi_anr_plain_color {
> + __u16 reg_w_gr[16];
> + __u16 reg_w_r[16];
> + __u16 reg_w_b[16];
> + __u16 reg_w_gb[16];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_transform_config - Advanced noise reduction transform
> + *
> + * @enable: advanced noise reduction enabled.
> + * @adaptive_treshhold_en: On IPU3, adaptive threshold is always enabled.
> + * @__reserved1: reserved
> + * @__reserved2: reserved
> + * @alpha: using following defaults:
> + * 13, 13, 13, 13, 0, 0, 0, 0
> + * 11, 11, 11, 11, 0, 0, 0, 0
> + * 14, 14, 14, 14, 0, 0, 0, 0
> + * @beta: use following defaults:
> + * 24, 24, 24, 24
> + * 21, 20, 20, 21
> + * 25, 25, 25, 25
> + * @color: use defaults defined in driver/media/pci/intel/ipu3-tables.c
> + * @sqrt_lut: 11 bits per element, values =
> + * [724 768 810 849 887
> + * 923 958 991 1024 1056
> + * 1116 1145 1173 1201 1086
> + * 1228 1254 1280 1305 1330
> + * 1355 1379 1402 1425 1448]
> + * @xreset: Reset value of X for r^2 calculation Value: col_start-X_center
> + * Constraint: Xreset + FrameWdith=4095 Xreset= -4095, default -1632.
> + * @__reserved3: reserved
> + * @yreset: Reset value of Y for r^2 calculation Value: row_start-Y_center
> + * Constraint: Yreset + FrameHeight=4095 Yreset= -4095, default -1224.
> + * @__reserved4: reserved
> + * @x_sqr_reset: Reset value of X^2 for r^2 calculation Value = (Xreset)^2
> + * @r_normfactor: Normalization factor for R. Default 14.
> + * @__reserved5: reserved
> + * @y_sqr_reset: Reset value of Y^2 for r^2 calculation Value = (Yreset)^2
> + * @gain_scale: Parameter describing shading gain as a function of distance
> + * from the image center.
> + * A single value per frame, loaded by the driver. Default 115.
> + */
> +struct ipu3_uapi_anr_transform_config {
> + __u32 enable:1; /* 0 or 1, disabled or enabled */
> + __u32 adaptive_treshhold_en:1; /* On IPU3, always enabled */
> +
> + __u32 __reserved1:30;
> + __u8 __reserved2[44];
> +
> + struct ipu3_uapi_anr_alpha alpha[3];
> + struct ipu3_uapi_anr_beta beta[3];
> + struct ipu3_uapi_anr_plain_color color[3];
> +
> + __u16 sqrt_lut[IPU3_UAPI_ANR_LUT_SIZE]; /* 11 bits per element */
> +
> + __s16 xreset:13;
> + __u16 __reserved3:3;
> + __s16 yreset:13;
> + __u16 __reserved4:3;
> +
> + __u32 x_sqr_reset:24;
> + __u32 r_normfactor:5;
> + __u32 __reserved5:3;
> +
> + __u32 y_sqr_reset:24;
> + __u32 gain_scale:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_stitch_pyramid - ANR stitch pyramid
> + *
> + * @entry0: pyramid LUT entry0, range [0x0, 0x3f]
> + * @entry1: pyramid LUT entry1, range [0x0, 0x3f]
> + * @entry2: pyramid LUT entry2, range [0x0, 0x3f]
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_anr_stitch_pyramid {
> + __u32 entry0:6;
> + __u32 entry1:6;
> + __u32 entry2:6;
> + __u32 __reserved:14;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_stitch_config - ANR stitch config
> + *
> + * @anr_stitch_en: enable stitch. Enabled with 1.
> + * @__reserved: reserved
> + * @pyramid: pyramid table as defined by &ipu3_uapi_anr_stitch_pyramid
> + * default values:
> + * { 1, 3, 5 }, { 7, 7, 5 }, { 3, 1, 3 },
> + * { 9, 15, 21 }, { 21, 15, 9 }, { 3, 5, 15 },
> + * { 25, 35, 35 }, { 25, 15, 5 }, { 7, 21, 35 },
> + * { 49, 49, 35 }, { 21, 7, 7 }, { 21, 35, 49 },
> + * { 49, 35, 21 }, { 7, 5, 15 }, { 25, 35, 35 },
> + * { 25, 15, 5 }, { 3, 9, 15 }, { 21, 21, 15 },
> + * { 9, 3, 1 }, { 3, 5, 7 }, { 7, 5, 3}, { 1 }
> + */
> +struct ipu3_uapi_anr_stitch_config {
> + __u32 anr_stitch_en;
> + __u8 __reserved[44];
> + struct ipu3_uapi_anr_stitch_pyramid pyramid[IPU3_UAPI_ANR_PYRAMID_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_config - ANR config
> + *
> + * @transform: advanced noise reduction transform config as specified by
> + * &ipu3_uapi_anr_transform_config
> + * @stitch: create 4x4 patch from 4 surrounding 8x8 patches.
> + */
> +struct ipu3_uapi_anr_config {
> + struct ipu3_uapi_anr_transform_config transform __attribute__((aligned(32)));
> + struct ipu3_uapi_anr_stitch_config stitch __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_acc_param - Accelerator cluster parameters
> + *
> + * ACC refers to the HW cluster containing all Fixed Functions(FFs). Each FF
> + * implements a specific algorithm.
> + *
> + * @bnr: parameters for bayer noise reduction static config. See
> + * &ipu3_uapi_bnr_static_config
> + * @green_disparity: disparity static config between gr and gb channel.
> + * See &ipu3_uapi_bnr_static_config_green_disparity
> + * @dm: de-mosaic config. See &ipu3_uapi_dm_config
> + * @ccm: color correction matrix. See &ipu3_uapi_ccm_mat_config
> + * @gamma: gamma correction config. See &ipu3_uapi_gamma_config
> + * @csc: color space conversion matrix. See &ipu3_uapi_csc_mat_config
> + * @cds: color down sample config. See &ipu3_uapi_cds_params
> + * @shd: lens shading correction config. See &ipu3_uapi_shd_config
> + * @iefd: Image enhancement filter and denoise config.
> + * &ipu3_uapi_yuvp1_iefd_config
> + * @yds_c0: y down scaler config. &ipu3_uapi_yuvp1_yds_config
> + * @chnr_c0: chroma noise reduction config. &ipu3_uapi_yuvp1_chnr_config
> + * @y_ee_nr: y edge enhancement and noise reduction config.
> + * &ipu3_uapi_yuvp1_y_ee_nr_config
> + * @yds: y down scaler config. See &ipu3_uapi_yuvp1_yds_config
> + * @chnr: chroma noise reduction config. See &ipu3_uapi_yuvp1_chnr_config
> + * @__reserved1: reserved
> + * @yds2: y channel down scaler config. See &ipu3_uapi_yuvp1_yds_config
> + * @tcc: total color correction config as defined in struct
> + * &ipu3_uapi_yuvp2_tcc_static_config
> + * @__reserved2: reserved
> + * @anr: advanced noise reduction config.See &ipu3_uapi_anr_config
> + * @awb_fr: AWB filter response config. See ipu3_uapi_awb_fr_config
> + * @ae: auto exposure config As specified by &ipu3_uapi_ae_config
> + * @af: auto focus config. As specified by &ipu3_uapi_af_config
> + * @awb: auto white balance config. As specified by &ipu3_uapi_awb_config
> + */
> +struct ipu3_uapi_acc_param {
> + struct ipu3_uapi_bnr_static_config bnr;
> + struct ipu3_uapi_bnr_static_config_green_disparity
> + green_disparity __attribute__((aligned(32)));
> + struct ipu3_uapi_dm_config dm __attribute__((aligned(32)));
> + struct ipu3_uapi_ccm_mat_config ccm __attribute__((aligned(32)));
> + struct ipu3_uapi_gamma_config gamma __attribute__((aligned(32)));
> + struct ipu3_uapi_csc_mat_config csc __attribute__((aligned(32)));
> + struct ipu3_uapi_cds_params cds __attribute__((aligned(32)));
> + struct ipu3_uapi_shd_config shd __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_iefd_config iefd __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_yds_config yds_c0 __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_chnr_config chnr_c0 __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_y_ee_nr_config y_ee_nr __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_yds_config yds __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_chnr_config chnr __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_yds_config yds2 __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp2_tcc_static_config tcc __attribute__((aligned(32)));
> + struct ipu3_uapi_anr_config anr;
> + struct ipu3_uapi_awb_fr_config awb_fr;
> + struct ipu3_uapi_ae_config ae;
> + struct ipu3_uapi_af_config af;
> + struct ipu3_uapi_awb_config awb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
> + *
> + * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
> + * @lin_lutlow_r: linearization look-up table for R channel interpolation.
> + * @lin_lutlow_b: linearization look-up table for B channel interpolation.
> + * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
> + * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
> + * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
> + * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
> + * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
> + * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
> + * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
> + */
> +struct ipu3_uapi_isp_lin_vmem_params {
> + __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
> +} __packed;
> +
> +/* Temporal Noise Reduction */
> +
> +/**
> + * struct ipu3_uapi_isp_tnr3_vmem_params - Temporal noise reduction vector
> + * memory parameters
> + *
> + * @slope: slope setting in interpolation curve for temporal noise reduction.
> + * @__reserved1: reserved
> + * @sigma: knee point setting in interpolation curve for temporal
> + * noise reduction.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_isp_tnr3_vmem_params {
> + __u16 slope[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> + __u16 __reserved1[IPU3_UAPI_ISP_VEC_ELEMS
> + - IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> + __u16 sigma[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> + __u16 __reserved2[IPU3_UAPI_ISP_VEC_ELEMS
> + - IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_isp_tnr3_params - Temporal noise reduction v3 parameters
> + *
> + * @knee_y1: Knee point TNR3 assumes standard deviation of Y,U and
> + * V at Y1 are TnrY1_Sigma_Y, U and V.
> + * @knee_y2: Knee point TNR3 assumes standard deviation of Y,U and
> + * V at Y2 are TnrY2_Sigma_Y, U and V.
> + * @maxfb_y: Max feedback gain for Y
> + * @maxfb_u: Max feedback gain for U
> + * @maxfb_v: Max feedback gain for V
> + * @round_adj_y: rounding Adjust for Y
> + * @round_adj_u: rounding Adjust for U
> + * @round_adj_v: rounding Adjust for V
> + * @ref_buf_select: selection of the reference frame buffer to be used.
> + */
> +struct ipu3_uapi_isp_tnr3_params {
> + __u32 knee_y1;
> + __u32 knee_y2;
> + __u32 maxfb_y;
> + __u32 maxfb_u;
> + __u32 maxfb_v;
> + __u32 round_adj_y;
> + __u32 round_adj_u;
> + __u32 round_adj_v;
> + __u32 ref_buf_select;
> +} __packed;
> +
> +/* Extreme Noise Reduction version 3 */
> +
> +/**
> + * struct ipu3_uapi_isp_xnr3_vmem_params - Extreme noise reduction v3
> + * vector memory parameters
> + *
> + * @x: xnr3 parameters.
> + * @a: xnr3 parameters.
> + * @b: xnr3 parameters.
> + * @c: xnr3 parameters.
> + */
> +struct ipu3_uapi_isp_xnr3_vmem_params {
> + __u16 x[IPU3_UAPI_ISP_VEC_ELEMS];
> + __u16 a[IPU3_UAPI_ISP_VEC_ELEMS];
> + __u16 b[IPU3_UAPI_ISP_VEC_ELEMS];
> + __u16 c[IPU3_UAPI_ISP_VEC_ELEMS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_xnr3_alpha_params - Extreme noise reduction v3
> + * alpha tuning parameters
> + *
> + * @y0: Sigma for Y range similarity in dark area.
> + * @u0: Sigma for U range similarity in dark area.
> + * @v0: Sigma for V range similarity in dark area.
> + * @ydiff: Sigma difference for Y between bright area and dark area.
> + * @udiff: Sigma difference for U between bright area and dark area.
> + * @vdiff: Sigma difference for V between bright area and dark area.
> + */
> +struct ipu3_uapi_xnr3_alpha_params {
> + __u32 y0;
> + __u32 u0;
> + __u32 v0;
> + __u32 ydiff;
> + __u32 udiff;
> + __u32 vdiff;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_xnr3_coring_params - Extreme noise reduction v3
> + * coring parameters
> + *
> + * @u0: Coring Threshold of U channel in dark area.
> + * @v0: Coring Threshold of V channel in dark area.
> + * @udiff: Threshold difference of U channel between bright and dark area.
> + * @vdiff: Threshold difference of V channel between bright and dark area.
> + */
> +struct ipu3_uapi_xnr3_coring_params {
> + __u32 u0;
> + __u32 v0;
> + __u32 udiff;
> + __u32 vdiff;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_xnr3_blending_params - Blending factor
> + *
> + * @strength: The factor for blending output with input. This is tuning
> + * parameterHigher values lead to more aggressive XNR operation.
> + */
> +struct ipu3_uapi_xnr3_blending_params {
> + __u32 strength;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_isp_xnr3_params - Extreme noise reduction v3 parameters
> + *
> + * @alpha: parameters for xnr3 alpha. See &ipu3_uapi_xnr3_alpha_params
> + * @coring: parameters for xnr3 coring. See &ipu3_uapi_xnr3_coring_params
> + * @blending: parameters for xnr3 blending. See &ipu3_uapi_xnr3_blending_params
> + */
> +struct ipu3_uapi_isp_xnr3_params {
> + struct ipu3_uapi_xnr3_alpha_params alpha;
> + struct ipu3_uapi_xnr3_coring_params coring;
> + struct ipu3_uapi_xnr3_blending_params blending;
> +} __packed;
> +
> +/***** Obgrid (optical black level compensation) table entry *****/
> +
> +/**
> + * struct ipu3_uapi_obgrid_param - Optical black level compensation parameters
> + *
> + * @gr: Grid table values for color GR
> + * @r: Grid table values for color R
> + * @b: Grid table values for color B
> + * @gb: Grid table values for color GB
> + *
> + * Black level is different for red, green, and blue channels. So black level
> + * compensation is different per channel.
> + */
> +struct ipu3_uapi_obgrid_param {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> +} __packed;
> +
> +/******************* V4L2_META_FMT_IPU3_PARAMS *******************/
> +
> +/**
> + * struct ipu3_uapi_flags - bits to indicate which pipeline needs update
> + *
> + * @gdc: 0 = no update, 1 = update.
> + * @obgrid: 0 = no update, 1 = update.
> + * @__reserved1: Not used.
> + * @acc_bnr: 0 = no update, 1 = update.
> + * @acc_green_disparity: 0 = no update, 1 = update.
> + * @acc_dm: 0 = no update, 1 = update.
> + * @acc_ccm: 0 = no update, 1 = update.
> + * @acc_gamma: 0 = no update, 1 = update.
> + * @acc_csc: 0 = no update, 1 = update.
> + * @acc_cds: 0 = no update, 1 = update.
> + * @acc_shd: 0 = no update, 1 = update.
> + * @__reserved2: Not used.
> + * @acc_iefd: 0 = no update, 1 = update.
> + * @acc_yds_c0: 0 = no update, 1 = update.
> + * @acc_chnr_c0: 0 = no update, 1 = update.
> + * @acc_y_ee_nr: 0 = no update, 1 = update.
> + * @acc_yds: 0 = no update, 1 = update.
> + * @acc_chnr: 0 = no update, 1 = update.
> + * @acc_ytm: 0 = no update, 1 = update.
> + * @acc_yds2: 0 = no update, 1 = update.
> + * @acc_tcc: 0 = no update, 1 = update.
> + * @acc_dpc: 0 = no update, 1 = update.
> + * @acc_bds: 0 = no update, 1 = update.
> + * @acc_anr: 0 = no update, 1 = update.
> + * @acc_awb_fr: 0 = no update, 1 = update.
> + * @acc_ae: 0 = no update, 1 = update.
> + * @acc_af: 0 = no update, 1 = update.
> + * @acc_awb: 0 = no update, 1 = update.
> + * @__acc_osys: 0 = no update, 1 = update.
> + * @__reserved3: Not used.
> + * @lin_vmem_params: 0 = no update, 1 = update.
> + * @tnr3_vmem_params: 0 = no update, 1 = update.
> + * @xnr3_vmem_params: 0 = no update, 1 = update.
> + * @tnr3_dmem_params: 0 = no update, 1 = update.
> + * @xnr3_dmem_params: 0 = no update, 1 = update.
> + * @__reserved4: Not used.
> + * @obgrid_param: 0 = no update, 1 = update.
> + * @__reserved5: Not used.
> + */
> +struct ipu3_uapi_flags {
> + __u32 gdc:1;
> + __u32 obgrid:1;
> + __u32 __reserved1:30;
> +
> + __u32 acc_bnr:1;
> + __u32 acc_green_disparity:1;
> + __u32 acc_dm:1;
> + __u32 acc_ccm:1;
> + __u32 acc_gamma:1;
> + __u32 acc_csc:1;
> + __u32 acc_cds:1;
> + __u32 acc_shd:1;
> + __u32 __reserved2:2;
> + __u32 acc_iefd:1;
> + __u32 acc_yds_c0:1;
> + __u32 acc_chnr_c0:1;
> + __u32 acc_y_ee_nr:1;
> + __u32 acc_yds:1;
> + __u32 acc_chnr:1;
> + __u32 acc_ytm:1;
> + __u32 acc_yds2:1;
> + __u32 acc_tcc:1;
> + __u32 acc_dpc:1;
> + __u32 acc_bds:1;
> + __u32 acc_anr:1;
> + __u32 acc_awb_fr:1;
> + __u32 acc_ae:1;
> + __u32 acc_af:1;
> + __u32 acc_awb:1;
> + __u32 __reserved3:4;
> +
> + __u32 lin_vmem_params:1;
> + __u32 tnr3_vmem_params:1;
> + __u32 xnr3_vmem_params:1;
> + __u32 tnr3_dmem_params:1;
> + __u32 xnr3_dmem_params:1;
> + __u32 __reserved4:1;
> + __u32 obgrid_param:1;
> + __u32 __reserved5:25;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_params - V4L2_META_FMT_IPU3_PARAMS
> + *
> + * @use: select which parameters to apply, see &ipu3_uapi_flags
> + * @acc_param: ACC parameters, as specified by &ipu3_uapi_acc_param
> + * @lin_vmem_params: linearization VMEM, as specified by
> + * &ipu3_uapi_isp_lin_vmem_params
> + * @tnr3_vmem_params: tnr3 VMEM as specified by
> + * &ipu3_uapi_isp_tnr3_vmem_params
> + * @xnr3_vmem_params: xnr3 VMEM as specified by
> + * &ipu3_uapi_isp_xnr3_vmem_params
> + * @tnr3_dmem_params: tnr3 DMEM as specified by &ipu3_uapi_isp_tnr3_params
> + * @xnr3_dmem_params: xnr3 DMEM as specified by &ipu3_uapi_isp_xnr3_params
> + * @obgrid_param: obgrid parameters as specified by
> + * &ipu3_uapi_obgrid_param
> + *
> + * The video queue "parameters" is of format V4L2_META_FMT_IPU3_PARAMS.
> + * This is a "single plane" v4l2_meta_format using V4L2_BUF_TYPE_META_OUTPUT.
> + *
> + * struct ipu3_uapi_params as defined below contains a lot of parameters and
> + * ipu3_uapi_flags selects which parameters to apply.
> + */
> +struct ipu3_uapi_params {
> + /* Flags which of the settings below are to be applied */
> + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> +
> + /* Accelerator cluster parameters */
> + struct ipu3_uapi_acc_param acc_param;
> +
> + /* ISP vector address space parameters */
> + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> +
> + /* ISP data memory (DMEM) parameters */
> + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> +
> + /* Optical black level compensation */
> + struct ipu3_uapi_obgrid_param obgrid_param;
> +} __packed;
> +#endif
Thanks,
Mauro
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-02 13:49 ` Mauro Carvalho Chehab
@ 2018-11-02 14:04 ` Tomasz Figa
2018-11-06 23:27 ` Mani, Rajmohan
2018-11-06 18:25 ` Zhi, Yong
1 sibling, 1 reply; 123+ messages in thread
From: Tomasz Figa @ 2018-11-02 14:04 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Yong Zhi, Linux Media Mailing List, Sakari Ailus, Hans Verkuil,
Laurent Pinchart, Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao Bing Bu, chao.c.li
Hi Mauro,
On Fri, Nov 2, 2018 at 10:49 PM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> Em Mon, 29 Oct 2018 15:22:57 -0700
> Yong Zhi <yong.zhi@intel.com> escreveu:
[snip]
> > +struct ipu3_uapi_awb_config_s {
> > + __u16 rgbs_thr_gr;
> > + __u16 rgbs_thr_r;
> > + __u16 rgbs_thr_gb;
> > + __u16 rgbs_thr_b;
> > + struct ipu3_uapi_grid_config grid;
> > +} __attribute__((aligned(32))) __packed;
>
> Hmm... Kernel defines a macro for aligned attribute:
>
> include/linux/compiler_types.h:#define __aligned(x) __attribute__((aligned(x)))
>
First, thanks for review!
Maybe I missed something, but last time I checked, it wasn't
accessible from UAPI headers in userspace.
> I'm not a gcc expert, but it sounds weird to first ask it to align
> with 32 bits and then have __packed (with means that pads should be
> removed).
>
> In other words, I *guess* is it should either be __packed
> or __aligned(32).
>
> Not that it would do any difference, in practice, as this
> specific struct has a size with is multiple of 32 bits, but
> let's do the right annotation here, not mixing two incompatible
> alignment requirements.
>
My understanding was that __packed makes the compiler not insert any
alignment between particular fields of the struct, while __aligned
makes the whole struct be aligned at given boundary, if placed in
another struct. If I didn't miss anything, having both should make
perfect sense here.
Best regards,
Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-02 14:04 ` Tomasz Figa
@ 2018-11-06 23:27 ` Mani, Rajmohan
2018-11-15 10:52 ` Hans Verkuil
0 siblings, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2018-11-06 23:27 UTC (permalink / raw)
To: Tomasz Figa, Mauro Carvalho Chehab
Cc: Zhi, Yong, Linux Media Mailing List, Sakari Ailus, Hans Verkuil,
Laurent Pinchart, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu, Li, Chao C
Hi Mauro,
Thanks for the reviews.
> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
>
> Hi Mauro,
>
> On Fri, Nov 2, 2018 at 10:49 PM Mauro Carvalho Chehab
> <mchehab+samsung@kernel.org> wrote:
> >
> > Em Mon, 29 Oct 2018 15:22:57 -0700
> > Yong Zhi <yong.zhi@intel.com> escreveu:
> [snip]
> > > +struct ipu3_uapi_awb_config_s {
> > > + __u16 rgbs_thr_gr;
> > > + __u16 rgbs_thr_r;
> > > + __u16 rgbs_thr_gb;
> > > + __u16 rgbs_thr_b;
> > > + struct ipu3_uapi_grid_config grid; }
> > > +__attribute__((aligned(32))) __packed;
> >
> > Hmm... Kernel defines a macro for aligned attribute:
> >
> > include/linux/compiler_types.h:#define __aligned(x)
> __attribute__((aligned(x)))
> >
>
> First, thanks for review!
>
> Maybe I missed something, but last time I checked, it wasn't accessible from
> UAPI headers in userspace.
Ack. We see that's still the case.
>
> > I'm not a gcc expert, but it sounds weird to first ask it to align
> > with 32 bits and then have __packed (with means that pads should be
> > removed).
> >
> > In other words, I *guess* is it should either be __packed or
> > __aligned(32).
> >
> > Not that it would do any difference, in practice, as this specific
> > struct has a size with is multiple of 32 bits, but let's do the right
> > annotation here, not mixing two incompatible alignment requirements.
> >
>
> My understanding was that __packed makes the compiler not insert any
> alignment between particular fields of the struct, while __aligned makes the
> whole struct be aligned at given boundary, if placed in another struct. If I
> didn't miss anything, having both should make perfect sense here.
Ack
I also recall that as part of addressing review comments (from Hans and Sakari),
on earlier versions of this patch series, we added __packed attribute to all structs
to ensure the size of the structs remains the same between 32 and 64 bit builds.
The addition of structure members of the name padding[x] in some of the structs
ensures that respective members are aligned at 32 byte boundaries, while the
overall size of the structs remain the same between 32 and 64 bit builds.
Thanks
Raj
>
> Best regards,
> Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-06 23:27 ` Mani, Rajmohan
@ 2018-11-15 10:52 ` Hans Verkuil
2018-11-29 0:41 ` Mani, Rajmohan
0 siblings, 1 reply; 123+ messages in thread
From: Hans Verkuil @ 2018-11-15 10:52 UTC (permalink / raw)
To: Mani, Rajmohan, Tomasz Figa, Mauro Carvalho Chehab
Cc: Zhi, Yong, Linux Media Mailing List, Sakari Ailus, Hans Verkuil,
Laurent Pinchart, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu, Li, Chao C
On 11/07/18 00:27, Mani, Rajmohan wrote:
> Hi Mauro,
>
> Thanks for the reviews.
>
>> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
>>
>> Hi Mauro,
>>
>> On Fri, Nov 2, 2018 at 10:49 PM Mauro Carvalho Chehab
>> <mchehab+samsung@kernel.org> wrote:
>>>
>>> Em Mon, 29 Oct 2018 15:22:57 -0700
>>> Yong Zhi <yong.zhi@intel.com> escreveu:
>> [snip]
>>>> +struct ipu3_uapi_awb_config_s {
>>>> + __u16 rgbs_thr_gr;
>>>> + __u16 rgbs_thr_r;
>>>> + __u16 rgbs_thr_gb;
>>>> + __u16 rgbs_thr_b;
>>>> + struct ipu3_uapi_grid_config grid; }
>>>> +__attribute__((aligned(32))) __packed;
>>>
>>> Hmm... Kernel defines a macro for aligned attribute:
>>>
>>> include/linux/compiler_types.h:#define __aligned(x)
>> __attribute__((aligned(x)))
>>>
>>
>> First, thanks for review!
>>
>> Maybe I missed something, but last time I checked, it wasn't accessible from
>> UAPI headers in userspace.
>
> Ack. We see that's still the case.
>
>>
>>> I'm not a gcc expert, but it sounds weird to first ask it to align
>>> with 32 bits and then have __packed (with means that pads should be
>>> removed).
>>>
>>> In other words, I *guess* is it should either be __packed or
>>> __aligned(32).
>>>
>>> Not that it would do any difference, in practice, as this specific
>>> struct has a size with is multiple of 32 bits, but let's do the right
>>> annotation here, not mixing two incompatible alignment requirements.
>>>
>>
>> My understanding was that __packed makes the compiler not insert any
>> alignment between particular fields of the struct, while __aligned makes the
>> whole struct be aligned at given boundary, if placed in another struct. If I
>> didn't miss anything, having both should make perfect sense here.
>
> Ack
>
> I also recall that as part of addressing review comments (from Hans and Sakari),
> on earlier versions of this patch series, we added __packed attribute to all structs
> to ensure the size of the structs remains the same between 32 and 64 bit builds.
>
> The addition of structure members of the name padding[x] in some of the structs
> ensures that respective members are aligned at 32 byte boundaries, while the
> overall size of the structs remain the same between 32 and 64 bit builds.
I recommend that this is documented in the header. It's not a common construction
so an explanation will help.
Regards,
Hans
>
> Thanks
> Raj
>
>>
>> Best regards,
>> Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-15 10:52 ` Hans Verkuil
@ 2018-11-29 0:41 ` Mani, Rajmohan
0 siblings, 0 replies; 123+ messages in thread
From: Mani, Rajmohan @ 2018-11-29 0:41 UTC (permalink / raw)
To: Hans Verkuil, Tomasz Figa, Mauro Carvalho Chehab
Cc: Zhi, Yong, Linux Media Mailing List, Sakari Ailus, Hans Verkuil,
Laurent Pinchart, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu, Li, Chao C
Hi Hans,
> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
>
> On 11/07/18 00:27, Mani, Rajmohan wrote:
> > Hi Mauro,
> >
> > Thanks for the reviews.
> >
> >> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
> >>
> >> Hi Mauro,
> >>
> >> On Fri, Nov 2, 2018 at 10:49 PM Mauro Carvalho Chehab
> >> <mchehab+samsung@kernel.org> wrote:
> >>>
> >>> Em Mon, 29 Oct 2018 15:22:57 -0700
> >>> Yong Zhi <yong.zhi@intel.com> escreveu:
> >> [snip]
> >>>> +struct ipu3_uapi_awb_config_s {
> >>>> + __u16 rgbs_thr_gr;
> >>>> + __u16 rgbs_thr_r;
> >>>> + __u16 rgbs_thr_gb;
> >>>> + __u16 rgbs_thr_b;
> >>>> + struct ipu3_uapi_grid_config grid; }
> >>>> +__attribute__((aligned(32))) __packed;
> >>>
> >>> Hmm... Kernel defines a macro for aligned attribute:
> >>>
> >>> include/linux/compiler_types.h:#define __aligned(x)
> >> __attribute__((aligned(x)))
> >>>
> >>
> >> First, thanks for review!
> >>
> >> Maybe I missed something, but last time I checked, it wasn't
> >> accessible from UAPI headers in userspace.
> >
> > Ack. We see that's still the case.
> >
> >>
> >>> I'm not a gcc expert, but it sounds weird to first ask it to align
> >>> with 32 bits and then have __packed (with means that pads should be
> >>> removed).
> >>>
> >>> In other words, I *guess* is it should either be __packed or
> >>> __aligned(32).
> >>>
> >>> Not that it would do any difference, in practice, as this specific
> >>> struct has a size with is multiple of 32 bits, but let's do the
> >>> right annotation here, not mixing two incompatible alignment
> requirements.
> >>>
> >>
> >> My understanding was that __packed makes the compiler not insert any
> >> alignment between particular fields of the struct, while __aligned
> >> makes the whole struct be aligned at given boundary, if placed in
> >> another struct. If I didn't miss anything, having both should make perfect
> sense here.
> >
> > Ack
> >
> > I also recall that as part of addressing review comments (from Hans
> > and Sakari), on earlier versions of this patch series, we added
> > __packed attribute to all structs to ensure the size of the structs remains the
> same between 32 and 64 bit builds.
> >
> > The addition of structure members of the name padding[x] in some of
> > the structs ensures that respective members are aligned at 32 byte
> > boundaries, while the overall size of the structs remain the same between 32
> and 64 bit builds.
>
> I recommend that this is documented in the header. It's not a common
> construction so an explanation will help.
Ack.
>
> Regards,
>
> Hans
>
> >
> > Thanks
> > Raj
> >
> >>
> >> Best regards,
> >> Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-02 13:49 ` Mauro Carvalho Chehab
2018-11-02 14:04 ` Tomasz Figa
@ 2018-11-06 18:25 ` Zhi, Yong
1 sibling, 0 replies; 123+ messages in thread
From: Zhi, Yong @ 2018-11-06 18:25 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: linux-media, sakari.ailus, tfiga, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu, Li, Chao C
Hi, Mauro,
Thanks for your review.
> -----Original Message-----
> From: Mauro Carvalho Chehab [mailto:mchehab+samsung@kernel.org]
> Sent: Friday, November 2, 2018 6:49 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> tfiga@chromium.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>; Li, Chao C <chao.c.li@intel.com>
> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
>
> Em Mon, 29 Oct 2018 15:22:57 -0700
> Yong Zhi <yong.zhi@intel.com> escreveu:
>
> > These meta formats are used on Intel IPU3 ImgU video queues
> > to carry 3A statistics and ISP pipeline parameters.
>
> Just minor things. See below.
>
> >
> > V4L2_META_FMT_IPU3_3A
> > V4L2_META_FMT_IPU3_PARAMS
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > Signed-off-by: Chao C Li <chao.c.li@intel.com>
> > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> > ---
> > Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> > .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
>
> I would actually prefer to have those two changes merged together with
> patch 1, as it makes easier for review.
>
> > include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
>
> This one makes sense to have a separate patch.
>
Ack, will re-group the three files as suggested.
> > 3 files changed, 3001 insertions(+)
> > create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-
> ipu3.rst
> > create mode 100644 include/uapi/linux/intel-ipu3.h
> >
> > diff --git a/Documentation/media/uapi/v4l/meta-formats.rst
> b/Documentation/media/uapi/v4l/meta-formats.rst
> > index cf971d5..eafc534 100644
> > --- a/Documentation/media/uapi/v4l/meta-formats.rst
> > +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> > @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata`
> interface only.
> > .. toctree::
> > :maxdepth: 1
> >
> > + pixfmt-meta-intel-ipu3
> > pixfmt-meta-d4xx
> > pixfmt-meta-uvc
> > pixfmt-meta-vsp1-hgo
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > new file mode 100644
> > index 0000000..23b945b
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > @@ -0,0 +1,181 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _intel-ipu3:
> > +
> >
> +***************************************************************
> ***
> > +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A
> ('ip3s')
> >
> +***************************************************************
> ***
> > +
> > +.. c:type:: ipu3_uapi_stats_3a
> > +
> > +3A statistics
> > +=============
> > +
> > +For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
> > +an input bayer frame. Those statistics, defined in data struct
> > +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu
> 3a stat"
> > +video node, which are then passed to user space for statistics analysis
> > +using :c:type:`v4l2_meta_format` interface.
> > +
> > +The statistics collected are AWB (Auto-white balance) RGBS (Red, Green,
> Blue and
> > +Saturation measure) cells, AWB filter response, AF (Auto-focus) filter
> response,
> > +and AE (Auto-exposure) histogram.
> > +
> > +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all
> above.
> > +
> > +
> > +.. code-block:: c
> > +
> > +
> > + struct ipu3_uapi_stats_3a {
> > + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> > + __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_raw_buffer_aligned
> > + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> > + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> > + struct ipu3_uapi_4a_config stats_4a_config;
> > + __u32 ae_join_buffers;
> > + __u8 padding[28];
> > + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + stats_3a_bubble_per_stripe;
> > + struct ipu3_uapi_ff_status stats_3a_status;
> > + } __packed;
> > +
> > +
> > +.. c:type:: ipu3_uapi_params
> > +
> > +Pipeline parameters
> > +===================
> > +
> > +IPU3 pipeline has a number of image processing stages, each of which
> takes a
> > +set of parameters as input. The major stages of pipelines are shown here:
> > +
> > +Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
> > +
> > +Linearization -> Lens Shading Correction -> White Balance / Exposure /
> > +
> > +Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
> > +
> > +Correction Matrix -> Gamma correction -> Color Space Conversion ->
> > +
> > +Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
> > +
> > +Correction -> XNR3 -> TNR -> DDR
> > +
> > +The table below presents a description of the above algorithms.
> > +
> > +========================
> =======================================================
> > +Name Description
> > +========================
> =======================================================
> > +Optical Black Correction Optical Black Correction block subtracts a pre-
> defined
> > + value from the respective pixel values to obtain
> better
> > + image quality.
> > + Defined in :c:type:`ipu3_uapi_obgrid_param`.
> > +Linearization This algo block uses linearization parameters
> to
> > + address non-linearity sensor effects. The Lookup
> table
> > + table is defined in
> > + :c:type:`ipu3_uapi_isp_lin_vmem_params`.
> > +SHD Lens shading correction is used to correct spatial
> > + non-uniformity of the pixel response due to optical
> > + lens shading. This is done by applying a different
> gain
> > + for each pixel. The gain, black level etc are
> > + configured in :c:type:`ipu3_uapi_shd_config_static`.
> > +BNR Bayer noise reduction block removes image noise by
> > + applying a bilateral filter.
> > + See :c:type:`ipu3_uapi_bnr_static_config` for details.
> > +ANR Advanced Noise Reduction is a block based algorithm
> > + that performs noise reduction in the Bayer domain.
> The
> > + convolution matrix etc can be found in
> > + :c:type:`ipu3_uapi_anr_config`.
> > +Demosaicing Demosaicing converts raw sensor data in
> Bayer format
> > + into RGB (Red, Green, Blue) presentation. Then add
> > + outputs of estimation of Y channel for following
> stream
> > + processing by Firmware. The struct is defined as
> > + :c:type:`ipu3_uapi_dm_config`.
> > +Color Correction Color Correction algo transforms sensor specific
> color
> > + space to the standard "sRGB" color space. This is
> done
> > + by applying 3x3 matrix defined in
> > + :c:type:`ipu3_uapi_ccm_mat_config`.
> > +Gamma correction Gamma
> correction :c:type:`ipu3_uapi_gamma_config` is a
> > + basic non-linear tone mapping correction that is
> > + applied per pixel for each pixel component.
> > +CSC Color space conversion transforms each pixel from
> the
> > + RGB primary presentation to YUV (Y - brightness,
> > + UV - Luminance) presentation. This is done by
> applying
> > + a 3x3 matrix defined in
> > + :c:type:`ipu3_uapi_csc_mat_config`
> > +CDS Chroma down sampling
> > + After the CSC is performed, the Chroma Down
> Sampling
> > + is applied for a UV plane down sampling by a factor
> > + of 2 in each direction for YUV 4:2:0 using a 4x2
> > + configurable filter :c:type:`ipu3_uapi_cds_params`.
> > +CHNR Chroma noise reduction
> > + This block processes only the chrominance pixels
> and
> > + performs noise reduction by cleaning the high
> > + frequency noise.
> > + See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
> > +TCC Total color correction as defined in struct
> > + :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
> > +XNR3 eXtreme Noise Reduction V3 is the third
> revision of
> > + noise reduction algorithm used to improve image
> > + quality. This removes the low frequency noise in the
> > + captured image. Two related structs are being
> defined,
> > + :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data
> memory
> > + and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for
> vector
> > + memory.
> > +TNR Temporal Noise Reduction block compares
> successive
> > + frames in time to remove anomalies / noise in pixel
> > + values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params`
> and
> > + :c:type:`ipu3_uapi_isp_tnr3_params` are defined for
> ISP
> > + vector and data memory respectively.
> > +========================
> =======================================================
> > +
> > +A few stages of the pipeline will be executed by firmware running on the
> ISP
> > +processor, while many others will use a set of fixed hardware blocks also
> > +called accelerator cluster (ACC) to crunch pixel data and produce statistics.
> > +
> > +ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be
> selectively
> > +enabled / disabled by the user space through
> struct :c:type:`ipu3_uapi_flags`
> > +embedded in :c:type:`ipu3_uapi_params` structure. For parameters that
> are not
> > +enabled by the user space, corresponding structs are ignored by the ISP.
> > +
> > +Both 3A statistics and pipeline parameters described here are closely tied
> to
> > +the underlying camera sub-system (CSS) APIs. They are usually consumed
> and
> > +produced by dedicated user space libraries that comprise the important
> tuning
> > +tools, thus freeing the developers from being bothered with the low level
> > +hardware and algorithm details.
> > +
> > +It should be noted that IPU3 DMA operations require the addresses of all
> data
> > +structures (that includes both input and output) to be aligned on 32 byte
> > +boundaries.
> > +
> > +The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu
> parameters"
> > +video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
> > +
> > +.. code-block:: c
> > +
> > + struct ipu3_uapi_params {
> > + /* Flags which of the settings below are to be applied */
> > + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> > +
> > + /* Accelerator cluster parameters */
> > + struct ipu3_uapi_acc_param acc_param;
> > +
> > + /* ISP vector address space parameters */
> > + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> > + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> > + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> > +
> > + /* ISP data memory (DMEM) parameters */
> > + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> > + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> > +
> > + /* Optical black level compensation */
> > + struct ipu3_uapi_obgrid_param obgrid_param;
> > + } __packed;
> > +
> > +Intel IPU3 ImgU uAPI data types
> > +===============================
> > +
> > +.. kernel-doc:: include/uapi/linux/intel-ipu3.h
> > diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
> > new file mode 100644
> > index 0000000..c2608b6
> > --- /dev/null
> > +++ b/include/uapi/linux/intel-ipu3.h
> > @@ -0,0 +1,2819 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2017 - 2018 Intel Corporation */
> > +
> > +#ifndef __IPU3_UAPI_H
> > +#define __IPU3_UAPI_H
> > +
> > +#include <linux/types.h>
> > +
> > +/********************* Key Acronyms *************************/
> > +/*
> > + * ACC - Accelerator cluster
> > + * ANR - Adaptive noise reduction
> > + * AWB_FR- Auto white balance filter response statistics
> > + * BNR - Bayer noise reduction parameters
> > + * BDS - Bayer downscaler parameters
> > + * CCM - Color correction matrix coefficients
> > + * CDS - Chroma down sample
> > + * CHNR - Chroma noise reduction
> > + * CSC - Color space conversion
> > + * DM - De-mosaic
> > + * IEFd - Image enhancement filter directed
> > + * Obgrid - Optical black level compensation
> > + * OSYS - Output system configuration
> > + * ROI - Region of interest
> > + * SHD - Lens shading correction table
> > + * TCC - Total color correction
> > + * YDS - Y down sampling
> > + * YTM - Y-tone mapping
> > + */
>
> Hmm... It probably makes sense to convert this into a documentation
> block, e. g.:
>
> /**
> * DOC: Key Acronyms used by IPU3 ImgU driver
> *
> ...
> */
>
> And then include this header inside Documentation/media/v4l-
> drivers/ipu3.rst.
>
Will discuss with Raj on how to include above acronyms into ipu3.rst.
>
> > +
> > +/*
> > + * IPU3 DMA operations require buffers to be aligned at
> > + * 32 byte boundaries
> > + */
> > +
> > +/******************* ipu3_uapi_stats_3a *******************/
> > +
> > +#define IPU3_UAPI_MAX_STRIPES 2
> > +#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
> > +
> > +#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
> > +#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
> > +
> > +/* controls generation of meta_data (like FF enable/disable) */
> > +#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
> > +#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
> > +
> > +/**
> > + * struct ipu3_uapi_grid_config - Grid plane config
> > + *
> > + * @width: Grid horizontal dimensions, in number of grid blocks(cells).
> > + * @height: Grid vertical dimensions, in number of grid cells.
> > + * @block_width_log2: Log2 of the width of each cell in pixels.
> > + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> > + * @block_height_log2: Log2 of the height of each cell in pixels.
> > + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> > + * @height_per_slice: The number of blocks in vertical axis per slice.
> > + * Default 2.
> > + * @x_start: X value of top left corner of Region of Interest(ROI).
> > + * @y_start: Y value of top left corner of ROI
> > + * @x_end: X value of bottom right corner of ROI
> > + * @y_end: Y value of bottom right corner of ROI
> > + *
> > + * Due to the size of total amount of collected data, most statistics
> > + * create a grid-based output, and the data is then divided into "slices".
> > + */
> > +struct ipu3_uapi_grid_config {
> > + __u8 width;
> > + __u8 height;
> > + __u16 block_width_log2:3;
> > + __u16 block_height_log2:3;
> > + __u16 height_per_slice:8;
> > + __u16 x_start;
> > + __u16 y_start;
> > + __u16 x_end;
> > + __u16 y_end;
> > +} __packed;
> > +
> > +/*
> > + * The grid based data is divided into "slices" called set, each slice of setX
> > + * refers to ipu3_uapi_grid_config width * height_per_slice.
> > + */
> > +#define IPU3_UAPI_AWB_MAX_SETS 60
> > +/* Based on grid size 80 * 60 and cell size 16 x 16 */
> > +#define IPU3_UAPI_AWB_SET_SIZE 1280
> > +#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
> > +#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AWB_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
> > + (IPU3_UAPI_AWB_MAX_SETS * \
> > + (IPU3_UAPI_AWB_SET_SIZE +
> IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
> > +/**
> > + * struct ipu3_uapi_awb_meta_data - AWB meta data
> > + *
> > + * @meta_data_buffer: Average values for each color channel
> > + */
> > +struct ipu3_uapi_awb_meta_data {
> > + __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
> > + *
> > + * @meta_data: buffer to hold auto white balance meta data.
> > + */
> > +struct ipu3_uapi_awb_raw_buffer {
> > + struct ipu3_uapi_awb_meta_data meta_data;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_config_s - AWB config
> > + *
> > + * @rgbs_thr_gr: gr threshold value.
> > + * @rgbs_thr_r: Red threshold value.
> > + * @rgbs_thr_gb: gb threshold value.
> > + * @rgbs_thr_b: Blue threshold value.
> > + * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
> > + *
> > + * The threshold is a saturation measure range [0, 8191], 8191 is default.
> > + * Values over threshold may be optionally rejected for averaging.
> > + */
> > +struct ipu3_uapi_awb_config_s {
> > + __u16 rgbs_thr_gr;
> > + __u16 rgbs_thr_r;
> > + __u16 rgbs_thr_gb;
> > + __u16 rgbs_thr_b;
> > + struct ipu3_uapi_grid_config grid;
> > +} __attribute__((aligned(32))) __packed;
>
> Hmm... Kernel defines a macro for aligned attribute:
>
> include/linux/compiler_types.h:#define __aligned(x)
> __attribute__((aligned(x)))
>
> I'm not a gcc expert, but it sounds weird to first ask it to align
> with 32 bits and then have __packed (with means that pads should be
> removed).
>
> In other words, I *guess* is it should either be __packed
> or __aligned(32).
>
> Not that it would do any difference, in practice, as this
> specific struct has a size with is multiple of 32 bits, but
> let's do the right annotation here, not mixing two incompatible
> alignment requirements.
>
I saw Tomasz has replied to above. Will follow the reached consensus if update is needed, for this and other places in this file.
Thanks!!
Yong
> > +
> > +/**
> > + * struct ipu3_uapi_awb_config - AWB config wrapper
> > + *
> > + * @config: config for auto white balance as defined by
> &ipu3_uapi_awb_config_s
> > + */
> > +struct ipu3_uapi_awb_config {
> > + struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AE_COLORS 4 /* R,
> G, B, Y */
> > +#define IPU3_UAPI_AE_BINS 256
> > +#define IPU3_UAPI_AE_WEIGHTS 96
> > +
> > +/**
> > + * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
> > + *
> > + * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
> > + *
> > + * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit
> unsigned
> > + * for counting the number of the pixel.
> > + */
> > +struct ipu3_uapi_ae_raw_buffer {
> > + __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
> > + *
> > + * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
> > + */
> > +struct ipu3_uapi_ae_raw_buffer_aligned {
> > + struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
>
> Please use __aligned(32).
>
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_grid_config - AE weight grid
> > + *
> > + * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
> > + * @height: Grid vertical dimensions. Value: [16, 24], default 16.
> > + * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
> > + * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
> > + * @__reserved0: reserved
> > + * @ae_en: 0: does not write to meta-data array, 1: write normally.
> > + * @rst_hist_array: write 1 to trigger histogram array reset.
> > + * @done_rst_hist_array: flag for histogram array reset done.
> > + * @x_start: X value of top left corner of ROI, default 0.
> > + * @y_start: Y value of top left corner of ROI, default 0.
> > + * @x_end: X value of bottom right corner of ROI
> > + * @y_end: Y value of bottom right corner of ROI
> > + *
> > + * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
> > + * a defined ROI within the frame. The contribution of each pixel into the
> > + * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a
> grid.
> > + */
> > +struct ipu3_uapi_ae_grid_config {
> > + __u8 width;
> > + __u8 height;
> > + __u8 block_width_log2:4;
> > + __u8 block_height_log2:4;
> > + __u8 __reserved0:5;
> > + __u8 ae_en:1;
> > + __u8 rst_hist_array:1;
> > + __u8 done_rst_hist_array:1;
> > + __u16 x_start;
> > + __u16 y_start;
> > + __u16 x_end;
> > + __u16 y_end;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_weight_elem - AE weights LUT
> > + *
> > + * @cell0: weighted histogram grid value.
> > + * @cell1: weighted histogram grid value.
> > + * @cell2: weighted histogram grid value.
> > + * @cell3: weighted histogram grid value.
> > + * @cell4: weighted histogram grid value.
> > + * @cell5: weighted histogram grid value.
> > + * @cell6: weighted histogram grid value.
> > + * @cell7: weighted histogram grid value.
> > + *
> > + * Use weighted grid value to give a different contribution factor to each
> cell.
> > + * Precision u4, range [0, 15].
> > + */
> > +struct ipu3_uapi_ae_weight_elem {
> > + __u32 cell0:4;
> > + __u32 cell1:4;
> > + __u32 cell2:4;
> > + __u32 cell3:4;
> > + __u32 cell4:4;
> > + __u32 cell5:4;
> > + __u32 cell6:4;
> > + __u32 cell7:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
> > + *
> > + * @gain_gr: WB gain factor for the gr channels. Default 256.
> > + * @gain_r: WB gain factor for the r channel. Default 256.
> > + * @gain_b: WB gain factor for the b channel. Default 256.
> > + * @gain_gb: WB gain factor for the gb channels. Default 256.
> > + * @mat: 4x4 matrix that transforms Bayer quad output from WB to
> RGB+Y.
> > + *
> > + * Default:
> > + * 128, 0, 0, 0,
> > + * 0, 128, 0, 0,
> > + * 0, 0, 128, 0,
> > + * 0, 0, 0, 128,
> > + *
> > + * As part of the raw frame pre-process stage, the WB and color
> conversion need
> > + * to be applied to expose the impact of these gain operations.
> > + */
> > +struct ipu3_uapi_ae_ccm {
> > + __u16 gain_gr;
> > + __u16 gain_r;
> > + __u16 gain_b;
> > + __u16 gain_gb;
> > + __s16 mat[16];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_config - AE config
> > + *
> > + * @grid_cfg: config for auto exposure statistics grid. See struct
> > + * &ipu3_uapi_ae_grid_config
> > + * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks
> in the grid.
> > + * Each grid cell has a corresponding value in weights LUT called
> > + * grid value, global histogram is updated based on grid value
> and
> > + * pixel value.
> > + * @ae_ccm: Color convert matrix pre-processing block.
> > + *
> > + * Calculate AE grid from image resolution, resample ae weights.
> > + */
> > +struct ipu3_uapi_ae_config {
> > + struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_weight_elem weights[
> > + IPU3_UAPI_AE_WEIGHTS]
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
>
> Same above: __aligned(32). Please review the remaining of this header,
> as there are other occurrences of this pattern.
>
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast
> measurements
> > + *
> > + * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
> > + * anti-symmetry type. A12 is center, A1-A11 are neighbours.
> > + * for analyzing low frequency content, used to calculate sum
> > + * of gradients in x direction.
> > + * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
> > + * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
> > + * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
> > + * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
> > + * @y1_coeff_1: Struct
> > + * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
> > + * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
> > + * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
> > + * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
> > + * @y1_coeff_2: Struct
> > + * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> > + * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> > + * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> > + * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> > + * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
> > + * 0: positive, 1: negative, default 0.
> > + * @y2_coeff_0: Y2, same structure as Y1. For analyzing high
> frequency content.
> > + * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
> > + * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
> > + * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
> > + * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
> > + * @y2_coeff_1: Struct
> > + * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
> > + * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
> > + * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
> > + * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
> > + * @y2_coeff_2: Struct
> > + * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> > + * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> > + * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> > + * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> > + * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
> > + * 0: positive, 1: negative, default 0.
> > + * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to
> be
> > + * used for building histogram. Range [0, 32], default 8.
> > + * Rule:
> > + * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b +
> y_gen_rate_gb = 32
> > + * A single Y is calculated based on sum of Gr/R/B/Gb based on
> > + * their contribution ratio.
> > + * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
> > + * @y_calc.y_gen_rate_r: Contribution ratio R for Y
> > + * @y_calc.y_gen_rate_b: Contribution ratio B for Y
> > + * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
> > + * @nf: The shift right value that should be applied during the Y1/Y2
> filter to
> > + * make sure the total memory needed is 2 bytes per grid cell.
> > + * @nf.__reserved0: reserved
> > + * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
> > + * should be log2 of the sum of the abs values of the filter
> > + * coeffs, default 7 (2^7 = 128).
> > + * @nf.__reserved1: reserved
> > + * @nf.y2_nf: Normalization factor for y2, should be log2 of the
> sum of the
> > + * abs values of the filter coeffs.
> > + * @nf.__reserved2: reserved
> > + */
> > +struct ipu3_uapi_af_filter_config {
> > + struct {
> > + __u8 a1;
> > + __u8 a2;
> > + __u8 a3;
> > + __u8 a4;
> > + } y1_coeff_0;
> > + struct {
> > + __u8 a5;
> > + __u8 a6;
> > + __u8 a7;
> > + __u8 a8;
> > + } y1_coeff_1;
> > + struct {
> > + __u8 a9;
> > + __u8 a10;
> > + __u8 a11;
> > + __u8 a12;
> > + } y1_coeff_2;
> > +
> > + __u32 y1_sign_vec;
> > +
> > + struct {
> > + __u8 a1;
> > + __u8 a2;
> > + __u8 a3;
> > + __u8 a4;
> > + } y2_coeff_0;
> > + struct {
> > + __u8 a5;
> > + __u8 a6;
> > + __u8 a7;
> > + __u8 a8;
> > + } y2_coeff_1;
> > + struct {
> > + __u8 a9;
> > + __u8 a10;
> > + __u8 a11;
> > + __u8 a12;
> > + } y2_coeff_2;
> > +
> > + __u32 y2_sign_vec;
> > +
> > + struct {
> > + __u8 y_gen_rate_gr;
> > + __u8 y_gen_rate_r;
> > + __u8 y_gen_rate_b;
> > + __u8 y_gen_rate_gb;
> > + } y_calc;
> > +
> > + struct {
> > + __u32 __reserved0:8;
> > + __u32 y1_nf:4;
> > + __u32 __reserved1:4;
> > + __u32 y2_nf:4;
> > + __u32 __reserved2:12;
> > + } nf;
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AF_MAX_SETS 24
> > +#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
> > +#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AF_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
> > +#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
> > + (IPU3_UAPI_AF_MAX_SETS * \
> > + (IPU3_UAPI_AF_Y_TABLE_SET_SIZE +
> IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
> > + IPU3_UAPI_MAX_STRIPES)
> > +
> > +/**
> > + * struct ipu3_uapi_af_meta_data - AF meta data
> > + *
> > + * @y_table: Each color component will be convolved separately with
> filter1
> > + * and filter2 and the result will be summed out and averaged
> for
> > + * each cell.
> > + */
> > +struct ipu3_uapi_af_meta_data {
> > + __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_raw_buffer - AF raw buffer
> > + *
> > + * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus
> meta data.
> > + */
> > +struct ipu3_uapi_af_raw_buffer {
> > + struct ipu3_uapi_af_meta_data meta_data
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_config_s - AF config
> > + *
> > + * @filter_config: AF uses Y1 and Y2 filters as configured in
> > + * &ipu3_uapi_af_filter_config
> > + * @padding: paddings
> > + * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use
> large
> > + * grid size for large image and vice versa.
> > + */
> > +struct ipu3_uapi_af_config_s {
> > + struct ipu3_uapi_af_filter_config filter_config
> __attribute__((aligned(32)));
> > + __u8 padding[4];
> > + struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_config - AF config wrapper
> > + *
> > + * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
> > + */
> > +struct ipu3_uapi_af_config {
> > + struct ipu3_uapi_af_config_s config;
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AWB_FR_MAX_SETS 24
> > +#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
> > +#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
> > +#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
> > + (IPU3_UAPI_AWB_FR_MAX_SETS * \
> > + (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
> > + IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) *
> IPU3_UAPI_MAX_STRIPES)
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
> > + *
> > + * @bayer_table: Statistics output on the grid after convolving with 1D
> filter.
> > + */
> > +struct ipu3_uapi_awb_fr_meta_data {
> > + __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
> > + *
> > + * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
> > + */
> > +struct ipu3_uapi_awb_fr_raw_buffer {
> > + struct ipu3_uapi_awb_fr_meta_data meta_data;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
> > + *
> > + * @grid_cfg: grid config, default 16x16.
> > + * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
> > + * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
> > + * Applied on whole image for each Bayer channel
> separately
> > + * by a weighted sum of its 11x1 neighbors.
> > + * @__reserved1: reserved
> > + * @bayer_sign: sign of filter coeffcients, default 0.
> > + * @bayer_nf: normalization factor for the convolution coeffs, to
> make sure
> > + * total memory needed is within pre-determined range.
> > + * NF should be the log2 of the sum of the abs values of the
> > + * filter coeffs, range [7, 14], default 7.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_awb_fr_config_s {
> > + struct ipu3_uapi_grid_config grid_cfg;
> > + __u8 bayer_coeff[6];
> > + __u16 __reserved1;
> > + __u32 bayer_sign;
> > + __u8 bayer_nf;
> > + __u8 __reserved2[3];
> > +} __attribute__((aligned(32))) __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
> > + *
> > + * @config: See &ipu3_uapi_awb_fr_config_s.
> > + */
> > +struct ipu3_uapi_awb_fr_config {
> > + struct ipu3_uapi_awb_fr_config_s config;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_4a_config - 4A config
> > + *
> > + * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
> > + * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
> > + * @padding: paddings
> > + * @af_config: auto focus config &ipu3_uapi_af_config_s
> > + * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution
> 16x16
> > + */
> > +struct ipu3_uapi_4a_config {
> > + struct ipu3_uapi_awb_config_s awb_config
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_grid_config ae_grd_config;
> > + __u8 padding[20];
> > + struct ipu3_uapi_af_config_s af_config;
> > + struct ipu3_uapi_awb_fr_config_s awb_fr_config;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
> > + *
> > + * @num_of_stripes: A single frame is divided into several parts called
> stripes
> > + * due to limitation on line buffer memory.
> > + * The separation between the stripes is vertical. Each such
> > + * stripe is processed as a single frame by the ISP pipe.
> > + * @padding: padding bytes.
> > + * @num_sets: number of sets.
> > + * @padding1: padding bytes.
> > + * @size_of_set: set size.
> > + * @padding2: padding bytes.
> > + * @bubble_size: is the amount of padding in the bubble expressed in
> "sets".
> > + * @padding3: padding bytes.
> > + */
> > +struct ipu3_uapi_bubble_info {
> > + __u32 num_of_stripes __attribute__((aligned(32)));
> > + __u8 padding[28];
> > + __u32 num_sets;
> > + __u8 padding1[28];
> > + __u32 size_of_set;
> > + __u8 padding2[28];
> > + __u32 bubble_size;
> > + __u8 padding3[28];
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + */
> > +struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
> > + struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
> > + *
> > + * @awb_en: auto white balance enable
> > + * @padding: padding config
> > + * @ae_en: auto exposure enable
> > + * @padding1: padding config
> > + * @af_en: auto focus enable
> > + * @padding2: padding config
> > + * @awb_fr_en: awb filter response enable bit
> > + * @padding3: padding config
> > + */
> > +struct ipu3_uapi_ff_status {
> > + __u32 awb_en __attribute__((aligned(32)));
> > + __u8 padding[28];
> > + __u32 ae_en;
> > + __u8 padding1[28];
> > + __u32 af_en;
> > + __u8 padding2[28];
> > + __u32 awb_fr_en;
> > + __u8 padding3[28];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_stats_3a - 3A statistics
> > + *
> > + * @awb_raw_buffer: auto white balance meta data
> &ipu3_uapi_awb_raw_buffer
> > + * @ae_raw_buffer: auto exposure raw data
> &ipu3_uapi_ae_raw_buffer_aligned
> > + * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
> > + * @awb_fr_raw_buffer: value as specified by
> &ipu3_uapi_awb_fr_raw_buffer
> > + * @stats_4a_config: 4a statistics config as defined by
> &ipu3_uapi_4a_config.
> > + * @ae_join_buffers: 1 to use ae_raw_buffer.
> > + * @padding: padding config
> > + * @stats_3a_bubble_per_stripe: a
> &ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
> > + */
> > +struct ipu3_uapi_stats_3a {
> > + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_raw_buffer_aligned
> > + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> > + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> > + struct ipu3_uapi_4a_config stats_4a_config;
> > + __u32 ae_join_buffers;
> > + __u8 padding[28];
> > + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + stats_3a_bubble_per_stripe;
> > + struct ipu3_uapi_ff_status stats_3a_status;
> > +} __packed;
> > +
> > +/******************* ipu3_uapi_acc_param *******************/
> > +
> > +#define IPU3_UAPI_ISP_VEC_ELEMS 64
> > +#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
> > +
> > +#define IPU3_UAPI_BNR_LUT_SIZE 32
> > +
> > +/* number of elements in gamma correction LUT */
> > +#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
> > +
> > +/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
> > +#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
> > +#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
> > +/* Normalization shift aka nf */
> > +#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
> > +#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
> > +
> > +#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
> > +#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
> > +#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
> > +#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
> > +
> > +#define IPU3_UAPI_ANR_LUT_SIZE 26
> > +#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
> > +
> > +#define IPU3_UAPI_LIN_LUT_SIZE 64
> > +
> > +/* Bayer Noise Reduction related structs */
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance
> gains
> > + *
> > + * @gr: white balance gain for Gr channel.
> > + * @r: white balance gain for R channel.
> > + * @b: white balance gain for B channel.
> > + * @gb: white balance gain for Gb channel.
> > + *
> > + * Precision u3.13, range [0, 8]. White balance correction is done by
> applying
> > + * a multiplicative gain to each color channels prior to BNR.
> > + */
> > +struct ipu3_uapi_bnr_static_config_wb_gains_config {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold
> config
> > + *
> > + * @gr: white balance threshold gain for Gr channel.
> > + * @r: white balance threshold gain for R channel.
> > + * @b: white balance threshold gain for B channel.
> > + * @gb: white balance threshold gain for Gb channel.
> > + *
> > + * Defines the threshold that specifies how different a defect pixel can be
> from
> > + * its neighbors.(used by dynamic defect pixel correction sub block)
> > + * Precision u4.4 range [0, 8].
> > + */
> > +struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
> > + __u8 gr;
> > + __u8 r;
> > + __u8 b;
> > + __u8 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
> > + * coefficients that controls noise threshold
> > + *
> > + * @cf: Free coefficient for threshold calculation, range [0, 8191],
> default 0.
> > + * @__reserved0: reserved
> > + * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
> > + * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
> > + * default 6.
> > + * format: u3.2 (3 most significant bits represent whole number,
> > + * 2 least significant bits represent the fractional part
> > + * with each count representing 0.25)
> > + * e.g 6 in binary format is 00110, that translates to 1.5
> > + * @__reserved1: reserved
> > + * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
> > + * where r is a radius of pixel [row, col] from centor of sensor.
> > + * default 14.
> > + *
> > + * Threshold used to distinguish between noise and details.
> > + */
> > +struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
> > + __u32 cf:13;
> > + __u32 __reserved0:3;
> > + __u32 cg:5;
> > + __u32 ci:5;
> > + __u32 __reserved1:1;
> > + __u32 r_nf:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
> > + *
> > + * @gr: Coefficient defines lens shading gain approximation for gr
> channel
> > + * @r: Coefficient defines lens shading gain approximation for r
> channel
> > + * @b: Coefficient defines lens shading gain approximation for b
> channel
> > + * @gb: Coefficient defines lens shading gain approximation for gb
> channel
> > + *
> > + * Parameters for noise model (NM) adaptation of BNR due to shading
> correction.
> > + * All above have precision of u3.3, default to 0.
> > + */
> > +struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
> > + __u8 gr;
> > + __u8 r;
> > + __u8 b;
> > + __u8 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center
> config
> > + *
> > + * @x_reset: Reset value of X (col start - X center). Precision s12.0.
> > + * @__reserved0: reserved
> > + * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
> > + * @__reserved2: reserved
> > + *
> > + * Distance from corner to optical center for NM adaptation due to
> shading
> > + * correction (should be calculated based on shading tables)
> > + */
> > +struct ipu3_uapi_bnr_static_config_opt_center_config {
> > + __s32 x_reset:13;
> > + __u32 __reserved0:3;
> > + __s32 y_reset:13;
> > + __u32 __reserved2:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup
> table
> > + *
> > + * @values: pre-calculated values of square root function.
> > + *
> > + * LUT implementation of square root operation.
> > + */
> > +struct ipu3_uapi_bnr_static_config_lut_config {
> > + __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels
> (bp)
> > + *
> > + * @bp_thr_gain: Defines the threshold that specifies how different a
> > + * defect pixel can be from its neighbors. Threshold is
> > + * dependent on de-noise threshold calculated by
> algorithm.
> > + * Range [4, 31], default 4.
> > + * @__reserved0: reserved
> > + * @defect_mode: Mode of addressed defect pixels,
> > + * 0 - single defect pixel is expected,
> > + * 1 - 2 adjacent defect pixels are expected, default 1.
> > + * @bp_gain: Defines how 2nd derivation that passes through a
> defect pixel
> > + * is different from 2nd derivations that pass through
> > + * neighbor pixels. u4.2, range [0, 256], default 8.
> > + * @__reserved1: reserved
> > + * @w0_coeff: Blending coefficient of defect pixel correction.
> > + * Precision u4, range [0, 8], default 8.
> > + * @__reserved2: reserved
> > + * @w1_coeff: Enable influence of incorrect defect pixel correction
> to be
> > + * avoided. Precision u4, range [1, 8], default 8.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
> > + __u32 bp_thr_gain:5;
> > + __u32 __reserved0:2;
> > + __u32 defect_mode:1;
> > + __u32 bp_gain:6;
> > + __u32 __reserved1:18;
> > + __u32 w0_coeff:4;
> > + __u32 __reserved2:4;
> > + __u32 w1_coeff:4;
> > + __u32 __reserved3:20;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising
> config
> > + *
> > + * @alpha: Weight of central element of smoothing filter.
> > + * @beta: Weight of peripheral elements of smoothing filter, default 4.
> > + * @gamma: Weight of diagonal elements of smoothing filter, default 4.
> > + *
> > + * beta and gamma parameter define the strength of the noise removal
> filter.
> > + * All above has precision u0.4, range [0, 0xf]
> > + * format: u0.4 (no / zero bits represent whole number,
> > + * 4 bits represent the fractional part
> > + * with each count representing 0.0625)
> > + * e.g 0xf translates to 0.0625x15 = 0.9375
> > + *
> > + * @__reserved0: reserved
> > + * @max_inf: Maximum increase of peripheral or diagonal element
> influence
> > + * relative to the pre-defined value range: [0x5, 0xa]
> > + * @__reserved1: reserved
> > + * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
> > + * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 -
> enable.
> > + * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 -
> enable.
> > + * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
> > + __u32 alpha:4;
> > + __u32 beta:4;
> > + __u32 gamma:4;
> > + __u32 __reserved0:4;
> > + __u32 max_inf:4;
> > + __u32 __reserved1:7;
> > + __u32 gd_enable:1;
> > + __u32 bpc_enable:1;
> > + __u32 bnr_enable:1;
> > + __u32 ff_enable:1;
> > + __u32 __reserved2:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical
> square
> > + *
> > + * @x_sqr_reset: Reset value of X^2.
> > + * @y_sqr_reset: Reset value of Y^2.
> > + *
> > + * Please note:
> > + *
> > + * #. X and Y ref to
> > + * &ipu3_uapi_bnr_static_config_opt_center_config
> > + * #. Both structs are used in threshold formula to calculate r^2, where r
> > + * is a radius of pixel [row, col] from centor of sensor.
> > + */
> > +struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
> > + __u32 x_sqr_reset;
> > + __u32 y_sqr_reset;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config - BNR static config
> > + *
> > + * @wb_gains: white balance gains
> &ipu3_uapi_bnr_static_config_wb_gains_config
> > + * @wb_gains_thr: white balance gains threshold as defined by
> > + * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
> > + * @thr_coeffs: coefficients of threshold
> > + * &ipu3_uapi_bnr_static_config_thr_coeffs_config
> > + * @thr_ctrl_shd: control of shading threshold
> > + * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
> > + * @opt_center: optical center
> &ipu3_uapi_bnr_static_config_opt_center_config
> > + *
> > + * Above parameters and opt_center_sqr are used for white balance and
> shading.
> > + *
> > + * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
> > + * @bp_ctrl: detect and remove bad pixels as defined in struct
> > + * &ipu3_uapi_bnr_static_config_bp_ctrl_config
> > + * @dn_detect_ctrl: detect and remove noise.
> > + * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> > + * @column_size: The number of pixels in column.
> > + * @opt_center_sqr: Reset value of r^2 to optical center, see
> > + *
> &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
> > + */
> > +struct ipu3_uapi_bnr_static_config {
> > + struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
> > + struct ipu3_uapi_bnr_static_config_wb_gains_thr_config
> wb_gains_thr;
> > + struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
> > + struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
> > + struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
> > + struct ipu3_uapi_bnr_static_config_lut_config lut;
> > + struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
> > + struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> dn_detect_ctrl;
> > + __u32 column_size;
> > + struct ipu3_uapi_bnr_static_config_opt_center_sqr_config
> opt_center_sqr;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green
> disparity
> > + *
> > + * @gd_red: Shading gain coeff for gr disparity level in bright red region.
> > + * Precision u0.6, default 4(0.0625).
> > + * @__reserved0: reserved
> > + * @gd_green: Shading gain coeff for gr disparity level in bright
> green
> > + * region. Precision u0.6, default 4(0.0625).
> > + * @__reserved1: reserved
> > + * @gd_blue: Shading gain coeff for gr disparity level in bright blue
> region.
> > + * Precision u0.6, default 4(0.0625).
> > + * @__reserved2: reserved
> > + * @gd_black: Maximal green disparity level in dark region (stronger
> disparity
> > + * assumed to be image detail). Precision u14, default 80.
> > + * @__reserved3: reserved
> > + * @gd_shading: Change maximal green disparity level according to
> square
> > + * distance from image center.
> > + * @__reserved4: reserved
> > + * @gd_support: Lower bound for the number of second green color
> pixels in
> > + * current pixel neighborhood with less than threshold
> difference
> > + * from it.
> > + *
> > + * The shading gain coeff of red, green, blue and black are used to
> calculate
> > + * threshold given a pixel's color value and its coordinates in the image.
> > + *
> > + * @__reserved5: reserved
> > + * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
> > + * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
> > + */
> > +struct ipu3_uapi_bnr_static_config_green_disparity {
> > + __u32 gd_red:6;
> > + __u32 __reserved0:2;
> > + __u32 gd_green:6;
> > + __u32 __reserved1:2;
> > + __u32 gd_blue:6;
> > + __u32 __reserved2:10;
> > + __u32 gd_black:14;
> > + __u32 __reserved3:2;
> > + __u32 gd_shading:7;
> > + __u32 __reserved4:1;
> > + __u32 gd_support:2;
> > + __u32 __reserved5:1;
> > + __u32 gd_clip:1;
> > + __u32 gd_central_weight:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_dm_config - De-mosaic parameters
> > + *
> > + * @dm_en: de-mosaic enable.
> > + * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
> > + * @fcc_en: False color correction (FCC) enable flag. Default 0.
> > + * @__reserved0: reserved
> > + * @frame_width: do not care
> > + * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation
> of
> > + * complementary color in Hamilton-Adams interpolation).
> > + * u5, range [0, 31], default 8.
> > + * @__reserved1: reserved
> > + * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity
> metric
> > + * in calculation of final homogeneity metric.
> > + * u5, range [0, 31], default 7.
> > + * @__reserved2: reserved
> > + * @cr_param1: First parameter that defines Checker artifact removal
> > + * feature gain.Precision u5, range [0, 31], default 8.
> > + * @__reserved3: reserved
> > + * @cr_param2: Second parameter that defines Checker artifact
> removal
> > + * feature gain. Precision u5, range [0, 31], default 8.
> > + * @__reserved4: reserved
> > + * @coring_param: Defines power of false color correction operation.
> > + * low for preserving edge colors, high for preserving
> gray
> > + * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
> > + * @__reserved5: reserved
> > + *
> > + * The demosaic fixed function block is responsible to covert
> Bayer(mosaiced)
> > + * images into color images based on demosaicing algorithm.
> > + */
> > +struct ipu3_uapi_dm_config {
> > + __u32 dm_en:1;
> > + __u32 ch_ar_en:1;
> > + __u32 fcc_en:1;
> > + __u32 __reserved0:13;
> > + __u32 frame_width:16;
> > +
> > + __u32 gamma_sc:5;
> > + __u32 __reserved1:3;
> > + __u32 lc_ctrl:5;
> > + __u32 __reserved2:3;
> > + __u32 cr_param1:5;
> > + __u32 __reserved3:3;
> > + __u32 cr_param2:5;
> > + __u32 __reserved4:3;
> > +
> > + __u32 coring_param:5;
> > + __u32 __reserved5:27;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ccm_mat_config - Color correction matrix
> > + *
> > + * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
> > + * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
> > + * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
> > + * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
> > + * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
> > + *
> > + * Transform sensor specific color space to standard sRGB by applying 3x3
> matrix
> > + * and adding a bias vector O. The transformation is basically a rotation
> and
> > + * translation in the 3-dimensional color spaces. Here are the defaults:
> > + *
> > + * 9775, -2671, 1087, 0
> > + * -1071, 8303, 815, 0
> > + * -23, -7887, 16103, 0
> > + */
> > +struct ipu3_uapi_ccm_mat_config {
> > + __s16 coeff_m11;
> > + __s16 coeff_m12;
> > + __s16 coeff_m13;
> > + __s16 coeff_o_r;
> > + __s16 coeff_m21;
> > + __s16 coeff_m22;
> > + __s16 coeff_m23;
> > + __s16 coeff_o_g;
> > + __s16 coeff_m31;
> > + __s16 coeff_m32;
> > + __s16 coeff_m33;
> > + __s16 coeff_o_b;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
> > + *
> > + * @enable: gamma correction enable.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_gamma_corr_ctrl {
> > + __u32 enable:1;
> > + __u32 __reserved:31;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping
> implemented as LUT.
> > + *
> > + * @lut: 256 tabulated values of the gamma function. LUT[1]..
> LUT[256]
> > + * format u13.0, range [0, 8191].
> > + *
> > + * The tone mapping operation is done by a Piece wise linear graph
> > + * that is implemented as a lookup table(LUT). The pixel component input
> > + * intensity is the X-axis of the graph which is the table entry.
> > + */
> > +struct ipu3_uapi_gamma_corr_lut {
> > + __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_config - Gamma config
> > + *
> > + * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
> > + * @gc_lut: lookup table of gamma correction
> &ipu3_uapi_gamma_corr_lut
> > + */
> > +struct ipu3_uapi_gamma_config {
> > + struct ipu3_uapi_gamma_corr_ctrl gc_ctrl
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
> > + *
> > + * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + *
> > + * To transform each pixel from RGB to YUV (Y - brightness/luminance,
> > + * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding
> an
> > + * optional bias 3x1 vector.
> > + */
> > +struct ipu3_uapi_csc_mat_config {
> > + __s16 coeff_c11;
> > + __s16 coeff_c12;
> > + __s16 coeff_c13;
> > + __s16 coeff_b1;
> > + __s16 coeff_c21;
> > + __s16 coeff_c22;
> > + __s16 coeff_c23;
> > + __s16 coeff_b2;
> > + __s16 coeff_c31;
> > + __s16 coeff_c32;
> > + __s16 coeff_c33;
> > + __s16 coeff_b3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cds_params - Chroma down-scaling
> > + *
> > + * @ds_c00: range [0, 3]
> > + * @ds_c01: range [0, 3]
> > + * @ds_c02: range [0, 3]
> > + * @ds_c03: range [0, 3]
> > + * @ds_c10: range [0, 3]
> > + * @ds_c11: range [0, 3]
> > + * @ds_c12: range [0, 3]
> > + * @ds_c13: range [0, 3]
> > + *
> > + * In case user does not provide, above 4x2 filter will use following
> defaults:
> > + * 1, 3, 3, 1,
> > + * 1, 3, 3, 1,
> > + *
> > + * @ds_nf: Normalization factor for Chroma output downscaling filter,
> > + * range 0,4, default 2.
> > + * @__reserved0: reserved
> > + * @csc_en: Color space conversion enable
> > + * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_cds_params {
> > + __u32 ds_c00:2;
> > + __u32 ds_c01:2;
> > + __u32 ds_c02:2;
> > + __u32 ds_c03:2;
> > + __u32 ds_c10:2;
> > + __u32 ds_c11:2;
> > + __u32 ds_c12:2;
> > + __u32 ds_c13:2;
> > + __u32 ds_nf:5;
> > + __u32 __reserved0:3;
> > + __u32 csc_en:1;
> > + __u32 uv_bin_output:1;
> > + __u32 __reserved1:6;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
> > + *
> > + * @width: Grid horizontal dimensions, u8, [8, 128], default 73
> > + * @height: Grid vertical dimensions, u8, [8, 128], default 56
> > + * @block_width_log2: Log2 of the width of the grid cell in pixel
> count
> > + * u4, [0, 15], default value 5.
> > + * @__reserved0: reserved
> > + * @block_height_log2: Log2 of the height of the grid cell in pixel
> count
> > + * u4, [0, 15], default value 6.
> > + * @__reserved1: reserved
> > + * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
> > + * (with SHD_MAX_CELLS_PER_SET = 146).
> > + * @x_start: X value of top left corner of sensor relative to ROI
> > + * u12, [-4096, 0]. default 0, only negative values.
> > + * @y_start: Y value of top left corner of sensor relative to ROI
> > + * u12, [-4096, 0]. default 0, only negative values.
> > + */
> > +struct ipu3_uapi_shd_grid_config {
> > + /* reg 0 */
> > + __u8 width;
> > + __u8 height;
> > + __u8 block_width_log2:3;
> > + __u8 __reserved0:1;
> > + __u8 block_height_log2:3;
> > + __u8 __reserved1:1;
> > + __u8 grid_height_per_slice;
> > + /* reg 1 */
> > + __s16 x_start;
> > + __s16 y_start;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_general_config - Shading general config
> > + *
> > + * @init_set_vrt_offst_ul: set vertical offset,
> > + * y_start >> block_height_log2 % grid_height_per_slice.
> > + * @shd_enable: shading enable.
> > + * @gain_factor: Gain factor. Shift calculated anti shading value. Precision
> u2.
> > + * 0x0 - gain factor [1, 5], means no shift interpolated value.
> > + * 0x1 - gain factor [1, 9], means shift interpolated by 1.
> > + * 0x2 - gain factor [1, 17], means shift interpolated by 2.
> > + * @__reserved: reserved
> > + *
> > + * Correction is performed by multiplying a gain factor for each of the 4
> Bayer
> > + * channels as a function of the pixel location in the sensor.
> > + */
> > +struct ipu3_uapi_shd_general_config {
> > + __u32 init_set_vrt_offst_ul:8;
> > + __u32 shd_enable:1;
> > + __u32 gain_factor:2;
> > + __u32 __reserved:21;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_black_level_config - Black level correction
> > + *
> > + * @bl_r: Bios values for green red. s11 range [-2048, 2047].
> > + * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
> > + * @bl_gb: Bios values for red. s11 range [-2048, 2047].
> > + * @bl_b: Bios values for blue. s11 range [-2048, 2047].
> > + */
> > +struct ipu3_uapi_shd_black_level_config {
> > + __s16 bl_r;
> > + __s16 bl_gr;
> > + __s16 bl_gb;
> > + __s16 bl_b;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_config_static - Shading config static
> > + *
> > + * @grid: shading grid config &ipu3_uapi_shd_grid_config
> > + * @general: shading general config &ipu3_uapi_shd_general_config
> > + * @black_level: black level config for shading correction as defined by
> > + * &ipu3_uapi_shd_black_level_config
> > + */
> > +struct ipu3_uapi_shd_config_static {
> > + struct ipu3_uapi_shd_grid_config grid;
> > + struct ipu3_uapi_shd_general_config general;
> > + struct ipu3_uapi_shd_black_level_config black_level;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
> > + *
> > + * @sets: array
> > + * @sets.r_and_gr: Red and GreenR Lookup table.
> > + * @sets.r_and_gr.r: Red shading factor.
> > + * @sets.r_and_gr.gr: GreenR shading factor.
> > + * @sets.__reserved1: reserved
> > + * @sets.gb_and_b: GreenB and Blue Lookup table.
> > + * @sets.gb_and_b.gb: GreenB shading factor.
> > + * @sets.gb_and_b.b: Blue shading factor.
> > + * @sets.__reserved2: reserved
> > + *
> > + * Map to shading correction LUT register set.
> > + */
> > +struct ipu3_uapi_shd_lut {
> > + struct {
> > + struct {
> > + __u16 r;
> > + __u16 gr;
> > + } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> > + __u8 __reserved1[24];
> > + struct {
> > + __u16 gb;
> > + __u16 b;
> > + } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> > + __u8 __reserved2[24];
> > + } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_config - Shading config
> > + *
> > + * @shd: shading static config, see &ipu3_uapi_shd_config_static
> > + * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
> > + */
> > +struct ipu3_uapi_shd_config {
> > + struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
> > + struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/* Image Enhancement Filter directed */
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
> > + *
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A of Config Unit, s4.4, default 0.
> > + * @b01: Always 0.
> > + *
> > + * Calculate weight for blending directed and non-directed denoise
> elements
> > + *
> > + * Note:
> > + * Each instance of Config Unit needs X coordinate of n points and
> > + * slope A factor between points calculated by driver based on calibration
> > + * parameters.
> > + */
> > +struct ipu3_uapi_iefd_cux2 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 a01:9;
> > + __u32 b01:5; /* NOTE: hardcoded to zero */
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed
> sharpening
> > + * element, Config Unit 6 for edge detail (ED).
> > + *
> > + * @x0: X coordinate of point 0, u9.0, default 0.
> > + * @x1: X coordinate of point 1, u9.0, default 0.
> > + * @x2: X coordinate of point 2, u9.0, default 0.
> > + * @__reserved0: reserved
> > + * @x3: X coordinate of point 3, u9.0, default 0.
> > + * @x4: X coordinate of point 4, u9.0, default 0.
> > + * @x5: X coordinate of point 5, u9.0, default 0.
> > + * @__reserved1: reserved
> > + * @a01: slope A points 01, s4.4, default 0.
> > + * @a12: slope A points 12, s4.4, default 0.
> > + * @a23: slope A points 23, s4.4, default 0.
> > + * @__reserved2: reserved
> > + * @a34: slope A points 34, s4.4, default 0.
> > + * @a45: slope A points 45, s4.4, default 0.
> > + * @__reserved3: reserved
> > + * @b01: slope B points 01, s4.4, default 0.
> > + * @b12: slope B points 12, s4.4, default 0.
> > + * @b23: slope B points 23, s4.4, default 0.
> > + * @__reserved4: reserved
> > + * @b34: slope B points 34, s4.4, default 0.
> > + * @b45: slope B points 45, s4.4, default 0.
> > + * @__reserved5: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux6_ed {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 x2:9;
> > + __u32 __reserved0:5;
> > +
> > + __u32 x3:9;
> > + __u32 x4:9;
> > + __u32 x5:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 a01:9;
> > + __u32 a12:9;
> > + __u32 a23:9;
> > + __u32 __reserved2:5;
> > +
> > + __u32 a34:9;
> > + __u32 a45:9;
> > + __u32 __reserved3:14;
> > +
> > + __u32 b01:9;
> > + __u32 b12:9;
> > + __u32 b23:9;
> > + __u32 __reserved4:5;
> > +
> > + __u32 b34:9;
> > + __u32 b45:9;
> > + __u32 __reserved5:14;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
> > + * element apply.
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A of Config Unit, s4.4, default 0.
> > + * @__reserved1: reserved
> > + * @b01: offset B0 of Config Unit, u7.0, default 0.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux2_1 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 a01:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 b01:8;
> > + __u32 __reserved2:24;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed
> sharpening
> > + * element.
> > + *
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @x2: X2 point of Config Unit, u9.0, default 0.
> > + * @__reserved0: reserved
> > + * @x3: X3 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A0 of Config Unit, s4.4, default 0.
> > + * @a12: Slope A1 of Config Unit, s4.4, default 0.
> > + * @__reserved1: reserved
> > + * @a23: Slope A2 of Config Unit, s4.4, default 0.
> > + * @b01: Offset B0 of Config Unit, s7.0, default 0.
> > + * @b12: Offset B1 of Config Unit, s7.0, default 0.
> > + * @__reserved2: reserved
> > + * @b23: Offset B2 of Config Unit, s7.0, default 0.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux4 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 x2:9;
> > + __u32 __reserved0:5;
> > +
> > + __u32 x3:9;
> > + __u32 a01:9;
> > + __u32 a12:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 a23:9;
> > + __u32 b01:8;
> > + __u32 b12:8;
> > + __u32 __reserved2:7;
> > +
> > + __u32 b23:8;
> > + __u32 __reserved3:24;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
> > + *
> > + * @x0: x0 points of Config Unit radial, u8.0
> > + * @x1: x1 points of Config Unit radial, u8.0
> > + * @x2: x2 points of Config Unit radial, u8.0
> > + * @x3: x3 points of Config Unit radial, u8.0
> > + * @x4: x4 points of Config Unit radial, u8.0
> > + * @x5: x5 points of Config Unit radial, u8.0
> > + * @__reserved1: reserved
> > + * @a01: Slope A of Config Unit radial, s7.8
> > + * @a12: Slope A of Config Unit radial, s7.8
> > + * @a23: Slope A of Config Unit radial, s7.8
> > + * @a34: Slope A of Config Unit radial, s7.8
> > + * @a45: Slope A of Config Unit radial, s7.8
> > + * @__reserved2: reserved
> > + * @b01: Slope B of Config Unit radial, s9.0
> > + * @b12: Slope B of Config Unit radial, s9.0
> > + * @b23: Slope B of Config Unit radial, s9.0
> > + * @__reserved4: reserved
> > + * @b34: Slope B of Config Unit radial, s9.0
> > + * @b45: Slope B of Config Unit radial, s9.0
> > + * @__reserved5: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux6_rad {
> > + __u32 x0:8;
> > + __u32 x1:8;
> > + __u32 x2:8;
> > + __u32 x3:8;
> > +
> > + __u32 x4:8;
> > + __u32 x5:8;
> > + __u32 __reserved1:16;
> > +
> > + __u32 a01:16;
> > + __u32 a12:16;
> > +
> > + __u32 a23:16;
> > + __u32 a34:16;
> > +
> > + __u32 a45:16;
> > + __u32 __reserved2:16;
> > +
> > + __u32 b01:10;
> > + __u32 b12:10;
> > + __u32 b23:10;
> > + __u32 __reserved4:2;
> > +
> > + __u32 b34:10;
> > + __u32 b45:10;
> > + __u32 __reserved5:12;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
> > + *
> > + * @cu_1: calculate weight for blending directed and
> > + * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
> > + * @cu_ed: calculate power of non-directed sharpening element, see
> > + * &ipu3_uapi_iefd_cux6_ed
> > + * @cu_3: calculate weight for blending directed and
> > + * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
> > + * @cu_5: calculate power of non-directed denoise element apply, use
> > + * &ipu3_uapi_iefd_cux2_1
> > + * @cu_6: calculate power of non-directed sharpening element. See
> > + * &ipu3_uapi_iefd_cux4
> > + * @cu_7: calculate weight for blending directed and
> > + * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
> > + * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
> > + * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
> > + * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_cfg_units {
> > + struct ipu3_uapi_iefd_cux2 cu_1;
> > + struct ipu3_uapi_iefd_cux6_ed cu_ed;
> > + struct ipu3_uapi_iefd_cux2 cu_3;
> > + struct ipu3_uapi_iefd_cux2_1 cu_5;
> > + struct ipu3_uapi_iefd_cux4 cu_6;
> > + struct ipu3_uapi_iefd_cux2 cu_7;
> > + struct ipu3_uapi_iefd_cux4 cu_unsharp;
> > + struct ipu3_uapi_iefd_cux6_rad cu_radial;
> > + struct ipu3_uapi_iefd_cux2 cu_vssnlm;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
> > + *
> > + * @horver_diag_coeff: Gradiant compensation, coefficient that
> compensates for
> > + * different distance for vertical / horizontal and diagonal
> > + * * gradient calculation (~1/sqrt(2)).
> > + * @__reserved0: reserved
> > + * @clamp_stitch: Slope to stitch between clamped and unclamped edge
> values
> > + * @__reserved1: reserved
> > + * @direct_metric_update: Update coeff for direction metric
> > + * @__reserved2: reserved
> > + * @ed_horver_diag_coeff: Radial Coefficient that compensates for
> > + * different distance for vertical/horizontal and
> > + * diagonal gradient calculation (~1/sqrt(2))
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_config_s {
> > + __u32 horver_diag_coeff:7;
> > + __u32 __reserved0:1;
> > + __u32 clamp_stitch:6;
> > + __u32 __reserved1:2;
> > + __u32 direct_metric_update:5;
> > + __u32 __reserved2:3;
> > + __u32 ed_horver_diag_coeff:7;
> > + __u32 __reserved3:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
> > + *
> > + * @iefd_en: Enable IEFd
> > + * @denoise_en: Enable denoise
> > + * @direct_smooth_en: Enable directional smooth
> > + * @rad_en: Enable radial update
> > + * @vssnlm_en: Enable VSSNLM output filter
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_control {
> > + __u32 iefd_en:1;
> > + __u32 denoise_en:1;
> > + __u32 direct_smooth_en:1;
> > + __u32 rad_en:1;
> > + __u32 vssnlm_en:1;
> > + __u32 __reserved:27;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_sharp_cfg - Sharpening config
> > + *
> > + * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
> > + * @__reserved0: reserved
> > + * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
> > + * @__reserved1: reserved
> > + * @nega_lmt_dir: Sharpening limit for negative overshoots for direction
> (edge).
> > + * @__reserved2: reserved
> > + * @posi_lmt_dir: Sharpening limit for positive overshoots for direction
> (edge).
> > + * @__reserved3: reserved
> > + *
> > + * Fixed point type u13.0, range [0, 8191].
> > + */
> > +struct ipu3_uapi_sharp_cfg {
> > + __u32 nega_lmt_txt:13;
> > + __u32 __reserved0:19;
> > + __u32 posi_lmt_txt:13;
> > + __u32 __reserved1:19;
> > + __u32 nega_lmt_dir:13;
> > + __u32 __reserved2:19;
> > + __u32 posi_lmt_dir:13;
> > + __u32 __reserved3:19;
> > +} __packed;
> > +
> > +/**
> > + * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
> > + *
> > + * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64],
> default 64.
> > + * @__reserved0: reserved
> > + * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64],
> default 0.
> > + * @__reserved1: reserved
> > + * @ndir_dns_powr: Power of non-direct denoising,
> > + * Precision u1.6, range [0, 64], default 64.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_far_w {
> > + __u32 dir_shrp:7;
> > + __u32 __reserved0:1;
> > + __u32 dir_dns:7;
> > + __u32 __reserved1:1;
> > + __u32 ndir_dns_powr:7;
> > + __u32 __reserved2:9;
> > +} __packed;
> > +
> > +/**
> > + * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
> > + *
> > + * @unsharp_weight: Unsharp mask blending weight.
> > + * u1.6, range [0, 64], default 16.
> > + * 0 - disabled, 64 - use only unsharp.
> > + * @__reserved0: reserved
> > + * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511],
> default 0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_unsharp_cfg {
> > + __u32 unsharp_weight:7;
> > + __u32 __reserved0:1;
> > + __u32 unsharp_amount:9;
> > + __u32 __reserved1:15;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
> > + *
> > + * @cfg: sharpness config &ipu3_uapi_sharp_cfg
> > + * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
> > + * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
> > + * neighbors (8 pixels called Near), and the second order neighborhood
> > + * around them (16 pixels called Far).
> > + * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
> > + struct ipu3_uapi_sharp_cfg cfg;
> > + struct ipu3_uapi_far_w far_w;
> > + struct ipu3_uapi_unsharp_cfg unshrp_cfg;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
> > + *
> > + * @c00: Coeff11, s0.8, range [-255, 255], default 1.
> > + * @c01: Coeff12, s0.8, range [-255, 255], default 5.
> > + * @c02: Coeff13, s0.8, range [-255, 255], default 9.
> > + * @__reserved: reserved
> > + *
> > + * Configurable registers for common sharpening support.
> > + */
> > +struct ipu3_uapi_unsharp_coef0 {
> > + __u32 c00:9;
> > + __u32 c01:9;
> > + __u32 c02:9;
> > + __u32 __reserved:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
> > + *
> > + * @c11: Coeff22, s0.8, range [-255, 255], default 29.
> > + * @c12: Coeff23, s0.8, range [-255, 255], default 55.
> > + * @c22: Coeff33, s0.8, range [-255, 255], default 96.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_unsharp_coef1 {
> > + __u32 c11:9;
> > + __u32 c12:9;
> > + __u32 c22:9;
> > + __u32 __reserved:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
> > + *
> > + * @unsharp_coef0: unsharp coefficient 0 config. See
> &ipu3_uapi_unsharp_coef0
> > + * @unsharp_coef1: unsharp coefficient 1 config. See
> &ipu3_uapi_unsharp_coef1
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
> > + struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
> > + struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_reset_xy - Radial coordinate reset
> > + *
> > + * @x: Radial reset of x coordinate. Precision s12, [-4095, 4095],
> default 0.
> > + * @__reserved0: reserved
> > + * @y: Radial center y coordinate. Precision s12, [-4095, 4095],
> default 0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_radial_reset_xy {
> > + __s32 x:13;
> > + __u32 __reserved0:3;
> > + __s32 y:13;
> > + __u32 __reserved1:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_reset_x2 - Radial X^2 reset
> > + *
> > + * @x2: Radial reset of x^2 coordinate. Precision u24, default 0.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_radial_reset_x2 {
> > + __u32 x2:24;
> > + __u32 __reserved:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_reset_y2 - Radial Y^2 reset
> > + *
> > + * @y2: Radial reset of y^2 coordinate. Precision u24, default 0.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_radial_reset_y2 {
> > + __u32 y2:24;
> > + __u32 __reserved:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_cfg - Radial config
> > + *
> > + * @rad_nf: Radial. R^2 normalization factor is scale down by 2^ - (15 +
> scale)
> > + * @__reserved0: reserved
> > + * @rad_inv_r2: Radial R^-2 normelized to (0.5..1), Prec' u7, range [0, 127].
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_radial_cfg {
> > + __u32 rad_nf:4;
> > + __u32 __reserved0:4;
> > + __u32 rad_inv_r2:7;
> > + __u32 __reserved1:17;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_rad_far_w - Radial FAR sub-group
> > + *
> > + * @rad_dir_far_sharp_w: Weight of wide direct sharpening, u1.6, range
> [0, 64],
> > + * default 64.
> > + * @rad_dir_far_dns_w: Weight of wide direct denoising, u1.6, range [0,
> 64],
> > + * default 0.
> > + * @rad_ndir_far_dns_power: power of non-direct sharpening, u1.6,
> range [0, 64],
> > + * default 0.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_rad_far_w {
> > + __u32 rad_dir_far_sharp_w:8;
> > + __u32 rad_dir_far_dns_w:8;
> > + __u32 rad_ndir_far_dns_power:8;
> > + __u32 __reserved:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cu_cfg0 - Radius Config Unit cfg0 register
> > + *
> > + * @cu6_pow: Power of CU6. Power of non-direct sharpening, u3.4.
> > + * @__reserved0: reserved
> > + * @cu_unsharp_pow: Power of unsharp mask, u2.4.
> > + * @__reserved1: reserved
> > + * @rad_cu6_pow: Radial/corner CU6. Directed sharpening power, u3.4.
> > + * @__reserved2: reserved
> > + * @rad_cu_unsharp_pow: Radial power of unsharp mask, u2.4.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_cu_cfg0 {
> > + __u32 cu6_pow:7;
> > + __u32 __reserved0:1;
> > + __u32 cu_unsharp_pow:7;
> > + __u32 __reserved1:1;
> > + __u32 rad_cu6_pow:7;
> > + __u32 __reserved2:1;
> > + __u32 rad_cu_unsharp_pow:6;
> > + __u32 __reserved3:2;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cu_cfg1 - Radius Config Unit cfg1 register
> > + *
> > + * @rad_cu6_x1: X1 point of Config Unit 6, precision u9.0.
> > + * @__reserved0: reserved
> > + * @rad_cu_unsharp_x1: X1 point for Config Unit unsharp for
> radial/corner point
> > + * precision u9.0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_cu_cfg1 {
> > + __u32 rad_cu6_x1:9;
> > + __u32 __reserved0:1;
> > + __u32 rad_cu_unsharp_x1:9;
> > + __u32 __reserved1:13;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_rad_cfg - IEFd parameters changed
> radially over
> > + * the picture plain.
> > + *
> > + * @reset_xy: reset xy value in radial calculation.
> &ipu3_uapi_radial_reset_xy
> > + * @reset_x2: reset x square value in radial calculation. See struct
> > + * &ipu3_uapi_radial_reset_x2
> > + * @reset_y2: reset y square value in radial calculation. See struct
> > + * &ipu3_uapi_radial_reset_y2
> > + * @cfg: radial config defined in &ipu3_uapi_radial_cfg
> > + * @rad_far_w: weight for wide range radial. &ipu3_uapi_rad_far_w
> > + * @cu_cfg0: configuration unit 0. See &ipu3_uapi_cu_cfg0
> > + * @cu_cfg1: configuration unit 1. See &ipu3_uapi_cu_cfg1
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_rad_cfg {
> > + struct ipu3_uapi_radial_reset_xy reset_xy;
> > + struct ipu3_uapi_radial_reset_x2 reset_x2;
> > + struct ipu3_uapi_radial_reset_y2 reset_y2;
> > + struct ipu3_uapi_radial_cfg cfg;
> > + struct ipu3_uapi_rad_far_w rad_far_w;
> > + struct ipu3_uapi_cu_cfg0 cu_cfg0;
> > + struct ipu3_uapi_cu_cfg1 cu_cfg1;
> > +} __packed;
> > +
> > +/* Vssnlm - Very small scale non-local mean algorithm */
> > +
> > +/**
> > + * struct ipu3_uapi_vss_lut_x - Vssnlm LUT x0/x1/x2
> > + *
> > + * @vs_x0: Vssnlm LUT x0, precision u8, range [0, 255], default 16.
> > + * @vs_x1: Vssnlm LUT x1, precision u8, range [0, 255], default 32.
> > + * @vs_x2: Vssnlm LUT x2, precision u8, range [0, 255], default 64.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_vss_lut_x {
> > + __u32 vs_x0:8;
> > + __u32 vs_x1:8;
> > + __u32 vs_x2:8;
> > + __u32 __reserved2:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_vss_lut_y - Vssnlm LUT y0/y1/y2
> > + *
> > + * @vs_y1: Vssnlm LUT y1, precision u4, range [0, 8], default 1.
> > + * @__reserved0: reserved
> > + * @vs_y2: Vssnlm LUT y2, precision u4, range [0, 8], default 3.
> > + * @__reserved1: reserved
> > + * @vs_y3: Vssnlm LUT y3, precision u4, range [0, 8], default 8.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_vss_lut_y {
> > + __u32 vs_y1:4;
> > + __u32 __reserved0:4;
> > + __u32 vs_y2:4;
> > + __u32 __reserved1:4;
> > + __u32 vs_y3:4;
> > + __u32 __reserved2:12;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_vssnlm_cf - IEFd Vssnlm Lookup table
> > + *
> > + * @vss_lut_x: vss lookup table. See &ipu3_uapi_vss_lut_x description
> > + * @vss_lut_y: vss lookup table. See &ipu3_uapi_vss_lut_y description
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg {
> > + struct ipu3_uapi_vss_lut_x vss_lut_x;
> > + struct ipu3_uapi_vss_lut_y vss_lut_y;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_config - IEFd config
> > + *
> > + * @units: configuration unit setting, &ipu3_uapi_yuvp1_iefd_cfg_units
> > + * @config: configuration, as defined by &ipu3_uapi_yuvp1_iefd_config_s
> > + * @control: control setting, as defined by
> &ipu3_uapi_yuvp1_iefd_control
> > + * @sharp: sharpness setting, as defined by
> &ipu3_uapi_yuvp1_iefd_shrp_cfg
> > + * @unsharp: unsharpness setting, as defined by
> &ipu3_uapi_yuvp1_iefd_unshrp_cfg
> > + * @rad: radial setting, as defined by &ipu3_uapi_yuvp1_iefd_rad_cfg
> > + * @vsslnm: vsslnm setting, as defined by
> &ipu3_uapi_yuvp1_iefd_vssnlm_cfg
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_config {
> > + struct ipu3_uapi_yuvp1_iefd_cfg_units units;
> > + struct ipu3_uapi_yuvp1_iefd_config_s config;
> > + struct ipu3_uapi_yuvp1_iefd_control control;
> > + struct ipu3_uapi_yuvp1_iefd_shrp_cfg sharp;
> > + struct ipu3_uapi_yuvp1_iefd_unshrp_cfg unsharp;
> > + struct ipu3_uapi_yuvp1_iefd_rad_cfg rad;
> > + struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg vsslnm;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_yds_config - Y Down-Sampling config
> > + *
> > + * @c00: range [0, 3], default 0x0
> > + * @c01: range [0, 3], default 0x1
> > + * @c02: range [0, 3], default 0x1
> > + * @c03: range [0, 3], default 0x0
> > + * @c10: range [0, 3], default 0x0
> > + * @c11: range [0, 3], default 0x1
> > + * @c12: range [0, 3], default 0x1
> > + * @c13: range [0, 3], default 0x0
> > + *
> > + * Above are 4x2 filter coefficients for chroma output downscaling.
> > + *
> > + * @norm_factor: Normalization factor, range [0, 4], default 2
> > + * 0 - divide by 1
> > + * 1 - divide by 2
> > + * 2 - divide by 4
> > + * 3 - divide by 8
> > + * 4 - divide by 16
> > + * @__reserved0: reserved
> > + * @bin_output: Down sampling on Luma channel in two optional modes
> > + * 0 - Bin output 4.2.0 (default), 1 output 4.2.2.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_yds_config {
> > + __u32 c00:2;
> > + __u32 c01:2;
> > + __u32 c02:2;
> > + __u32 c03:2;
> > + __u32 c10:2;
> > + __u32 c11:2;
> > + __u32 c12:2;
> > + __u32 c13:2;
> > + __u32 norm_factor:5;
> > + __u32 __reserved0:4;
> > + __u32 bin_output:1;
> > + __u32 __reserved1:6;
> > +} __packed;
> > +
> > +/* Chroma Noise Reduction */
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_enable_config - Chroma noise reduction
> enable
> > + *
> > + * @enable: enable/disable chroma noise reduction
> > + * @yuv_mode: 0 - YUV420, 1 - YUV422
> > + * @__reserved0: reserved
> > + * @col_size: number of columns in the frame, max width is 2560
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_enable_config {
> > + __u32 enable:1;
> > + __u32 yuv_mode:1;
> > + __u32 __reserved0:14;
> > + __u32 col_size:12;
> > + __u32 __reserved1:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_coring_config - Coring thresholds for UV
> > + *
> > + * @u: U coring level, u0.13, range [0.0, 1.0], default 0.0
> > + * @__reserved0: reserved
> > + * @v: V coring level, u0.13, range [0.0, 1.0], default 0.0
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_coring_config {
> > + __u32 u:13;
> > + __u32 __reserved0:3;
> > + __u32 v:13;
> > + __u32 __reserved1:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_sense_gain_config - Chroma noise
> reduction gains
> > + *
> > + * All sensitivity gain parameters have precision u13.0, range [0, 8191].
> > + *
> > + * @vy: Sensitivity of horizontal edge of Y, default 100
> > + * @vu: Sensitivity of horizontal edge of U, default 100
> > + * @vv: Sensitivity of horizontal edge of V, default 100
> > + * @__reserved0: reserved
> > + * @hy: Sensitivity of vertical edge of Y, default 50
> > + * @hu: Sensitivity of vertical edge of U, default 50
> > + * @hv: Sensitivity of vertical edge of V, default 50
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_sense_gain_config {
> > + __u32 vy:8;
> > + __u32 vu:8;
> > + __u32 vv:8;
> > + __u32 __reserved0:8;
> > +
> > + __u32 hy:8;
> > + __u32 hu:8;
> > + __u32 hv:8;
> > + __u32 __reserved1:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_iir_fir_config - Chroma IIR/FIR filter config
> > + *
> > + * @fir_0h: Value of center tap in horizontal FIR, range [0, 32], default 8.
> > + * @__reserved0: reserved
> > + * @fir_1h: Value of distance 1 in horizontal FIR, range [0, 32], default 12.
> > + * @__reserved1: reserved
> > + * @fir_2h: Value of distance 2 tap in horizontal FIR, range [0, 32], default
> 0.
> > + * @dalpha_clip_val: weight for previous row in IIR, range [1, 256], default
> 0.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_iir_fir_config {
> > + __u32 fir_0h:6;
> > + __u32 __reserved0:2;
> > + __u32 fir_1h:6;
> > + __u32 __reserved1:2;
> > + __u32 fir_2h:6;
> > + __u32 dalpha_clip_val:9;
> > + __u32 __reserved2:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_config - Chroma noise reduction config
> > + *
> > + * @enable: chroma noise reduction enable, see
> > + * &ipu3_uapi_yuvp1_chnr_enable_config
> > + * @coring: coring config for chroma noise reduction, see
> > + * &ipu3_uapi_yuvp1_chnr_coring_config
> > + * @sense_gain: sensitivity config for chroma noise reduction, see
> > + * ipu3_uapi_yuvp1_chnr_sense_gain_config
> > + * @iir_fir: iir and fir config for chroma noise reduction, see
> > + * ipu3_uapi_yuvp1_chnr_iir_fir_config
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_config {
> > + struct ipu3_uapi_yuvp1_chnr_enable_config enable;
> > + struct ipu3_uapi_yuvp1_chnr_coring_config coring;
> > + struct ipu3_uapi_yuvp1_chnr_sense_gain_config sense_gain;
> > + struct ipu3_uapi_yuvp1_chnr_iir_fir_config iir_fir;
> > +} __packed;
> > +
> > +/* Edge Enhancement and Noise Reduction */
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config - Luma(Y) edge
> enhancement low-pass
> > + * filter coefficients
> > + *
> > + * @a_diag: Smoothing diagonal coefficient, u5.0.
> > + * @__reserved0: reserved
> > + * @a_periph: Image smoothing perpherial, u5.0.
> > + * @__reserved1: reserved
> > + * @a_cent: Image Smoothing center coefficient, u5.0.
> > + * @__reserved2: reserved
> > + * @enable: 0: Y_EE_NR disabled, output = input; 1: Y_EE_NR enabled.
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config {
> > + __u32 a_diag:5;
> > + __u32 __reserved0:3;
> > + __u32 a_periph:5;
> > + __u32 __reserved1:3;
> > + __u32 a_cent:5;
> > + __u32 __reserved2:9;
> > + __u32 enable:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_sense_config - Luma(Y) edge
> enhancement
> > + * noise reduction sensitivity gains
> > + *
> > + * @edge_sense_0: Sensitivity of edge in dark area. u13.0, default 8191.
> > + * @__reserved0: reserved
> > + * @delta_edge_sense: Difference in the sensitivity of edges between
> > + * the bright and dark areas. u13.0, default 0.
> > + * @__reserved1: reserved
> > + * @corner_sense_0: Sensitivity of corner in dark area. u13.0, default 0.
> > + * @__reserved2: reserved
> > + * @delta_corner_sense: Difference in the sensitivity of corners between
> > + * the bright and dark areas. u13.0, default 8191.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_sense_config {
> > + __u32 edge_sense_0:13;
> > + __u32 __reserved0:3;
> > + __u32 delta_edge_sense:13;
> > + __u32 __reserved1:3;
> > + __u32 corner_sense_0:13;
> > + __u32 __reserved2:3;
> > + __u32 delta_corner_sense:13;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_gain_config - Luma(Y) edge
> enhancement
> > + * noise reduction gain config
> > + *
> > + * @gain_pos_0: Gain for positive edge in dark area. u5.0, [0, 16], default
> 2.
> > + * @__reserved0: reserved
> > + * @delta_gain_posi: Difference in the gain of edges between the bright
> and
> > + * dark areas for positive edges. u5.0, [0, 16], default 0.
> > + * @__reserved1: reserved
> > + * @gain_neg_0: Gain for negative edge in dark area. u5.0, [0, 16], default
> 8.
> > + * @__reserved2: reserved
> > + * @delta_gain_neg: Difference in the gain of edges between the bright
> and
> > + * dark areas for negative edges. u5.0, [0, 16], default 0.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_gain_config {
> > + __u32 gain_pos_0:5;
> > + __u32 __reserved0:3;
> > + __u32 delta_gain_posi:5;
> > + __u32 __reserved1:3;
> > + __u32 gain_neg_0:5;
> > + __u32 __reserved2:3;
> > + __u32 delta_gain_neg:5;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_clip_config - Luma(Y) edge
> enhancement
> > + * noise reduction clipping config
> > + *
> > + * @clip_pos_0: Limit of positive edge in dark area
> > + * u5, value [0, 16], default 8.
> > + * @__reserved0: reserved
> > + * @delta_clip_posi: Difference in the limit of edges between the bright
> > + * and dark areas for positive edges.
> > + * u5, value [0, 16], default 8.
> > + * @__reserved1: reserved
> > + * @clip_neg_0: Limit of negative edge in dark area
> > + * u5, value [0, 16], default 8.
> > + * @__reserved2: reserved
> > + * @delta_clip_neg: Difference in the limit of edges between the bright
> > + * and dark areas for negative edges.
> > + * u5, value [0, 16], default 8.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_clip_config {
> > + __u32 clip_pos_0:5;
> > + __u32 __reserved0:3;
> > + __u32 delta_clip_posi:5;
> > + __u32 __reserved1:3;
> > + __u32 clip_neg_0:5;
> > + __u32 __reserved2:3;
> > + __u32 delta_clip_neg:5;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_frng_config - Luma(Y) edge
> enhancement
> > + * noise reduction fringe config
> > + *
> > + * @gain_exp: Common exponent of gains, u4, [0, 8], default 2.
> > + * @__reserved0: reserved
> > + * @min_edge: Threshold for edge and smooth stitching, u13.
> > + * @__reserved1: reserved
> > + * @lin_seg_param: Power of LinSeg, u4.
> > + * @__reserved2: reserved
> > + * @t1: Parameter for enabling/disabling the edge enhancement, u1.0, [0,
> 1],
> > + * default 1.
> > + * @t2: Parameter for enabling/disabling the smoothing, u1.0, [0, 1],
> > + * default 1.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_frng_config {
> > + __u32 gain_exp:4;
> > + __u32 __reserved0:28;
> > + __u32 min_edge:13;
> > + __u32 __reserved1:3;
> > + __u32 lin_seg_param:4;
> > + __u32 __reserved2:4;
> > + __u32 t1:1;
> > + __u32 t2:1;
> > + __u32 __reserved3:6;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_diag_config - Luma(Y) edge
> enhancement
> > + * noise reduction diagonal config
> > + *
> > + * @diag_disc_g: Coefficient that prioritize diagonal edge direction on
> > + * horizontal or vertical for final enhancement.
> > + * u4.0, [1, 15], default 1.
> > + * @__reserved0: reserved
> > + * @hvw_hor: Weight of horizontal/vertical edge enhancement for hv
> edge.
> > + * u2.2, [1, 15], default 4.
> > + * @dw_hor: Weight of diagonal edge enhancement for hv edge.
> > + * u2.2, [1, 15], default 1.
> > + * @hvw_diag: Weight of horizontal/vertical edge enhancement for
> diagonal edge.
> > + * u2.2, [1, 15], default 1.
> > + * @dw_diag: Weight of diagonal edge enhancement for diagonal edge.
> > + * u2.2, [1, 15], default 4.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_diag_config {
> > + __u32 diag_disc_g:4;
> > + __u32 __reserved0:4;
> > + __u32 hvw_hor:4;
> > + __u32 dw_hor:4;
> > + __u32 hvw_diag:4;
> > + __u32 dw_diag:4;
> > + __u32 __reserved1:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config - Luma(Y) edge
> enhancement
> > + * noise reduction false color correction (FCC) coring config
> > + *
> > + * @pos_0: Gain for positive edge in dark, u13.0, [0, 16], default 0.
> > + * @__reserved0: reserved
> > + * @pos_delta: Gain for positive edge in bright, value: pos_0 + pos_delta
> <=16
> > + * u13.0, default 0.
> > + * @__reserved1: reserved
> > + * @neg_0: Gain for negative edge in dark area, u13.0, range [0, 16],
> default 0.
> > + * @__reserved2: reserved
> > + * @neg_delta: Gain for negative edge in bright area. neg_0 + neg_delta
> <=16
> > + * u13.0, default 0.
> > + * @__reserved3: reserved
> > + *
> > + * Coring is a simple soft thresholding technique.
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config {
> > + __u32 pos_0:13;
> > + __u32 __reserved0:3;
> > + __u32 pos_delta:13;
> > + __u32 __reserved1:3;
> > + __u32 neg_0:13;
> > + __u32 __reserved2:3;
> > + __u32 neg_delta:13;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_config - Edge enhancement and noise
> reduction
> > + *
> > + * @lpf: low-pass filter config. See &ipu3_uapi_yuvp1_y_ee_nr_lpf_config
> > + * @sense: sensitivity config. See
> &ipu3_uapi_yuvp1_y_ee_nr_sense_config
> > + * @gain: gain config as defined in
> &ipu3_uapi_yuvp1_y_ee_nr_gain_config
> > + * @clip: clip config as defined in &ipu3_uapi_yuvp1_y_ee_nr_clip_config
> > + * @frng: fringe config as defined in
> &ipu3_uapi_yuvp1_y_ee_nr_frng_config
> > + * @diag: diagonal edge config. See
> &ipu3_uapi_yuvp1_y_ee_nr_diag_config
> > + * @fc_coring: coring config for fringe control. See
> > + * &ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_config {
> > + struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config lpf;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_sense_config sense;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_gain_config gain;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_clip_config clip;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_frng_config frng;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_diag_config diag;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config fc_coring;
> > +} __packed;
> > +
> > +/* Total Color Correction */
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_gen_control_static_config - Total color
> correction
> > + * general control config
> > + *
> > + * @en: 0 - TCC disabled. Output = input 1 - TCC enabled.
> > + * @blend_shift: blend shift, Range[3, 4], default NA.
> > + * @gain_according_to_y_only: 0: Gain is calculated according to YUV,
> > + * 1: Gain is calculated according to Y only
> > + * @__reserved0: reserved
> > + * @gamma: Final blending coefficients. Values[-16, 16], default NA.
> > + * @__reserved1: reserved
> > + * @delta: Final blending coefficients. Values[-16, 16], default NA.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_gen_control_static_config {
> > + __u32 en:1;
> > + __u32 blend_shift:3;
> > + __u32 gain_according_to_y_only:1;
> > + __u32 __reserved0:11;
> > + __s32 gamma:5;
> > + __u32 __reserved1:3;
> > + __s32 delta:5;
> > + __u32 __reserved2:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config - Total color
> correction
> > + * multi-axis color control (MACC) config
> > + *
> > + * @a: a coefficient for 2x2 MACC conversion matrix.
> > + * @__reserved0: reserved
> > + * @b: b coefficient 2x2 MACC conversion matrix.
> > + * @__reserved1: reserved
> > + * @c: c coefficient for 2x2 MACC conversion matrix.
> > + * @__reserved2: reserved
> > + * @d: d coefficient for 2x2 MACC conversion matrix.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config {
> > + __s32 a:12;
> > + __u32 __reserved0:4;
> > + __s32 b:12;
> > + __u32 __reserved1:4;
> > + __s32 c:12;
> > + __u32 __reserved2:4;
> > + __s32 d:12;
> > + __u32 __reserved3:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_macc_table_static_config - Total color
> correction
> > + * multi-axis color control (MACC) table array
> > + *
> > + * @entries: config for multi axis color correction, as specified by
> > + * &ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_macc_table_static_config {
> > + struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> > + entries[IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config - Total color
> correction
> > + * inverse y lookup table
> > + *
> > + * @entries: lookup table for inverse y estimation, and use it to estimate
> the
> > + * ratio between luma and chroma. Chroma by approximate the
> absolute
> > + * value of the radius on the chroma plane (R = sqrt(u^2+v^2) ) and
> > + * luma by approximate by 1/Y.
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config {
> > + __u16 entries[IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config - Total color
> > + * correction lookup table for PCWL
> > + *
> > + * @entries: lookup table for gain piece wise linear transformation (PCWL)
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config {
> > + __u16 entries[IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config - Total color
> correction
> > + * lookup table for r square root
> > + *
> > + * @entries: lookup table for r square root estimation
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config {
> > + __s16 entries[IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_static_config- Total color correction static
> > + *
> > + * @gen_control: general config for Total Color Correction
> > + * @macc_table: config for multi axis color correction
> > + * @inv_y_lut: lookup table for inverse y estimation
> > + * @gain_pcwl: lookup table for gain PCWL
> > + * @r_sqr_lut: lookup table for r square root estimation.
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_static_config {
> > + struct ipu3_uapi_yuvp2_tcc_gen_control_static_config gen_control;
> > + struct ipu3_uapi_yuvp2_tcc_macc_table_static_config macc_table;
> > + struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config inv_y_lut;
> > + struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config gain_pcwl;
> > + struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config r_sqr_lut;
> > +} __packed;
> > +
> > +/* Advanced Noise Reduction related structs */
> > +
> > +/*
> > + * struct ipu3_uapi_anr_alpha - Advanced noise reduction alpha
> > + *
> > + * Tunable parameters that are subject to modification according to the
> > + * total gain used.
> > + */
> > +struct ipu3_uapi_anr_alpha {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > + __u16 dc_gr;
> > + __u16 dc_r;
> > + __u16 dc_b;
> > + __u16 dc_gb;
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_anr_beta - Advanced noise reduction beta
> > + *
> > + * Tunable parameters that are subject to modification according to the
> > + * total gain used.
> > + */
> > +struct ipu3_uapi_anr_beta {
> > + __u16 beta_gr;
> > + __u16 beta_r;
> > + __u16 beta_b;
> > + __u16 beta_gb;
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_anr_plain_color - Advanced noise reduction plain
> color with
> > + * 4x4 matrix
> > + *
> > + * Tunable parameters that are subject to modification according to the
> > + * total gain used.
> > + */
> > +struct ipu3_uapi_anr_plain_color {
> > + __u16 reg_w_gr[16];
> > + __u16 reg_w_r[16];
> > + __u16 reg_w_b[16];
> > + __u16 reg_w_gb[16];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_transform_config - Advanced noise reduction
> transform
> > + *
> > + * @enable: advanced noise reduction enabled.
> > + * @adaptive_treshhold_en: On IPU3, adaptive threshold is always
> enabled.
> > + * @__reserved1: reserved
> > + * @__reserved2: reserved
> > + * @alpha: using following defaults:
> > + * 13, 13, 13, 13, 0, 0, 0, 0
> > + * 11, 11, 11, 11, 0, 0, 0, 0
> > + * 14, 14, 14, 14, 0, 0, 0, 0
> > + * @beta: use following defaults:
> > + * 24, 24, 24, 24
> > + * 21, 20, 20, 21
> > + * 25, 25, 25, 25
> > + * @color: use defaults defined in driver/media/pci/intel/ipu3-tables.c
> > + * @sqrt_lut: 11 bits per element, values =
> > + * [724 768 810 849 887
> > + * 923 958 991 1024 1056
> > + * 1116 1145 1173 1201 1086
> > + * 1228 1254 1280 1305 1330
> > + * 1355 1379 1402 1425 1448]
> > + * @xreset: Reset value of X for r^2 calculation Value: col_start-X_center
> > + * Constraint: Xreset + FrameWdith=4095 Xreset= -4095, default -1632.
> > + * @__reserved3: reserved
> > + * @yreset: Reset value of Y for r^2 calculation Value: row_start-Y_center
> > + * Constraint: Yreset + FrameHeight=4095 Yreset= -4095, default -1224.
> > + * @__reserved4: reserved
> > + * @x_sqr_reset: Reset value of X^2 for r^2 calculation Value = (Xreset)^2
> > + * @r_normfactor: Normalization factor for R. Default 14.
> > + * @__reserved5: reserved
> > + * @y_sqr_reset: Reset value of Y^2 for r^2 calculation Value = (Yreset)^2
> > + * @gain_scale: Parameter describing shading gain as a function of
> distance
> > + * from the image center.
> > + * A single value per frame, loaded by the driver. Default 115.
> > + */
> > +struct ipu3_uapi_anr_transform_config {
> > + __u32 enable:1; /* 0 or 1, disabled or enabled
> */
> > + __u32 adaptive_treshhold_en:1; /* On IPU3, always enabled
> */
> > +
> > + __u32 __reserved1:30;
> > + __u8 __reserved2[44];
> > +
> > + struct ipu3_uapi_anr_alpha alpha[3];
> > + struct ipu3_uapi_anr_beta beta[3];
> > + struct ipu3_uapi_anr_plain_color color[3];
> > +
> > + __u16 sqrt_lut[IPU3_UAPI_ANR_LUT_SIZE]; /* 11 bits per element
> */
> > +
> > + __s16 xreset:13;
> > + __u16 __reserved3:3;
> > + __s16 yreset:13;
> > + __u16 __reserved4:3;
> > +
> > + __u32 x_sqr_reset:24;
> > + __u32 r_normfactor:5;
> > + __u32 __reserved5:3;
> > +
> > + __u32 y_sqr_reset:24;
> > + __u32 gain_scale:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_stitch_pyramid - ANR stitch pyramid
> > + *
> > + * @entry0: pyramid LUT entry0, range [0x0, 0x3f]
> > + * @entry1: pyramid LUT entry1, range [0x0, 0x3f]
> > + * @entry2: pyramid LUT entry2, range [0x0, 0x3f]
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_anr_stitch_pyramid {
> > + __u32 entry0:6;
> > + __u32 entry1:6;
> > + __u32 entry2:6;
> > + __u32 __reserved:14;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_stitch_config - ANR stitch config
> > + *
> > + * @anr_stitch_en: enable stitch. Enabled with 1.
> > + * @__reserved: reserved
> > + * @pyramid: pyramid table as defined by
> &ipu3_uapi_anr_stitch_pyramid
> > + * default values:
> > + * { 1, 3, 5 }, { 7, 7, 5 }, { 3, 1, 3 },
> > + * { 9, 15, 21 }, { 21, 15, 9 }, { 3, 5, 15 },
> > + * { 25, 35, 35 }, { 25, 15, 5 }, { 7, 21, 35 },
> > + * { 49, 49, 35 }, { 21, 7, 7 }, { 21, 35, 49 },
> > + * { 49, 35, 21 }, { 7, 5, 15 }, { 25, 35, 35 },
> > + * { 25, 15, 5 }, { 3, 9, 15 }, { 21, 21, 15 },
> > + * { 9, 3, 1 }, { 3, 5, 7 }, { 7, 5, 3}, { 1 }
> > + */
> > +struct ipu3_uapi_anr_stitch_config {
> > + __u32 anr_stitch_en;
> > + __u8 __reserved[44];
> > + struct ipu3_uapi_anr_stitch_pyramid
> pyramid[IPU3_UAPI_ANR_PYRAMID_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_config - ANR config
> > + *
> > + * @transform: advanced noise reduction transform config as
> specified by
> > + * &ipu3_uapi_anr_transform_config
> > + * @stitch: create 4x4 patch from 4 surrounding 8x8 patches.
> > + */
> > +struct ipu3_uapi_anr_config {
> > + struct ipu3_uapi_anr_transform_config transform
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_anr_stitch_config stitch __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_acc_param - Accelerator cluster parameters
> > + *
> > + * ACC refers to the HW cluster containing all Fixed Functions(FFs). Each FF
> > + * implements a specific algorithm.
> > + *
> > + * @bnr: parameters for bayer noise reduction static config. See
> > + * &ipu3_uapi_bnr_static_config
> > + * @green_disparity: disparity static config between gr and gb
> channel.
> > + * See &ipu3_uapi_bnr_static_config_green_disparity
> > + * @dm: de-mosaic config. See &ipu3_uapi_dm_config
> > + * @ccm: color correction matrix. See &ipu3_uapi_ccm_mat_config
> > + * @gamma: gamma correction config. See &ipu3_uapi_gamma_config
> > + * @csc: color space conversion matrix. See
> &ipu3_uapi_csc_mat_config
> > + * @cds: color down sample config. See &ipu3_uapi_cds_params
> > + * @shd: lens shading correction config. See &ipu3_uapi_shd_config
> > + * @iefd: Image enhancement filter and denoise config.
> > + * &ipu3_uapi_yuvp1_iefd_config
> > + * @yds_c0: y down scaler config. &ipu3_uapi_yuvp1_yds_config
> > + * @chnr_c0: chroma noise reduction config.
> &ipu3_uapi_yuvp1_chnr_config
> > + * @y_ee_nr: y edge enhancement and noise reduction config.
> > + * &ipu3_uapi_yuvp1_y_ee_nr_config
> > + * @yds: y down scaler config. See &ipu3_uapi_yuvp1_yds_config
> > + * @chnr: chroma noise reduction config. See
> &ipu3_uapi_yuvp1_chnr_config
> > + * @__reserved1: reserved
> > + * @yds2: y channel down scaler config. See
> &ipu3_uapi_yuvp1_yds_config
> > + * @tcc: total color correction config as defined in struct
> > + * &ipu3_uapi_yuvp2_tcc_static_config
> > + * @__reserved2: reserved
> > + * @anr: advanced noise reduction config.See &ipu3_uapi_anr_config
> > + * @awb_fr: AWB filter response config. See ipu3_uapi_awb_fr_config
> > + * @ae: auto exposure config As specified by &ipu3_uapi_ae_config
> > + * @af: auto focus config. As specified by &ipu3_uapi_af_config
> > + * @awb: auto white balance config. As specified by
> &ipu3_uapi_awb_config
> > + */
> > +struct ipu3_uapi_acc_param {
> > + struct ipu3_uapi_bnr_static_config bnr;
> > + struct ipu3_uapi_bnr_static_config_green_disparity
> > + green_disparity __attribute__((aligned(32)));
> > + struct ipu3_uapi_dm_config dm __attribute__((aligned(32)));
> > + struct ipu3_uapi_ccm_mat_config ccm __attribute__((aligned(32)));
> > + struct ipu3_uapi_gamma_config gamma __attribute__((aligned(32)));
> > + struct ipu3_uapi_csc_mat_config csc __attribute__((aligned(32)));
> > + struct ipu3_uapi_cds_params cds __attribute__((aligned(32)));
> > + struct ipu3_uapi_shd_config shd __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_iefd_config iefd __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_yds_config yds_c0
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_chnr_config chnr_c0
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_y_ee_nr_config y_ee_nr
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_yds_config yds __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_chnr_config chnr
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_yds_config yds2 __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp2_tcc_static_config tcc
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_anr_config anr;
> > + struct ipu3_uapi_awb_fr_config awb_fr;
> > + struct ipu3_uapi_ae_config ae;
> > + struct ipu3_uapi_af_config af;
> > + struct ipu3_uapi_awb_config awb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
> > + *
> > + * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
> > + * @lin_lutlow_r: linearization look-up table for R channel interpolation.
> > + * @lin_lutlow_b: linearization look-up table for B channel interpolation.
> > + * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
> > + * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
> > + * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
> > + * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
> > + * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
> > + * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
> > + * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
> > + */
> > +struct ipu3_uapi_isp_lin_vmem_params {
> > + __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
> > +} __packed;
> > +
> > +/* Temporal Noise Reduction */
> > +
> > +/**
> > + * struct ipu3_uapi_isp_tnr3_vmem_params - Temporal noise reduction
> vector
> > + * memory parameters
> > + *
> > + * @slope: slope setting in interpolation curve for temporal noise
> reduction.
> > + * @__reserved1: reserved
> > + * @sigma: knee point setting in interpolation curve for temporal
> > + * noise reduction.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_isp_tnr3_vmem_params {
> > + __u16 slope[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > + __u16 __reserved1[IPU3_UAPI_ISP_VEC_ELEMS
> > + -
> IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > + __u16 sigma[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > + __u16 __reserved2[IPU3_UAPI_ISP_VEC_ELEMS
> > + -
> IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_isp_tnr3_params - Temporal noise reduction v3
> parameters
> > + *
> > + * @knee_y1: Knee point TNR3 assumes standard deviation of Y,U and
> > + * V at Y1 are TnrY1_Sigma_Y, U and V.
> > + * @knee_y2: Knee point TNR3 assumes standard deviation of Y,U and
> > + * V at Y2 are TnrY2_Sigma_Y, U and V.
> > + * @maxfb_y: Max feedback gain for Y
> > + * @maxfb_u: Max feedback gain for U
> > + * @maxfb_v: Max feedback gain for V
> > + * @round_adj_y: rounding Adjust for Y
> > + * @round_adj_u: rounding Adjust for U
> > + * @round_adj_v: rounding Adjust for V
> > + * @ref_buf_select: selection of the reference frame buffer to be used.
> > + */
> > +struct ipu3_uapi_isp_tnr3_params {
> > + __u32 knee_y1;
> > + __u32 knee_y2;
> > + __u32 maxfb_y;
> > + __u32 maxfb_u;
> > + __u32 maxfb_v;
> > + __u32 round_adj_y;
> > + __u32 round_adj_u;
> > + __u32 round_adj_v;
> > + __u32 ref_buf_select;
> > +} __packed;
> > +
> > +/* Extreme Noise Reduction version 3 */
> > +
> > +/**
> > + * struct ipu3_uapi_isp_xnr3_vmem_params - Extreme noise reduction v3
> > + * vector memory parameters
> > + *
> > + * @x: xnr3 parameters.
> > + * @a: xnr3 parameters.
> > + * @b: xnr3 parameters.
> > + * @c: xnr3 parameters.
> > + */
> > +struct ipu3_uapi_isp_xnr3_vmem_params {
> > + __u16 x[IPU3_UAPI_ISP_VEC_ELEMS];
> > + __u16 a[IPU3_UAPI_ISP_VEC_ELEMS];
> > + __u16 b[IPU3_UAPI_ISP_VEC_ELEMS];
> > + __u16 c[IPU3_UAPI_ISP_VEC_ELEMS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_xnr3_alpha_params - Extreme noise reduction v3
> > + * alpha tuning parameters
> > + *
> > + * @y0: Sigma for Y range similarity in dark area.
> > + * @u0: Sigma for U range similarity in dark area.
> > + * @v0: Sigma for V range similarity in dark area.
> > + * @ydiff: Sigma difference for Y between bright area and dark area.
> > + * @udiff: Sigma difference for U between bright area and dark area.
> > + * @vdiff: Sigma difference for V between bright area and dark area.
> > + */
> > +struct ipu3_uapi_xnr3_alpha_params {
> > + __u32 y0;
> > + __u32 u0;
> > + __u32 v0;
> > + __u32 ydiff;
> > + __u32 udiff;
> > + __u32 vdiff;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_xnr3_coring_params - Extreme noise reduction v3
> > + * coring parameters
> > + *
> > + * @u0: Coring Threshold of U channel in dark area.
> > + * @v0: Coring Threshold of V channel in dark area.
> > + * @udiff: Threshold difference of U channel between bright and dark
> area.
> > + * @vdiff: Threshold difference of V channel between bright and dark area.
> > + */
> > +struct ipu3_uapi_xnr3_coring_params {
> > + __u32 u0;
> > + __u32 v0;
> > + __u32 udiff;
> > + __u32 vdiff;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_xnr3_blending_params - Blending factor
> > + *
> > + * @strength: The factor for blending output with input. This is tuning
> > + * parameterHigher values lead to more aggressive XNR operation.
> > + */
> > +struct ipu3_uapi_xnr3_blending_params {
> > + __u32 strength;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_isp_xnr3_params - Extreme noise reduction v3
> parameters
> > + *
> > + * @alpha: parameters for xnr3 alpha. See
> &ipu3_uapi_xnr3_alpha_params
> > + * @coring: parameters for xnr3 coring. See
> &ipu3_uapi_xnr3_coring_params
> > + * @blending: parameters for xnr3 blending. See
> &ipu3_uapi_xnr3_blending_params
> > + */
> > +struct ipu3_uapi_isp_xnr3_params {
> > + struct ipu3_uapi_xnr3_alpha_params alpha;
> > + struct ipu3_uapi_xnr3_coring_params coring;
> > + struct ipu3_uapi_xnr3_blending_params blending;
> > +} __packed;
> > +
> > +/***** Obgrid (optical black level compensation) table entry *****/
> > +
> > +/**
> > + * struct ipu3_uapi_obgrid_param - Optical black level compensation
> parameters
> > + *
> > + * @gr: Grid table values for color GR
> > + * @r: Grid table values for color R
> > + * @b: Grid table values for color B
> > + * @gb: Grid table values for color GB
> > + *
> > + * Black level is different for red, green, and blue channels. So black level
> > + * compensation is different per channel.
> > + */
> > +struct ipu3_uapi_obgrid_param {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > +} __packed;
> > +
> > +/******************* V4L2_META_FMT_IPU3_PARAMS
> *******************/
> > +
> > +/**
> > + * struct ipu3_uapi_flags - bits to indicate which pipeline needs update
> > + *
> > + * @gdc: 0 = no update, 1 = update.
> > + * @obgrid: 0 = no update, 1 = update.
> > + * @__reserved1: Not used.
> > + * @acc_bnr: 0 = no update, 1 = update.
> > + * @acc_green_disparity: 0 = no update, 1 = update.
> > + * @acc_dm: 0 = no update, 1 = update.
> > + * @acc_ccm: 0 = no update, 1 = update.
> > + * @acc_gamma: 0 = no update, 1 = update.
> > + * @acc_csc: 0 = no update, 1 = update.
> > + * @acc_cds: 0 = no update, 1 = update.
> > + * @acc_shd: 0 = no update, 1 = update.
> > + * @__reserved2: Not used.
> > + * @acc_iefd: 0 = no update, 1 = update.
> > + * @acc_yds_c0: 0 = no update, 1 = update.
> > + * @acc_chnr_c0: 0 = no update, 1 = update.
> > + * @acc_y_ee_nr: 0 = no update, 1 = update.
> > + * @acc_yds: 0 = no update, 1 = update.
> > + * @acc_chnr: 0 = no update, 1 = update.
> > + * @acc_ytm: 0 = no update, 1 = update.
> > + * @acc_yds2: 0 = no update, 1 = update.
> > + * @acc_tcc: 0 = no update, 1 = update.
> > + * @acc_dpc: 0 = no update, 1 = update.
> > + * @acc_bds: 0 = no update, 1 = update.
> > + * @acc_anr: 0 = no update, 1 = update.
> > + * @acc_awb_fr: 0 = no update, 1 = update.
> > + * @acc_ae: 0 = no update, 1 = update.
> > + * @acc_af: 0 = no update, 1 = update.
> > + * @acc_awb: 0 = no update, 1 = update.
> > + * @__acc_osys: 0 = no update, 1 = update.
> > + * @__reserved3: Not used.
> > + * @lin_vmem_params: 0 = no update, 1 = update.
> > + * @tnr3_vmem_params: 0 = no update, 1 = update.
> > + * @xnr3_vmem_params: 0 = no update, 1 = update.
> > + * @tnr3_dmem_params: 0 = no update, 1 = update.
> > + * @xnr3_dmem_params: 0 = no update, 1 = update.
> > + * @__reserved4: Not used.
> > + * @obgrid_param: 0 = no update, 1 = update.
> > + * @__reserved5: Not used.
> > + */
> > +struct ipu3_uapi_flags {
> > + __u32 gdc:1;
> > + __u32 obgrid:1;
> > + __u32 __reserved1:30;
> > +
> > + __u32 acc_bnr:1;
> > + __u32 acc_green_disparity:1;
> > + __u32 acc_dm:1;
> > + __u32 acc_ccm:1;
> > + __u32 acc_gamma:1;
> > + __u32 acc_csc:1;
> > + __u32 acc_cds:1;
> > + __u32 acc_shd:1;
> > + __u32 __reserved2:2;
> > + __u32 acc_iefd:1;
> > + __u32 acc_yds_c0:1;
> > + __u32 acc_chnr_c0:1;
> > + __u32 acc_y_ee_nr:1;
> > + __u32 acc_yds:1;
> > + __u32 acc_chnr:1;
> > + __u32 acc_ytm:1;
> > + __u32 acc_yds2:1;
> > + __u32 acc_tcc:1;
> > + __u32 acc_dpc:1;
> > + __u32 acc_bds:1;
> > + __u32 acc_anr:1;
> > + __u32 acc_awb_fr:1;
> > + __u32 acc_ae:1;
> > + __u32 acc_af:1;
> > + __u32 acc_awb:1;
> > + __u32 __reserved3:4;
> > +
> > + __u32 lin_vmem_params:1;
> > + __u32 tnr3_vmem_params:1;
> > + __u32 xnr3_vmem_params:1;
> > + __u32 tnr3_dmem_params:1;
> > + __u32 xnr3_dmem_params:1;
> > + __u32 __reserved4:1;
> > + __u32 obgrid_param:1;
> > + __u32 __reserved5:25;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_params - V4L2_META_FMT_IPU3_PARAMS
> > + *
> > + * @use: select which parameters to apply, see &ipu3_uapi_flags
> > + * @acc_param: ACC parameters, as specified by
> &ipu3_uapi_acc_param
> > + * @lin_vmem_params: linearization VMEM, as specified by
> > + * &ipu3_uapi_isp_lin_vmem_params
> > + * @tnr3_vmem_params: tnr3 VMEM as specified by
> > + * &ipu3_uapi_isp_tnr3_vmem_params
> > + * @xnr3_vmem_params: xnr3 VMEM as specified by
> > + * &ipu3_uapi_isp_xnr3_vmem_params
> > + * @tnr3_dmem_params: tnr3 DMEM as specified by
> &ipu3_uapi_isp_tnr3_params
> > + * @xnr3_dmem_params: xnr3 DMEM as specified by
> &ipu3_uapi_isp_xnr3_params
> > + * @obgrid_param: obgrid parameters as specified by
> > + * &ipu3_uapi_obgrid_param
> > + *
> > + * The video queue "parameters" is of format
> V4L2_META_FMT_IPU3_PARAMS.
> > + * This is a "single plane" v4l2_meta_format using
> V4L2_BUF_TYPE_META_OUTPUT.
> > + *
> > + * struct ipu3_uapi_params as defined below contains a lot of parameters
> and
> > + * ipu3_uapi_flags selects which parameters to apply.
> > + */
> > +struct ipu3_uapi_params {
> > + /* Flags which of the settings below are to be applied */
> > + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> > +
> > + /* Accelerator cluster parameters */
> > + struct ipu3_uapi_acc_param acc_param;
> > +
> > + /* ISP vector address space parameters */
> > + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> > + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> > + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> > +
> > + /* ISP data memory (DMEM) parameters */
> > + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> > + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> > +
> > + /* Optical black level compensation */
> > + struct ipu3_uapi_obgrid_param obgrid_param;
> > +} __packed;
> > +#endif
>
>
>
> Thanks,
> Mauro
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-10-29 22:22 ` [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI Yong Zhi
2018-11-02 13:02 ` Sakari Ailus
2018-11-02 13:49 ` Mauro Carvalho Chehab
@ 2018-11-15 12:51 ` Hans Verkuil
2018-11-21 18:45 ` Zhi, Yong
2 siblings, 1 reply; 123+ messages in thread
From: Hans Verkuil @ 2018-11-15 12:51 UTC (permalink / raw)
To: Yong Zhi, linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Chao C Li
On 10/29/18 23:22, Yong Zhi wrote:
> These meta formats are used on Intel IPU3 ImgU video queues
> to carry 3A statistics and ISP pipeline parameters.
>
> V4L2_META_FMT_IPU3_3A
> V4L2_META_FMT_IPU3_PARAMS
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> Signed-off-by: Chao C Li <chao.c.li@intel.com>
> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> ---
> Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
> include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
> 3 files changed, 3001 insertions(+)
> create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> create mode 100644 include/uapi/linux/intel-ipu3.h
>
> diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
> index cf971d5..eafc534 100644
> --- a/Documentation/media/uapi/v4l/meta-formats.rst
> +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
> .. toctree::
> :maxdepth: 1
>
> + pixfmt-meta-intel-ipu3
> pixfmt-meta-d4xx
> pixfmt-meta-uvc
> pixfmt-meta-vsp1-hgo
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> new file mode 100644
> index 0000000..23b945b
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> @@ -0,0 +1,181 @@
> +.. -*- coding: utf-8; mode: rst -*-
> +
> +.. _intel-ipu3:
> +
> +******************************************************************
> +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
> +******************************************************************
> +
> +.. c:type:: ipu3_uapi_stats_3a
> +
> +3A statistics
> +=============
> +
> +For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
> +an input bayer frame. Those statistics, defined in data struct
> +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu 3a stat"
I'd rephrase this:
are obtained from the "ipu3-imgu 3a stat" metadata capture video node,
> +video node, which are then passed to user space for statistics analysis
> +using :c:type:`v4l2_meta_format` interface.
> +
> +The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
> +Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
> +and AE (Auto-exposure) histogram.
> +
> +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.
> +
> +
> +.. code-block:: c
> +
> +
> + struct ipu3_uapi_stats_3a {
> + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> + __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_raw_buffer_aligned
> + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> + struct ipu3_uapi_4a_config stats_4a_config;
> + __u32 ae_join_buffers;
> + __u8 padding[28];
> + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + stats_3a_bubble_per_stripe;
> + struct ipu3_uapi_ff_status stats_3a_status;
> + } __packed;
> +
> +
> +.. c:type:: ipu3_uapi_params
> +
> +Pipeline parameters
> +===================
> +
> +IPU3 pipeline has a number of image processing stages, each of which takes a
> +set of parameters as input. The major stages of pipelines are shown here:
> +
> +Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
> +
> +Linearization -> Lens Shading Correction -> White Balance / Exposure /
> +
> +Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
> +
> +Correction Matrix -> Gamma correction -> Color Space Conversion ->
> +
> +Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
> +
> +Correction -> XNR3 -> TNR -> DDR
> +
> +The table below presents a description of the above algorithms.
> +
> +======================== =======================================================
> +Name Description
> +======================== =======================================================
> +Optical Black Correction Optical Black Correction block subtracts a pre-defined
> + value from the respective pixel values to obtain better
> + image quality.
> + Defined in :c:type:`ipu3_uapi_obgrid_param`.
> +Linearization This algo block uses linearization parameters to
> + address non-linearity sensor effects. The Lookup table
> + table is defined in
> + :c:type:`ipu3_uapi_isp_lin_vmem_params`.
> +SHD Lens shading correction is used to correct spatial
> + non-uniformity of the pixel response due to optical
> + lens shading. This is done by applying a different gain
> + for each pixel. The gain, black level etc are
> + configured in :c:type:`ipu3_uapi_shd_config_static`.
> +BNR Bayer noise reduction block removes image noise by
> + applying a bilateral filter.
> + See :c:type:`ipu3_uapi_bnr_static_config` for details.
> +ANR Advanced Noise Reduction is a block based algorithm
> + that performs noise reduction in the Bayer domain. The
> + convolution matrix etc can be found in
> + :c:type:`ipu3_uapi_anr_config`.
> +Demosaicing Demosaicing converts raw sensor data in Bayer format
> + into RGB (Red, Green, Blue) presentation. Then add
> + outputs of estimation of Y channel for following stream
"Then add outputs of estimation..."
Seems a bit garbled, I'm not quite sure what is meant here.
> + processing by Firmware. The struct is defined as
> + :c:type:`ipu3_uapi_dm_config`.
> +Color Correction Color Correction algo transforms sensor specific color
> + space to the standard "sRGB" color space. This is done
> + by applying 3x3 matrix defined in
> + :c:type:`ipu3_uapi_ccm_mat_config`.
> +Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
> + basic non-linear tone mapping correction that is
> + applied per pixel for each pixel component.
> +CSC Color space conversion transforms each pixel from the
> + RGB primary presentation to YUV (Y - brightness,
> + UV - Luminance) presentation. This is done by applying
Use Y: and UV: instead of '-' to avoid interpreting this as substraction.
> + a 3x3 matrix defined in
> + :c:type:`ipu3_uapi_csc_mat_config`
> +CDS Chroma down sampling
> + After the CSC is performed, the Chroma Down Sampling
> + is applied for a UV plane down sampling by a factor
> + of 2 in each direction for YUV 4:2:0 using a 4x2
> + configurable filter :c:type:`ipu3_uapi_cds_params`.
> +CHNR Chroma noise reduction
> + This block processes only the chrominance pixels and
> + performs noise reduction by cleaning the high
> + frequency noise.
> + See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
> +TCC Total color correction as defined in struct
> + :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
> +XNR3 eXtreme Noise Reduction V3 is the third revision of
> + noise reduction algorithm used to improve image
> + quality. This removes the low frequency noise in the
> + captured image. Two related structs are being defined,
> + :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
> + and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
> + memory.
> +TNR Temporal Noise Reduction block compares successive
> + frames in time to remove anomalies / noise in pixel
> + values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
> + :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
> + vector and data memory respectively.
> +======================== =======================================================
> +
> +A few stages of the pipeline will be executed by firmware running on the ISP
> +processor, while many others will use a set of fixed hardware blocks also
> +called accelerator cluster (ACC) to crunch pixel data and produce statistics.
> +
> +ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be selectively
> +enabled / disabled by the user space through struct :c:type:`ipu3_uapi_flags`
> +embedded in :c:type:`ipu3_uapi_params` structure. For parameters that are not
> +enabled by the user space, corresponding structs are ignored by the ISP.
> +
> +Both 3A statistics and pipeline parameters described here are closely tied to
> +the underlying camera sub-system (CSS) APIs. They are usually consumed and
> +produced by dedicated user space libraries that comprise the important tuning
> +tools, thus freeing the developers from being bothered with the low level
> +hardware and algorithm details.
> +
> +It should be noted that IPU3 DMA operations require the addresses of all data
> +structures (that includes both input and output) to be aligned on 32 byte
> +boundaries.
> +
> +The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
> +video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
> +
> +.. code-block:: c
> +
> + struct ipu3_uapi_params {
> + /* Flags which of the settings below are to be applied */
> + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> +
> + /* Accelerator cluster parameters */
> + struct ipu3_uapi_acc_param acc_param;
> +
> + /* ISP vector address space parameters */
> + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> +
> + /* ISP data memory (DMEM) parameters */
> + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> +
> + /* Optical black level compensation */
> + struct ipu3_uapi_obgrid_param obgrid_param;
> + } __packed;
> +
> +Intel IPU3 ImgU uAPI data types
> +===============================
> +
> +.. kernel-doc:: include/uapi/linux/intel-ipu3.h
> diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
> new file mode 100644
> index 0000000..c2608b6
> --- /dev/null
> +++ b/include/uapi/linux/intel-ipu3.h
> @@ -0,0 +1,2819 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2017 - 2018 Intel Corporation */
> +
> +#ifndef __IPU3_UAPI_H
> +#define __IPU3_UAPI_H
> +
> +#include <linux/types.h>
> +
> +/********************* Key Acronyms *************************/
> +/*
> + * ACC - Accelerator cluster
> + * ANR - Adaptive noise reduction
> + * AWB_FR- Auto white balance filter response statistics
> + * BNR - Bayer noise reduction parameters
> + * BDS - Bayer downscaler parameters
> + * CCM - Color correction matrix coefficients
> + * CDS - Chroma down sample
> + * CHNR - Chroma noise reduction
> + * CSC - Color space conversion
> + * DM - De-mosaic
> + * IEFd - Image enhancement filter directed
> + * Obgrid - Optical black level compensation
> + * OSYS - Output system configuration
> + * ROI - Region of interest
> + * SHD - Lens shading correction table
> + * TCC - Total color correction
> + * YDS - Y down sampling
> + * YTM - Y-tone mapping
> + */
> +
> +/*
> + * IPU3 DMA operations require buffers to be aligned at
> + * 32 byte boundaries
> + */
> +
> +/******************* ipu3_uapi_stats_3a *******************/
> +
> +#define IPU3_UAPI_MAX_STRIPES 2
> +#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
> +
> +#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
> +#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
> +
> +/* controls generation of meta_data (like FF enable/disable) */
> +#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
> +#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
> +
> +/**
> + * struct ipu3_uapi_grid_config - Grid plane config
> + *
> + * @width: Grid horizontal dimensions, in number of grid blocks(cells).
> + * @height: Grid vertical dimensions, in number of grid cells.
> + * @block_width_log2: Log2 of the width of each cell in pixels.
> + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> + * @block_height_log2: Log2 of the height of each cell in pixels.
> + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> + * @height_per_slice: The number of blocks in vertical axis per slice.
> + * Default 2.
> + * @x_start: X value of top left corner of Region of Interest(ROI).
> + * @y_start: Y value of top left corner of ROI
> + * @x_end: X value of bottom right corner of ROI
> + * @y_end: Y value of bottom right corner of ROI
> + *
> + * Due to the size of total amount of collected data, most statistics
> + * create a grid-based output, and the data is then divided into "slices".
> + */
> +struct ipu3_uapi_grid_config {
> + __u8 width;
> + __u8 height;
> + __u16 block_width_log2:3;
> + __u16 block_height_log2:3;
> + __u16 height_per_slice:8;
> + __u16 x_start;
> + __u16 y_start;
> + __u16 x_end;
> + __u16 y_end;
> +} __packed;
> +
> +/*
> + * The grid based data is divided into "slices" called set, each slice of setX
> + * refers to ipu3_uapi_grid_config width * height_per_slice.
> + */
> +#define IPU3_UAPI_AWB_MAX_SETS 60
> +/* Based on grid size 80 * 60 and cell size 16 x 16 */
> +#define IPU3_UAPI_AWB_SET_SIZE 1280
> +#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
> +#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AWB_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
> + (IPU3_UAPI_AWB_MAX_SETS * \
> + (IPU3_UAPI_AWB_SET_SIZE + IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
Add an empty line here.
> +/**
> + * struct ipu3_uapi_awb_meta_data - AWB meta data
> + *
> + * @meta_data_buffer: Average values for each color channel
> + */
> +struct ipu3_uapi_awb_meta_data {
> + __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
> + *
> + * @meta_data: buffer to hold auto white balance meta data.
> + */
> +struct ipu3_uapi_awb_raw_buffer {
> + struct ipu3_uapi_awb_meta_data meta_data;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_config_s - AWB config
> + *
> + * @rgbs_thr_gr: gr threshold value.
> + * @rgbs_thr_r: Red threshold value.
> + * @rgbs_thr_gb: gb threshold value.
> + * @rgbs_thr_b: Blue threshold value.
> + * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
> + *
> + * The threshold is a saturation measure range [0, 8191], 8191 is default.
> + * Values over threshold may be optionally rejected for averaging.
> + */
> +struct ipu3_uapi_awb_config_s {
> + __u16 rgbs_thr_gr;
> + __u16 rgbs_thr_r;
> + __u16 rgbs_thr_gb;
> + __u16 rgbs_thr_b;
> + struct ipu3_uapi_grid_config grid;
> +} __attribute__((aligned(32))) __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_config - AWB config wrapper
> + *
> + * @config: config for auto white balance as defined by &ipu3_uapi_awb_config_s
> + */
> +struct ipu3_uapi_awb_config {
> + struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
> +} __packed;
> +
> +#define IPU3_UAPI_AE_COLORS 4 /* R, G, B, Y */
> +#define IPU3_UAPI_AE_BINS 256
> +#define IPU3_UAPI_AE_WEIGHTS 96
> +
> +/**
> + * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
> + *
> + * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
> + *
> + * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit unsigned
> + * for counting the number of the pixel.
> + */
> +struct ipu3_uapi_ae_raw_buffer {
> + __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
> + *
> + * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
> + */
> +struct ipu3_uapi_ae_raw_buffer_aligned {
> + struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_grid_config - AE weight grid
> + *
> + * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
> + * @height: Grid vertical dimensions. Value: [16, 24], default 16.
> + * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
> + * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
> + * @__reserved0: reserved
> + * @ae_en: 0: does not write to meta-data array, 1: write normally.
> + * @rst_hist_array: write 1 to trigger histogram array reset.
> + * @done_rst_hist_array: flag for histogram array reset done.
> + * @x_start: X value of top left corner of ROI, default 0.
> + * @y_start: Y value of top left corner of ROI, default 0.
> + * @x_end: X value of bottom right corner of ROI
> + * @y_end: Y value of bottom right corner of ROI
> + *
> + * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
> + * a defined ROI within the frame. The contribution of each pixel into the
> + * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a grid.
> + */
> +struct ipu3_uapi_ae_grid_config {
> + __u8 width;
> + __u8 height;
> + __u8 block_width_log2:4;
> + __u8 block_height_log2:4;
> + __u8 __reserved0:5;
> + __u8 ae_en:1;
> + __u8 rst_hist_array:1;
> + __u8 done_rst_hist_array:1;
> + __u16 x_start;
> + __u16 y_start;
> + __u16 x_end;
> + __u16 y_end;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_weight_elem - AE weights LUT
> + *
> + * @cell0: weighted histogram grid value.
> + * @cell1: weighted histogram grid value.
> + * @cell2: weighted histogram grid value.
> + * @cell3: weighted histogram grid value.
> + * @cell4: weighted histogram grid value.
> + * @cell5: weighted histogram grid value.
> + * @cell6: weighted histogram grid value.
> + * @cell7: weighted histogram grid value.
> + *
> + * Use weighted grid value to give a different contribution factor to each cell.
> + * Precision u4, range [0, 15].
> + */
> +struct ipu3_uapi_ae_weight_elem {
> + __u32 cell0:4;
> + __u32 cell1:4;
> + __u32 cell2:4;
> + __u32 cell3:4;
> + __u32 cell4:4;
> + __u32 cell5:4;
> + __u32 cell6:4;
> + __u32 cell7:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
> + *
> + * @gain_gr: WB gain factor for the gr channels. Default 256.
> + * @gain_r: WB gain factor for the r channel. Default 256.
> + * @gain_b: WB gain factor for the b channel. Default 256.
> + * @gain_gb: WB gain factor for the gb channels. Default 256.
> + * @mat: 4x4 matrix that transforms Bayer quad output from WB to RGB+Y.
> + *
> + * Default:
> + * 128, 0, 0, 0,
> + * 0, 128, 0, 0,
> + * 0, 0, 128, 0,
> + * 0, 0, 0, 128,
> + *
> + * As part of the raw frame pre-process stage, the WB and color conversion need
> + * to be applied to expose the impact of these gain operations.
> + */
> +struct ipu3_uapi_ae_ccm {
> + __u16 gain_gr;
> + __u16 gain_r;
> + __u16 gain_b;
> + __u16 gain_gb;
> + __s16 mat[16];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ae_config - AE config
> + *
> + * @grid_cfg: config for auto exposure statistics grid. See struct
> + * &ipu3_uapi_ae_grid_config
> + * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks in the grid.
> + * Each grid cell has a corresponding value in weights LUT called
> + * grid value, global histogram is updated based on grid value and
> + * pixel value.
> + * @ae_ccm: Color convert matrix pre-processing block.
> + *
> + * Calculate AE grid from image resolution, resample ae weights.
> + */
> +struct ipu3_uapi_ae_config {
> + struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_weight_elem weights[
> + IPU3_UAPI_AE_WEIGHTS] __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast measurements
> + *
> + * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
> + * anti-symmetry type. A12 is center, A1-A11 are neighbours.
> + * for analyzing low frequency content, used to calculate sum
> + * of gradients in x direction.
> + * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
> + * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
> + * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
> + * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
> + * @y1_coeff_1: Struct
> + * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
> + * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
> + * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
> + * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
> + * @y1_coeff_2: Struct
> + * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> + * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> + * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> + * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> + * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
> + * 0: positive, 1: negative, default 0.
> + * @y2_coeff_0: Y2, same structure as Y1. For analyzing high frequency content.
> + * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
> + * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
> + * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
> + * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
> + * @y2_coeff_1: Struct
> + * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
> + * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
> + * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
> + * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
> + * @y2_coeff_2: Struct
> + * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> + * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> + * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> + * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> + * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
> + * 0: positive, 1: negative, default 0.
> + * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to be
> + * used for building histogram. Range [0, 32], default 8.
> + * Rule:
> + * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b + y_gen_rate_gb = 32
> + * A single Y is calculated based on sum of Gr/R/B/Gb based on
> + * their contribution ratio.
> + * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
> + * @y_calc.y_gen_rate_r: Contribution ratio R for Y
> + * @y_calc.y_gen_rate_b: Contribution ratio B for Y
> + * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
> + * @nf: The shift right value that should be applied during the Y1/Y2 filter to
> + * make sure the total memory needed is 2 bytes per grid cell.
> + * @nf.__reserved0: reserved
> + * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
> + * should be log2 of the sum of the abs values of the filter
> + * coeffs, default 7 (2^7 = 128).
> + * @nf.__reserved1: reserved
> + * @nf.y2_nf: Normalization factor for y2, should be log2 of the sum of the
> + * abs values of the filter coeffs.
> + * @nf.__reserved2: reserved
> + */
> +struct ipu3_uapi_af_filter_config {
> + struct {
> + __u8 a1;
> + __u8 a2;
> + __u8 a3;
> + __u8 a4;
> + } y1_coeff_0;
> + struct {
> + __u8 a5;
> + __u8 a6;
> + __u8 a7;
> + __u8 a8;
> + } y1_coeff_1;
> + struct {
> + __u8 a9;
> + __u8 a10;
> + __u8 a11;
> + __u8 a12;
> + } y1_coeff_2;
> +
> + __u32 y1_sign_vec;
> +
> + struct {
> + __u8 a1;
> + __u8 a2;
> + __u8 a3;
> + __u8 a4;
> + } y2_coeff_0;
> + struct {
> + __u8 a5;
> + __u8 a6;
> + __u8 a7;
> + __u8 a8;
> + } y2_coeff_1;
> + struct {
> + __u8 a9;
> + __u8 a10;
> + __u8 a11;
> + __u8 a12;
> + } y2_coeff_2;
> +
> + __u32 y2_sign_vec;
> +
> + struct {
> + __u8 y_gen_rate_gr;
> + __u8 y_gen_rate_r;
> + __u8 y_gen_rate_b;
> + __u8 y_gen_rate_gb;
> + } y_calc;
> +
> + struct {
> + __u32 __reserved0:8;
> + __u32 y1_nf:4;
> + __u32 __reserved1:4;
> + __u32 y2_nf:4;
> + __u32 __reserved2:12;
> + } nf;
> +} __packed;
> +
> +#define IPU3_UAPI_AF_MAX_SETS 24
> +#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
> +#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AF_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
> +#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
> + (IPU3_UAPI_AF_MAX_SETS * \
> + (IPU3_UAPI_AF_Y_TABLE_SET_SIZE + IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
> + IPU3_UAPI_MAX_STRIPES)
> +
> +/**
> + * struct ipu3_uapi_af_meta_data - AF meta data
> + *
> + * @y_table: Each color component will be convolved separately with filter1
> + * and filter2 and the result will be summed out and averaged for
> + * each cell.
> + */
> +struct ipu3_uapi_af_meta_data {
> + __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE] __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_raw_buffer - AF raw buffer
> + *
> + * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus meta data.
> + */
> +struct ipu3_uapi_af_raw_buffer {
> + struct ipu3_uapi_af_meta_data meta_data __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_config_s - AF config
> + *
> + * @filter_config: AF uses Y1 and Y2 filters as configured in
> + * &ipu3_uapi_af_filter_config
> + * @padding: paddings
> + * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use large
> + * grid size for large image and vice versa.
> + */
> +struct ipu3_uapi_af_config_s {
> + struct ipu3_uapi_af_filter_config filter_config __attribute__((aligned(32)));
> + __u8 padding[4];
> + struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_af_config - AF config wrapper
> + *
> + * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
> + */
> +struct ipu3_uapi_af_config {
> + struct ipu3_uapi_af_config_s config;
> +} __packed;
> +
> +#define IPU3_UAPI_AWB_FR_MAX_SETS 24
> +#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
> +#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
> +#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
> + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> + IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
> +#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
> + (IPU3_UAPI_AWB_FR_MAX_SETS * \
> + (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
> + IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) * IPU3_UAPI_MAX_STRIPES)
> +
> +/**
> + * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
> + *
> + * @bayer_table: Statistics output on the grid after convolving with 1D filter.
> + */
> +struct ipu3_uapi_awb_fr_meta_data {
> + __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE] __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
> + *
> + * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
> + */
> +struct ipu3_uapi_awb_fr_raw_buffer {
> + struct ipu3_uapi_awb_fr_meta_data meta_data;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
> + *
> + * @grid_cfg: grid config, default 16x16.
> + * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
> + * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
> + * Applied on whole image for each Bayer channel separately
> + * by a weighted sum of its 11x1 neighbors.
> + * @__reserved1: reserved
> + * @bayer_sign: sign of filter coeffcients, default 0.
> + * @bayer_nf: normalization factor for the convolution coeffs, to make sure
> + * total memory needed is within pre-determined range.
> + * NF should be the log2 of the sum of the abs values of the
> + * filter coeffs, range [7, 14], default 7.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_awb_fr_config_s {
> + struct ipu3_uapi_grid_config grid_cfg;
> + __u8 bayer_coeff[6];
> + __u16 __reserved1;
> + __u32 bayer_sign;
> + __u8 bayer_nf;
> + __u8 __reserved2[3];
> +} __attribute__((aligned(32))) __packed;
> +
> +/**
> + * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
> + *
> + * @config: See &ipu3_uapi_awb_fr_config_s.
> + */
> +struct ipu3_uapi_awb_fr_config {
> + struct ipu3_uapi_awb_fr_config_s config;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_4a_config - 4A config
> + *
> + * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
> + * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
> + * @padding: paddings
> + * @af_config: auto focus config &ipu3_uapi_af_config_s
> + * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution 16x16
> + */
> +struct ipu3_uapi_4a_config {
> + struct ipu3_uapi_awb_config_s awb_config __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_grid_config ae_grd_config;
> + __u8 padding[20];
> + struct ipu3_uapi_af_config_s af_config;
> + struct ipu3_uapi_awb_fr_config_s awb_fr_config;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
> + *
> + * @num_of_stripes: A single frame is divided into several parts called stripes
> + * due to limitation on line buffer memory.
> + * The separation between the stripes is vertical. Each such
> + * stripe is processed as a single frame by the ISP pipe.
> + * @padding: padding bytes.
> + * @num_sets: number of sets.
> + * @padding1: padding bytes.
> + * @size_of_set: set size.
> + * @padding2: padding bytes.
> + * @bubble_size: is the amount of padding in the bubble expressed in "sets".
> + * @padding3: padding bytes.
> + */
> +struct ipu3_uapi_bubble_info {
> + __u32 num_of_stripes __attribute__((aligned(32)));
> + __u8 padding[28];
> + __u32 num_sets;
> + __u8 padding1[28];
> + __u32 size_of_set;
> + __u8 padding2[28];
> + __u32 bubble_size;
> + __u8 padding3[28];
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + */
> +struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
> + struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
> + *
> + * @awb_en: auto white balance enable
> + * @padding: padding config
> + * @ae_en: auto exposure enable
> + * @padding1: padding config
> + * @af_en: auto focus enable
> + * @padding2: padding config
> + * @awb_fr_en: awb filter response enable bit
> + * @padding3: padding config
> + */
> +struct ipu3_uapi_ff_status {
> + __u32 awb_en __attribute__((aligned(32)));
> + __u8 padding[28];
> + __u32 ae_en;
> + __u8 padding1[28];
> + __u32 af_en;
> + __u8 padding2[28];
> + __u32 awb_fr_en;
> + __u8 padding3[28];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_stats_3a - 3A statistics
> + *
> + * @awb_raw_buffer: auto white balance meta data &ipu3_uapi_awb_raw_buffer
> + * @ae_raw_buffer: auto exposure raw data &ipu3_uapi_ae_raw_buffer_aligned
> + * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
> + * @awb_fr_raw_buffer: value as specified by &ipu3_uapi_awb_fr_raw_buffer
> + * @stats_4a_config: 4a statistics config as defined by &ipu3_uapi_4a_config.
> + * @ae_join_buffers: 1 to use ae_raw_buffer.
> + * @padding: padding config
> + * @stats_3a_bubble_per_stripe: a &ipu3_uapi_stats_3a_bubble_info_per_stripe
> + * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
> + */
> +struct ipu3_uapi_stats_3a {
> + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer __attribute__((aligned(32)));
> + struct ipu3_uapi_ae_raw_buffer_aligned
> + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> + struct ipu3_uapi_4a_config stats_4a_config;
> + __u32 ae_join_buffers;
> + __u8 padding[28];
> + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> + stats_3a_bubble_per_stripe;
> + struct ipu3_uapi_ff_status stats_3a_status;
> +} __packed;
> +
> +/******************* ipu3_uapi_acc_param *******************/
> +
> +#define IPU3_UAPI_ISP_VEC_ELEMS 64
> +#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
> +
> +#define IPU3_UAPI_BNR_LUT_SIZE 32
> +
> +/* number of elements in gamma correction LUT */
> +#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
> +
> +/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
> +#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
> +#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
> +/* Normalization shift aka nf */
> +#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
> +#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
> +
> +#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
> +#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
> +#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
> +#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
> +
> +#define IPU3_UAPI_ANR_LUT_SIZE 26
> +#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
> +
> +#define IPU3_UAPI_LIN_LUT_SIZE 64
> +
> +/* Bayer Noise Reduction related structs */
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance gains
> + *
> + * @gr: white balance gain for Gr channel.
> + * @r: white balance gain for R channel.
> + * @b: white balance gain for B channel.
> + * @gb: white balance gain for Gb channel.
> + *
> + * Precision u3.13, range [0, 8]. White balance correction is done by applying
> + * a multiplicative gain to each color channels prior to BNR.
> + */
> +struct ipu3_uapi_bnr_static_config_wb_gains_config {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold config
> + *
> + * @gr: white balance threshold gain for Gr channel.
> + * @r: white balance threshold gain for R channel.
> + * @b: white balance threshold gain for B channel.
> + * @gb: white balance threshold gain for Gb channel.
> + *
> + * Defines the threshold that specifies how different a defect pixel can be from
> + * its neighbors.(used by dynamic defect pixel correction sub block)
> + * Precision u4.4 range [0, 8].
> + */
> +struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
> + __u8 gr;
> + __u8 r;
> + __u8 b;
> + __u8 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
> + * coefficients that controls noise threshold
> + *
> + * @cf: Free coefficient for threshold calculation, range [0, 8191], default 0.
> + * @__reserved0: reserved
> + * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
> + * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
> + * default 6.
> + * format: u3.2 (3 most significant bits represent whole number,
> + * 2 least significant bits represent the fractional part
> + * with each count representing 0.25)
> + * e.g 6 in binary format is 00110, that translates to 1.5
> + * @__reserved1: reserved
Why use '__' prefix? It seems to me that calling it 'reserved1' is just fine.
Unless there is some good reason for this, please remove the __ from such
bitfields in this header. The use of '__' is reserved for compilers etc.
> + * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
> + * where r is a radius of pixel [row, col] from centor of sensor.
> + * default 14.
> + *
> + * Threshold used to distinguish between noise and details.
> + */
> +struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
> + __u32 cf:13;
> + __u32 __reserved0:3;
> + __u32 cg:5;
> + __u32 ci:5;
> + __u32 __reserved1:1;
> + __u32 r_nf:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
> + *
> + * @gr: Coefficient defines lens shading gain approximation for gr channel
> + * @r: Coefficient defines lens shading gain approximation for r channel
> + * @b: Coefficient defines lens shading gain approximation for b channel
> + * @gb: Coefficient defines lens shading gain approximation for gb channel
> + *
> + * Parameters for noise model (NM) adaptation of BNR due to shading correction.
> + * All above have precision of u3.3, default to 0.
> + */
> +struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
> + __u8 gr;
> + __u8 r;
> + __u8 b;
> + __u8 gb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center config
> + *
> + * @x_reset: Reset value of X (col start - X center). Precision s12.0.
> + * @__reserved0: reserved
> + * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
> + * @__reserved2: reserved
> + *
> + * Distance from corner to optical center for NM adaptation due to shading
> + * correction (should be calculated based on shading tables)
> + */
> +struct ipu3_uapi_bnr_static_config_opt_center_config {
> + __s32 x_reset:13;
> + __u32 __reserved0:3;
> + __s32 y_reset:13;
> + __u32 __reserved2:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup table
> + *
> + * @values: pre-calculated values of square root function.
> + *
> + * LUT implementation of square root operation.
> + */
> +struct ipu3_uapi_bnr_static_config_lut_config {
> + __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels (bp)
> + *
> + * @bp_thr_gain: Defines the threshold that specifies how different a
> + * defect pixel can be from its neighbors. Threshold is
> + * dependent on de-noise threshold calculated by algorithm.
> + * Range [4, 31], default 4.
> + * @__reserved0: reserved
> + * @defect_mode: Mode of addressed defect pixels,
> + * 0 - single defect pixel is expected,
> + * 1 - 2 adjacent defect pixels are expected, default 1.
> + * @bp_gain: Defines how 2nd derivation that passes through a defect pixel
> + * is different from 2nd derivations that pass through
> + * neighbor pixels. u4.2, range [0, 256], default 8.
> + * @__reserved1: reserved
> + * @w0_coeff: Blending coefficient of defect pixel correction.
> + * Precision u4, range [0, 8], default 8.
> + * @__reserved2: reserved
> + * @w1_coeff: Enable influence of incorrect defect pixel correction to be
> + * avoided. Precision u4, range [1, 8], default 8.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
> + __u32 bp_thr_gain:5;
> + __u32 __reserved0:2;
> + __u32 defect_mode:1;
> + __u32 bp_gain:6;
> + __u32 __reserved1:18;
> + __u32 w0_coeff:4;
> + __u32 __reserved2:4;
> + __u32 w1_coeff:4;
> + __u32 __reserved3:20;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising config
> + *
> + * @alpha: Weight of central element of smoothing filter.
> + * @beta: Weight of peripheral elements of smoothing filter, default 4.
> + * @gamma: Weight of diagonal elements of smoothing filter, default 4.
> + *
> + * beta and gamma parameter define the strength of the noise removal filter.
> + * All above has precision u0.4, range [0, 0xf]
> + * format: u0.4 (no / zero bits represent whole number,
> + * 4 bits represent the fractional part
> + * with each count representing 0.0625)
> + * e.g 0xf translates to 0.0625x15 = 0.9375
e.g -> e.g.
> + *
> + * @__reserved0: reserved
> + * @max_inf: Maximum increase of peripheral or diagonal element influence
> + * relative to the pre-defined value range: [0x5, 0xa]
> + * @__reserved1: reserved
> + * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
> + * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 - enable.
> + * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 - enable.
> + * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
> + __u32 alpha:4;
> + __u32 beta:4;
> + __u32 gamma:4;
> + __u32 __reserved0:4;
> + __u32 max_inf:4;
> + __u32 __reserved1:7;
> + __u32 gd_enable:1;
> + __u32 bpc_enable:1;
> + __u32 bnr_enable:1;
> + __u32 ff_enable:1;
> + __u32 __reserved2:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical square
> + *
> + * @x_sqr_reset: Reset value of X^2.
> + * @y_sqr_reset: Reset value of Y^2.
> + *
> + * Please note:
> + *
> + * #. X and Y ref to
> + * &ipu3_uapi_bnr_static_config_opt_center_config
> + * #. Both structs are used in threshold formula to calculate r^2, where r
> + * is a radius of pixel [row, col] from centor of sensor.
> + */
> +struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
> + __u32 x_sqr_reset;
> + __u32 y_sqr_reset;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config - BNR static config
> + *
> + * @wb_gains: white balance gains &ipu3_uapi_bnr_static_config_wb_gains_config
> + * @wb_gains_thr: white balance gains threshold as defined by
> + * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
> + * @thr_coeffs: coefficients of threshold
> + * &ipu3_uapi_bnr_static_config_thr_coeffs_config
> + * @thr_ctrl_shd: control of shading threshold
> + * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
> + * @opt_center: optical center &ipu3_uapi_bnr_static_config_opt_center_config
> + *
> + * Above parameters and opt_center_sqr are used for white balance and shading.
> + *
> + * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
> + * @bp_ctrl: detect and remove bad pixels as defined in struct
> + * &ipu3_uapi_bnr_static_config_bp_ctrl_config
> + * @dn_detect_ctrl: detect and remove noise.
> + * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> + * @column_size: The number of pixels in column.
> + * @opt_center_sqr: Reset value of r^2 to optical center, see
> + * &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
> + */
> +struct ipu3_uapi_bnr_static_config {
> + struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
> + struct ipu3_uapi_bnr_static_config_wb_gains_thr_config wb_gains_thr;
> + struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
> + struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
> + struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
> + struct ipu3_uapi_bnr_static_config_lut_config lut;
> + struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
> + struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config dn_detect_ctrl;
> + __u32 column_size;
> + struct ipu3_uapi_bnr_static_config_opt_center_sqr_config opt_center_sqr;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green disparity
> + *
> + * @gd_red: Shading gain coeff for gr disparity level in bright red region.
> + * Precision u0.6, default 4(0.0625).
> + * @__reserved0: reserved
> + * @gd_green: Shading gain coeff for gr disparity level in bright green
> + * region. Precision u0.6, default 4(0.0625).
> + * @__reserved1: reserved
> + * @gd_blue: Shading gain coeff for gr disparity level in bright blue region.
> + * Precision u0.6, default 4(0.0625).
> + * @__reserved2: reserved
> + * @gd_black: Maximal green disparity level in dark region (stronger disparity
> + * assumed to be image detail). Precision u14, default 80.
> + * @__reserved3: reserved
> + * @gd_shading: Change maximal green disparity level according to square
> + * distance from image center.
> + * @__reserved4: reserved
> + * @gd_support: Lower bound for the number of second green color pixels in
> + * current pixel neighborhood with less than threshold difference
> + * from it.
> + *
> + * The shading gain coeff of red, green, blue and black are used to calculate
> + * threshold given a pixel's color value and its coordinates in the image.
> + *
> + * @__reserved5: reserved
> + * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
> + * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
> + */
> +struct ipu3_uapi_bnr_static_config_green_disparity {
> + __u32 gd_red:6;
> + __u32 __reserved0:2;
> + __u32 gd_green:6;
> + __u32 __reserved1:2;
> + __u32 gd_blue:6;
> + __u32 __reserved2:10;
> + __u32 gd_black:14;
> + __u32 __reserved3:2;
> + __u32 gd_shading:7;
> + __u32 __reserved4:1;
> + __u32 gd_support:2;
> + __u32 __reserved5:1;
> + __u32 gd_clip:1;
> + __u32 gd_central_weight:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_dm_config - De-mosaic parameters
> + *
> + * @dm_en: de-mosaic enable.
> + * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
> + * @fcc_en: False color correction (FCC) enable flag. Default 0.
> + * @__reserved0: reserved
> + * @frame_width: do not care
> + * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation of
> + * complementary color in Hamilton-Adams interpolation).
> + * u5, range [0, 31], default 8.
> + * @__reserved1: reserved
> + * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity metric
> + * in calculation of final homogeneity metric.
> + * u5, range [0, 31], default 7.
> + * @__reserved2: reserved
> + * @cr_param1: First parameter that defines Checker artifact removal
> + * feature gain.Precision u5, range [0, 31], default 8.
Add space before 'Precision'
> + * @__reserved3: reserved
> + * @cr_param2: Second parameter that defines Checker artifact removal
> + * feature gain. Precision u5, range [0, 31], default 8.
> + * @__reserved4: reserved
> + * @coring_param: Defines power of false color correction operation.
> + * low for preserving edge colors, high for preserving gray
> + * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
Add space after "default 4".
> + * @__reserved5: reserved
> + *
> + * The demosaic fixed function block is responsible to covert Bayer(mosaiced)
> + * images into color images based on demosaicing algorithm.
> + */
> +struct ipu3_uapi_dm_config {
> + __u32 dm_en:1;
> + __u32 ch_ar_en:1;
> + __u32 fcc_en:1;
> + __u32 __reserved0:13;
> + __u32 frame_width:16;
> +
> + __u32 gamma_sc:5;
> + __u32 __reserved1:3;
> + __u32 lc_ctrl:5;
> + __u32 __reserved2:3;
> + __u32 cr_param1:5;
> + __u32 __reserved3:3;
> + __u32 cr_param2:5;
> + __u32 __reserved4:3;
> +
> + __u32 coring_param:5;
> + __u32 __reserved5:27;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_ccm_mat_config - Color correction matrix
> + *
> + * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
> + * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
> + * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
> + * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
> + * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
> + * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
> + * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
> + *
> + * Transform sensor specific color space to standard sRGB by applying 3x3 matrix
> + * and adding a bias vector O. The transformation is basically a rotation and
> + * translation in the 3-dimensional color spaces. Here are the defaults:
> + *
> + * 9775, -2671, 1087, 0
> + * -1071, 8303, 815, 0
> + * -23, -7887, 16103, 0
> + */
> +struct ipu3_uapi_ccm_mat_config {
> + __s16 coeff_m11;
> + __s16 coeff_m12;
> + __s16 coeff_m13;
> + __s16 coeff_o_r;
> + __s16 coeff_m21;
> + __s16 coeff_m22;
> + __s16 coeff_m23;
> + __s16 coeff_o_g;
> + __s16 coeff_m31;
> + __s16 coeff_m32;
> + __s16 coeff_m33;
> + __s16 coeff_o_b;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
> + *
> + * @enable: gamma correction enable.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_gamma_corr_ctrl {
> + __u32 enable:1;
> + __u32 __reserved:31;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping implemented as LUT.
> + *
> + * @lut: 256 tabulated values of the gamma function. LUT[1].. LUT[256]
> + * format u13.0, range [0, 8191].
> + *
> + * The tone mapping operation is done by a Piece wise linear graph
> + * that is implemented as a lookup table(LUT). The pixel component input
> + * intensity is the X-axis of the graph which is the table entry.
> + */
> +struct ipu3_uapi_gamma_corr_lut {
> + __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_gamma_config - Gamma config
> + *
> + * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
> + * @gc_lut: lookup table of gamma correction &ipu3_uapi_gamma_corr_lut
> + */
> +struct ipu3_uapi_gamma_config {
> + struct ipu3_uapi_gamma_corr_ctrl gc_ctrl __attribute__((aligned(32)));
> + struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
> + *
> + * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1], default 0.
> + * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1], default 1.
> + * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181], default 0.
> + *
> + * To transform each pixel from RGB to YUV (Y - brightness/luminance,
> + * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding an
> + * optional bias 3x1 vector.
> + */
> +struct ipu3_uapi_csc_mat_config {
> + __s16 coeff_c11;
> + __s16 coeff_c12;
> + __s16 coeff_c13;
> + __s16 coeff_b1;
> + __s16 coeff_c21;
> + __s16 coeff_c22;
> + __s16 coeff_c23;
> + __s16 coeff_b2;
> + __s16 coeff_c31;
> + __s16 coeff_c32;
> + __s16 coeff_c33;
> + __s16 coeff_b3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cds_params - Chroma down-scaling
> + *
> + * @ds_c00: range [0, 3]
> + * @ds_c01: range [0, 3]
> + * @ds_c02: range [0, 3]
> + * @ds_c03: range [0, 3]
> + * @ds_c10: range [0, 3]
> + * @ds_c11: range [0, 3]
> + * @ds_c12: range [0, 3]
> + * @ds_c13: range [0, 3]
> + *
> + * In case user does not provide, above 4x2 filter will use following defaults:
> + * 1, 3, 3, 1,
> + * 1, 3, 3, 1,
> + *
> + * @ds_nf: Normalization factor for Chroma output downscaling filter,
> + * range 0,4, default 2.
> + * @__reserved0: reserved
> + * @csc_en: Color space conversion enable
> + * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_cds_params {
> + __u32 ds_c00:2;
> + __u32 ds_c01:2;
> + __u32 ds_c02:2;
> + __u32 ds_c03:2;
> + __u32 ds_c10:2;
> + __u32 ds_c11:2;
> + __u32 ds_c12:2;
> + __u32 ds_c13:2;
> + __u32 ds_nf:5;
> + __u32 __reserved0:3;
> + __u32 csc_en:1;
> + __u32 uv_bin_output:1;
> + __u32 __reserved1:6;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
> + *
> + * @width: Grid horizontal dimensions, u8, [8, 128], default 73
> + * @height: Grid vertical dimensions, u8, [8, 128], default 56
> + * @block_width_log2: Log2 of the width of the grid cell in pixel count
> + * u4, [0, 15], default value 5.
> + * @__reserved0: reserved
> + * @block_height_log2: Log2 of the height of the grid cell in pixel count
> + * u4, [0, 15], default value 6.
> + * @__reserved1: reserved
> + * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
> + * (with SHD_MAX_CELLS_PER_SET = 146).
> + * @x_start: X value of top left corner of sensor relative to ROI
> + * u12, [-4096, 0]. default 0, only negative values.
. default -> , default
> + * @y_start: Y value of top left corner of sensor relative to ROI
> + * u12, [-4096, 0]. default 0, only negative values.
ditto.
> + */
> +struct ipu3_uapi_shd_grid_config {
> + /* reg 0 */
> + __u8 width;
> + __u8 height;
> + __u8 block_width_log2:3;
> + __u8 __reserved0:1;
> + __u8 block_height_log2:3;
> + __u8 __reserved1:1;
> + __u8 grid_height_per_slice;
> + /* reg 1 */
> + __s16 x_start;
> + __s16 y_start;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_general_config - Shading general config
> + *
> + * @init_set_vrt_offst_ul: set vertical offset,
> + * y_start >> block_height_log2 % grid_height_per_slice.
> + * @shd_enable: shading enable.
> + * @gain_factor: Gain factor. Shift calculated anti shading value. Precision u2.
> + * 0x0 - gain factor [1, 5], means no shift interpolated value.
> + * 0x1 - gain factor [1, 9], means shift interpolated by 1.
> + * 0x2 - gain factor [1, 17], means shift interpolated by 2.
> + * @__reserved: reserved
> + *
> + * Correction is performed by multiplying a gain factor for each of the 4 Bayer
> + * channels as a function of the pixel location in the sensor.
> + */
> +struct ipu3_uapi_shd_general_config {
> + __u32 init_set_vrt_offst_ul:8;
> + __u32 shd_enable:1;
> + __u32 gain_factor:2;
> + __u32 __reserved:21;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_black_level_config - Black level correction
> + *
> + * @bl_r: Bios values for green red. s11 range [-2048, 2047].
> + * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
> + * @bl_gb: Bios values for red. s11 range [-2048, 2047].
> + * @bl_b: Bios values for blue. s11 range [-2048, 2047].
> + */
> +struct ipu3_uapi_shd_black_level_config {
> + __s16 bl_r;
> + __s16 bl_gr;
> + __s16 bl_gb;
> + __s16 bl_b;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_config_static - Shading config static
> + *
> + * @grid: shading grid config &ipu3_uapi_shd_grid_config
> + * @general: shading general config &ipu3_uapi_shd_general_config
> + * @black_level: black level config for shading correction as defined by
> + * &ipu3_uapi_shd_black_level_config
> + */
> +struct ipu3_uapi_shd_config_static {
> + struct ipu3_uapi_shd_grid_config grid;
> + struct ipu3_uapi_shd_general_config general;
> + struct ipu3_uapi_shd_black_level_config black_level;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
> + *
> + * @sets: array
> + * @sets.r_and_gr: Red and GreenR Lookup table.
> + * @sets.r_and_gr.r: Red shading factor.
> + * @sets.r_and_gr.gr: GreenR shading factor.
> + * @sets.__reserved1: reserved
> + * @sets.gb_and_b: GreenB and Blue Lookup table.
> + * @sets.gb_and_b.gb: GreenB shading factor.
> + * @sets.gb_and_b.b: Blue shading factor.
> + * @sets.__reserved2: reserved
> + *
> + * Map to shading correction LUT register set.
> + */
> +struct ipu3_uapi_shd_lut {
> + struct {
> + struct {
> + __u16 r;
> + __u16 gr;
> + } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> + __u8 __reserved1[24];
> + struct {
> + __u16 gb;
> + __u16 b;
> + } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> + __u8 __reserved2[24];
> + } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_shd_config - Shading config
> + *
> + * @shd: shading static config, see &ipu3_uapi_shd_config_static
> + * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
> + */
> +struct ipu3_uapi_shd_config {
> + struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
> + struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
> +} __packed;
> +
> +/* Image Enhancement Filter directed */
> +
> +/**
> + * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
> + *
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A of Config Unit, s4.4, default 0.
> + * @b01: Always 0.
> + *
> + * Calculate weight for blending directed and non-directed denoise elements
> + *
> + * Note:
> + * Each instance of Config Unit needs X coordinate of n points and
> + * slope A factor between points calculated by driver based on calibration
> + * parameters.
> + */
> +struct ipu3_uapi_iefd_cux2 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 a01:9;
> + __u32 b01:5; /* NOTE: hardcoded to zero */
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed sharpening
> + * element, Config Unit 6 for edge detail (ED).
> + *
> + * @x0: X coordinate of point 0, u9.0, default 0.
> + * @x1: X coordinate of point 1, u9.0, default 0.
> + * @x2: X coordinate of point 2, u9.0, default 0.
> + * @__reserved0: reserved
> + * @x3: X coordinate of point 3, u9.0, default 0.
> + * @x4: X coordinate of point 4, u9.0, default 0.
> + * @x5: X coordinate of point 5, u9.0, default 0.
> + * @__reserved1: reserved
> + * @a01: slope A points 01, s4.4, default 0.
> + * @a12: slope A points 12, s4.4, default 0.
> + * @a23: slope A points 23, s4.4, default 0.
> + * @__reserved2: reserved
> + * @a34: slope A points 34, s4.4, default 0.
> + * @a45: slope A points 45, s4.4, default 0.
> + * @__reserved3: reserved
> + * @b01: slope B points 01, s4.4, default 0.
> + * @b12: slope B points 12, s4.4, default 0.
> + * @b23: slope B points 23, s4.4, default 0.
> + * @__reserved4: reserved
> + * @b34: slope B points 34, s4.4, default 0.
> + * @b45: slope B points 45, s4.4, default 0.
> + * @__reserved5: reserved
> + */
> +struct ipu3_uapi_iefd_cux6_ed {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 x2:9;
> + __u32 __reserved0:5;
> +
> + __u32 x3:9;
> + __u32 x4:9;
> + __u32 x5:9;
> + __u32 __reserved1:5;
> +
> + __u32 a01:9;
> + __u32 a12:9;
> + __u32 a23:9;
> + __u32 __reserved2:5;
> +
> + __u32 a34:9;
> + __u32 a45:9;
> + __u32 __reserved3:14;
> +
> + __u32 b01:9;
> + __u32 b12:9;
> + __u32 b23:9;
> + __u32 __reserved4:5;
> +
> + __u32 b34:9;
> + __u32 b45:9;
> + __u32 __reserved5:14;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
> + * element apply.
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A of Config Unit, s4.4, default 0.
> + * @__reserved1: reserved
> + * @b01: offset B0 of Config Unit, u7.0, default 0.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_iefd_cux2_1 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 a01:9;
> + __u32 __reserved1:5;
> +
> + __u32 b01:8;
> + __u32 __reserved2:24;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed sharpening
> + * element.
> + *
> + * @x0: X0 point of Config Unit, u9.0, default 0.
> + * @x1: X1 point of Config Unit, u9.0, default 0.
> + * @x2: X2 point of Config Unit, u9.0, default 0.
> + * @__reserved0: reserved
> + * @x3: X3 point of Config Unit, u9.0, default 0.
> + * @a01: Slope A0 of Config Unit, s4.4, default 0.
> + * @a12: Slope A1 of Config Unit, s4.4, default 0.
> + * @__reserved1: reserved
> + * @a23: Slope A2 of Config Unit, s4.4, default 0.
> + * @b01: Offset B0 of Config Unit, s7.0, default 0.
> + * @b12: Offset B1 of Config Unit, s7.0, default 0.
> + * @__reserved2: reserved
> + * @b23: Offset B2 of Config Unit, s7.0, default 0.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_iefd_cux4 {
> + __u32 x0:9;
> + __u32 x1:9;
> + __u32 x2:9;
> + __u32 __reserved0:5;
> +
> + __u32 x3:9;
> + __u32 a01:9;
> + __u32 a12:9;
> + __u32 __reserved1:5;
> +
> + __u32 a23:9;
> + __u32 b01:8;
> + __u32 b12:8;
> + __u32 __reserved2:7;
> +
> + __u32 b23:8;
> + __u32 __reserved3:24;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
> + *
> + * @x0: x0 points of Config Unit radial, u8.0
> + * @x1: x1 points of Config Unit radial, u8.0
> + * @x2: x2 points of Config Unit radial, u8.0
> + * @x3: x3 points of Config Unit radial, u8.0
> + * @x4: x4 points of Config Unit radial, u8.0
> + * @x5: x5 points of Config Unit radial, u8.0
> + * @__reserved1: reserved
> + * @a01: Slope A of Config Unit radial, s7.8
> + * @a12: Slope A of Config Unit radial, s7.8
> + * @a23: Slope A of Config Unit radial, s7.8
> + * @a34: Slope A of Config Unit radial, s7.8
> + * @a45: Slope A of Config Unit radial, s7.8
> + * @__reserved2: reserved
> + * @b01: Slope B of Config Unit radial, s9.0
> + * @b12: Slope B of Config Unit radial, s9.0
> + * @b23: Slope B of Config Unit radial, s9.0
> + * @__reserved4: reserved
> + * @b34: Slope B of Config Unit radial, s9.0
> + * @b45: Slope B of Config Unit radial, s9.0
> + * @__reserved5: reserved
> + */
> +struct ipu3_uapi_iefd_cux6_rad {
> + __u32 x0:8;
> + __u32 x1:8;
> + __u32 x2:8;
> + __u32 x3:8;
> +
> + __u32 x4:8;
> + __u32 x5:8;
> + __u32 __reserved1:16;
> +
> + __u32 a01:16;
> + __u32 a12:16;
> +
> + __u32 a23:16;
> + __u32 a34:16;
> +
> + __u32 a45:16;
> + __u32 __reserved2:16;
> +
> + __u32 b01:10;
> + __u32 b12:10;
> + __u32 b23:10;
> + __u32 __reserved4:2;
> +
> + __u32 b34:10;
> + __u32 b45:10;
> + __u32 __reserved5:12;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
> + *
> + * @cu_1: calculate weight for blending directed and
> + * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
> + * @cu_ed: calculate power of non-directed sharpening element, see
> + * &ipu3_uapi_iefd_cux6_ed
> + * @cu_3: calculate weight for blending directed and
> + * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
> + * @cu_5: calculate power of non-directed denoise element apply, use
> + * &ipu3_uapi_iefd_cux2_1
> + * @cu_6: calculate power of non-directed sharpening element. See
> + * &ipu3_uapi_iefd_cux4
> + * @cu_7: calculate weight for blending directed and
> + * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
> + * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
> + * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
> + * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
> + */
> +struct ipu3_uapi_yuvp1_iefd_cfg_units {
> + struct ipu3_uapi_iefd_cux2 cu_1;
> + struct ipu3_uapi_iefd_cux6_ed cu_ed;
> + struct ipu3_uapi_iefd_cux2 cu_3;
> + struct ipu3_uapi_iefd_cux2_1 cu_5;
> + struct ipu3_uapi_iefd_cux4 cu_6;
> + struct ipu3_uapi_iefd_cux2 cu_7;
> + struct ipu3_uapi_iefd_cux4 cu_unsharp;
> + struct ipu3_uapi_iefd_cux6_rad cu_radial;
> + struct ipu3_uapi_iefd_cux2 cu_vssnlm;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
> + *
> + * @horver_diag_coeff: Gradiant compensation, coefficient that compensates for
> + * different distance for vertical / horizontal and diagonal
> + * * gradient calculation (~1/sqrt(2)).
Uh, is this "diagonal * gradient" or "diagonal gradient"? It's probably a
spurious '*'.
And what does ~1/sqrt(2) refer to? The default value of the coefficient?
And does '~' stand for 'approximate'? Slightly confusing given the
C ~ operator. It's probably better to just write 'approx. 1/sqrt(2)'.
> + * @__reserved0: reserved
> + * @clamp_stitch: Slope to stitch between clamped and unclamped edge values
> + * @__reserved1: reserved
> + * @direct_metric_update: Update coeff for direction metric
> + * @__reserved2: reserved
> + * @ed_horver_diag_coeff: Radial Coefficient that compensates for
> + * different distance for vertical/horizontal and
> + * diagonal gradient calculation (~1/sqrt(2))
ditto
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_iefd_config_s {
> + __u32 horver_diag_coeff:7;
> + __u32 __reserved0:1;
> + __u32 clamp_stitch:6;
> + __u32 __reserved1:2;
> + __u32 direct_metric_update:5;
> + __u32 __reserved2:3;
> + __u32 ed_horver_diag_coeff:7;
> + __u32 __reserved3:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
> + *
> + * @iefd_en: Enable IEFd
> + * @denoise_en: Enable denoise
> + * @direct_smooth_en: Enable directional smooth
> + * @rad_en: Enable radial update
> + * @vssnlm_en: Enable VSSNLM output filter
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_yuvp1_iefd_control {
> + __u32 iefd_en:1;
> + __u32 denoise_en:1;
> + __u32 direct_smooth_en:1;
> + __u32 rad_en:1;
> + __u32 vssnlm_en:1;
> + __u32 __reserved:27;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_sharp_cfg - Sharpening config
> + *
> + * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
> + * @__reserved0: reserved
> + * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
> + * @__reserved1: reserved
> + * @nega_lmt_dir: Sharpening limit for negative overshoots for direction (edge).
> + * @__reserved2: reserved
> + * @posi_lmt_dir: Sharpening limit for positive overshoots for direction (edge).
> + * @__reserved3: reserved
> + *
> + * Fixed point type u13.0, range [0, 8191].
> + */
> +struct ipu3_uapi_sharp_cfg {
> + __u32 nega_lmt_txt:13;
> + __u32 __reserved0:19;
> + __u32 posi_lmt_txt:13;
> + __u32 __reserved1:19;
> + __u32 nega_lmt_dir:13;
> + __u32 __reserved2:19;
> + __u32 posi_lmt_dir:13;
> + __u32 __reserved3:19;
> +} __packed;
> +
> +/**
> + * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
> + *
> + * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64], default 64.
> + * @__reserved0: reserved
> + * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64], default 0.
> + * @__reserved1: reserved
> + * @ndir_dns_powr: Power of non-direct denoising,
> + * Precision u1.6, range [0, 64], default 64.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_far_w {
> + __u32 dir_shrp:7;
> + __u32 __reserved0:1;
> + __u32 dir_dns:7;
> + __u32 __reserved1:1;
> + __u32 ndir_dns_powr:7;
> + __u32 __reserved2:9;
> +} __packed;
> +
> +/**
> + * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
> + *
> + * @unsharp_weight: Unsharp mask blending weight.
> + * u1.6, range [0, 64], default 16.
> + * 0 - disabled, 64 - use only unsharp.
> + * @__reserved0: reserved
> + * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511], default 0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_unsharp_cfg {
> + __u32 unsharp_weight:7;
> + __u32 __reserved0:1;
> + __u32 unsharp_amount:9;
> + __u32 __reserved1:15;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
> + *
> + * @cfg: sharpness config &ipu3_uapi_sharp_cfg
> + * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
> + * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
> + * neighbors (8 pixels called Near), and the second order neighborhood
> + * around them (16 pixels called Far).
> + * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
> + */
> +struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
> + struct ipu3_uapi_sharp_cfg cfg;
> + struct ipu3_uapi_far_w far_w;
> + struct ipu3_uapi_unsharp_cfg unshrp_cfg;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
> + *
> + * @c00: Coeff11, s0.8, range [-255, 255], default 1.
> + * @c01: Coeff12, s0.8, range [-255, 255], default 5.
> + * @c02: Coeff13, s0.8, range [-255, 255], default 9.
> + * @__reserved: reserved
> + *
> + * Configurable registers for common sharpening support.
> + */
> +struct ipu3_uapi_unsharp_coef0 {
> + __u32 c00:9;
> + __u32 c01:9;
> + __u32 c02:9;
> + __u32 __reserved:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
> + *
> + * @c11: Coeff22, s0.8, range [-255, 255], default 29.
> + * @c12: Coeff23, s0.8, range [-255, 255], default 55.
> + * @c22: Coeff33, s0.8, range [-255, 255], default 96.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_unsharp_coef1 {
> + __u32 c11:9;
> + __u32 c12:9;
> + __u32 c22:9;
> + __u32 __reserved:5;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
> + *
> + * @unsharp_coef0: unsharp coefficient 0 config. See &ipu3_uapi_unsharp_coef0
> + * @unsharp_coef1: unsharp coefficient 1 config. See &ipu3_uapi_unsharp_coef1
> + */
> +struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
> + struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
> + struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_reset_xy - Radial coordinate reset
> + *
> + * @x: Radial reset of x coordinate. Precision s12, [-4095, 4095], default 0.
> + * @__reserved0: reserved
> + * @y: Radial center y coordinate. Precision s12, [-4095, 4095], default 0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_radial_reset_xy {
> + __s32 x:13;
> + __u32 __reserved0:3;
> + __s32 y:13;
> + __u32 __reserved1:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_reset_x2 - Radial X^2 reset
> + *
> + * @x2: Radial reset of x^2 coordinate. Precision u24, default 0.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_radial_reset_x2 {
> + __u32 x2:24;
> + __u32 __reserved:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_reset_y2 - Radial Y^2 reset
> + *
> + * @y2: Radial reset of y^2 coordinate. Precision u24, default 0.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_radial_reset_y2 {
> + __u32 y2:24;
> + __u32 __reserved:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_radial_cfg - Radial config
> + *
> + * @rad_nf: Radial. R^2 normalization factor is scale down by 2^ - (15 + scale)
> + * @__reserved0: reserved
> + * @rad_inv_r2: Radial R^-2 normelized to (0.5..1), Prec' u7, range [0, 127].
Prec' -> Precision
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_radial_cfg {
> + __u32 rad_nf:4;
> + __u32 __reserved0:4;
> + __u32 rad_inv_r2:7;
> + __u32 __reserved1:17;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_rad_far_w - Radial FAR sub-group
> + *
> + * @rad_dir_far_sharp_w: Weight of wide direct sharpening, u1.6, range [0, 64],
> + * default 64.
> + * @rad_dir_far_dns_w: Weight of wide direct denoising, u1.6, range [0, 64],
> + * default 0.
> + * @rad_ndir_far_dns_power: power of non-direct sharpening, u1.6, range [0, 64],
> + * default 0.
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_rad_far_w {
> + __u32 rad_dir_far_sharp_w:8;
> + __u32 rad_dir_far_dns_w:8;
> + __u32 rad_ndir_far_dns_power:8;
> + __u32 __reserved:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cu_cfg0 - Radius Config Unit cfg0 register
> + *
> + * @cu6_pow: Power of CU6. Power of non-direct sharpening, u3.4.
> + * @__reserved0: reserved
> + * @cu_unsharp_pow: Power of unsharp mask, u2.4.
> + * @__reserved1: reserved
> + * @rad_cu6_pow: Radial/corner CU6. Directed sharpening power, u3.4.
> + * @__reserved2: reserved
> + * @rad_cu_unsharp_pow: Radial power of unsharp mask, u2.4.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_cu_cfg0 {
> + __u32 cu6_pow:7;
> + __u32 __reserved0:1;
> + __u32 cu_unsharp_pow:7;
> + __u32 __reserved1:1;
> + __u32 rad_cu6_pow:7;
> + __u32 __reserved2:1;
> + __u32 rad_cu_unsharp_pow:6;
> + __u32 __reserved3:2;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_cu_cfg1 - Radius Config Unit cfg1 register
> + *
> + * @rad_cu6_x1: X1 point of Config Unit 6, precision u9.0.
> + * @__reserved0: reserved
> + * @rad_cu_unsharp_x1: X1 point for Config Unit unsharp for radial/corner point
> + * precision u9.0.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_cu_cfg1 {
> + __u32 rad_cu6_x1:9;
> + __u32 __reserved0:1;
> + __u32 rad_cu_unsharp_x1:9;
> + __u32 __reserved1:13;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_rad_cfg - IEFd parameters changed radially over
> + * the picture plain.
I guess you mean 'plane' instead of 'plain'?
> + *
> + * @reset_xy: reset xy value in radial calculation. &ipu3_uapi_radial_reset_xy
> + * @reset_x2: reset x square value in radial calculation. See struct
> + * &ipu3_uapi_radial_reset_x2
> + * @reset_y2: reset y square value in radial calculation. See struct
> + * &ipu3_uapi_radial_reset_y2
> + * @cfg: radial config defined in &ipu3_uapi_radial_cfg
> + * @rad_far_w: weight for wide range radial. &ipu3_uapi_rad_far_w
> + * @cu_cfg0: configuration unit 0. See &ipu3_uapi_cu_cfg0
> + * @cu_cfg1: configuration unit 1. See &ipu3_uapi_cu_cfg1
> + */
> +struct ipu3_uapi_yuvp1_iefd_rad_cfg {
> + struct ipu3_uapi_radial_reset_xy reset_xy;
> + struct ipu3_uapi_radial_reset_x2 reset_x2;
> + struct ipu3_uapi_radial_reset_y2 reset_y2;
> + struct ipu3_uapi_radial_cfg cfg;
> + struct ipu3_uapi_rad_far_w rad_far_w;
> + struct ipu3_uapi_cu_cfg0 cu_cfg0;
> + struct ipu3_uapi_cu_cfg1 cu_cfg1;
> +} __packed;
> +
> +/* Vssnlm - Very small scale non-local mean algorithm */
> +
> +/**
> + * struct ipu3_uapi_vss_lut_x - Vssnlm LUT x0/x1/x2
> + *
> + * @vs_x0: Vssnlm LUT x0, precision u8, range [0, 255], default 16.
> + * @vs_x1: Vssnlm LUT x1, precision u8, range [0, 255], default 32.
> + * @vs_x2: Vssnlm LUT x2, precision u8, range [0, 255], default 64.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_vss_lut_x {
> + __u32 vs_x0:8;
> + __u32 vs_x1:8;
> + __u32 vs_x2:8;
> + __u32 __reserved2:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_vss_lut_y - Vssnlm LUT y0/y1/y2
> + *
> + * @vs_y1: Vssnlm LUT y1, precision u4, range [0, 8], default 1.
> + * @__reserved0: reserved
> + * @vs_y2: Vssnlm LUT y2, precision u4, range [0, 8], default 3.
> + * @__reserved1: reserved
> + * @vs_y3: Vssnlm LUT y3, precision u4, range [0, 8], default 8.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_vss_lut_y {
> + __u32 vs_y1:4;
> + __u32 __reserved0:4;
> + __u32 vs_y2:4;
> + __u32 __reserved1:4;
> + __u32 vs_y3:4;
> + __u32 __reserved2:12;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_vssnlm_cf - IEFd Vssnlm Lookup table
> + *
> + * @vss_lut_x: vss lookup table. See &ipu3_uapi_vss_lut_x description
> + * @vss_lut_y: vss lookup table. See &ipu3_uapi_vss_lut_y description
> + */
> +struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg {
> + struct ipu3_uapi_vss_lut_x vss_lut_x;
> + struct ipu3_uapi_vss_lut_y vss_lut_y;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_iefd_config - IEFd config
> + *
> + * @units: configuration unit setting, &ipu3_uapi_yuvp1_iefd_cfg_units
> + * @config: configuration, as defined by &ipu3_uapi_yuvp1_iefd_config_s
> + * @control: control setting, as defined by &ipu3_uapi_yuvp1_iefd_control
> + * @sharp: sharpness setting, as defined by &ipu3_uapi_yuvp1_iefd_shrp_cfg
> + * @unsharp: unsharpness setting, as defined by &ipu3_uapi_yuvp1_iefd_unshrp_cfg
> + * @rad: radial setting, as defined by &ipu3_uapi_yuvp1_iefd_rad_cfg
> + * @vsslnm: vsslnm setting, as defined by &ipu3_uapi_yuvp1_iefd_vssnlm_cfg
> + */
> +struct ipu3_uapi_yuvp1_iefd_config {
> + struct ipu3_uapi_yuvp1_iefd_cfg_units units;
> + struct ipu3_uapi_yuvp1_iefd_config_s config;
> + struct ipu3_uapi_yuvp1_iefd_control control;
> + struct ipu3_uapi_yuvp1_iefd_shrp_cfg sharp;
> + struct ipu3_uapi_yuvp1_iefd_unshrp_cfg unsharp;
> + struct ipu3_uapi_yuvp1_iefd_rad_cfg rad;
> + struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg vsslnm;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_yds_config - Y Down-Sampling config
> + *
> + * @c00: range [0, 3], default 0x0
> + * @c01: range [0, 3], default 0x1
> + * @c02: range [0, 3], default 0x1
> + * @c03: range [0, 3], default 0x0
> + * @c10: range [0, 3], default 0x0
> + * @c11: range [0, 3], default 0x1
> + * @c12: range [0, 3], default 0x1
> + * @c13: range [0, 3], default 0x0
> + *
> + * Above are 4x2 filter coefficients for chroma output downscaling.
> + *
> + * @norm_factor: Normalization factor, range [0, 4], default 2
> + * 0 - divide by 1
> + * 1 - divide by 2
> + * 2 - divide by 4
> + * 3 - divide by 8
> + * 4 - divide by 16
> + * @__reserved0: reserved
> + * @bin_output: Down sampling on Luma channel in two optional modes
> + * 0 - Bin output 4.2.0 (default), 1 output 4.2.2.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_yds_config {
> + __u32 c00:2;
> + __u32 c01:2;
> + __u32 c02:2;
> + __u32 c03:2;
> + __u32 c10:2;
> + __u32 c11:2;
> + __u32 c12:2;
> + __u32 c13:2;
> + __u32 norm_factor:5;
> + __u32 __reserved0:4;
> + __u32 bin_output:1;
> + __u32 __reserved1:6;
> +} __packed;
> +
> +/* Chroma Noise Reduction */
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_enable_config - Chroma noise reduction enable
> + *
> + * @enable: enable/disable chroma noise reduction
> + * @yuv_mode: 0 - YUV420, 1 - YUV422
> + * @__reserved0: reserved
> + * @col_size: number of columns in the frame, max width is 2560
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_enable_config {
> + __u32 enable:1;
> + __u32 yuv_mode:1;
> + __u32 __reserved0:14;
> + __u32 col_size:12;
> + __u32 __reserved1:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_coring_config - Coring thresholds for UV
> + *
> + * @u: U coring level, u0.13, range [0.0, 1.0], default 0.0
> + * @__reserved0: reserved
> + * @v: V coring level, u0.13, range [0.0, 1.0], default 0.0
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_coring_config {
> + __u32 u:13;
> + __u32 __reserved0:3;
> + __u32 v:13;
> + __u32 __reserved1:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_sense_gain_config - Chroma noise reduction gains
> + *
> + * All sensitivity gain parameters have precision u13.0, range [0, 8191].
> + *
> + * @vy: Sensitivity of horizontal edge of Y, default 100
> + * @vu: Sensitivity of horizontal edge of U, default 100
> + * @vv: Sensitivity of horizontal edge of V, default 100
> + * @__reserved0: reserved
> + * @hy: Sensitivity of vertical edge of Y, default 50
> + * @hu: Sensitivity of vertical edge of U, default 50
> + * @hv: Sensitivity of vertical edge of V, default 50
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_sense_gain_config {
> + __u32 vy:8;
> + __u32 vu:8;
> + __u32 vv:8;
> + __u32 __reserved0:8;
> +
> + __u32 hy:8;
> + __u32 hu:8;
> + __u32 hv:8;
> + __u32 __reserved1:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_iir_fir_config - Chroma IIR/FIR filter config
> + *
> + * @fir_0h: Value of center tap in horizontal FIR, range [0, 32], default 8.
> + * @__reserved0: reserved
> + * @fir_1h: Value of distance 1 in horizontal FIR, range [0, 32], default 12.
> + * @__reserved1: reserved
> + * @fir_2h: Value of distance 2 tap in horizontal FIR, range [0, 32], default 0.
> + * @dalpha_clip_val: weight for previous row in IIR, range [1, 256], default 0.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_yuvp1_chnr_iir_fir_config {
> + __u32 fir_0h:6;
> + __u32 __reserved0:2;
> + __u32 fir_1h:6;
> + __u32 __reserved1:2;
> + __u32 fir_2h:6;
> + __u32 dalpha_clip_val:9;
> + __u32 __reserved2:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_chnr_config - Chroma noise reduction config
> + *
> + * @enable: chroma noise reduction enable, see
> + * &ipu3_uapi_yuvp1_chnr_enable_config
> + * @coring: coring config for chroma noise reduction, see
> + * &ipu3_uapi_yuvp1_chnr_coring_config
> + * @sense_gain: sensitivity config for chroma noise reduction, see
> + * ipu3_uapi_yuvp1_chnr_sense_gain_config
> + * @iir_fir: iir and fir config for chroma noise reduction, see
> + * ipu3_uapi_yuvp1_chnr_iir_fir_config
> + */
> +struct ipu3_uapi_yuvp1_chnr_config {
> + struct ipu3_uapi_yuvp1_chnr_enable_config enable;
> + struct ipu3_uapi_yuvp1_chnr_coring_config coring;
> + struct ipu3_uapi_yuvp1_chnr_sense_gain_config sense_gain;
> + struct ipu3_uapi_yuvp1_chnr_iir_fir_config iir_fir;
> +} __packed;
> +
> +/* Edge Enhancement and Noise Reduction */
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config - Luma(Y) edge enhancement low-pass
> + * filter coefficients
> + *
> + * @a_diag: Smoothing diagonal coefficient, u5.0.
> + * @__reserved0: reserved
> + * @a_periph: Image smoothing perpherial, u5.0.
> + * @__reserved1: reserved
> + * @a_cent: Image Smoothing center coefficient, u5.0.
> + * @__reserved2: reserved
> + * @enable: 0: Y_EE_NR disabled, output = input; 1: Y_EE_NR enabled.
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config {
> + __u32 a_diag:5;
> + __u32 __reserved0:3;
> + __u32 a_periph:5;
> + __u32 __reserved1:3;
> + __u32 a_cent:5;
> + __u32 __reserved2:9;
> + __u32 enable:1;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_sense_config - Luma(Y) edge enhancement
> + * noise reduction sensitivity gains
> + *
> + * @edge_sense_0: Sensitivity of edge in dark area. u13.0, default 8191.
> + * @__reserved0: reserved
> + * @delta_edge_sense: Difference in the sensitivity of edges between
> + * the bright and dark areas. u13.0, default 0.
> + * @__reserved1: reserved
> + * @corner_sense_0: Sensitivity of corner in dark area. u13.0, default 0.
> + * @__reserved2: reserved
> + * @delta_corner_sense: Difference in the sensitivity of corners between
> + * the bright and dark areas. u13.0, default 8191.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_sense_config {
> + __u32 edge_sense_0:13;
> + __u32 __reserved0:3;
> + __u32 delta_edge_sense:13;
> + __u32 __reserved1:3;
> + __u32 corner_sense_0:13;
> + __u32 __reserved2:3;
> + __u32 delta_corner_sense:13;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_gain_config - Luma(Y) edge enhancement
> + * noise reduction gain config
> + *
> + * @gain_pos_0: Gain for positive edge in dark area. u5.0, [0, 16], default 2.
> + * @__reserved0: reserved
> + * @delta_gain_posi: Difference in the gain of edges between the bright and
> + * dark areas for positive edges. u5.0, [0, 16], default 0.
> + * @__reserved1: reserved
> + * @gain_neg_0: Gain for negative edge in dark area. u5.0, [0, 16], default 8.
> + * @__reserved2: reserved
> + * @delta_gain_neg: Difference in the gain of edges between the bright and
> + * dark areas for negative edges. u5.0, [0, 16], default 0.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_gain_config {
> + __u32 gain_pos_0:5;
> + __u32 __reserved0:3;
> + __u32 delta_gain_posi:5;
> + __u32 __reserved1:3;
> + __u32 gain_neg_0:5;
> + __u32 __reserved2:3;
> + __u32 delta_gain_neg:5;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_clip_config - Luma(Y) edge enhancement
> + * noise reduction clipping config
> + *
> + * @clip_pos_0: Limit of positive edge in dark area
> + * u5, value [0, 16], default 8.
> + * @__reserved0: reserved
> + * @delta_clip_posi: Difference in the limit of edges between the bright
> + * and dark areas for positive edges.
> + * u5, value [0, 16], default 8.
> + * @__reserved1: reserved
> + * @clip_neg_0: Limit of negative edge in dark area
> + * u5, value [0, 16], default 8.
> + * @__reserved2: reserved
> + * @delta_clip_neg: Difference in the limit of edges between the bright
> + * and dark areas for negative edges.
> + * u5, value [0, 16], default 8.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_clip_config {
> + __u32 clip_pos_0:5;
> + __u32 __reserved0:3;
> + __u32 delta_clip_posi:5;
> + __u32 __reserved1:3;
> + __u32 clip_neg_0:5;
> + __u32 __reserved2:3;
> + __u32 delta_clip_neg:5;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_frng_config - Luma(Y) edge enhancement
> + * noise reduction fringe config
> + *
> + * @gain_exp: Common exponent of gains, u4, [0, 8], default 2.
> + * @__reserved0: reserved
> + * @min_edge: Threshold for edge and smooth stitching, u13.
> + * @__reserved1: reserved
> + * @lin_seg_param: Power of LinSeg, u4.
> + * @__reserved2: reserved
> + * @t1: Parameter for enabling/disabling the edge enhancement, u1.0, [0, 1],
> + * default 1.
> + * @t2: Parameter for enabling/disabling the smoothing, u1.0, [0, 1],
> + * default 1.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_frng_config {
> + __u32 gain_exp:4;
> + __u32 __reserved0:28;
> + __u32 min_edge:13;
> + __u32 __reserved1:3;
> + __u32 lin_seg_param:4;
> + __u32 __reserved2:4;
> + __u32 t1:1;
> + __u32 t2:1;
> + __u32 __reserved3:6;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_diag_config - Luma(Y) edge enhancement
> + * noise reduction diagonal config
> + *
> + * @diag_disc_g: Coefficient that prioritize diagonal edge direction on
> + * horizontal or vertical for final enhancement.
> + * u4.0, [1, 15], default 1.
> + * @__reserved0: reserved
> + * @hvw_hor: Weight of horizontal/vertical edge enhancement for hv edge.
> + * u2.2, [1, 15], default 4.
> + * @dw_hor: Weight of diagonal edge enhancement for hv edge.
> + * u2.2, [1, 15], default 1.
> + * @hvw_diag: Weight of horizontal/vertical edge enhancement for diagonal edge.
> + * u2.2, [1, 15], default 1.
> + * @dw_diag: Weight of diagonal edge enhancement for diagonal edge.
> + * u2.2, [1, 15], default 4.
> + * @__reserved1: reserved
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_diag_config {
> + __u32 diag_disc_g:4;
> + __u32 __reserved0:4;
> + __u32 hvw_hor:4;
> + __u32 dw_hor:4;
> + __u32 hvw_diag:4;
> + __u32 dw_diag:4;
> + __u32 __reserved1:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config - Luma(Y) edge enhancement
> + * noise reduction false color correction (FCC) coring config
> + *
> + * @pos_0: Gain for positive edge in dark, u13.0, [0, 16], default 0.
> + * @__reserved0: reserved
> + * @pos_delta: Gain for positive edge in bright, value: pos_0 + pos_delta <=16
> + * u13.0, default 0.
> + * @__reserved1: reserved
> + * @neg_0: Gain for negative edge in dark area, u13.0, range [0, 16], default 0.
> + * @__reserved2: reserved
> + * @neg_delta: Gain for negative edge in bright area. neg_0 + neg_delta <=16
> + * u13.0, default 0.
> + * @__reserved3: reserved
> + *
> + * Coring is a simple soft thresholding technique.
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config {
> + __u32 pos_0:13;
> + __u32 __reserved0:3;
> + __u32 pos_delta:13;
> + __u32 __reserved1:3;
> + __u32 neg_0:13;
> + __u32 __reserved2:3;
> + __u32 neg_delta:13;
> + __u32 __reserved3:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp1_y_ee_nr_config - Edge enhancement and noise reduction
> + *
> + * @lpf: low-pass filter config. See &ipu3_uapi_yuvp1_y_ee_nr_lpf_config
> + * @sense: sensitivity config. See &ipu3_uapi_yuvp1_y_ee_nr_sense_config
> + * @gain: gain config as defined in &ipu3_uapi_yuvp1_y_ee_nr_gain_config
> + * @clip: clip config as defined in &ipu3_uapi_yuvp1_y_ee_nr_clip_config
> + * @frng: fringe config as defined in &ipu3_uapi_yuvp1_y_ee_nr_frng_config
> + * @diag: diagonal edge config. See &ipu3_uapi_yuvp1_y_ee_nr_diag_config
> + * @fc_coring: coring config for fringe control. See
> + * &ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config
> + */
> +struct ipu3_uapi_yuvp1_y_ee_nr_config {
> + struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config lpf;
> + struct ipu3_uapi_yuvp1_y_ee_nr_sense_config sense;
> + struct ipu3_uapi_yuvp1_y_ee_nr_gain_config gain;
> + struct ipu3_uapi_yuvp1_y_ee_nr_clip_config clip;
> + struct ipu3_uapi_yuvp1_y_ee_nr_frng_config frng;
> + struct ipu3_uapi_yuvp1_y_ee_nr_diag_config diag;
> + struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config fc_coring;
> +} __packed;
> +
> +/* Total Color Correction */
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_gen_control_static_config - Total color correction
> + * general control config
> + *
> + * @en: 0 - TCC disabled. Output = input 1 - TCC enabled.
> + * @blend_shift: blend shift, Range[3, 4], default NA.
> + * @gain_according_to_y_only: 0: Gain is calculated according to YUV,
> + * 1: Gain is calculated according to Y only
> + * @__reserved0: reserved
> + * @gamma: Final blending coefficients. Values[-16, 16], default NA.
> + * @__reserved1: reserved
> + * @delta: Final blending coefficients. Values[-16, 16], default NA.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_yuvp2_tcc_gen_control_static_config {
> + __u32 en:1;
> + __u32 blend_shift:3;
> + __u32 gain_according_to_y_only:1;
> + __u32 __reserved0:11;
> + __s32 gamma:5;
> + __u32 __reserved1:3;
> + __s32 delta:5;
> + __u32 __reserved2:3;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config - Total color correction
> + * multi-axis color control (MACC) config
> + *
> + * @a: a coefficient for 2x2 MACC conversion matrix.
> + * @__reserved0: reserved
> + * @b: b coefficient 2x2 MACC conversion matrix.
> + * @__reserved1: reserved
> + * @c: c coefficient for 2x2 MACC conversion matrix.
> + * @__reserved2: reserved
> + * @d: d coefficient for 2x2 MACC conversion matrix.
> + * @__reserved3: reserved
> + */
> +struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config {
> + __s32 a:12;
> + __u32 __reserved0:4;
> + __s32 b:12;
> + __u32 __reserved1:4;
> + __s32 c:12;
> + __u32 __reserved2:4;
> + __s32 d:12;
> + __u32 __reserved3:4;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_macc_table_static_config - Total color correction
> + * multi-axis color control (MACC) table array
> + *
> + * @entries: config for multi axis color correction, as specified by
> + * &ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> + */
> +struct ipu3_uapi_yuvp2_tcc_macc_table_static_config {
> + struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> + entries[IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config - Total color correction
> + * inverse y lookup table
> + *
> + * @entries: lookup table for inverse y estimation, and use it to estimate the
> + * ratio between luma and chroma. Chroma by approximate the absolute
> + * value of the radius on the chroma plane (R = sqrt(u^2+v^2) ) and
> + * luma by approximate by 1/Y.
> + */
> +struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config {
> + __u16 entries[IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config - Total color
> + * correction lookup table for PCWL
> + *
> + * @entries: lookup table for gain piece wise linear transformation (PCWL)
> + */
> +struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config {
> + __u16 entries[IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config - Total color correction
> + * lookup table for r square root
> + *
> + * @entries: lookup table for r square root estimation
> + */
> +struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config {
> + __s16 entries[IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_yuvp2_tcc_static_config- Total color correction static
> + *
> + * @gen_control: general config for Total Color Correction
> + * @macc_table: config for multi axis color correction
> + * @inv_y_lut: lookup table for inverse y estimation
> + * @gain_pcwl: lookup table for gain PCWL
> + * @r_sqr_lut: lookup table for r square root estimation.
> + */
> +struct ipu3_uapi_yuvp2_tcc_static_config {
> + struct ipu3_uapi_yuvp2_tcc_gen_control_static_config gen_control;
> + struct ipu3_uapi_yuvp2_tcc_macc_table_static_config macc_table;
> + struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config inv_y_lut;
> + struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config gain_pcwl;
> + struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config r_sqr_lut;
> +} __packed;
> +
> +/* Advanced Noise Reduction related structs */
> +
> +/*
> + * struct ipu3_uapi_anr_alpha - Advanced noise reduction alpha
> + *
> + * Tunable parameters that are subject to modification according to the
> + * total gain used.
> + */
> +struct ipu3_uapi_anr_alpha {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> + __u16 dc_gr;
> + __u16 dc_r;
> + __u16 dc_b;
> + __u16 dc_gb;
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_anr_beta - Advanced noise reduction beta
> + *
> + * Tunable parameters that are subject to modification according to the
> + * total gain used.
> + */
> +struct ipu3_uapi_anr_beta {
> + __u16 beta_gr;
> + __u16 beta_r;
> + __u16 beta_b;
> + __u16 beta_gb;
> +} __packed;
> +
> +/*
> + * struct ipu3_uapi_anr_plain_color - Advanced noise reduction plain color with
> + * 4x4 matrix
What is a "plain color"? Or should this have been "plane color"? But then I
don't know what a "plane color" is either :-)
> + *
> + * Tunable parameters that are subject to modification according to the
> + * total gain used.
> + */
> +struct ipu3_uapi_anr_plain_color {
> + __u16 reg_w_gr[16];
> + __u16 reg_w_r[16];
> + __u16 reg_w_b[16];
> + __u16 reg_w_gb[16];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_transform_config - Advanced noise reduction transform
> + *
> + * @enable: advanced noise reduction enabled.
> + * @adaptive_treshhold_en: On IPU3, adaptive threshold is always enabled.
> + * @__reserved1: reserved
> + * @__reserved2: reserved
> + * @alpha: using following defaults:
> + * 13, 13, 13, 13, 0, 0, 0, 0
> + * 11, 11, 11, 11, 0, 0, 0, 0
> + * 14, 14, 14, 14, 0, 0, 0, 0
> + * @beta: use following defaults:
> + * 24, 24, 24, 24
> + * 21, 20, 20, 21
> + * 25, 25, 25, 25
> + * @color: use defaults defined in driver/media/pci/intel/ipu3-tables.c
> + * @sqrt_lut: 11 bits per element, values =
> + * [724 768 810 849 887
> + * 923 958 991 1024 1056
> + * 1116 1145 1173 1201 1086
> + * 1228 1254 1280 1305 1330
> + * 1355 1379 1402 1425 1448]
> + * @xreset: Reset value of X for r^2 calculation Value: col_start-X_center
> + * Constraint: Xreset + FrameWdith=4095 Xreset= -4095, default -1632.
> + * @__reserved3: reserved
> + * @yreset: Reset value of Y for r^2 calculation Value: row_start-Y_center
> + * Constraint: Yreset + FrameHeight=4095 Yreset= -4095, default -1224.
> + * @__reserved4: reserved
> + * @x_sqr_reset: Reset value of X^2 for r^2 calculation Value = (Xreset)^2
> + * @r_normfactor: Normalization factor for R. Default 14.
> + * @__reserved5: reserved
> + * @y_sqr_reset: Reset value of Y^2 for r^2 calculation Value = (Yreset)^2
> + * @gain_scale: Parameter describing shading gain as a function of distance
> + * from the image center.
> + * A single value per frame, loaded by the driver. Default 115.
> + */
> +struct ipu3_uapi_anr_transform_config {
> + __u32 enable:1; /* 0 or 1, disabled or enabled */
> + __u32 adaptive_treshhold_en:1; /* On IPU3, always enabled */
> +
> + __u32 __reserved1:30;
> + __u8 __reserved2[44];
> +
> + struct ipu3_uapi_anr_alpha alpha[3];
> + struct ipu3_uapi_anr_beta beta[3];
> + struct ipu3_uapi_anr_plain_color color[3];
> +
> + __u16 sqrt_lut[IPU3_UAPI_ANR_LUT_SIZE]; /* 11 bits per element */
> +
> + __s16 xreset:13;
> + __u16 __reserved3:3;
> + __s16 yreset:13;
> + __u16 __reserved4:3;
> +
> + __u32 x_sqr_reset:24;
> + __u32 r_normfactor:5;
> + __u32 __reserved5:3;
> +
> + __u32 y_sqr_reset:24;
> + __u32 gain_scale:8;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_stitch_pyramid - ANR stitch pyramid
> + *
> + * @entry0: pyramid LUT entry0, range [0x0, 0x3f]
> + * @entry1: pyramid LUT entry1, range [0x0, 0x3f]
> + * @entry2: pyramid LUT entry2, range [0x0, 0x3f]
> + * @__reserved: reserved
> + */
> +struct ipu3_uapi_anr_stitch_pyramid {
> + __u32 entry0:6;
> + __u32 entry1:6;
> + __u32 entry2:6;
> + __u32 __reserved:14;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_stitch_config - ANR stitch config
> + *
> + * @anr_stitch_en: enable stitch. Enabled with 1.
> + * @__reserved: reserved
> + * @pyramid: pyramid table as defined by &ipu3_uapi_anr_stitch_pyramid
> + * default values:
> + * { 1, 3, 5 }, { 7, 7, 5 }, { 3, 1, 3 },
> + * { 9, 15, 21 }, { 21, 15, 9 }, { 3, 5, 15 },
> + * { 25, 35, 35 }, { 25, 15, 5 }, { 7, 21, 35 },
> + * { 49, 49, 35 }, { 21, 7, 7 }, { 21, 35, 49 },
> + * { 49, 35, 21 }, { 7, 5, 15 }, { 25, 35, 35 },
> + * { 25, 15, 5 }, { 3, 9, 15 }, { 21, 21, 15 },
> + * { 9, 3, 1 }, { 3, 5, 7 }, { 7, 5, 3}, { 1 }
> + */
> +struct ipu3_uapi_anr_stitch_config {
> + __u32 anr_stitch_en;
> + __u8 __reserved[44];
> + struct ipu3_uapi_anr_stitch_pyramid pyramid[IPU3_UAPI_ANR_PYRAMID_SIZE];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_anr_config - ANR config
> + *
> + * @transform: advanced noise reduction transform config as specified by
> + * &ipu3_uapi_anr_transform_config
> + * @stitch: create 4x4 patch from 4 surrounding 8x8 patches.
> + */
> +struct ipu3_uapi_anr_config {
> + struct ipu3_uapi_anr_transform_config transform __attribute__((aligned(32)));
> + struct ipu3_uapi_anr_stitch_config stitch __attribute__((aligned(32)));
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_acc_param - Accelerator cluster parameters
> + *
> + * ACC refers to the HW cluster containing all Fixed Functions(FFs). Each FF
Add space before '(FFs)'.
> + * implements a specific algorithm.
> + *
> + * @bnr: parameters for bayer noise reduction static config. See
> + * &ipu3_uapi_bnr_static_config
> + * @green_disparity: disparity static config between gr and gb channel.
> + * See &ipu3_uapi_bnr_static_config_green_disparity
> + * @dm: de-mosaic config. See &ipu3_uapi_dm_config
> + * @ccm: color correction matrix. See &ipu3_uapi_ccm_mat_config
> + * @gamma: gamma correction config. See &ipu3_uapi_gamma_config
> + * @csc: color space conversion matrix. See &ipu3_uapi_csc_mat_config
> + * @cds: color down sample config. See &ipu3_uapi_cds_params
> + * @shd: lens shading correction config. See &ipu3_uapi_shd_config
> + * @iefd: Image enhancement filter and denoise config.
> + * &ipu3_uapi_yuvp1_iefd_config
> + * @yds_c0: y down scaler config. &ipu3_uapi_yuvp1_yds_config
> + * @chnr_c0: chroma noise reduction config. &ipu3_uapi_yuvp1_chnr_config
> + * @y_ee_nr: y edge enhancement and noise reduction config.
> + * &ipu3_uapi_yuvp1_y_ee_nr_config
> + * @yds: y down scaler config. See &ipu3_uapi_yuvp1_yds_config
> + * @chnr: chroma noise reduction config. See &ipu3_uapi_yuvp1_chnr_config
> + * @__reserved1: reserved
> + * @yds2: y channel down scaler config. See &ipu3_uapi_yuvp1_yds_config
> + * @tcc: total color correction config as defined in struct
> + * &ipu3_uapi_yuvp2_tcc_static_config
> + * @__reserved2: reserved
> + * @anr: advanced noise reduction config.See &ipu3_uapi_anr_config
> + * @awb_fr: AWB filter response config. See ipu3_uapi_awb_fr_config
> + * @ae: auto exposure config As specified by &ipu3_uapi_ae_config
> + * @af: auto focus config. As specified by &ipu3_uapi_af_config
> + * @awb: auto white balance config. As specified by &ipu3_uapi_awb_config
> + */
> +struct ipu3_uapi_acc_param {
> + struct ipu3_uapi_bnr_static_config bnr;
> + struct ipu3_uapi_bnr_static_config_green_disparity
> + green_disparity __attribute__((aligned(32)));
> + struct ipu3_uapi_dm_config dm __attribute__((aligned(32)));
> + struct ipu3_uapi_ccm_mat_config ccm __attribute__((aligned(32)));
> + struct ipu3_uapi_gamma_config gamma __attribute__((aligned(32)));
> + struct ipu3_uapi_csc_mat_config csc __attribute__((aligned(32)));
> + struct ipu3_uapi_cds_params cds __attribute__((aligned(32)));
> + struct ipu3_uapi_shd_config shd __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_iefd_config iefd __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_yds_config yds_c0 __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_chnr_config chnr_c0 __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_y_ee_nr_config y_ee_nr __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_yds_config yds __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_chnr_config chnr __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp1_yds_config yds2 __attribute__((aligned(32)));
> + struct ipu3_uapi_yuvp2_tcc_static_config tcc __attribute__((aligned(32)));
> + struct ipu3_uapi_anr_config anr;
> + struct ipu3_uapi_awb_fr_config awb_fr;
> + struct ipu3_uapi_ae_config ae;
> + struct ipu3_uapi_af_config af;
> + struct ipu3_uapi_awb_config awb;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
> + *
> + * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
> + * @lin_lutlow_r: linearization look-up table for R channel interpolation.
> + * @lin_lutlow_b: linearization look-up table for B channel interpolation.
> + * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
> + * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
> + * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
> + * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
> + * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
> + * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
> + * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
> + */
> +struct ipu3_uapi_isp_lin_vmem_params {
> + __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
> + __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
> +} __packed;
> +
> +/* Temporal Noise Reduction */
> +
> +/**
> + * struct ipu3_uapi_isp_tnr3_vmem_params - Temporal noise reduction vector
> + * memory parameters
> + *
> + * @slope: slope setting in interpolation curve for temporal noise reduction.
> + * @__reserved1: reserved
> + * @sigma: knee point setting in interpolation curve for temporal
> + * noise reduction.
> + * @__reserved2: reserved
> + */
> +struct ipu3_uapi_isp_tnr3_vmem_params {
> + __u16 slope[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> + __u16 __reserved1[IPU3_UAPI_ISP_VEC_ELEMS
> + - IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> + __u16 sigma[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> + __u16 __reserved2[IPU3_UAPI_ISP_VEC_ELEMS
> + - IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_isp_tnr3_params - Temporal noise reduction v3 parameters
> + *
> + * @knee_y1: Knee point TNR3 assumes standard deviation of Y,U and
> + * V at Y1 are TnrY1_Sigma_Y, U and V.
> + * @knee_y2: Knee point TNR3 assumes standard deviation of Y,U and
> + * V at Y2 are TnrY2_Sigma_Y, U and V.
> + * @maxfb_y: Max feedback gain for Y
> + * @maxfb_u: Max feedback gain for U
> + * @maxfb_v: Max feedback gain for V
> + * @round_adj_y: rounding Adjust for Y
> + * @round_adj_u: rounding Adjust for U
> + * @round_adj_v: rounding Adjust for V
> + * @ref_buf_select: selection of the reference frame buffer to be used.
> + */
> +struct ipu3_uapi_isp_tnr3_params {
> + __u32 knee_y1;
> + __u32 knee_y2;
> + __u32 maxfb_y;
> + __u32 maxfb_u;
> + __u32 maxfb_v;
> + __u32 round_adj_y;
> + __u32 round_adj_u;
> + __u32 round_adj_v;
> + __u32 ref_buf_select;
> +} __packed;
> +
> +/* Extreme Noise Reduction version 3 */
> +
> +/**
> + * struct ipu3_uapi_isp_xnr3_vmem_params - Extreme noise reduction v3
> + * vector memory parameters
> + *
> + * @x: xnr3 parameters.
> + * @a: xnr3 parameters.
> + * @b: xnr3 parameters.
> + * @c: xnr3 parameters.
> + */
> +struct ipu3_uapi_isp_xnr3_vmem_params {
> + __u16 x[IPU3_UAPI_ISP_VEC_ELEMS];
> + __u16 a[IPU3_UAPI_ISP_VEC_ELEMS];
> + __u16 b[IPU3_UAPI_ISP_VEC_ELEMS];
> + __u16 c[IPU3_UAPI_ISP_VEC_ELEMS];
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_xnr3_alpha_params - Extreme noise reduction v3
> + * alpha tuning parameters
> + *
> + * @y0: Sigma for Y range similarity in dark area.
> + * @u0: Sigma for U range similarity in dark area.
> + * @v0: Sigma for V range similarity in dark area.
> + * @ydiff: Sigma difference for Y between bright area and dark area.
> + * @udiff: Sigma difference for U between bright area and dark area.
> + * @vdiff: Sigma difference for V between bright area and dark area.
> + */
> +struct ipu3_uapi_xnr3_alpha_params {
> + __u32 y0;
> + __u32 u0;
> + __u32 v0;
> + __u32 ydiff;
> + __u32 udiff;
> + __u32 vdiff;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_xnr3_coring_params - Extreme noise reduction v3
> + * coring parameters
> + *
> + * @u0: Coring Threshold of U channel in dark area.
> + * @v0: Coring Threshold of V channel in dark area.
> + * @udiff: Threshold difference of U channel between bright and dark area.
> + * @vdiff: Threshold difference of V channel between bright and dark area.
> + */
> +struct ipu3_uapi_xnr3_coring_params {
> + __u32 u0;
> + __u32 v0;
> + __u32 udiff;
> + __u32 vdiff;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_xnr3_blending_params - Blending factor
> + *
> + * @strength: The factor for blending output with input. This is tuning
> + * parameterHigher values lead to more aggressive XNR operation.
> + */
> +struct ipu3_uapi_xnr3_blending_params {
> + __u32 strength;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_isp_xnr3_params - Extreme noise reduction v3 parameters
> + *
> + * @alpha: parameters for xnr3 alpha. See &ipu3_uapi_xnr3_alpha_params
> + * @coring: parameters for xnr3 coring. See &ipu3_uapi_xnr3_coring_params
> + * @blending: parameters for xnr3 blending. See &ipu3_uapi_xnr3_blending_params
> + */
> +struct ipu3_uapi_isp_xnr3_params {
> + struct ipu3_uapi_xnr3_alpha_params alpha;
> + struct ipu3_uapi_xnr3_coring_params coring;
> + struct ipu3_uapi_xnr3_blending_params blending;
> +} __packed;
> +
> +/***** Obgrid (optical black level compensation) table entry *****/
> +
> +/**
> + * struct ipu3_uapi_obgrid_param - Optical black level compensation parameters
> + *
> + * @gr: Grid table values for color GR
> + * @r: Grid table values for color R
> + * @b: Grid table values for color B
> + * @gb: Grid table values for color GB
> + *
> + * Black level is different for red, green, and blue channels. So black level
> + * compensation is different per channel.
> + */
> +struct ipu3_uapi_obgrid_param {
> + __u16 gr;
> + __u16 r;
> + __u16 b;
> + __u16 gb;
> +} __packed;
> +
> +/******************* V4L2_META_FMT_IPU3_PARAMS *******************/
> +
> +/**
> + * struct ipu3_uapi_flags - bits to indicate which pipeline needs update
> + *
> + * @gdc: 0 = no update, 1 = update.
> + * @obgrid: 0 = no update, 1 = update.
> + * @__reserved1: Not used.
> + * @acc_bnr: 0 = no update, 1 = update.
> + * @acc_green_disparity: 0 = no update, 1 = update.
> + * @acc_dm: 0 = no update, 1 = update.
> + * @acc_ccm: 0 = no update, 1 = update.
> + * @acc_gamma: 0 = no update, 1 = update.
> + * @acc_csc: 0 = no update, 1 = update.
> + * @acc_cds: 0 = no update, 1 = update.
> + * @acc_shd: 0 = no update, 1 = update.
> + * @__reserved2: Not used.
> + * @acc_iefd: 0 = no update, 1 = update.
> + * @acc_yds_c0: 0 = no update, 1 = update.
> + * @acc_chnr_c0: 0 = no update, 1 = update.
> + * @acc_y_ee_nr: 0 = no update, 1 = update.
> + * @acc_yds: 0 = no update, 1 = update.
> + * @acc_chnr: 0 = no update, 1 = update.
> + * @acc_ytm: 0 = no update, 1 = update.
> + * @acc_yds2: 0 = no update, 1 = update.
> + * @acc_tcc: 0 = no update, 1 = update.
> + * @acc_dpc: 0 = no update, 1 = update.
> + * @acc_bds: 0 = no update, 1 = update.
> + * @acc_anr: 0 = no update, 1 = update.
> + * @acc_awb_fr: 0 = no update, 1 = update.
> + * @acc_ae: 0 = no update, 1 = update.
> + * @acc_af: 0 = no update, 1 = update.
> + * @acc_awb: 0 = no update, 1 = update.
> + * @__acc_osys: 0 = no update, 1 = update.
> + * @__reserved3: Not used.
> + * @lin_vmem_params: 0 = no update, 1 = update.
> + * @tnr3_vmem_params: 0 = no update, 1 = update.
> + * @xnr3_vmem_params: 0 = no update, 1 = update.
> + * @tnr3_dmem_params: 0 = no update, 1 = update.
> + * @xnr3_dmem_params: 0 = no update, 1 = update.
> + * @__reserved4: Not used.
> + * @obgrid_param: 0 = no update, 1 = update.
> + * @__reserved5: Not used.
> + */
> +struct ipu3_uapi_flags {
> + __u32 gdc:1;
> + __u32 obgrid:1;
> + __u32 __reserved1:30;
> +
> + __u32 acc_bnr:1;
> + __u32 acc_green_disparity:1;
> + __u32 acc_dm:1;
> + __u32 acc_ccm:1;
> + __u32 acc_gamma:1;
> + __u32 acc_csc:1;
> + __u32 acc_cds:1;
> + __u32 acc_shd:1;
> + __u32 __reserved2:2;
> + __u32 acc_iefd:1;
> + __u32 acc_yds_c0:1;
> + __u32 acc_chnr_c0:1;
> + __u32 acc_y_ee_nr:1;
> + __u32 acc_yds:1;
> + __u32 acc_chnr:1;
> + __u32 acc_ytm:1;
> + __u32 acc_yds2:1;
> + __u32 acc_tcc:1;
> + __u32 acc_dpc:1;
> + __u32 acc_bds:1;
> + __u32 acc_anr:1;
> + __u32 acc_awb_fr:1;
> + __u32 acc_ae:1;
> + __u32 acc_af:1;
> + __u32 acc_awb:1;
> + __u32 __reserved3:4;
> +
> + __u32 lin_vmem_params:1;
> + __u32 tnr3_vmem_params:1;
> + __u32 xnr3_vmem_params:1;
> + __u32 tnr3_dmem_params:1;
> + __u32 xnr3_dmem_params:1;
> + __u32 __reserved4:1;
> + __u32 obgrid_param:1;
> + __u32 __reserved5:25;
> +} __packed;
> +
> +/**
> + * struct ipu3_uapi_params - V4L2_META_FMT_IPU3_PARAMS
> + *
> + * @use: select which parameters to apply, see &ipu3_uapi_flags
> + * @acc_param: ACC parameters, as specified by &ipu3_uapi_acc_param
> + * @lin_vmem_params: linearization VMEM, as specified by
> + * &ipu3_uapi_isp_lin_vmem_params
> + * @tnr3_vmem_params: tnr3 VMEM as specified by
> + * &ipu3_uapi_isp_tnr3_vmem_params
> + * @xnr3_vmem_params: xnr3 VMEM as specified by
> + * &ipu3_uapi_isp_xnr3_vmem_params
> + * @tnr3_dmem_params: tnr3 DMEM as specified by &ipu3_uapi_isp_tnr3_params
> + * @xnr3_dmem_params: xnr3 DMEM as specified by &ipu3_uapi_isp_xnr3_params
> + * @obgrid_param: obgrid parameters as specified by
> + * &ipu3_uapi_obgrid_param
> + *
> + * The video queue "parameters" is of format V4L2_META_FMT_IPU3_PARAMS.
> + * This is a "single plane" v4l2_meta_format using V4L2_BUF_TYPE_META_OUTPUT.
> + *
> + * struct ipu3_uapi_params as defined below contains a lot of parameters and
> + * ipu3_uapi_flags selects which parameters to apply.
> + */
> +struct ipu3_uapi_params {
> + /* Flags which of the settings below are to be applied */
> + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> +
> + /* Accelerator cluster parameters */
> + struct ipu3_uapi_acc_param acc_param;
> +
> + /* ISP vector address space parameters */
> + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> +
> + /* ISP data memory (DMEM) parameters */
> + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> +
> + /* Optical black level compensation */
> + struct ipu3_uapi_obgrid_param obgrid_param;
> +} __packed;
> +#endif
>
This looks very good. I'm sure you've been less than happy with us for having
to document all this, but the end result is very impressive.
Based on my comment and those of others I would expect that v8 will be the
final version.
Regards,
Hans
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
2018-11-15 12:51 ` Hans Verkuil
@ 2018-11-21 18:45 ` Zhi, Yong
0 siblings, 0 replies; 123+ messages in thread
From: Zhi, Yong @ 2018-11-21 18:45 UTC (permalink / raw)
To: 'Hans Verkuil', linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, Mani, Rajmohan,
Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu,
Cao, Bingbu, Li, Chao C
Hi, Hans,
Thanks for the review.
> -----Original Message-----
> From: Hans Verkuil [mailto:hverkuil@xs4all.nl]
> Sent: Thursday, November 15, 2018 6:51 AM
> To: Zhi, Yong <yong.zhi@intel.com>; linux-media@vger.kernel.org;
> sakari.ailus@linux.intel.com
> Cc: tfiga@chromium.org; mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>; Li, Chao C <chao.c.li@intel.com>
> Subject: Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
>
> On 10/29/18 23:22, Yong Zhi wrote:
> > These meta formats are used on Intel IPU3 ImgU video queues
> > to carry 3A statistics and ISP pipeline parameters.
> >
> > V4L2_META_FMT_IPU3_3A
> > V4L2_META_FMT_IPU3_PARAMS
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > Signed-off-by: Chao C Li <chao.c.li@intel.com>
> > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> > ---
> > Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> > .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 ++
> > include/uapi/linux/intel-ipu3.h | 2819 ++++++++++++++++++++
> > 3 files changed, 3001 insertions(+)
> > create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-
> ipu3.rst
> > create mode 100644 include/uapi/linux/intel-ipu3.h
> >
> > diff --git a/Documentation/media/uapi/v4l/meta-formats.rst
> b/Documentation/media/uapi/v4l/meta-formats.rst
> > index cf971d5..eafc534 100644
> > --- a/Documentation/media/uapi/v4l/meta-formats.rst
> > +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> > @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata`
> interface only.
> > .. toctree::
> > :maxdepth: 1
> >
> > + pixfmt-meta-intel-ipu3
> > pixfmt-meta-d4xx
> > pixfmt-meta-uvc
> > pixfmt-meta-vsp1-hgo
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > new file mode 100644
> > index 0000000..23b945b
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > @@ -0,0 +1,181 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _intel-ipu3:
> > +
> >
> +***************************************************************
> ***
> > +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A
> ('ip3s')
> >
> +***************************************************************
> ***
> > +
> > +.. c:type:: ipu3_uapi_stats_3a
> > +
> > +3A statistics
> > +=============
> > +
> > +For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
> > +an input bayer frame. Those statistics, defined in data struct
> > +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu
> 3a stat"
>
> I'd rephrase this:
>
> are obtained from the "ipu3-imgu 3a stat" metadata capture video node,
>
Ack, that's more clear and natural.
> > +video node, which are then passed to user space for statistics analysis
> > +using :c:type:`v4l2_meta_format` interface.
> > +
> > +The statistics collected are AWB (Auto-white balance) RGBS (Red, Green,
> Blue and
> > +Saturation measure) cells, AWB filter response, AF (Auto-focus) filter
> response,
> > +and AE (Auto-exposure) histogram.
> > +
> > +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all
> above.
> > +
> > +
> > +.. code-block:: c
> > +
> > +
> > + struct ipu3_uapi_stats_3a {
> > + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> > + __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_raw_buffer_aligned
> > + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> > + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> > + struct ipu3_uapi_4a_config stats_4a_config;
> > + __u32 ae_join_buffers;
> > + __u8 padding[28];
> > + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + stats_3a_bubble_per_stripe;
> > + struct ipu3_uapi_ff_status stats_3a_status;
> > + } __packed;
> > +
> > +
> > +.. c:type:: ipu3_uapi_params
> > +
> > +Pipeline parameters
> > +===================
> > +
> > +IPU3 pipeline has a number of image processing stages, each of which
> takes a
> > +set of parameters as input. The major stages of pipelines are shown here:
> > +
> > +Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
> > +
> > +Linearization -> Lens Shading Correction -> White Balance / Exposure /
> > +
> > +Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
> > +
> > +Correction Matrix -> Gamma correction -> Color Space Conversion ->
> > +
> > +Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
> > +
> > +Correction -> XNR3 -> TNR -> DDR
> > +
> > +The table below presents a description of the above algorithms.
> > +
> > +========================
> =======================================================
> > +Name Description
> > +========================
> =======================================================
> > +Optical Black Correction Optical Black Correction block subtracts a pre-
> defined
> > + value from the respective pixel values to obtain
> better
> > + image quality.
> > + Defined in :c:type:`ipu3_uapi_obgrid_param`.
> > +Linearization This algo block uses linearization parameters
> to
> > + address non-linearity sensor effects. The Lookup
> table
> > + table is defined in
> > + :c:type:`ipu3_uapi_isp_lin_vmem_params`.
> > +SHD Lens shading correction is used to correct spatial
> > + non-uniformity of the pixel response due to optical
> > + lens shading. This is done by applying a different
> gain
> > + for each pixel. The gain, black level etc are
> > + configured in :c:type:`ipu3_uapi_shd_config_static`.
> > +BNR Bayer noise reduction block removes image noise by
> > + applying a bilateral filter.
> > + See :c:type:`ipu3_uapi_bnr_static_config` for details.
> > +ANR Advanced Noise Reduction is a block based algorithm
> > + that performs noise reduction in the Bayer domain.
> The
> > + convolution matrix etc can be found in
> > + :c:type:`ipu3_uapi_anr_config`.
> > +Demosaicing Demosaicing converts raw sensor data in
> Bayer format
> > + into RGB (Red, Green, Blue) presentation. Then add
> > + outputs of estimation of Y channel for following
> stream
>
> "Then add outputs of estimation..."
>
> Seems a bit garbled, I'm not quite sure what is meant here.
>
Will rework the above description.
> > + processing by Firmware. The struct is defined as
> > + :c:type:`ipu3_uapi_dm_config`.
> > +Color Correction Color Correction algo transforms sensor specific
> color
> > + space to the standard "sRGB" color space. This is
> done
> > + by applying 3x3 matrix defined in
> > + :c:type:`ipu3_uapi_ccm_mat_config`.
> > +Gamma correction Gamma
> correction :c:type:`ipu3_uapi_gamma_config` is a
> > + basic non-linear tone mapping correction that is
> > + applied per pixel for each pixel component.
> > +CSC Color space conversion transforms each pixel from
> the
> > + RGB primary presentation to YUV (Y - brightness,
> > + UV - Luminance) presentation. This is done by
> applying
>
> Use Y: and UV: instead of '-' to avoid interpreting this as substraction.
>
> > + a 3x3 matrix defined in
> > + :c:type:`ipu3_uapi_csc_mat_config`
> > +CDS Chroma down sampling
> > + After the CSC is performed, the Chroma Down
> Sampling
> > + is applied for a UV plane down sampling by a factor
> > + of 2 in each direction for YUV 4:2:0 using a 4x2
> > + configurable filter :c:type:`ipu3_uapi_cds_params`.
> > +CHNR Chroma noise reduction
> > + This block processes only the chrominance pixels
> and
> > + performs noise reduction by cleaning the high
> > + frequency noise.
> > + See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
> > +TCC Total color correction as defined in struct
> > + :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
> > +XNR3 eXtreme Noise Reduction V3 is the third
> revision of
> > + noise reduction algorithm used to improve image
> > + quality. This removes the low frequency noise in the
> > + captured image. Two related structs are being
> defined,
> > + :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data
> memory
> > + and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for
> vector
> > + memory.
> > +TNR Temporal Noise Reduction block compares
> successive
> > + frames in time to remove anomalies / noise in pixel
> > + values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params`
> and
> > + :c:type:`ipu3_uapi_isp_tnr3_params` are defined for
> ISP
> > + vector and data memory respectively.
> > +========================
> =======================================================
> > +
> > +A few stages of the pipeline will be executed by firmware running on the
> ISP
> > +processor, while many others will use a set of fixed hardware blocks also
> > +called accelerator cluster (ACC) to crunch pixel data and produce statistics.
> > +
> > +ACC parameters as defined by :c:type:`ipu3_uapi_acc_param`, can be
> selectively
> > +enabled / disabled by the user space through
> struct :c:type:`ipu3_uapi_flags`
> > +embedded in :c:type:`ipu3_uapi_params` structure. For parameters that
> are not
> > +enabled by the user space, corresponding structs are ignored by the ISP.
> > +
> > +Both 3A statistics and pipeline parameters described here are closely tied
> to
> > +the underlying camera sub-system (CSS) APIs. They are usually consumed
> and
> > +produced by dedicated user space libraries that comprise the important
> tuning
> > +tools, thus freeing the developers from being bothered with the low level
> > +hardware and algorithm details.
> > +
> > +It should be noted that IPU3 DMA operations require the addresses of all
> data
> > +structures (that includes both input and output) to be aligned on 32 byte
> > +boundaries.
> > +
> > +The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu
> parameters"
> > +video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
> > +
> > +.. code-block:: c
> > +
> > + struct ipu3_uapi_params {
> > + /* Flags which of the settings below are to be applied */
> > + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> > +
> > + /* Accelerator cluster parameters */
> > + struct ipu3_uapi_acc_param acc_param;
> > +
> > + /* ISP vector address space parameters */
> > + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> > + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> > + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> > +
> > + /* ISP data memory (DMEM) parameters */
> > + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> > + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> > +
> > + /* Optical black level compensation */
> > + struct ipu3_uapi_obgrid_param obgrid_param;
> > + } __packed;
> > +
> > +Intel IPU3 ImgU uAPI data types
> > +===============================
> > +
> > +.. kernel-doc:: include/uapi/linux/intel-ipu3.h
> > diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
> > new file mode 100644
> > index 0000000..c2608b6
> > --- /dev/null
> > +++ b/include/uapi/linux/intel-ipu3.h
> > @@ -0,0 +1,2819 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2017 - 2018 Intel Corporation */
> > +
> > +#ifndef __IPU3_UAPI_H
> > +#define __IPU3_UAPI_H
> > +
> > +#include <linux/types.h>
> > +
> > +/********************* Key Acronyms *************************/
> > +/*
> > + * ACC - Accelerator cluster
> > + * ANR - Adaptive noise reduction
> > + * AWB_FR- Auto white balance filter response statistics
> > + * BNR - Bayer noise reduction parameters
> > + * BDS - Bayer downscaler parameters
> > + * CCM - Color correction matrix coefficients
> > + * CDS - Chroma down sample
> > + * CHNR - Chroma noise reduction
> > + * CSC - Color space conversion
> > + * DM - De-mosaic
> > + * IEFd - Image enhancement filter directed
> > + * Obgrid - Optical black level compensation
> > + * OSYS - Output system configuration
> > + * ROI - Region of interest
> > + * SHD - Lens shading correction table
> > + * TCC - Total color correction
> > + * YDS - Y down sampling
> > + * YTM - Y-tone mapping
> > + */
> > +
> > +/*
> > + * IPU3 DMA operations require buffers to be aligned at
> > + * 32 byte boundaries
> > + */
> > +
> > +/******************* ipu3_uapi_stats_3a *******************/
> > +
> > +#define IPU3_UAPI_MAX_STRIPES 2
> > +#define IPU3_UAPI_MAX_BUBBLE_SIZE 10
> > +
> > +#define IPU3_UAPI_GRID_START_MASK ((1 << 12) - 1)
> > +#define IPU3_UAPI_GRID_Y_START_EN (1 << 15)
> > +
> > +/* controls generation of meta_data (like FF enable/disable) */
> > +#define IPU3_UAPI_AWB_RGBS_THR_B_EN (1 << 14)
> > +#define IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT (1 << 15)
> > +
> > +/**
> > + * struct ipu3_uapi_grid_config - Grid plane config
> > + *
> > + * @width: Grid horizontal dimensions, in number of grid blocks(cells).
> > + * @height: Grid vertical dimensions, in number of grid cells.
> > + * @block_width_log2: Log2 of the width of each cell in pixels.
> > + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> > + * @block_height_log2: Log2 of the height of each cell in pixels.
> > + * for (2^3, 2^4, 2^5, 2^6, 2^7), values [3, 7].
> > + * @height_per_slice: The number of blocks in vertical axis per slice.
> > + * Default 2.
> > + * @x_start: X value of top left corner of Region of Interest(ROI).
> > + * @y_start: Y value of top left corner of ROI
> > + * @x_end: X value of bottom right corner of ROI
> > + * @y_end: Y value of bottom right corner of ROI
> > + *
> > + * Due to the size of total amount of collected data, most statistics
> > + * create a grid-based output, and the data is then divided into "slices".
> > + */
> > +struct ipu3_uapi_grid_config {
> > + __u8 width;
> > + __u8 height;
> > + __u16 block_width_log2:3;
> > + __u16 block_height_log2:3;
> > + __u16 height_per_slice:8;
> > + __u16 x_start;
> > + __u16 y_start;
> > + __u16 x_end;
> > + __u16 y_end;
> > +} __packed;
> > +
> > +/*
> > + * The grid based data is divided into "slices" called set, each slice of setX
> > + * refers to ipu3_uapi_grid_config width * height_per_slice.
> > + */
> > +#define IPU3_UAPI_AWB_MAX_SETS 60
> > +/* Based on grid size 80 * 60 and cell size 16 x 16 */
> > +#define IPU3_UAPI_AWB_SET_SIZE 1280
> > +#define IPU3_UAPI_AWB_MD_ITEM_SIZE 8
> > +#define IPU3_UAPI_AWB_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AWB_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AWB_MAX_BUFFER_SIZE \
> > + (IPU3_UAPI_AWB_MAX_SETS * \
> > + (IPU3_UAPI_AWB_SET_SIZE +
> IPU3_UAPI_AWB_SPARE_FOR_BUBBLES))
>
> Add an empty line here.
>
Ack.
> > +/**
> > + * struct ipu3_uapi_awb_meta_data - AWB meta data
> > + *
> > + * @meta_data_buffer: Average values for each color channel
> > + */
> > +struct ipu3_uapi_awb_meta_data {
> > + __u8 meta_data_buffer[IPU3_UAPI_AWB_MAX_BUFFER_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_raw_buffer - AWB raw buffer
> > + *
> > + * @meta_data: buffer to hold auto white balance meta data.
> > + */
> > +struct ipu3_uapi_awb_raw_buffer {
> > + struct ipu3_uapi_awb_meta_data meta_data;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_config_s - AWB config
> > + *
> > + * @rgbs_thr_gr: gr threshold value.
> > + * @rgbs_thr_r: Red threshold value.
> > + * @rgbs_thr_gb: gb threshold value.
> > + * @rgbs_thr_b: Blue threshold value.
> > + * @grid: &ipu3_uapi_grid_config, the default grid resolution is 16x16 cells.
> > + *
> > + * The threshold is a saturation measure range [0, 8191], 8191 is default.
> > + * Values over threshold may be optionally rejected for averaging.
> > + */
> > +struct ipu3_uapi_awb_config_s {
> > + __u16 rgbs_thr_gr;
> > + __u16 rgbs_thr_r;
> > + __u16 rgbs_thr_gb;
> > + __u16 rgbs_thr_b;
> > + struct ipu3_uapi_grid_config grid;
> > +} __attribute__((aligned(32))) __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_config - AWB config wrapper
> > + *
> > + * @config: config for auto white balance as defined by
> &ipu3_uapi_awb_config_s
> > + */
> > +struct ipu3_uapi_awb_config {
> > + struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AE_COLORS 4 /* R,
> G, B, Y */
> > +#define IPU3_UAPI_AE_BINS 256
> > +#define IPU3_UAPI_AE_WEIGHTS 96
> > +
> > +/**
> > + * struct ipu3_uapi_ae_raw_buffer - AE global weighted histogram
> > + *
> > + * @vals: Sum of IPU3_UAPI_AE_COLORS in cell
> > + *
> > + * Each histogram contains IPU3_UAPI_AE_BINS bins. Each bin has 24 bit
> unsigned
> > + * for counting the number of the pixel.
> > + */
> > +struct ipu3_uapi_ae_raw_buffer {
> > + __u32 vals[IPU3_UAPI_AE_BINS * IPU3_UAPI_AE_COLORS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_raw_buffer_aligned - AE raw buffer
> > + *
> > + * @buff: &ipu3_uapi_ae_raw_buffer to hold full frame meta data.
> > + */
> > +struct ipu3_uapi_ae_raw_buffer_aligned {
> > + struct ipu3_uapi_ae_raw_buffer buff __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_grid_config - AE weight grid
> > + *
> > + * @width: Grid horizontal dimensions. Value: [16, 32], default 16.
> > + * @height: Grid vertical dimensions. Value: [16, 24], default 16.
> > + * @block_width_log2: Log2 of the width of the grid cell, 2^3 = 16.
> > + * @block_height_log2: Log2 of the height of the grid cell, 2^3 = 16.
> > + * @__reserved0: reserved
> > + * @ae_en: 0: does not write to meta-data array, 1: write normally.
> > + * @rst_hist_array: write 1 to trigger histogram array reset.
> > + * @done_rst_hist_array: flag for histogram array reset done.
> > + * @x_start: X value of top left corner of ROI, default 0.
> > + * @y_start: Y value of top left corner of ROI, default 0.
> > + * @x_end: X value of bottom right corner of ROI
> > + * @y_end: Y value of bottom right corner of ROI
> > + *
> > + * The AE block accumulates 4 global weighted histograms(R, G, B, Y) over
> > + * a defined ROI within the frame. The contribution of each pixel into the
> > + * histogram, defined by &ipu3_uapi_ae_weight_elem LUT, is indexed by a
> grid.
> > + */
> > +struct ipu3_uapi_ae_grid_config {
> > + __u8 width;
> > + __u8 height;
> > + __u8 block_width_log2:4;
> > + __u8 block_height_log2:4;
> > + __u8 __reserved0:5;
> > + __u8 ae_en:1;
> > + __u8 rst_hist_array:1;
> > + __u8 done_rst_hist_array:1;
> > + __u16 x_start;
> > + __u16 y_start;
> > + __u16 x_end;
> > + __u16 y_end;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_weight_elem - AE weights LUT
> > + *
> > + * @cell0: weighted histogram grid value.
> > + * @cell1: weighted histogram grid value.
> > + * @cell2: weighted histogram grid value.
> > + * @cell3: weighted histogram grid value.
> > + * @cell4: weighted histogram grid value.
> > + * @cell5: weighted histogram grid value.
> > + * @cell6: weighted histogram grid value.
> > + * @cell7: weighted histogram grid value.
> > + *
> > + * Use weighted grid value to give a different contribution factor to each
> cell.
> > + * Precision u4, range [0, 15].
> > + */
> > +struct ipu3_uapi_ae_weight_elem {
> > + __u32 cell0:4;
> > + __u32 cell1:4;
> > + __u32 cell2:4;
> > + __u32 cell3:4;
> > + __u32 cell4:4;
> > + __u32 cell5:4;
> > + __u32 cell6:4;
> > + __u32 cell7:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_ccm - AE coefficients for WB and CCM
> > + *
> > + * @gain_gr: WB gain factor for the gr channels. Default 256.
> > + * @gain_r: WB gain factor for the r channel. Default 256.
> > + * @gain_b: WB gain factor for the b channel. Default 256.
> > + * @gain_gb: WB gain factor for the gb channels. Default 256.
> > + * @mat: 4x4 matrix that transforms Bayer quad output from WB to
> RGB+Y.
> > + *
> > + * Default:
> > + * 128, 0, 0, 0,
> > + * 0, 128, 0, 0,
> > + * 0, 0, 128, 0,
> > + * 0, 0, 0, 128,
> > + *
> > + * As part of the raw frame pre-process stage, the WB and color
> conversion need
> > + * to be applied to expose the impact of these gain operations.
> > + */
> > +struct ipu3_uapi_ae_ccm {
> > + __u16 gain_gr;
> > + __u16 gain_r;
> > + __u16 gain_b;
> > + __u16 gain_gb;
> > + __s16 mat[16];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ae_config - AE config
> > + *
> > + * @grid_cfg: config for auto exposure statistics grid. See struct
> > + * &ipu3_uapi_ae_grid_config
> > + * @weights: &IPU3_UAPI_AE_WEIGHTS is based on 32x24 blocks
> in the grid.
> > + * Each grid cell has a corresponding value in weights LUT called
> > + * grid value, global histogram is updated based on grid value
> and
> > + * pixel value.
> > + * @ae_ccm: Color convert matrix pre-processing block.
> > + *
> > + * Calculate AE grid from image resolution, resample ae weights.
> > + */
> > +struct ipu3_uapi_ae_config {
> > + struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_weight_elem weights[
> > + IPU3_UAPI_AE_WEIGHTS]
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_filter_config - AF 2D filter for contrast
> measurements
> > + *
> > + * @y1_coeff_0: filter Y1, structure: 3x11, support both symmetry and
> > + * anti-symmetry type. A12 is center, A1-A11 are neighbours.
> > + * for analyzing low frequency content, used to calculate sum
> > + * of gradients in x direction.
> > + * @y1_coeff_0.a1: filter1 coefficients A1, u8, default 0.
> > + * @y1_coeff_0.a2: filter1 coefficients A2, u8, default 0.
> > + * @y1_coeff_0.a3: filter1 coefficients A3, u8, default 0.
> > + * @y1_coeff_0.a4: filter1 coefficients A4, u8, default 0.
> > + * @y1_coeff_1: Struct
> > + * @y1_coeff_1.a5: filter1 coefficients A5, u8, default 0.
> > + * @y1_coeff_1.a6: filter1 coefficients A6, u8, default 0.
> > + * @y1_coeff_1.a7: filter1 coefficients A7, u8, default 0.
> > + * @y1_coeff_1.a8: filter1 coefficients A8, u8, default 0.
> > + * @y1_coeff_2: Struct
> > + * @y1_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> > + * @y1_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> > + * @y1_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> > + * @y1_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> > + * @y1_sign_vec: Each bit corresponds to one coefficient sign bit,
> > + * 0: positive, 1: negative, default 0.
> > + * @y2_coeff_0: Y2, same structure as Y1. For analyzing high
> frequency content.
> > + * @y2_coeff_0.a1: filter2 coefficients A1, u8, default 0.
> > + * @y2_coeff_0.a2: filter2 coefficients A2, u8, default 0.
> > + * @y2_coeff_0.a3: filter2 coefficients A3, u8, default 0.
> > + * @y2_coeff_0.a4: filter2 coefficients A4, u8, default 0.
> > + * @y2_coeff_1: Struct
> > + * @y2_coeff_1.a5: filter2 coefficients A5, u8, default 0.
> > + * @y2_coeff_1.a6: filter2 coefficients A6, u8, default 0.
> > + * @y2_coeff_1.a7: filter2 coefficients A7, u8, default 0.
> > + * @y2_coeff_1.a8: filter2 coefficients A8, u8, default 0.
> > + * @y2_coeff_2: Struct
> > + * @y2_coeff_2.a9: filter1 coefficients A9, u8, default 0.
> > + * @y2_coeff_2.a10: filter1 coefficients A10, u8, default 0.
> > + * @y2_coeff_2.a11: filter1 coefficients A11, u8, default 0.
> > + * @y2_coeff_2.a12: filter1 coefficients A12, u8, default 128.
> > + * @y2_sign_vec: Each bit corresponds to one coefficient sign bit,
> > + * 0: positive, 1: negative, default 0.
> > + * @y_calc: Pre-processing that converts Bayer quad to RGB+Y values to
> be
> > + * used for building histogram. Range [0, 32], default 8.
> > + * Rule:
> > + * y_gen_rate_gr + y_gen_rate_r + y_gen_rate_b +
> y_gen_rate_gb = 32
> > + * A single Y is calculated based on sum of Gr/R/B/Gb based on
> > + * their contribution ratio.
> > + * @y_calc.y_gen_rate_gr: Contribution ratio Gr for Y
> > + * @y_calc.y_gen_rate_r: Contribution ratio R for Y
> > + * @y_calc.y_gen_rate_b: Contribution ratio B for Y
> > + * @y_calc.y_gen_rate_gb: Contribution ratio Gb for Y
> > + * @nf: The shift right value that should be applied during the Y1/Y2
> filter to
> > + * make sure the total memory needed is 2 bytes per grid cell.
> > + * @nf.__reserved0: reserved
> > + * @nf.y1_nf: Normalization factor for the convolution coeffs of y1,
> > + * should be log2 of the sum of the abs values of the filter
> > + * coeffs, default 7 (2^7 = 128).
> > + * @nf.__reserved1: reserved
> > + * @nf.y2_nf: Normalization factor for y2, should be log2 of the
> sum of the
> > + * abs values of the filter coeffs.
> > + * @nf.__reserved2: reserved
> > + */
> > +struct ipu3_uapi_af_filter_config {
> > + struct {
> > + __u8 a1;
> > + __u8 a2;
> > + __u8 a3;
> > + __u8 a4;
> > + } y1_coeff_0;
> > + struct {
> > + __u8 a5;
> > + __u8 a6;
> > + __u8 a7;
> > + __u8 a8;
> > + } y1_coeff_1;
> > + struct {
> > + __u8 a9;
> > + __u8 a10;
> > + __u8 a11;
> > + __u8 a12;
> > + } y1_coeff_2;
> > +
> > + __u32 y1_sign_vec;
> > +
> > + struct {
> > + __u8 a1;
> > + __u8 a2;
> > + __u8 a3;
> > + __u8 a4;
> > + } y2_coeff_0;
> > + struct {
> > + __u8 a5;
> > + __u8 a6;
> > + __u8 a7;
> > + __u8 a8;
> > + } y2_coeff_1;
> > + struct {
> > + __u8 a9;
> > + __u8 a10;
> > + __u8 a11;
> > + __u8 a12;
> > + } y2_coeff_2;
> > +
> > + __u32 y2_sign_vec;
> > +
> > + struct {
> > + __u8 y_gen_rate_gr;
> > + __u8 y_gen_rate_r;
> > + __u8 y_gen_rate_b;
> > + __u8 y_gen_rate_gb;
> > + } y_calc;
> > +
> > + struct {
> > + __u32 __reserved0:8;
> > + __u32 y1_nf:4;
> > + __u32 __reserved1:4;
> > + __u32 y2_nf:4;
> > + __u32 __reserved2:12;
> > + } nf;
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AF_MAX_SETS 24
> > +#define IPU3_UAPI_AF_MD_ITEM_SIZE 4
> > +#define IPU3_UAPI_AF_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AF_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AF_Y_TABLE_SET_SIZE 128
> > +#define IPU3_UAPI_AF_Y_TABLE_MAX_SIZE \
> > + (IPU3_UAPI_AF_MAX_SETS * \
> > + (IPU3_UAPI_AF_Y_TABLE_SET_SIZE +
> IPU3_UAPI_AF_SPARE_FOR_BUBBLES) * \
> > + IPU3_UAPI_MAX_STRIPES)
> > +
> > +/**
> > + * struct ipu3_uapi_af_meta_data - AF meta data
> > + *
> > + * @y_table: Each color component will be convolved separately with
> filter1
> > + * and filter2 and the result will be summed out and averaged
> for
> > + * each cell.
> > + */
> > +struct ipu3_uapi_af_meta_data {
> > + __u8 y_table[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_raw_buffer - AF raw buffer
> > + *
> > + * @meta_data: raw buffer &ipu3_uapi_af_meta_data for auto focus
> meta data.
> > + */
> > +struct ipu3_uapi_af_raw_buffer {
> > + struct ipu3_uapi_af_meta_data meta_data
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_config_s - AF config
> > + *
> > + * @filter_config: AF uses Y1 and Y2 filters as configured in
> > + * &ipu3_uapi_af_filter_config
> > + * @padding: paddings
> > + * @grid_cfg: See &ipu3_uapi_grid_config, default resolution 16x16. Use
> large
> > + * grid size for large image and vice versa.
> > + */
> > +struct ipu3_uapi_af_config_s {
> > + struct ipu3_uapi_af_filter_config filter_config
> __attribute__((aligned(32)));
> > + __u8 padding[4];
> > + struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_af_config - AF config wrapper
> > + *
> > + * @config: config for auto focus as defined by &ipu3_uapi_af_config_s
> > + */
> > +struct ipu3_uapi_af_config {
> > + struct ipu3_uapi_af_config_s config;
> > +} __packed;
> > +
> > +#define IPU3_UAPI_AWB_FR_MAX_SETS 24
> > +#define IPU3_UAPI_AWB_FR_MD_ITEM_SIZE 8
> > +#define IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE 256
> > +#define IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES \
> > + (IPU3_UAPI_MAX_BUBBLE_SIZE * IPU3_UAPI_MAX_STRIPES * \
> > + IPU3_UAPI_AWB_FR_MD_ITEM_SIZE)
> > +#define IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE \
> > + (IPU3_UAPI_AWB_FR_MAX_SETS * \
> > + (IPU3_UAPI_AWB_FR_BAYER_TBL_SIZE + \
> > + IPU3_UAPI_AWB_FR_SPARE_FOR_BUBBLES) *
> IPU3_UAPI_MAX_STRIPES)
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_meta_data - AWB filter response meta data
> > + *
> > + * @bayer_table: Statistics output on the grid after convolving with 1D
> filter.
> > + */
> > +struct ipu3_uapi_awb_fr_meta_data {
> > + __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE]
> __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_raw_buffer - AWB filter response raw buffer
> > + *
> > + * @meta_data: See &ipu3_uapi_awb_fr_meta_data.
> > + */
> > +struct ipu3_uapi_awb_fr_raw_buffer {
> > + struct ipu3_uapi_awb_fr_meta_data meta_data;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_config_s - AWB filter response config
> > + *
> > + * @grid_cfg: grid config, default 16x16.
> > + * @bayer_coeff: 1D Filter 1x11 center symmetry/anti-symmetry.
> > + * coeffcients defaults { 0, 0, 0, 0, 0, 128 }.
> > + * Applied on whole image for each Bayer channel
> separately
> > + * by a weighted sum of its 11x1 neighbors.
> > + * @__reserved1: reserved
> > + * @bayer_sign: sign of filter coeffcients, default 0.
> > + * @bayer_nf: normalization factor for the convolution coeffs, to
> make sure
> > + * total memory needed is within pre-determined range.
> > + * NF should be the log2 of the sum of the abs values of the
> > + * filter coeffs, range [7, 14], default 7.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_awb_fr_config_s {
> > + struct ipu3_uapi_grid_config grid_cfg;
> > + __u8 bayer_coeff[6];
> > + __u16 __reserved1;
> > + __u32 bayer_sign;
> > + __u8 bayer_nf;
> > + __u8 __reserved2[3];
> > +} __attribute__((aligned(32))) __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_awb_fr_config - AWB filter response config wrapper
> > + *
> > + * @config: See &ipu3_uapi_awb_fr_config_s.
> > + */
> > +struct ipu3_uapi_awb_fr_config {
> > + struct ipu3_uapi_awb_fr_config_s config;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_4a_config - 4A config
> > + *
> > + * @awb_config: &ipu3_uapi_awb_config_s, default resolution 16x16
> > + * @ae_grd_config: auto exposure statistics &ipu3_uapi_ae_grid_config
> > + * @padding: paddings
> > + * @af_config: auto focus config &ipu3_uapi_af_config_s
> > + * @awb_fr_config: &ipu3_uapi_awb_fr_config_s, default resolution
> 16x16
> > + */
> > +struct ipu3_uapi_4a_config {
> > + struct ipu3_uapi_awb_config_s awb_config
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_grid_config ae_grd_config;
> > + __u8 padding[20];
> > + struct ipu3_uapi_af_config_s af_config;
> > + struct ipu3_uapi_awb_fr_config_s awb_fr_config;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bubble_info - Bubble info for host side debugging
> > + *
> > + * @num_of_stripes: A single frame is divided into several parts called
> stripes
> > + * due to limitation on line buffer memory.
> > + * The separation between the stripes is vertical. Each such
> > + * stripe is processed as a single frame by the ISP pipe.
> > + * @padding: padding bytes.
> > + * @num_sets: number of sets.
> > + * @padding1: padding bytes.
> > + * @size_of_set: set size.
> > + * @padding2: padding bytes.
> > + * @bubble_size: is the amount of padding in the bubble expressed in
> "sets".
> > + * @padding3: padding bytes.
> > + */
> > +struct ipu3_uapi_bubble_info {
> > + __u32 num_of_stripes __attribute__((aligned(32)));
> > + __u8 padding[28];
> > + __u32 num_sets;
> > + __u8 padding1[28];
> > + __u32 size_of_set;
> > + __u8 padding2[28];
> > + __u32 bubble_size;
> > + __u8 padding3[28];
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + */
> > +struct ipu3_uapi_stats_3a_bubble_info_per_stripe {
> > + struct ipu3_uapi_bubble_info awb[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_bubble_info af[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_bubble_info awb_fr[IPU3_UAPI_MAX_STRIPES];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ff_status - Enable bits for each 3A fixed function
> > + *
> > + * @awb_en: auto white balance enable
> > + * @padding: padding config
> > + * @ae_en: auto exposure enable
> > + * @padding1: padding config
> > + * @af_en: auto focus enable
> > + * @padding2: padding config
> > + * @awb_fr_en: awb filter response enable bit
> > + * @padding3: padding config
> > + */
> > +struct ipu3_uapi_ff_status {
> > + __u32 awb_en __attribute__((aligned(32)));
> > + __u8 padding[28];
> > + __u32 ae_en;
> > + __u8 padding1[28];
> > + __u32 af_en;
> > + __u8 padding2[28];
> > + __u32 awb_fr_en;
> > + __u8 padding3[28];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_stats_3a - 3A statistics
> > + *
> > + * @awb_raw_buffer: auto white balance meta data
> &ipu3_uapi_awb_raw_buffer
> > + * @ae_raw_buffer: auto exposure raw data
> &ipu3_uapi_ae_raw_buffer_aligned
> > + * @af_raw_buffer: &ipu3_uapi_af_raw_buffer for auto focus meta data
> > + * @awb_fr_raw_buffer: value as specified by
> &ipu3_uapi_awb_fr_raw_buffer
> > + * @stats_4a_config: 4a statistics config as defined by
> &ipu3_uapi_4a_config.
> > + * @ae_join_buffers: 1 to use ae_raw_buffer.
> > + * @padding: padding config
> > + * @stats_3a_bubble_per_stripe: a
> &ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + * @stats_3a_status: 3a statistics status set in &ipu3_uapi_ff_status
> > + */
> > +struct ipu3_uapi_stats_3a {
> > + struct ipu3_uapi_awb_raw_buffer awb_raw_buffer
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_ae_raw_buffer_aligned
> > + ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
> > + struct ipu3_uapi_af_raw_buffer af_raw_buffer;
> > + struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
> > + struct ipu3_uapi_4a_config stats_4a_config;
> > + __u32 ae_join_buffers;
> > + __u8 padding[28];
> > + struct ipu3_uapi_stats_3a_bubble_info_per_stripe
> > + stats_3a_bubble_per_stripe;
> > + struct ipu3_uapi_ff_status stats_3a_status;
> > +} __packed;
> > +
> > +/******************* ipu3_uapi_acc_param *******************/
> > +
> > +#define IPU3_UAPI_ISP_VEC_ELEMS 64
> > +#define IPU3_UAPI_ISP_TNR3_VMEM_LEN 9
> > +
> > +#define IPU3_UAPI_BNR_LUT_SIZE 32
> > +
> > +/* number of elements in gamma correction LUT */
> > +#define IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES 256
> > +
> > +/* largest grid is 73x56, for grid_height_per_slice of 2, 73x2 = 146 */
> > +#define IPU3_UAPI_SHD_MAX_CELLS_PER_SET 146
> > +#define IPU3_UAPI_SHD_MAX_CFG_SETS 28
> > +/* Normalization shift aka nf */
> > +#define IPU3_UAPI_SHD_BLGR_NF_SHIFT 13
> > +#define IPU3_UAPI_SHD_BLGR_NF_MASK 7
> > +
> > +#define IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS 16
> > +#define IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS 14
> > +#define IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS 258
> > +#define IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS 24
> > +
> > +#define IPU3_UAPI_ANR_LUT_SIZE 26
> > +#define IPU3_UAPI_ANR_PYRAMID_SIZE 22
> > +
> > +#define IPU3_UAPI_LIN_LUT_SIZE 64
> > +
> > +/* Bayer Noise Reduction related structs */
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_wb_gains_config - White balance
> gains
> > + *
> > + * @gr: white balance gain for Gr channel.
> > + * @r: white balance gain for R channel.
> > + * @b: white balance gain for B channel.
> > + * @gb: white balance gain for Gb channel.
> > + *
> > + * Precision u3.13, range [0, 8]. White balance correction is done by
> applying
> > + * a multiplicative gain to each color channels prior to BNR.
> > + */
> > +struct ipu3_uapi_bnr_static_config_wb_gains_config {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_wb_gains_thr_config - Threshold
> config
> > + *
> > + * @gr: white balance threshold gain for Gr channel.
> > + * @r: white balance threshold gain for R channel.
> > + * @b: white balance threshold gain for B channel.
> > + * @gb: white balance threshold gain for Gb channel.
> > + *
> > + * Defines the threshold that specifies how different a defect pixel can be
> from
> > + * its neighbors.(used by dynamic defect pixel correction sub block)
> > + * Precision u4.4 range [0, 8].
> > + */
> > +struct ipu3_uapi_bnr_static_config_wb_gains_thr_config {
> > + __u8 gr;
> > + __u8 r;
> > + __u8 b;
> > + __u8 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_thr_coeffs_config - Noise model
> > + * coefficients that controls noise threshold
> > + *
> > + * @cf: Free coefficient for threshold calculation, range [0, 8191],
> default 0.
> > + * @__reserved0: reserved
> > + * @cg: Gain coefficient for threshold calculation, [0, 31], default 8.
> > + * @ci: Intensity coefficient for threshold calculation. range [0, 0x1f]
> > + * default 6.
> > + * format: u3.2 (3 most significant bits represent whole number,
> > + * 2 least significant bits represent the fractional part
> > + * with each count representing 0.25)
> > + * e.g 6 in binary format is 00110, that translates to 1.5
> > + * @__reserved1: reserved
>
> Why use '__' prefix? It seems to me that calling it 'reserved1' is just fine.
> Unless there is some good reason for this, please remove the __ from such
> bitfields in this header. The use of '__' is reserved for compilers etc.
>
Ack, will remove '__' prefix for reserved in this file.
> > + * @r_nf: Normalization shift value for r^2 calculation, range [12, 20]
> > + * where r is a radius of pixel [row, col] from centor of sensor.
> > + * default 14.
> > + *
> > + * Threshold used to distinguish between noise and details.
> > + */
> > +struct ipu3_uapi_bnr_static_config_thr_coeffs_config {
> > + __u32 cf:13;
> > + __u32 __reserved0:3;
> > + __u32 cg:5;
> > + __u32 ci:5;
> > + __u32 __reserved1:1;
> > + __u32 r_nf:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config - Shading config
> > + *
> > + * @gr: Coefficient defines lens shading gain approximation for gr
> channel
> > + * @r: Coefficient defines lens shading gain approximation for r
> channel
> > + * @b: Coefficient defines lens shading gain approximation for b
> channel
> > + * @gb: Coefficient defines lens shading gain approximation for gb
> channel
> > + *
> > + * Parameters for noise model (NM) adaptation of BNR due to shading
> correction.
> > + * All above have precision of u3.3, default to 0.
> > + */
> > +struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config {
> > + __u8 gr;
> > + __u8 r;
> > + __u8 b;
> > + __u8 gb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_opt_center_config - Optical center
> config
> > + *
> > + * @x_reset: Reset value of X (col start - X center). Precision s12.0.
> > + * @__reserved0: reserved
> > + * @y_reset: Reset value of Y (row start - Y center). Precision s12.0.
> > + * @__reserved2: reserved
> > + *
> > + * Distance from corner to optical center for NM adaptation due to
> shading
> > + * correction (should be calculated based on shading tables)
> > + */
> > +struct ipu3_uapi_bnr_static_config_opt_center_config {
> > + __s32 x_reset:13;
> > + __u32 __reserved0:3;
> > + __s32 y_reset:13;
> > + __u32 __reserved2:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_lut_config - BNR square root lookup
> table
> > + *
> > + * @values: pre-calculated values of square root function.
> > + *
> > + * LUT implementation of square root operation.
> > + */
> > +struct ipu3_uapi_bnr_static_config_lut_config {
> > + __u8 values[IPU3_UAPI_BNR_LUT_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_bp_ctrl_config - Detect bad pixels
> (bp)
> > + *
> > + * @bp_thr_gain: Defines the threshold that specifies how different a
> > + * defect pixel can be from its neighbors. Threshold is
> > + * dependent on de-noise threshold calculated by
> algorithm.
> > + * Range [4, 31], default 4.
> > + * @__reserved0: reserved
> > + * @defect_mode: Mode of addressed defect pixels,
> > + * 0 - single defect pixel is expected,
> > + * 1 - 2 adjacent defect pixels are expected, default 1.
> > + * @bp_gain: Defines how 2nd derivation that passes through a
> defect pixel
> > + * is different from 2nd derivations that pass through
> > + * neighbor pixels. u4.2, range [0, 256], default 8.
> > + * @__reserved1: reserved
> > + * @w0_coeff: Blending coefficient of defect pixel correction.
> > + * Precision u4, range [0, 8], default 8.
> > + * @__reserved2: reserved
> > + * @w1_coeff: Enable influence of incorrect defect pixel correction
> to be
> > + * avoided. Precision u4, range [1, 8], default 8.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_bnr_static_config_bp_ctrl_config {
> > + __u32 bp_thr_gain:5;
> > + __u32 __reserved0:2;
> > + __u32 defect_mode:1;
> > + __u32 bp_gain:6;
> > + __u32 __reserved1:18;
> > + __u32 w0_coeff:4;
> > + __u32 __reserved2:4;
> > + __u32 w1_coeff:4;
> > + __u32 __reserved3:20;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config - Denoising
> config
> > + *
> > + * @alpha: Weight of central element of smoothing filter.
> > + * @beta: Weight of peripheral elements of smoothing filter, default 4.
> > + * @gamma: Weight of diagonal elements of smoothing filter, default 4.
> > + *
> > + * beta and gamma parameter define the strength of the noise removal
> filter.
> > + * All above has precision u0.4, range [0, 0xf]
> > + * format: u0.4 (no / zero bits represent whole number,
> > + * 4 bits represent the fractional part
> > + * with each count representing 0.0625)
> > + * e.g 0xf translates to 0.0625x15 = 0.9375
>
> e.g -> e.g.
>
> > + *
> > + * @__reserved0: reserved
> > + * @max_inf: Maximum increase of peripheral or diagonal element
> influence
> > + * relative to the pre-defined value range: [0x5, 0xa]
> > + * @__reserved1: reserved
> > + * @gd_enable: Green disparity enable control, 0 - disable, 1 - enable.
> > + * @bpc_enable: Bad pixel correction enable control, 0 - disable, 1 -
> enable.
> > + * @bnr_enable: Bayer noise removal enable control, 0 - disable, 1 -
> enable.
> > + * @ff_enable: Fixed function enable, 0 - disable, 1 - enable.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config {
> > + __u32 alpha:4;
> > + __u32 beta:4;
> > + __u32 gamma:4;
> > + __u32 __reserved0:4;
> > + __u32 max_inf:4;
> > + __u32 __reserved1:7;
> > + __u32 gd_enable:1;
> > + __u32 bpc_enable:1;
> > + __u32 bnr_enable:1;
> > + __u32 ff_enable:1;
> > + __u32 __reserved2:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_opt_center_sqr_config - BNR optical
> square
> > + *
> > + * @x_sqr_reset: Reset value of X^2.
> > + * @y_sqr_reset: Reset value of Y^2.
> > + *
> > + * Please note:
> > + *
> > + * #. X and Y ref to
> > + * &ipu3_uapi_bnr_static_config_opt_center_config
> > + * #. Both structs are used in threshold formula to calculate r^2, where r
> > + * is a radius of pixel [row, col] from centor of sensor.
> > + */
> > +struct ipu3_uapi_bnr_static_config_opt_center_sqr_config {
> > + __u32 x_sqr_reset;
> > + __u32 y_sqr_reset;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config - BNR static config
> > + *
> > + * @wb_gains: white balance gains
> &ipu3_uapi_bnr_static_config_wb_gains_config
> > + * @wb_gains_thr: white balance gains threshold as defined by
> > + * &ipu3_uapi_bnr_static_config_wb_gains_thr_config
> > + * @thr_coeffs: coefficients of threshold
> > + * &ipu3_uapi_bnr_static_config_thr_coeffs_config
> > + * @thr_ctrl_shd: control of shading threshold
> > + * &ipu3_uapi_bnr_static_config_thr_ctrl_shd_config
> > + * @opt_center: optical center
> &ipu3_uapi_bnr_static_config_opt_center_config
> > + *
> > + * Above parameters and opt_center_sqr are used for white balance and
> shading.
> > + *
> > + * @lut: lookup table &ipu3_uapi_bnr_static_config_lut_config
> > + * @bp_ctrl: detect and remove bad pixels as defined in struct
> > + * &ipu3_uapi_bnr_static_config_bp_ctrl_config
> > + * @dn_detect_ctrl: detect and remove noise.
> > + * &ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> > + * @column_size: The number of pixels in column.
> > + * @opt_center_sqr: Reset value of r^2 to optical center, see
> > + *
> &ipu3_uapi_bnr_static_config_opt_center_sqr_config.
> > + */
> > +struct ipu3_uapi_bnr_static_config {
> > + struct ipu3_uapi_bnr_static_config_wb_gains_config wb_gains;
> > + struct ipu3_uapi_bnr_static_config_wb_gains_thr_config
> wb_gains_thr;
> > + struct ipu3_uapi_bnr_static_config_thr_coeffs_config thr_coeffs;
> > + struct ipu3_uapi_bnr_static_config_thr_ctrl_shd_config thr_ctrl_shd;
> > + struct ipu3_uapi_bnr_static_config_opt_center_config opt_center;
> > + struct ipu3_uapi_bnr_static_config_lut_config lut;
> > + struct ipu3_uapi_bnr_static_config_bp_ctrl_config bp_ctrl;
> > + struct ipu3_uapi_bnr_static_config_dn_detect_ctrl_config
> dn_detect_ctrl;
> > + __u32 column_size;
> > + struct ipu3_uapi_bnr_static_config_opt_center_sqr_config
> opt_center_sqr;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_bnr_static_config_green_disparity - Correct green
> disparity
> > + *
> > + * @gd_red: Shading gain coeff for gr disparity level in bright red region.
> > + * Precision u0.6, default 4(0.0625).
> > + * @__reserved0: reserved
> > + * @gd_green: Shading gain coeff for gr disparity level in bright
> green
> > + * region. Precision u0.6, default 4(0.0625).
> > + * @__reserved1: reserved
> > + * @gd_blue: Shading gain coeff for gr disparity level in bright blue
> region.
> > + * Precision u0.6, default 4(0.0625).
> > + * @__reserved2: reserved
> > + * @gd_black: Maximal green disparity level in dark region (stronger
> disparity
> > + * assumed to be image detail). Precision u14, default 80.
> > + * @__reserved3: reserved
> > + * @gd_shading: Change maximal green disparity level according to
> square
> > + * distance from image center.
> > + * @__reserved4: reserved
> > + * @gd_support: Lower bound for the number of second green color
> pixels in
> > + * current pixel neighborhood with less than threshold
> difference
> > + * from it.
> > + *
> > + * The shading gain coeff of red, green, blue and black are used to
> calculate
> > + * threshold given a pixel's color value and its coordinates in the image.
> > + *
> > + * @__reserved5: reserved
> > + * @gd_clip: Turn green disparity clip on/off, [0, 1], default 1.
> > + * @gd_central_weight: Central pixel weight in 9 pixels weighted sum.
> > + */
> > +struct ipu3_uapi_bnr_static_config_green_disparity {
> > + __u32 gd_red:6;
> > + __u32 __reserved0:2;
> > + __u32 gd_green:6;
> > + __u32 __reserved1:2;
> > + __u32 gd_blue:6;
> > + __u32 __reserved2:10;
> > + __u32 gd_black:14;
> > + __u32 __reserved3:2;
> > + __u32 gd_shading:7;
> > + __u32 __reserved4:1;
> > + __u32 gd_support:2;
> > + __u32 __reserved5:1;
> > + __u32 gd_clip:1;
> > + __u32 gd_central_weight:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_dm_config - De-mosaic parameters
> > + *
> > + * @dm_en: de-mosaic enable.
> > + * @ch_ar_en: Checker artifacts removal enable flag. Default 0.
> > + * @fcc_en: False color correction (FCC) enable flag. Default 0.
> > + * @__reserved0: reserved
> > + * @frame_width: do not care
> > + * @gamma_sc: Sharpening coefficient (coefficient of 2-d derivation
> of
> > + * complementary color in Hamilton-Adams interpolation).
> > + * u5, range [0, 31], default 8.
> > + * @__reserved1: reserved
> > + * @lc_ctrl: Parameter that controls weights of Chroma Homogeneity
> metric
> > + * in calculation of final homogeneity metric.
> > + * u5, range [0, 31], default 7.
> > + * @__reserved2: reserved
> > + * @cr_param1: First parameter that defines Checker artifact removal
> > + * feature gain.Precision u5, range [0, 31], default 8.
>
> Add space before 'Precision'
Ack.
>
> > + * @__reserved3: reserved
> > + * @cr_param2: Second parameter that defines Checker artifact
> removal
> > + * feature gain. Precision u5, range [0, 31], default 8.
> > + * @__reserved4: reserved
> > + * @coring_param: Defines power of false color correction operation.
> > + * low for preserving edge colors, high for preserving
> gray
> > + * edge artifacts. u1.4, range [0, 1.9375], default 4(0.25).
>
> Add space after "default 4".
>
Ack.
> > + * @__reserved5: reserved
> > + *
> > + * The demosaic fixed function block is responsible to covert
> Bayer(mosaiced)
> > + * images into color images based on demosaicing algorithm.
> > + */
> > +struct ipu3_uapi_dm_config {
> > + __u32 dm_en:1;
> > + __u32 ch_ar_en:1;
> > + __u32 fcc_en:1;
> > + __u32 __reserved0:13;
> > + __u32 frame_width:16;
> > +
> > + __u32 gamma_sc:5;
> > + __u32 __reserved1:3;
> > + __u32 lc_ctrl:5;
> > + __u32 __reserved2:3;
> > + __u32 cr_param1:5;
> > + __u32 __reserved3:3;
> > + __u32 cr_param2:5;
> > + __u32 __reserved4:3;
> > +
> > + __u32 coring_param:5;
> > + __u32 __reserved5:27;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_ccm_mat_config - Color correction matrix
> > + *
> > + * @coeff_m11: CCM 3x3 coefficient, range [-65536, 65535]
> > + * @coeff_m12: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m13: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_r: Bias 3x1 coefficient, range [-8191, 8181]
> > + * @coeff_m21: CCM 3x3 coefficient, range [-32767, 32767]
> > + * @coeff_m22: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m23: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_g: Bias 3x1 coefficient, range [-8191, 8181]
> > + * @coeff_m31: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_m32: CCM 3x3 coefficient, range [-8192, 8191]
> > + * @coeff_m33: CCM 3x3 coefficient, range [-32768, 32767]
> > + * @coeff_o_b: Bias 3x1 coefficient, range [-8191, 8181]
> > + *
> > + * Transform sensor specific color space to standard sRGB by applying 3x3
> matrix
> > + * and adding a bias vector O. The transformation is basically a rotation
> and
> > + * translation in the 3-dimensional color spaces. Here are the defaults:
> > + *
> > + * 9775, -2671, 1087, 0
> > + * -1071, 8303, 815, 0
> > + * -23, -7887, 16103, 0
> > + */
> > +struct ipu3_uapi_ccm_mat_config {
> > + __s16 coeff_m11;
> > + __s16 coeff_m12;
> > + __s16 coeff_m13;
> > + __s16 coeff_o_r;
> > + __s16 coeff_m21;
> > + __s16 coeff_m22;
> > + __s16 coeff_m23;
> > + __s16 coeff_o_g;
> > + __s16 coeff_m31;
> > + __s16 coeff_m32;
> > + __s16 coeff_m33;
> > + __s16 coeff_o_b;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_corr_ctrl - Gamma correction
> > + *
> > + * @enable: gamma correction enable.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_gamma_corr_ctrl {
> > + __u32 enable:1;
> > + __u32 __reserved:31;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_corr_lut - Per-pixel tone mapping
> implemented as LUT.
> > + *
> > + * @lut: 256 tabulated values of the gamma function. LUT[1]..
> LUT[256]
> > + * format u13.0, range [0, 8191].
> > + *
> > + * The tone mapping operation is done by a Piece wise linear graph
> > + * that is implemented as a lookup table(LUT). The pixel component input
> > + * intensity is the X-axis of the graph which is the table entry.
> > + */
> > +struct ipu3_uapi_gamma_corr_lut {
> > + __u16 lut[IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_gamma_config - Gamma config
> > + *
> > + * @gc_ctrl: control of gamma correction &ipu3_uapi_gamma_corr_ctrl
> > + * @gc_lut: lookup table of gamma correction
> &ipu3_uapi_gamma_corr_lut
> > + */
> > +struct ipu3_uapi_gamma_config {
> > + struct ipu3_uapi_gamma_corr_ctrl gc_ctrl
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_gamma_corr_lut gc_lut __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_csc_mat_config - Color space conversion matrix config
> > + *
> > + * @coeff_c11: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_c12: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c13: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_b1: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + * @coeff_c21: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c22: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_c23: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_b2: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + * @coeff_c31: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c32: Conversion matrix value, format s0.14, range [-1, 1],
> default 0.
> > + * @coeff_c33: Conversion matrix value, format s0.14, range [-1, 1],
> default 1.
> > + * @coeff_b3: Bias 3x1 coefficient, s13,0 range [-8191, 8181],
> default 0.
> > + *
> > + * To transform each pixel from RGB to YUV (Y - brightness/luminance,
> > + * UV -chroma) by applying the pixel's values by a 3x3 matrix and adding
> an
> > + * optional bias 3x1 vector.
> > + */
> > +struct ipu3_uapi_csc_mat_config {
> > + __s16 coeff_c11;
> > + __s16 coeff_c12;
> > + __s16 coeff_c13;
> > + __s16 coeff_b1;
> > + __s16 coeff_c21;
> > + __s16 coeff_c22;
> > + __s16 coeff_c23;
> > + __s16 coeff_b2;
> > + __s16 coeff_c31;
> > + __s16 coeff_c32;
> > + __s16 coeff_c33;
> > + __s16 coeff_b3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cds_params - Chroma down-scaling
> > + *
> > + * @ds_c00: range [0, 3]
> > + * @ds_c01: range [0, 3]
> > + * @ds_c02: range [0, 3]
> > + * @ds_c03: range [0, 3]
> > + * @ds_c10: range [0, 3]
> > + * @ds_c11: range [0, 3]
> > + * @ds_c12: range [0, 3]
> > + * @ds_c13: range [0, 3]
> > + *
> > + * In case user does not provide, above 4x2 filter will use following
> defaults:
> > + * 1, 3, 3, 1,
> > + * 1, 3, 3, 1,
> > + *
> > + * @ds_nf: Normalization factor for Chroma output downscaling filter,
> > + * range 0,4, default 2.
> > + * @__reserved0: reserved
> > + * @csc_en: Color space conversion enable
> > + * @uv_bin_output: 0: output YUV 4.2.0, 1: output YUV 4.2.2(default).
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_cds_params {
> > + __u32 ds_c00:2;
> > + __u32 ds_c01:2;
> > + __u32 ds_c02:2;
> > + __u32 ds_c03:2;
> > + __u32 ds_c10:2;
> > + __u32 ds_c11:2;
> > + __u32 ds_c12:2;
> > + __u32 ds_c13:2;
> > + __u32 ds_nf:5;
> > + __u32 __reserved0:3;
> > + __u32 csc_en:1;
> > + __u32 uv_bin_output:1;
> > + __u32 __reserved1:6;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_grid_config - Bayer shading(darkening) correction
> > + *
> > + * @width: Grid horizontal dimensions, u8, [8, 128], default 73
> > + * @height: Grid vertical dimensions, u8, [8, 128], default 56
> > + * @block_width_log2: Log2 of the width of the grid cell in pixel
> count
> > + * u4, [0, 15], default value 5.
> > + * @__reserved0: reserved
> > + * @block_height_log2: Log2 of the height of the grid cell in pixel
> count
> > + * u4, [0, 15], default value 6.
> > + * @__reserved1: reserved
> > + * @grid_height_per_slice: SHD_MAX_CELLS_PER_SET/width.
> > + * (with SHD_MAX_CELLS_PER_SET = 146).
> > + * @x_start: X value of top left corner of sensor relative to ROI
> > + * u12, [-4096, 0]. default 0, only negative values.
>
> . default -> , default
>
> > + * @y_start: Y value of top left corner of sensor relative to ROI
> > + * u12, [-4096, 0]. default 0, only negative values.
>
> ditto.
>
Ack.
> > + */
> > +struct ipu3_uapi_shd_grid_config {
> > + /* reg 0 */
> > + __u8 width;
> > + __u8 height;
> > + __u8 block_width_log2:3;
> > + __u8 __reserved0:1;
> > + __u8 block_height_log2:3;
> > + __u8 __reserved1:1;
> > + __u8 grid_height_per_slice;
> > + /* reg 1 */
> > + __s16 x_start;
> > + __s16 y_start;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_general_config - Shading general config
> > + *
> > + * @init_set_vrt_offst_ul: set vertical offset,
> > + * y_start >> block_height_log2 % grid_height_per_slice.
> > + * @shd_enable: shading enable.
> > + * @gain_factor: Gain factor. Shift calculated anti shading value. Precision
> u2.
> > + * 0x0 - gain factor [1, 5], means no shift interpolated value.
> > + * 0x1 - gain factor [1, 9], means shift interpolated by 1.
> > + * 0x2 - gain factor [1, 17], means shift interpolated by 2.
> > + * @__reserved: reserved
> > + *
> > + * Correction is performed by multiplying a gain factor for each of the 4
> Bayer
> > + * channels as a function of the pixel location in the sensor.
> > + */
> > +struct ipu3_uapi_shd_general_config {
> > + __u32 init_set_vrt_offst_ul:8;
> > + __u32 shd_enable:1;
> > + __u32 gain_factor:2;
> > + __u32 __reserved:21;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_black_level_config - Black level correction
> > + *
> > + * @bl_r: Bios values for green red. s11 range [-2048, 2047].
> > + * @bl_gr: Bios values for green blue. s11 range [-2048, 2047].
> > + * @bl_gb: Bios values for red. s11 range [-2048, 2047].
> > + * @bl_b: Bios values for blue. s11 range [-2048, 2047].
> > + */
> > +struct ipu3_uapi_shd_black_level_config {
> > + __s16 bl_r;
> > + __s16 bl_gr;
> > + __s16 bl_gb;
> > + __s16 bl_b;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_config_static - Shading config static
> > + *
> > + * @grid: shading grid config &ipu3_uapi_shd_grid_config
> > + * @general: shading general config &ipu3_uapi_shd_general_config
> > + * @black_level: black level config for shading correction as defined by
> > + * &ipu3_uapi_shd_black_level_config
> > + */
> > +struct ipu3_uapi_shd_config_static {
> > + struct ipu3_uapi_shd_grid_config grid;
> > + struct ipu3_uapi_shd_general_config general;
> > + struct ipu3_uapi_shd_black_level_config black_level;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_lut - Shading gain factor lookup table.
> > + *
> > + * @sets: array
> > + * @sets.r_and_gr: Red and GreenR Lookup table.
> > + * @sets.r_and_gr.r: Red shading factor.
> > + * @sets.r_and_gr.gr: GreenR shading factor.
> > + * @sets.__reserved1: reserved
> > + * @sets.gb_and_b: GreenB and Blue Lookup table.
> > + * @sets.gb_and_b.gb: GreenB shading factor.
> > + * @sets.gb_and_b.b: Blue shading factor.
> > + * @sets.__reserved2: reserved
> > + *
> > + * Map to shading correction LUT register set.
> > + */
> > +struct ipu3_uapi_shd_lut {
> > + struct {
> > + struct {
> > + __u16 r;
> > + __u16 gr;
> > + } r_and_gr[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> > + __u8 __reserved1[24];
> > + struct {
> > + __u16 gb;
> > + __u16 b;
> > + } gb_and_b[IPU3_UAPI_SHD_MAX_CELLS_PER_SET];
> > + __u8 __reserved2[24];
> > + } sets[IPU3_UAPI_SHD_MAX_CFG_SETS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_shd_config - Shading config
> > + *
> > + * @shd: shading static config, see &ipu3_uapi_shd_config_static
> > + * @shd_lut: shading lookup table &ipu3_uapi_shd_lut
> > + */
> > +struct ipu3_uapi_shd_config {
> > + struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
> > + struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/* Image Enhancement Filter directed */
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux2 - IEFd Config Unit 2 parameters
> > + *
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A of Config Unit, s4.4, default 0.
> > + * @b01: Always 0.
> > + *
> > + * Calculate weight for blending directed and non-directed denoise
> elements
> > + *
> > + * Note:
> > + * Each instance of Config Unit needs X coordinate of n points and
> > + * slope A factor between points calculated by driver based on calibration
> > + * parameters.
> > + */
> > +struct ipu3_uapi_iefd_cux2 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 a01:9;
> > + __u32 b01:5; /* NOTE: hardcoded to zero */
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux6_ed - Calculate power of non-directed
> sharpening
> > + * element, Config Unit 6 for edge detail (ED).
> > + *
> > + * @x0: X coordinate of point 0, u9.0, default 0.
> > + * @x1: X coordinate of point 1, u9.0, default 0.
> > + * @x2: X coordinate of point 2, u9.0, default 0.
> > + * @__reserved0: reserved
> > + * @x3: X coordinate of point 3, u9.0, default 0.
> > + * @x4: X coordinate of point 4, u9.0, default 0.
> > + * @x5: X coordinate of point 5, u9.0, default 0.
> > + * @__reserved1: reserved
> > + * @a01: slope A points 01, s4.4, default 0.
> > + * @a12: slope A points 12, s4.4, default 0.
> > + * @a23: slope A points 23, s4.4, default 0.
> > + * @__reserved2: reserved
> > + * @a34: slope A points 34, s4.4, default 0.
> > + * @a45: slope A points 45, s4.4, default 0.
> > + * @__reserved3: reserved
> > + * @b01: slope B points 01, s4.4, default 0.
> > + * @b12: slope B points 12, s4.4, default 0.
> > + * @b23: slope B points 23, s4.4, default 0.
> > + * @__reserved4: reserved
> > + * @b34: slope B points 34, s4.4, default 0.
> > + * @b45: slope B points 45, s4.4, default 0.
> > + * @__reserved5: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux6_ed {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 x2:9;
> > + __u32 __reserved0:5;
> > +
> > + __u32 x3:9;
> > + __u32 x4:9;
> > + __u32 x5:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 a01:9;
> > + __u32 a12:9;
> > + __u32 a23:9;
> > + __u32 __reserved2:5;
> > +
> > + __u32 a34:9;
> > + __u32 a45:9;
> > + __u32 __reserved3:14;
> > +
> > + __u32 b01:9;
> > + __u32 b12:9;
> > + __u32 b23:9;
> > + __u32 __reserved4:5;
> > +
> > + __u32 b34:9;
> > + __u32 b45:9;
> > + __u32 __reserved5:14;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux2_1 - Calculate power of non-directed denoise
> > + * element apply.
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A of Config Unit, s4.4, default 0.
> > + * @__reserved1: reserved
> > + * @b01: offset B0 of Config Unit, u7.0, default 0.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux2_1 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 a01:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 b01:8;
> > + __u32 __reserved2:24;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux4 - Calculate power of non-directed
> sharpening
> > + * element.
> > + *
> > + * @x0: X0 point of Config Unit, u9.0, default 0.
> > + * @x1: X1 point of Config Unit, u9.0, default 0.
> > + * @x2: X2 point of Config Unit, u9.0, default 0.
> > + * @__reserved0: reserved
> > + * @x3: X3 point of Config Unit, u9.0, default 0.
> > + * @a01: Slope A0 of Config Unit, s4.4, default 0.
> > + * @a12: Slope A1 of Config Unit, s4.4, default 0.
> > + * @__reserved1: reserved
> > + * @a23: Slope A2 of Config Unit, s4.4, default 0.
> > + * @b01: Offset B0 of Config Unit, s7.0, default 0.
> > + * @b12: Offset B1 of Config Unit, s7.0, default 0.
> > + * @__reserved2: reserved
> > + * @b23: Offset B2 of Config Unit, s7.0, default 0.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux4 {
> > + __u32 x0:9;
> > + __u32 x1:9;
> > + __u32 x2:9;
> > + __u32 __reserved0:5;
> > +
> > + __u32 x3:9;
> > + __u32 a01:9;
> > + __u32 a12:9;
> > + __u32 __reserved1:5;
> > +
> > + __u32 a23:9;
> > + __u32 b01:8;
> > + __u32 b12:8;
> > + __u32 __reserved2:7;
> > +
> > + __u32 b23:8;
> > + __u32 __reserved3:24;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_iefd_cux6_rad - Radial Config Unit (CU)
> > + *
> > + * @x0: x0 points of Config Unit radial, u8.0
> > + * @x1: x1 points of Config Unit radial, u8.0
> > + * @x2: x2 points of Config Unit radial, u8.0
> > + * @x3: x3 points of Config Unit radial, u8.0
> > + * @x4: x4 points of Config Unit radial, u8.0
> > + * @x5: x5 points of Config Unit radial, u8.0
> > + * @__reserved1: reserved
> > + * @a01: Slope A of Config Unit radial, s7.8
> > + * @a12: Slope A of Config Unit radial, s7.8
> > + * @a23: Slope A of Config Unit radial, s7.8
> > + * @a34: Slope A of Config Unit radial, s7.8
> > + * @a45: Slope A of Config Unit radial, s7.8
> > + * @__reserved2: reserved
> > + * @b01: Slope B of Config Unit radial, s9.0
> > + * @b12: Slope B of Config Unit radial, s9.0
> > + * @b23: Slope B of Config Unit radial, s9.0
> > + * @__reserved4: reserved
> > + * @b34: Slope B of Config Unit radial, s9.0
> > + * @b45: Slope B of Config Unit radial, s9.0
> > + * @__reserved5: reserved
> > + */
> > +struct ipu3_uapi_iefd_cux6_rad {
> > + __u32 x0:8;
> > + __u32 x1:8;
> > + __u32 x2:8;
> > + __u32 x3:8;
> > +
> > + __u32 x4:8;
> > + __u32 x5:8;
> > + __u32 __reserved1:16;
> > +
> > + __u32 a01:16;
> > + __u32 a12:16;
> > +
> > + __u32 a23:16;
> > + __u32 a34:16;
> > +
> > + __u32 a45:16;
> > + __u32 __reserved2:16;
> > +
> > + __u32 b01:10;
> > + __u32 b12:10;
> > + __u32 b23:10;
> > + __u32 __reserved4:2;
> > +
> > + __u32 b34:10;
> > + __u32 b45:10;
> > + __u32 __reserved5:12;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_cfg_units - IEFd Config Units parameters
> > + *
> > + * @cu_1: calculate weight for blending directed and
> > + * non-directed denoise elements. See &ipu3_uapi_iefd_cux2
> > + * @cu_ed: calculate power of non-directed sharpening element, see
> > + * &ipu3_uapi_iefd_cux6_ed
> > + * @cu_3: calculate weight for blending directed and
> > + * non-directed denoise elements. A &ipu3_uapi_iefd_cux2
> > + * @cu_5: calculate power of non-directed denoise element apply, use
> > + * &ipu3_uapi_iefd_cux2_1
> > + * @cu_6: calculate power of non-directed sharpening element. See
> > + * &ipu3_uapi_iefd_cux4
> > + * @cu_7: calculate weight for blending directed and
> > + * non-directed denoise elements. Use &ipu3_uapi_iefd_cux2
> > + * @cu_unsharp: Config Unit of unsharp &ipu3_uapi_iefd_cux4
> > + * @cu_radial: Config Unit of radial &ipu3_uapi_iefd_cux6_rad
> > + * @cu_vssnlm: Config Unit of vssnlm &ipu3_uapi_iefd_cux2
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_cfg_units {
> > + struct ipu3_uapi_iefd_cux2 cu_1;
> > + struct ipu3_uapi_iefd_cux6_ed cu_ed;
> > + struct ipu3_uapi_iefd_cux2 cu_3;
> > + struct ipu3_uapi_iefd_cux2_1 cu_5;
> > + struct ipu3_uapi_iefd_cux4 cu_6;
> > + struct ipu3_uapi_iefd_cux2 cu_7;
> > + struct ipu3_uapi_iefd_cux4 cu_unsharp;
> > + struct ipu3_uapi_iefd_cux6_rad cu_radial;
> > + struct ipu3_uapi_iefd_cux2 cu_vssnlm;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_config_s - IEFd config
> > + *
> > + * @horver_diag_coeff: Gradiant compensation, coefficient that
> compensates for
> > + * different distance for vertical / horizontal and diagonal
> > + * * gradient calculation (~1/sqrt(2)).
>
> Uh, is this "diagonal * gradient" or "diagonal gradient"? It's probably a
> spurious '*'.
>
Yes, it's a spurious typing.
> And what does ~1/sqrt(2) refer to? The default value of the coefficient?
> And does '~' stand for 'approximate'? Slightly confusing given the
> C ~ operator. It's probably better to just write 'approx. 1/sqrt(2)'.
>
Ack, for 1/sqrt(2), it means comparing to horizontal(0 degree) or vertical(90 degree) direction, coefficient of diagonal(45 degree or 135 degree) direction should be corrected with 1/sqrt(2), to make all direction align with same gradient calculation/correction.
> > + * @__reserved0: reserved
> > + * @clamp_stitch: Slope to stitch between clamped and unclamped edge
> values
> > + * @__reserved1: reserved
> > + * @direct_metric_update: Update coeff for direction metric
> > + * @__reserved2: reserved
> > + * @ed_horver_diag_coeff: Radial Coefficient that compensates for
> > + * different distance for vertical/horizontal and
> > + * diagonal gradient calculation (~1/sqrt(2))
>
> ditto
>
Ack, will improve the clarity in next update.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_config_s {
> > + __u32 horver_diag_coeff:7;
> > + __u32 __reserved0:1;
> > + __u32 clamp_stitch:6;
> > + __u32 __reserved1:2;
> > + __u32 direct_metric_update:5;
> > + __u32 __reserved2:3;
> > + __u32 ed_horver_diag_coeff:7;
> > + __u32 __reserved3:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_control - IEFd control
> > + *
> > + * @iefd_en: Enable IEFd
> > + * @denoise_en: Enable denoise
> > + * @direct_smooth_en: Enable directional smooth
> > + * @rad_en: Enable radial update
> > + * @vssnlm_en: Enable VSSNLM output filter
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_control {
> > + __u32 iefd_en:1;
> > + __u32 denoise_en:1;
> > + __u32 direct_smooth_en:1;
> > + __u32 rad_en:1;
> > + __u32 vssnlm_en:1;
> > + __u32 __reserved:27;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_sharp_cfg - Sharpening config
> > + *
> > + * @nega_lmt_txt: Sharpening limit for negative overshoots for texture.
> > + * @__reserved0: reserved
> > + * @posi_lmt_txt: Sharpening limit for positive overshoots for texture.
> > + * @__reserved1: reserved
> > + * @nega_lmt_dir: Sharpening limit for negative overshoots for direction
> (edge).
> > + * @__reserved2: reserved
> > + * @posi_lmt_dir: Sharpening limit for positive overshoots for direction
> (edge).
> > + * @__reserved3: reserved
> > + *
> > + * Fixed point type u13.0, range [0, 8191].
> > + */
> > +struct ipu3_uapi_sharp_cfg {
> > + __u32 nega_lmt_txt:13;
> > + __u32 __reserved0:19;
> > + __u32 posi_lmt_txt:13;
> > + __u32 __reserved1:19;
> > + __u32 nega_lmt_dir:13;
> > + __u32 __reserved2:19;
> > + __u32 posi_lmt_dir:13;
> > + __u32 __reserved3:19;
> > +} __packed;
> > +
> > +/**
> > + * struct struct ipu3_uapi_far_w - Sharpening config for far sub-group
> > + *
> > + * @dir_shrp: Weight of wide direct sharpening, u1.6, range [0, 64],
> default 64.
> > + * @__reserved0: reserved
> > + * @dir_dns: Weight of wide direct denoising, u1.6, range [0, 64],
> default 0.
> > + * @__reserved1: reserved
> > + * @ndir_dns_powr: Power of non-direct denoising,
> > + * Precision u1.6, range [0, 64], default 64.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_far_w {
> > + __u32 dir_shrp:7;
> > + __u32 __reserved0:1;
> > + __u32 dir_dns:7;
> > + __u32 __reserved1:1;
> > + __u32 ndir_dns_powr:7;
> > + __u32 __reserved2:9;
> > +} __packed;
> > +
> > +/**
> > + * struct struct ipu3_uapi_unsharp_cfg - Unsharp config
> > + *
> > + * @unsharp_weight: Unsharp mask blending weight.
> > + * u1.6, range [0, 64], default 16.
> > + * 0 - disabled, 64 - use only unsharp.
> > + * @__reserved0: reserved
> > + * @unsharp_amount: Unsharp mask amount, u4.5, range [0, 511],
> default 0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_unsharp_cfg {
> > + __u32 unsharp_weight:7;
> > + __u32 __reserved0:1;
> > + __u32 unsharp_amount:9;
> > + __u32 __reserved1:15;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_shrp_cfg - IEFd sharpness config
> > + *
> > + * @cfg: sharpness config &ipu3_uapi_sharp_cfg
> > + * @far_w: wide range config, value as specified by &ipu3_uapi_far_w:
> > + * The 5x5 environment is separated into 2 sub-groups, the 3x3 nearest
> > + * neighbors (8 pixels called Near), and the second order neighborhood
> > + * around them (16 pixels called Far).
> > + * @unshrp_cfg: unsharpness config. &ipu3_uapi_unsharp_cfg
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_shrp_cfg {
> > + struct ipu3_uapi_sharp_cfg cfg;
> > + struct ipu3_uapi_far_w far_w;
> > + struct ipu3_uapi_unsharp_cfg unshrp_cfg;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_unsharp_coef0 - Unsharp mask coefficients
> > + *
> > + * @c00: Coeff11, s0.8, range [-255, 255], default 1.
> > + * @c01: Coeff12, s0.8, range [-255, 255], default 5.
> > + * @c02: Coeff13, s0.8, range [-255, 255], default 9.
> > + * @__reserved: reserved
> > + *
> > + * Configurable registers for common sharpening support.
> > + */
> > +struct ipu3_uapi_unsharp_coef0 {
> > + __u32 c00:9;
> > + __u32 c01:9;
> > + __u32 c02:9;
> > + __u32 __reserved:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_unsharp_coef1 - Unsharp mask coefficients
> > + *
> > + * @c11: Coeff22, s0.8, range [-255, 255], default 29.
> > + * @c12: Coeff23, s0.8, range [-255, 255], default 55.
> > + * @c22: Coeff33, s0.8, range [-255, 255], default 96.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_unsharp_coef1 {
> > + __u32 c11:9;
> > + __u32 c12:9;
> > + __u32 c22:9;
> > + __u32 __reserved:5;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_unshrp_cfg - Unsharp mask config
> > + *
> > + * @unsharp_coef0: unsharp coefficient 0 config. See
> &ipu3_uapi_unsharp_coef0
> > + * @unsharp_coef1: unsharp coefficient 1 config. See
> &ipu3_uapi_unsharp_coef1
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_unshrp_cfg {
> > + struct ipu3_uapi_unsharp_coef0 unsharp_coef0;
> > + struct ipu3_uapi_unsharp_coef1 unsharp_coef1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_reset_xy - Radial coordinate reset
> > + *
> > + * @x: Radial reset of x coordinate. Precision s12, [-4095, 4095],
> default 0.
> > + * @__reserved0: reserved
> > + * @y: Radial center y coordinate. Precision s12, [-4095, 4095],
> default 0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_radial_reset_xy {
> > + __s32 x:13;
> > + __u32 __reserved0:3;
> > + __s32 y:13;
> > + __u32 __reserved1:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_reset_x2 - Radial X^2 reset
> > + *
> > + * @x2: Radial reset of x^2 coordinate. Precision u24, default 0.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_radial_reset_x2 {
> > + __u32 x2:24;
> > + __u32 __reserved:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_reset_y2 - Radial Y^2 reset
> > + *
> > + * @y2: Radial reset of y^2 coordinate. Precision u24, default 0.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_radial_reset_y2 {
> > + __u32 y2:24;
> > + __u32 __reserved:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_radial_cfg - Radial config
> > + *
> > + * @rad_nf: Radial. R^2 normalization factor is scale down by 2^ - (15 +
> scale)
> > + * @__reserved0: reserved
> > + * @rad_inv_r2: Radial R^-2 normelized to (0.5..1), Prec' u7, range [0, 127].
>
> Prec' -> Precision
>
Ack.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_radial_cfg {
> > + __u32 rad_nf:4;
> > + __u32 __reserved0:4;
> > + __u32 rad_inv_r2:7;
> > + __u32 __reserved1:17;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_rad_far_w - Radial FAR sub-group
> > + *
> > + * @rad_dir_far_sharp_w: Weight of wide direct sharpening, u1.6, range
> [0, 64],
> > + * default 64.
> > + * @rad_dir_far_dns_w: Weight of wide direct denoising, u1.6, range [0,
> 64],
> > + * default 0.
> > + * @rad_ndir_far_dns_power: power of non-direct sharpening, u1.6,
> range [0, 64],
> > + * default 0.
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_rad_far_w {
> > + __u32 rad_dir_far_sharp_w:8;
> > + __u32 rad_dir_far_dns_w:8;
> > + __u32 rad_ndir_far_dns_power:8;
> > + __u32 __reserved:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cu_cfg0 - Radius Config Unit cfg0 register
> > + *
> > + * @cu6_pow: Power of CU6. Power of non-direct sharpening, u3.4.
> > + * @__reserved0: reserved
> > + * @cu_unsharp_pow: Power of unsharp mask, u2.4.
> > + * @__reserved1: reserved
> > + * @rad_cu6_pow: Radial/corner CU6. Directed sharpening power, u3.4.
> > + * @__reserved2: reserved
> > + * @rad_cu_unsharp_pow: Radial power of unsharp mask, u2.4.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_cu_cfg0 {
> > + __u32 cu6_pow:7;
> > + __u32 __reserved0:1;
> > + __u32 cu_unsharp_pow:7;
> > + __u32 __reserved1:1;
> > + __u32 rad_cu6_pow:7;
> > + __u32 __reserved2:1;
> > + __u32 rad_cu_unsharp_pow:6;
> > + __u32 __reserved3:2;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_cu_cfg1 - Radius Config Unit cfg1 register
> > + *
> > + * @rad_cu6_x1: X1 point of Config Unit 6, precision u9.0.
> > + * @__reserved0: reserved
> > + * @rad_cu_unsharp_x1: X1 point for Config Unit unsharp for
> radial/corner point
> > + * precision u9.0.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_cu_cfg1 {
> > + __u32 rad_cu6_x1:9;
> > + __u32 __reserved0:1;
> > + __u32 rad_cu_unsharp_x1:9;
> > + __u32 __reserved1:13;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_rad_cfg - IEFd parameters changed
> radially over
> > + * the picture plain.
>
> I guess you mean 'plane' instead of 'plain'?
>
It's 'plain' according to the specs, we are consulting internally for a more generic/descriptive term, same for struct ipu3_uapi_anr_plain_color.
> > + *
> > + * @reset_xy: reset xy value in radial calculation.
> &ipu3_uapi_radial_reset_xy
> > + * @reset_x2: reset x square value in radial calculation. See struct
> > + * &ipu3_uapi_radial_reset_x2
> > + * @reset_y2: reset y square value in radial calculation. See struct
> > + * &ipu3_uapi_radial_reset_y2
> > + * @cfg: radial config defined in &ipu3_uapi_radial_cfg
> > + * @rad_far_w: weight for wide range radial. &ipu3_uapi_rad_far_w
> > + * @cu_cfg0: configuration unit 0. See &ipu3_uapi_cu_cfg0
> > + * @cu_cfg1: configuration unit 1. See &ipu3_uapi_cu_cfg1
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_rad_cfg {
> > + struct ipu3_uapi_radial_reset_xy reset_xy;
> > + struct ipu3_uapi_radial_reset_x2 reset_x2;
> > + struct ipu3_uapi_radial_reset_y2 reset_y2;
> > + struct ipu3_uapi_radial_cfg cfg;
> > + struct ipu3_uapi_rad_far_w rad_far_w;
> > + struct ipu3_uapi_cu_cfg0 cu_cfg0;
> > + struct ipu3_uapi_cu_cfg1 cu_cfg1;
> > +} __packed;
> > +
> > +/* Vssnlm - Very small scale non-local mean algorithm */
> > +
> > +/**
> > + * struct ipu3_uapi_vss_lut_x - Vssnlm LUT x0/x1/x2
> > + *
> > + * @vs_x0: Vssnlm LUT x0, precision u8, range [0, 255], default 16.
> > + * @vs_x1: Vssnlm LUT x1, precision u8, range [0, 255], default 32.
> > + * @vs_x2: Vssnlm LUT x2, precision u8, range [0, 255], default 64.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_vss_lut_x {
> > + __u32 vs_x0:8;
> > + __u32 vs_x1:8;
> > + __u32 vs_x2:8;
> > + __u32 __reserved2:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_vss_lut_y - Vssnlm LUT y0/y1/y2
> > + *
> > + * @vs_y1: Vssnlm LUT y1, precision u4, range [0, 8], default 1.
> > + * @__reserved0: reserved
> > + * @vs_y2: Vssnlm LUT y2, precision u4, range [0, 8], default 3.
> > + * @__reserved1: reserved
> > + * @vs_y3: Vssnlm LUT y3, precision u4, range [0, 8], default 8.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_vss_lut_y {
> > + __u32 vs_y1:4;
> > + __u32 __reserved0:4;
> > + __u32 vs_y2:4;
> > + __u32 __reserved1:4;
> > + __u32 vs_y3:4;
> > + __u32 __reserved2:12;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_vssnlm_cf - IEFd Vssnlm Lookup table
> > + *
> > + * @vss_lut_x: vss lookup table. See &ipu3_uapi_vss_lut_x description
> > + * @vss_lut_y: vss lookup table. See &ipu3_uapi_vss_lut_y description
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg {
> > + struct ipu3_uapi_vss_lut_x vss_lut_x;
> > + struct ipu3_uapi_vss_lut_y vss_lut_y;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_iefd_config - IEFd config
> > + *
> > + * @units: configuration unit setting, &ipu3_uapi_yuvp1_iefd_cfg_units
> > + * @config: configuration, as defined by &ipu3_uapi_yuvp1_iefd_config_s
> > + * @control: control setting, as defined by
> &ipu3_uapi_yuvp1_iefd_control
> > + * @sharp: sharpness setting, as defined by
> &ipu3_uapi_yuvp1_iefd_shrp_cfg
> > + * @unsharp: unsharpness setting, as defined by
> &ipu3_uapi_yuvp1_iefd_unshrp_cfg
> > + * @rad: radial setting, as defined by &ipu3_uapi_yuvp1_iefd_rad_cfg
> > + * @vsslnm: vsslnm setting, as defined by
> &ipu3_uapi_yuvp1_iefd_vssnlm_cfg
> > + */
> > +struct ipu3_uapi_yuvp1_iefd_config {
> > + struct ipu3_uapi_yuvp1_iefd_cfg_units units;
> > + struct ipu3_uapi_yuvp1_iefd_config_s config;
> > + struct ipu3_uapi_yuvp1_iefd_control control;
> > + struct ipu3_uapi_yuvp1_iefd_shrp_cfg sharp;
> > + struct ipu3_uapi_yuvp1_iefd_unshrp_cfg unsharp;
> > + struct ipu3_uapi_yuvp1_iefd_rad_cfg rad;
> > + struct ipu3_uapi_yuvp1_iefd_vssnlm_cfg vsslnm;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_yds_config - Y Down-Sampling config
> > + *
> > + * @c00: range [0, 3], default 0x0
> > + * @c01: range [0, 3], default 0x1
> > + * @c02: range [0, 3], default 0x1
> > + * @c03: range [0, 3], default 0x0
> > + * @c10: range [0, 3], default 0x0
> > + * @c11: range [0, 3], default 0x1
> > + * @c12: range [0, 3], default 0x1
> > + * @c13: range [0, 3], default 0x0
> > + *
> > + * Above are 4x2 filter coefficients for chroma output downscaling.
> > + *
> > + * @norm_factor: Normalization factor, range [0, 4], default 2
> > + * 0 - divide by 1
> > + * 1 - divide by 2
> > + * 2 - divide by 4
> > + * 3 - divide by 8
> > + * 4 - divide by 16
> > + * @__reserved0: reserved
> > + * @bin_output: Down sampling on Luma channel in two optional modes
> > + * 0 - Bin output 4.2.0 (default), 1 output 4.2.2.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_yds_config {
> > + __u32 c00:2;
> > + __u32 c01:2;
> > + __u32 c02:2;
> > + __u32 c03:2;
> > + __u32 c10:2;
> > + __u32 c11:2;
> > + __u32 c12:2;
> > + __u32 c13:2;
> > + __u32 norm_factor:5;
> > + __u32 __reserved0:4;
> > + __u32 bin_output:1;
> > + __u32 __reserved1:6;
> > +} __packed;
> > +
> > +/* Chroma Noise Reduction */
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_enable_config - Chroma noise reduction
> enable
> > + *
> > + * @enable: enable/disable chroma noise reduction
> > + * @yuv_mode: 0 - YUV420, 1 - YUV422
> > + * @__reserved0: reserved
> > + * @col_size: number of columns in the frame, max width is 2560
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_enable_config {
> > + __u32 enable:1;
> > + __u32 yuv_mode:1;
> > + __u32 __reserved0:14;
> > + __u32 col_size:12;
> > + __u32 __reserved1:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_coring_config - Coring thresholds for UV
> > + *
> > + * @u: U coring level, u0.13, range [0.0, 1.0], default 0.0
> > + * @__reserved0: reserved
> > + * @v: V coring level, u0.13, range [0.0, 1.0], default 0.0
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_coring_config {
> > + __u32 u:13;
> > + __u32 __reserved0:3;
> > + __u32 v:13;
> > + __u32 __reserved1:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_sense_gain_config - Chroma noise
> reduction gains
> > + *
> > + * All sensitivity gain parameters have precision u13.0, range [0, 8191].
> > + *
> > + * @vy: Sensitivity of horizontal edge of Y, default 100
> > + * @vu: Sensitivity of horizontal edge of U, default 100
> > + * @vv: Sensitivity of horizontal edge of V, default 100
> > + * @__reserved0: reserved
> > + * @hy: Sensitivity of vertical edge of Y, default 50
> > + * @hu: Sensitivity of vertical edge of U, default 50
> > + * @hv: Sensitivity of vertical edge of V, default 50
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_sense_gain_config {
> > + __u32 vy:8;
> > + __u32 vu:8;
> > + __u32 vv:8;
> > + __u32 __reserved0:8;
> > +
> > + __u32 hy:8;
> > + __u32 hu:8;
> > + __u32 hv:8;
> > + __u32 __reserved1:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_iir_fir_config - Chroma IIR/FIR filter config
> > + *
> > + * @fir_0h: Value of center tap in horizontal FIR, range [0, 32], default 8.
> > + * @__reserved0: reserved
> > + * @fir_1h: Value of distance 1 in horizontal FIR, range [0, 32], default 12.
> > + * @__reserved1: reserved
> > + * @fir_2h: Value of distance 2 tap in horizontal FIR, range [0, 32], default
> 0.
> > + * @dalpha_clip_val: weight for previous row in IIR, range [1, 256], default
> 0.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_iir_fir_config {
> > + __u32 fir_0h:6;
> > + __u32 __reserved0:2;
> > + __u32 fir_1h:6;
> > + __u32 __reserved1:2;
> > + __u32 fir_2h:6;
> > + __u32 dalpha_clip_val:9;
> > + __u32 __reserved2:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_chnr_config - Chroma noise reduction config
> > + *
> > + * @enable: chroma noise reduction enable, see
> > + * &ipu3_uapi_yuvp1_chnr_enable_config
> > + * @coring: coring config for chroma noise reduction, see
> > + * &ipu3_uapi_yuvp1_chnr_coring_config
> > + * @sense_gain: sensitivity config for chroma noise reduction, see
> > + * ipu3_uapi_yuvp1_chnr_sense_gain_config
> > + * @iir_fir: iir and fir config for chroma noise reduction, see
> > + * ipu3_uapi_yuvp1_chnr_iir_fir_config
> > + */
> > +struct ipu3_uapi_yuvp1_chnr_config {
> > + struct ipu3_uapi_yuvp1_chnr_enable_config enable;
> > + struct ipu3_uapi_yuvp1_chnr_coring_config coring;
> > + struct ipu3_uapi_yuvp1_chnr_sense_gain_config sense_gain;
> > + struct ipu3_uapi_yuvp1_chnr_iir_fir_config iir_fir;
> > +} __packed;
> > +
> > +/* Edge Enhancement and Noise Reduction */
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config - Luma(Y) edge
> enhancement low-pass
> > + * filter coefficients
> > + *
> > + * @a_diag: Smoothing diagonal coefficient, u5.0.
> > + * @__reserved0: reserved
> > + * @a_periph: Image smoothing perpherial, u5.0.
> > + * @__reserved1: reserved
> > + * @a_cent: Image Smoothing center coefficient, u5.0.
> > + * @__reserved2: reserved
> > + * @enable: 0: Y_EE_NR disabled, output = input; 1: Y_EE_NR enabled.
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config {
> > + __u32 a_diag:5;
> > + __u32 __reserved0:3;
> > + __u32 a_periph:5;
> > + __u32 __reserved1:3;
> > + __u32 a_cent:5;
> > + __u32 __reserved2:9;
> > + __u32 enable:1;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_sense_config - Luma(Y) edge
> enhancement
> > + * noise reduction sensitivity gains
> > + *
> > + * @edge_sense_0: Sensitivity of edge in dark area. u13.0, default 8191.
> > + * @__reserved0: reserved
> > + * @delta_edge_sense: Difference in the sensitivity of edges between
> > + * the bright and dark areas. u13.0, default 0.
> > + * @__reserved1: reserved
> > + * @corner_sense_0: Sensitivity of corner in dark area. u13.0, default 0.
> > + * @__reserved2: reserved
> > + * @delta_corner_sense: Difference in the sensitivity of corners between
> > + * the bright and dark areas. u13.0, default 8191.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_sense_config {
> > + __u32 edge_sense_0:13;
> > + __u32 __reserved0:3;
> > + __u32 delta_edge_sense:13;
> > + __u32 __reserved1:3;
> > + __u32 corner_sense_0:13;
> > + __u32 __reserved2:3;
> > + __u32 delta_corner_sense:13;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_gain_config - Luma(Y) edge
> enhancement
> > + * noise reduction gain config
> > + *
> > + * @gain_pos_0: Gain for positive edge in dark area. u5.0, [0, 16], default
> 2.
> > + * @__reserved0: reserved
> > + * @delta_gain_posi: Difference in the gain of edges between the bright
> and
> > + * dark areas for positive edges. u5.0, [0, 16], default 0.
> > + * @__reserved1: reserved
> > + * @gain_neg_0: Gain for negative edge in dark area. u5.0, [0, 16], default
> 8.
> > + * @__reserved2: reserved
> > + * @delta_gain_neg: Difference in the gain of edges between the bright
> and
> > + * dark areas for negative edges. u5.0, [0, 16], default 0.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_gain_config {
> > + __u32 gain_pos_0:5;
> > + __u32 __reserved0:3;
> > + __u32 delta_gain_posi:5;
> > + __u32 __reserved1:3;
> > + __u32 gain_neg_0:5;
> > + __u32 __reserved2:3;
> > + __u32 delta_gain_neg:5;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_clip_config - Luma(Y) edge
> enhancement
> > + * noise reduction clipping config
> > + *
> > + * @clip_pos_0: Limit of positive edge in dark area
> > + * u5, value [0, 16], default 8.
> > + * @__reserved0: reserved
> > + * @delta_clip_posi: Difference in the limit of edges between the bright
> > + * and dark areas for positive edges.
> > + * u5, value [0, 16], default 8.
> > + * @__reserved1: reserved
> > + * @clip_neg_0: Limit of negative edge in dark area
> > + * u5, value [0, 16], default 8.
> > + * @__reserved2: reserved
> > + * @delta_clip_neg: Difference in the limit of edges between the bright
> > + * and dark areas for negative edges.
> > + * u5, value [0, 16], default 8.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_clip_config {
> > + __u32 clip_pos_0:5;
> > + __u32 __reserved0:3;
> > + __u32 delta_clip_posi:5;
> > + __u32 __reserved1:3;
> > + __u32 clip_neg_0:5;
> > + __u32 __reserved2:3;
> > + __u32 delta_clip_neg:5;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_frng_config - Luma(Y) edge
> enhancement
> > + * noise reduction fringe config
> > + *
> > + * @gain_exp: Common exponent of gains, u4, [0, 8], default 2.
> > + * @__reserved0: reserved
> > + * @min_edge: Threshold for edge and smooth stitching, u13.
> > + * @__reserved1: reserved
> > + * @lin_seg_param: Power of LinSeg, u4.
> > + * @__reserved2: reserved
> > + * @t1: Parameter for enabling/disabling the edge enhancement, u1.0, [0,
> 1],
> > + * default 1.
> > + * @t2: Parameter for enabling/disabling the smoothing, u1.0, [0, 1],
> > + * default 1.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_frng_config {
> > + __u32 gain_exp:4;
> > + __u32 __reserved0:28;
> > + __u32 min_edge:13;
> > + __u32 __reserved1:3;
> > + __u32 lin_seg_param:4;
> > + __u32 __reserved2:4;
> > + __u32 t1:1;
> > + __u32 t2:1;
> > + __u32 __reserved3:6;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_diag_config - Luma(Y) edge
> enhancement
> > + * noise reduction diagonal config
> > + *
> > + * @diag_disc_g: Coefficient that prioritize diagonal edge direction on
> > + * horizontal or vertical for final enhancement.
> > + * u4.0, [1, 15], default 1.
> > + * @__reserved0: reserved
> > + * @hvw_hor: Weight of horizontal/vertical edge enhancement for hv
> edge.
> > + * u2.2, [1, 15], default 4.
> > + * @dw_hor: Weight of diagonal edge enhancement for hv edge.
> > + * u2.2, [1, 15], default 1.
> > + * @hvw_diag: Weight of horizontal/vertical edge enhancement for
> diagonal edge.
> > + * u2.2, [1, 15], default 1.
> > + * @dw_diag: Weight of diagonal edge enhancement for diagonal edge.
> > + * u2.2, [1, 15], default 4.
> > + * @__reserved1: reserved
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_diag_config {
> > + __u32 diag_disc_g:4;
> > + __u32 __reserved0:4;
> > + __u32 hvw_hor:4;
> > + __u32 dw_hor:4;
> > + __u32 hvw_diag:4;
> > + __u32 dw_diag:4;
> > + __u32 __reserved1:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config - Luma(Y) edge
> enhancement
> > + * noise reduction false color correction (FCC) coring config
> > + *
> > + * @pos_0: Gain for positive edge in dark, u13.0, [0, 16], default 0.
> > + * @__reserved0: reserved
> > + * @pos_delta: Gain for positive edge in bright, value: pos_0 + pos_delta
> <=16
> > + * u13.0, default 0.
> > + * @__reserved1: reserved
> > + * @neg_0: Gain for negative edge in dark area, u13.0, range [0, 16],
> default 0.
> > + * @__reserved2: reserved
> > + * @neg_delta: Gain for negative edge in bright area. neg_0 + neg_delta
> <=16
> > + * u13.0, default 0.
> > + * @__reserved3: reserved
> > + *
> > + * Coring is a simple soft thresholding technique.
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config {
> > + __u32 pos_0:13;
> > + __u32 __reserved0:3;
> > + __u32 pos_delta:13;
> > + __u32 __reserved1:3;
> > + __u32 neg_0:13;
> > + __u32 __reserved2:3;
> > + __u32 neg_delta:13;
> > + __u32 __reserved3:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp1_y_ee_nr_config - Edge enhancement and noise
> reduction
> > + *
> > + * @lpf: low-pass filter config. See &ipu3_uapi_yuvp1_y_ee_nr_lpf_config
> > + * @sense: sensitivity config. See
> &ipu3_uapi_yuvp1_y_ee_nr_sense_config
> > + * @gain: gain config as defined in
> &ipu3_uapi_yuvp1_y_ee_nr_gain_config
> > + * @clip: clip config as defined in &ipu3_uapi_yuvp1_y_ee_nr_clip_config
> > + * @frng: fringe config as defined in
> &ipu3_uapi_yuvp1_y_ee_nr_frng_config
> > + * @diag: diagonal edge config. See
> &ipu3_uapi_yuvp1_y_ee_nr_diag_config
> > + * @fc_coring: coring config for fringe control. See
> > + * &ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config
> > + */
> > +struct ipu3_uapi_yuvp1_y_ee_nr_config {
> > + struct ipu3_uapi_yuvp1_y_ee_nr_lpf_config lpf;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_sense_config sense;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_gain_config gain;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_clip_config clip;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_frng_config frng;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_diag_config diag;
> > + struct ipu3_uapi_yuvp1_y_ee_nr_fc_coring_config fc_coring;
> > +} __packed;
> > +
> > +/* Total Color Correction */
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_gen_control_static_config - Total color
> correction
> > + * general control config
> > + *
> > + * @en: 0 - TCC disabled. Output = input 1 - TCC enabled.
> > + * @blend_shift: blend shift, Range[3, 4], default NA.
> > + * @gain_according_to_y_only: 0: Gain is calculated according to YUV,
> > + * 1: Gain is calculated according to Y only
> > + * @__reserved0: reserved
> > + * @gamma: Final blending coefficients. Values[-16, 16], default NA.
> > + * @__reserved1: reserved
> > + * @delta: Final blending coefficients. Values[-16, 16], default NA.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_gen_control_static_config {
> > + __u32 en:1;
> > + __u32 blend_shift:3;
> > + __u32 gain_according_to_y_only:1;
> > + __u32 __reserved0:11;
> > + __s32 gamma:5;
> > + __u32 __reserved1:3;
> > + __s32 delta:5;
> > + __u32 __reserved2:3;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config - Total color
> correction
> > + * multi-axis color control (MACC) config
> > + *
> > + * @a: a coefficient for 2x2 MACC conversion matrix.
> > + * @__reserved0: reserved
> > + * @b: b coefficient 2x2 MACC conversion matrix.
> > + * @__reserved1: reserved
> > + * @c: c coefficient for 2x2 MACC conversion matrix.
> > + * @__reserved2: reserved
> > + * @d: d coefficient for 2x2 MACC conversion matrix.
> > + * @__reserved3: reserved
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config {
> > + __s32 a:12;
> > + __u32 __reserved0:4;
> > + __s32 b:12;
> > + __u32 __reserved1:4;
> > + __s32 c:12;
> > + __u32 __reserved2:4;
> > + __s32 d:12;
> > + __u32 __reserved3:4;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_macc_table_static_config - Total color
> correction
> > + * multi-axis color control (MACC) table array
> > + *
> > + * @entries: config for multi axis color correction, as specified by
> > + * &ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_macc_table_static_config {
> > + struct ipu3_uapi_yuvp2_tcc_macc_elem_static_config
> > + entries[IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config - Total color
> correction
> > + * inverse y lookup table
> > + *
> > + * @entries: lookup table for inverse y estimation, and use it to estimate
> the
> > + * ratio between luma and chroma. Chroma by approximate the
> absolute
> > + * value of the radius on the chroma plane (R = sqrt(u^2+v^2) ) and
> > + * luma by approximate by 1/Y.
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config {
> > + __u16 entries[IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config - Total color
> > + * correction lookup table for PCWL
> > + *
> > + * @entries: lookup table for gain piece wise linear transformation (PCWL)
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config {
> > + __u16 entries[IPU3_UAPI_YUVP2_TCC_GAIN_PCWL_LUT_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config - Total color
> correction
> > + * lookup table for r square root
> > + *
> > + * @entries: lookup table for r square root estimation
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config {
> > + __s16 entries[IPU3_UAPI_YUVP2_TCC_R_SQR_LUT_ELEMENTS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_yuvp2_tcc_static_config- Total color correction static
> > + *
> > + * @gen_control: general config for Total Color Correction
> > + * @macc_table: config for multi axis color correction
> > + * @inv_y_lut: lookup table for inverse y estimation
> > + * @gain_pcwl: lookup table for gain PCWL
> > + * @r_sqr_lut: lookup table for r square root estimation.
> > + */
> > +struct ipu3_uapi_yuvp2_tcc_static_config {
> > + struct ipu3_uapi_yuvp2_tcc_gen_control_static_config gen_control;
> > + struct ipu3_uapi_yuvp2_tcc_macc_table_static_config macc_table;
> > + struct ipu3_uapi_yuvp2_tcc_inv_y_lut_static_config inv_y_lut;
> > + struct ipu3_uapi_yuvp2_tcc_gain_pcwl_lut_static_config gain_pcwl;
> > + struct ipu3_uapi_yuvp2_tcc_r_sqr_lut_static_config r_sqr_lut;
> > +} __packed;
> > +
> > +/* Advanced Noise Reduction related structs */
> > +
> > +/*
> > + * struct ipu3_uapi_anr_alpha - Advanced noise reduction alpha
> > + *
> > + * Tunable parameters that are subject to modification according to the
> > + * total gain used.
> > + */
> > +struct ipu3_uapi_anr_alpha {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > + __u16 dc_gr;
> > + __u16 dc_r;
> > + __u16 dc_b;
> > + __u16 dc_gb;
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_anr_beta - Advanced noise reduction beta
> > + *
> > + * Tunable parameters that are subject to modification according to the
> > + * total gain used.
> > + */
> > +struct ipu3_uapi_anr_beta {
> > + __u16 beta_gr;
> > + __u16 beta_r;
> > + __u16 beta_b;
> > + __u16 beta_gb;
> > +} __packed;
> > +
> > +/*
> > + * struct ipu3_uapi_anr_plain_color - Advanced noise reduction plain
> color with
> > + * 4x4 matrix
>
> What is a "plain color"? Or should this have been "plane color"? But then I
> don't know what a "plane color" is either :-)
>
Thanks for catching this, will find out the plain explanation :)
> > + *
> > + * Tunable parameters that are subject to modification according to the
> > + * total gain used.
> > + */
> > +struct ipu3_uapi_anr_plain_color {
> > + __u16 reg_w_gr[16];
> > + __u16 reg_w_r[16];
> > + __u16 reg_w_b[16];
> > + __u16 reg_w_gb[16];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_transform_config - Advanced noise reduction
> transform
> > + *
> > + * @enable: advanced noise reduction enabled.
> > + * @adaptive_treshhold_en: On IPU3, adaptive threshold is always
> enabled.
> > + * @__reserved1: reserved
> > + * @__reserved2: reserved
> > + * @alpha: using following defaults:
> > + * 13, 13, 13, 13, 0, 0, 0, 0
> > + * 11, 11, 11, 11, 0, 0, 0, 0
> > + * 14, 14, 14, 14, 0, 0, 0, 0
> > + * @beta: use following defaults:
> > + * 24, 24, 24, 24
> > + * 21, 20, 20, 21
> > + * 25, 25, 25, 25
> > + * @color: use defaults defined in driver/media/pci/intel/ipu3-tables.c
> > + * @sqrt_lut: 11 bits per element, values =
> > + * [724 768 810 849 887
> > + * 923 958 991 1024 1056
> > + * 1116 1145 1173 1201 1086
> > + * 1228 1254 1280 1305 1330
> > + * 1355 1379 1402 1425 1448]
> > + * @xreset: Reset value of X for r^2 calculation Value: col_start-X_center
> > + * Constraint: Xreset + FrameWdith=4095 Xreset= -4095, default -1632.
> > + * @__reserved3: reserved
> > + * @yreset: Reset value of Y for r^2 calculation Value: row_start-Y_center
> > + * Constraint: Yreset + FrameHeight=4095 Yreset= -4095, default -1224.
> > + * @__reserved4: reserved
> > + * @x_sqr_reset: Reset value of X^2 for r^2 calculation Value = (Xreset)^2
> > + * @r_normfactor: Normalization factor for R. Default 14.
> > + * @__reserved5: reserved
> > + * @y_sqr_reset: Reset value of Y^2 for r^2 calculation Value = (Yreset)^2
> > + * @gain_scale: Parameter describing shading gain as a function of
> distance
> > + * from the image center.
> > + * A single value per frame, loaded by the driver. Default 115.
> > + */
> > +struct ipu3_uapi_anr_transform_config {
> > + __u32 enable:1; /* 0 or 1, disabled or enabled
> */
> > + __u32 adaptive_treshhold_en:1; /* On IPU3, always enabled
> */
> > +
> > + __u32 __reserved1:30;
> > + __u8 __reserved2[44];
> > +
> > + struct ipu3_uapi_anr_alpha alpha[3];
> > + struct ipu3_uapi_anr_beta beta[3];
> > + struct ipu3_uapi_anr_plain_color color[3];
> > +
> > + __u16 sqrt_lut[IPU3_UAPI_ANR_LUT_SIZE]; /* 11 bits per element
> */
> > +
> > + __s16 xreset:13;
> > + __u16 __reserved3:3;
> > + __s16 yreset:13;
> > + __u16 __reserved4:3;
> > +
> > + __u32 x_sqr_reset:24;
> > + __u32 r_normfactor:5;
> > + __u32 __reserved5:3;
> > +
> > + __u32 y_sqr_reset:24;
> > + __u32 gain_scale:8;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_stitch_pyramid - ANR stitch pyramid
> > + *
> > + * @entry0: pyramid LUT entry0, range [0x0, 0x3f]
> > + * @entry1: pyramid LUT entry1, range [0x0, 0x3f]
> > + * @entry2: pyramid LUT entry2, range [0x0, 0x3f]
> > + * @__reserved: reserved
> > + */
> > +struct ipu3_uapi_anr_stitch_pyramid {
> > + __u32 entry0:6;
> > + __u32 entry1:6;
> > + __u32 entry2:6;
> > + __u32 __reserved:14;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_stitch_config - ANR stitch config
> > + *
> > + * @anr_stitch_en: enable stitch. Enabled with 1.
> > + * @__reserved: reserved
> > + * @pyramid: pyramid table as defined by
> &ipu3_uapi_anr_stitch_pyramid
> > + * default values:
> > + * { 1, 3, 5 }, { 7, 7, 5 }, { 3, 1, 3 },
> > + * { 9, 15, 21 }, { 21, 15, 9 }, { 3, 5, 15 },
> > + * { 25, 35, 35 }, { 25, 15, 5 }, { 7, 21, 35 },
> > + * { 49, 49, 35 }, { 21, 7, 7 }, { 21, 35, 49 },
> > + * { 49, 35, 21 }, { 7, 5, 15 }, { 25, 35, 35 },
> > + * { 25, 15, 5 }, { 3, 9, 15 }, { 21, 21, 15 },
> > + * { 9, 3, 1 }, { 3, 5, 7 }, { 7, 5, 3}, { 1 }
> > + */
> > +struct ipu3_uapi_anr_stitch_config {
> > + __u32 anr_stitch_en;
> > + __u8 __reserved[44];
> > + struct ipu3_uapi_anr_stitch_pyramid
> pyramid[IPU3_UAPI_ANR_PYRAMID_SIZE];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_anr_config - ANR config
> > + *
> > + * @transform: advanced noise reduction transform config as
> specified by
> > + * &ipu3_uapi_anr_transform_config
> > + * @stitch: create 4x4 patch from 4 surrounding 8x8 patches.
> > + */
> > +struct ipu3_uapi_anr_config {
> > + struct ipu3_uapi_anr_transform_config transform
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_anr_stitch_config stitch __attribute__((aligned(32)));
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_acc_param - Accelerator cluster parameters
> > + *
> > + * ACC refers to the HW cluster containing all Fixed Functions(FFs). Each FF
>
> Add space before '(FFs)'.
Ack.
>
> > + * implements a specific algorithm.
> > + *
> > + * @bnr: parameters for bayer noise reduction static config. See
> > + * &ipu3_uapi_bnr_static_config
> > + * @green_disparity: disparity static config between gr and gb
> channel.
> > + * See &ipu3_uapi_bnr_static_config_green_disparity
> > + * @dm: de-mosaic config. See &ipu3_uapi_dm_config
> > + * @ccm: color correction matrix. See &ipu3_uapi_ccm_mat_config
> > + * @gamma: gamma correction config. See &ipu3_uapi_gamma_config
> > + * @csc: color space conversion matrix. See
> &ipu3_uapi_csc_mat_config
> > + * @cds: color down sample config. See &ipu3_uapi_cds_params
> > + * @shd: lens shading correction config. See &ipu3_uapi_shd_config
> > + * @iefd: Image enhancement filter and denoise config.
> > + * &ipu3_uapi_yuvp1_iefd_config
> > + * @yds_c0: y down scaler config. &ipu3_uapi_yuvp1_yds_config
> > + * @chnr_c0: chroma noise reduction config.
> &ipu3_uapi_yuvp1_chnr_config
> > + * @y_ee_nr: y edge enhancement and noise reduction config.
> > + * &ipu3_uapi_yuvp1_y_ee_nr_config
> > + * @yds: y down scaler config. See &ipu3_uapi_yuvp1_yds_config
> > + * @chnr: chroma noise reduction config. See
> &ipu3_uapi_yuvp1_chnr_config
> > + * @__reserved1: reserved
> > + * @yds2: y channel down scaler config. See
> &ipu3_uapi_yuvp1_yds_config
> > + * @tcc: total color correction config as defined in struct
> > + * &ipu3_uapi_yuvp2_tcc_static_config
> > + * @__reserved2: reserved
> > + * @anr: advanced noise reduction config.See &ipu3_uapi_anr_config
> > + * @awb_fr: AWB filter response config. See ipu3_uapi_awb_fr_config
> > + * @ae: auto exposure config As specified by &ipu3_uapi_ae_config
> > + * @af: auto focus config. As specified by &ipu3_uapi_af_config
> > + * @awb: auto white balance config. As specified by
> &ipu3_uapi_awb_config
> > + */
> > +struct ipu3_uapi_acc_param {
> > + struct ipu3_uapi_bnr_static_config bnr;
> > + struct ipu3_uapi_bnr_static_config_green_disparity
> > + green_disparity __attribute__((aligned(32)));
> > + struct ipu3_uapi_dm_config dm __attribute__((aligned(32)));
> > + struct ipu3_uapi_ccm_mat_config ccm __attribute__((aligned(32)));
> > + struct ipu3_uapi_gamma_config gamma __attribute__((aligned(32)));
> > + struct ipu3_uapi_csc_mat_config csc __attribute__((aligned(32)));
> > + struct ipu3_uapi_cds_params cds __attribute__((aligned(32)));
> > + struct ipu3_uapi_shd_config shd __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_iefd_config iefd __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_yds_config yds_c0
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_chnr_config chnr_c0
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_y_ee_nr_config y_ee_nr
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_yds_config yds __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_chnr_config chnr
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp1_yds_config yds2 __attribute__((aligned(32)));
> > + struct ipu3_uapi_yuvp2_tcc_static_config tcc
> __attribute__((aligned(32)));
> > + struct ipu3_uapi_anr_config anr;
> > + struct ipu3_uapi_awb_fr_config awb_fr;
> > + struct ipu3_uapi_ae_config ae;
> > + struct ipu3_uapi_af_config af;
> > + struct ipu3_uapi_awb_config awb;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_isp_lin_vmem_params - Linearization parameters
> > + *
> > + * @lin_lutlow_gr: linearization look-up table for GR channel interpolation.
> > + * @lin_lutlow_r: linearization look-up table for R channel interpolation.
> > + * @lin_lutlow_b: linearization look-up table for B channel interpolation.
> > + * @lin_lutlow_gb: linearization look-up table for GB channel interpolation.
> > + * lin_lutlow_gr / lin_lutlow_gr / lin_lutlow_gr /
> > + * lin_lutlow_gr <= LIN_MAX_VALUE - 1.
> > + * @lin_lutdif_gr: lin_lutlow_gr[i+1] - lin_lutlow_gr[i].
> > + * @lin_lutdif_r: lin_lutlow_r[i+1] - lin_lutlow_r[i].
> > + * @lin_lutdif_b: lin_lutlow_b[i+1] - lin_lutlow_b[i].
> > + * @lin_lutdif_gb: lin_lutlow_gb[i+1] - lin_lutlow_gb[i].
> > + */
> > +struct ipu3_uapi_isp_lin_vmem_params {
> > + __s16 lin_lutlow_gr[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_r[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_b[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutlow_gb[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_gr[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_r[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_b[IPU3_UAPI_LIN_LUT_SIZE];
> > + __s16 lin_lutdif_gb[IPU3_UAPI_LIN_LUT_SIZE];
> > +} __packed;
> > +
> > +/* Temporal Noise Reduction */
> > +
> > +/**
> > + * struct ipu3_uapi_isp_tnr3_vmem_params - Temporal noise reduction
> vector
> > + * memory parameters
> > + *
> > + * @slope: slope setting in interpolation curve for temporal noise
> reduction.
> > + * @__reserved1: reserved
> > + * @sigma: knee point setting in interpolation curve for temporal
> > + * noise reduction.
> > + * @__reserved2: reserved
> > + */
> > +struct ipu3_uapi_isp_tnr3_vmem_params {
> > + __u16 slope[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > + __u16 __reserved1[IPU3_UAPI_ISP_VEC_ELEMS
> > + -
> IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > + __u16 sigma[IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > + __u16 __reserved2[IPU3_UAPI_ISP_VEC_ELEMS
> > + -
> IPU3_UAPI_ISP_TNR3_VMEM_LEN];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_isp_tnr3_params - Temporal noise reduction v3
> parameters
> > + *
> > + * @knee_y1: Knee point TNR3 assumes standard deviation of Y,U and
> > + * V at Y1 are TnrY1_Sigma_Y, U and V.
> > + * @knee_y2: Knee point TNR3 assumes standard deviation of Y,U and
> > + * V at Y2 are TnrY2_Sigma_Y, U and V.
> > + * @maxfb_y: Max feedback gain for Y
> > + * @maxfb_u: Max feedback gain for U
> > + * @maxfb_v: Max feedback gain for V
> > + * @round_adj_y: rounding Adjust for Y
> > + * @round_adj_u: rounding Adjust for U
> > + * @round_adj_v: rounding Adjust for V
> > + * @ref_buf_select: selection of the reference frame buffer to be used.
> > + */
> > +struct ipu3_uapi_isp_tnr3_params {
> > + __u32 knee_y1;
> > + __u32 knee_y2;
> > + __u32 maxfb_y;
> > + __u32 maxfb_u;
> > + __u32 maxfb_v;
> > + __u32 round_adj_y;
> > + __u32 round_adj_u;
> > + __u32 round_adj_v;
> > + __u32 ref_buf_select;
> > +} __packed;
> > +
> > +/* Extreme Noise Reduction version 3 */
> > +
> > +/**
> > + * struct ipu3_uapi_isp_xnr3_vmem_params - Extreme noise reduction v3
> > + * vector memory parameters
> > + *
> > + * @x: xnr3 parameters.
> > + * @a: xnr3 parameters.
> > + * @b: xnr3 parameters.
> > + * @c: xnr3 parameters.
> > + */
> > +struct ipu3_uapi_isp_xnr3_vmem_params {
> > + __u16 x[IPU3_UAPI_ISP_VEC_ELEMS];
> > + __u16 a[IPU3_UAPI_ISP_VEC_ELEMS];
> > + __u16 b[IPU3_UAPI_ISP_VEC_ELEMS];
> > + __u16 c[IPU3_UAPI_ISP_VEC_ELEMS];
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_xnr3_alpha_params - Extreme noise reduction v3
> > + * alpha tuning parameters
> > + *
> > + * @y0: Sigma for Y range similarity in dark area.
> > + * @u0: Sigma for U range similarity in dark area.
> > + * @v0: Sigma for V range similarity in dark area.
> > + * @ydiff: Sigma difference for Y between bright area and dark area.
> > + * @udiff: Sigma difference for U between bright area and dark area.
> > + * @vdiff: Sigma difference for V between bright area and dark area.
> > + */
> > +struct ipu3_uapi_xnr3_alpha_params {
> > + __u32 y0;
> > + __u32 u0;
> > + __u32 v0;
> > + __u32 ydiff;
> > + __u32 udiff;
> > + __u32 vdiff;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_xnr3_coring_params - Extreme noise reduction v3
> > + * coring parameters
> > + *
> > + * @u0: Coring Threshold of U channel in dark area.
> > + * @v0: Coring Threshold of V channel in dark area.
> > + * @udiff: Threshold difference of U channel between bright and dark
> area.
> > + * @vdiff: Threshold difference of V channel between bright and dark area.
> > + */
> > +struct ipu3_uapi_xnr3_coring_params {
> > + __u32 u0;
> > + __u32 v0;
> > + __u32 udiff;
> > + __u32 vdiff;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_xnr3_blending_params - Blending factor
> > + *
> > + * @strength: The factor for blending output with input. This is tuning
> > + * parameterHigher values lead to more aggressive XNR operation.
> > + */
> > +struct ipu3_uapi_xnr3_blending_params {
> > + __u32 strength;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_isp_xnr3_params - Extreme noise reduction v3
> parameters
> > + *
> > + * @alpha: parameters for xnr3 alpha. See
> &ipu3_uapi_xnr3_alpha_params
> > + * @coring: parameters for xnr3 coring. See
> &ipu3_uapi_xnr3_coring_params
> > + * @blending: parameters for xnr3 blending. See
> &ipu3_uapi_xnr3_blending_params
> > + */
> > +struct ipu3_uapi_isp_xnr3_params {
> > + struct ipu3_uapi_xnr3_alpha_params alpha;
> > + struct ipu3_uapi_xnr3_coring_params coring;
> > + struct ipu3_uapi_xnr3_blending_params blending;
> > +} __packed;
> > +
> > +/***** Obgrid (optical black level compensation) table entry *****/
> > +
> > +/**
> > + * struct ipu3_uapi_obgrid_param - Optical black level compensation
> parameters
> > + *
> > + * @gr: Grid table values for color GR
> > + * @r: Grid table values for color R
> > + * @b: Grid table values for color B
> > + * @gb: Grid table values for color GB
> > + *
> > + * Black level is different for red, green, and blue channels. So black level
> > + * compensation is different per channel.
> > + */
> > +struct ipu3_uapi_obgrid_param {
> > + __u16 gr;
> > + __u16 r;
> > + __u16 b;
> > + __u16 gb;
> > +} __packed;
> > +
> > +/******************* V4L2_META_FMT_IPU3_PARAMS
> *******************/
> > +
> > +/**
> > + * struct ipu3_uapi_flags - bits to indicate which pipeline needs update
> > + *
> > + * @gdc: 0 = no update, 1 = update.
> > + * @obgrid: 0 = no update, 1 = update.
> > + * @__reserved1: Not used.
> > + * @acc_bnr: 0 = no update, 1 = update.
> > + * @acc_green_disparity: 0 = no update, 1 = update.
> > + * @acc_dm: 0 = no update, 1 = update.
> > + * @acc_ccm: 0 = no update, 1 = update.
> > + * @acc_gamma: 0 = no update, 1 = update.
> > + * @acc_csc: 0 = no update, 1 = update.
> > + * @acc_cds: 0 = no update, 1 = update.
> > + * @acc_shd: 0 = no update, 1 = update.
> > + * @__reserved2: Not used.
> > + * @acc_iefd: 0 = no update, 1 = update.
> > + * @acc_yds_c0: 0 = no update, 1 = update.
> > + * @acc_chnr_c0: 0 = no update, 1 = update.
> > + * @acc_y_ee_nr: 0 = no update, 1 = update.
> > + * @acc_yds: 0 = no update, 1 = update.
> > + * @acc_chnr: 0 = no update, 1 = update.
> > + * @acc_ytm: 0 = no update, 1 = update.
> > + * @acc_yds2: 0 = no update, 1 = update.
> > + * @acc_tcc: 0 = no update, 1 = update.
> > + * @acc_dpc: 0 = no update, 1 = update.
> > + * @acc_bds: 0 = no update, 1 = update.
> > + * @acc_anr: 0 = no update, 1 = update.
> > + * @acc_awb_fr: 0 = no update, 1 = update.
> > + * @acc_ae: 0 = no update, 1 = update.
> > + * @acc_af: 0 = no update, 1 = update.
> > + * @acc_awb: 0 = no update, 1 = update.
> > + * @__acc_osys: 0 = no update, 1 = update.
> > + * @__reserved3: Not used.
> > + * @lin_vmem_params: 0 = no update, 1 = update.
> > + * @tnr3_vmem_params: 0 = no update, 1 = update.
> > + * @xnr3_vmem_params: 0 = no update, 1 = update.
> > + * @tnr3_dmem_params: 0 = no update, 1 = update.
> > + * @xnr3_dmem_params: 0 = no update, 1 = update.
> > + * @__reserved4: Not used.
> > + * @obgrid_param: 0 = no update, 1 = update.
> > + * @__reserved5: Not used.
> > + */
> > +struct ipu3_uapi_flags {
> > + __u32 gdc:1;
> > + __u32 obgrid:1;
> > + __u32 __reserved1:30;
> > +
> > + __u32 acc_bnr:1;
> > + __u32 acc_green_disparity:1;
> > + __u32 acc_dm:1;
> > + __u32 acc_ccm:1;
> > + __u32 acc_gamma:1;
> > + __u32 acc_csc:1;
> > + __u32 acc_cds:1;
> > + __u32 acc_shd:1;
> > + __u32 __reserved2:2;
> > + __u32 acc_iefd:1;
> > + __u32 acc_yds_c0:1;
> > + __u32 acc_chnr_c0:1;
> > + __u32 acc_y_ee_nr:1;
> > + __u32 acc_yds:1;
> > + __u32 acc_chnr:1;
> > + __u32 acc_ytm:1;
> > + __u32 acc_yds2:1;
> > + __u32 acc_tcc:1;
> > + __u32 acc_dpc:1;
> > + __u32 acc_bds:1;
> > + __u32 acc_anr:1;
> > + __u32 acc_awb_fr:1;
> > + __u32 acc_ae:1;
> > + __u32 acc_af:1;
> > + __u32 acc_awb:1;
> > + __u32 __reserved3:4;
> > +
> > + __u32 lin_vmem_params:1;
> > + __u32 tnr3_vmem_params:1;
> > + __u32 xnr3_vmem_params:1;
> > + __u32 tnr3_dmem_params:1;
> > + __u32 xnr3_dmem_params:1;
> > + __u32 __reserved4:1;
> > + __u32 obgrid_param:1;
> > + __u32 __reserved5:25;
> > +} __packed;
> > +
> > +/**
> > + * struct ipu3_uapi_params - V4L2_META_FMT_IPU3_PARAMS
> > + *
> > + * @use: select which parameters to apply, see &ipu3_uapi_flags
> > + * @acc_param: ACC parameters, as specified by
> &ipu3_uapi_acc_param
> > + * @lin_vmem_params: linearization VMEM, as specified by
> > + * &ipu3_uapi_isp_lin_vmem_params
> > + * @tnr3_vmem_params: tnr3 VMEM as specified by
> > + * &ipu3_uapi_isp_tnr3_vmem_params
> > + * @xnr3_vmem_params: xnr3 VMEM as specified by
> > + * &ipu3_uapi_isp_xnr3_vmem_params
> > + * @tnr3_dmem_params: tnr3 DMEM as specified by
> &ipu3_uapi_isp_tnr3_params
> > + * @xnr3_dmem_params: xnr3 DMEM as specified by
> &ipu3_uapi_isp_xnr3_params
> > + * @obgrid_param: obgrid parameters as specified by
> > + * &ipu3_uapi_obgrid_param
> > + *
> > + * The video queue "parameters" is of format
> V4L2_META_FMT_IPU3_PARAMS.
> > + * This is a "single plane" v4l2_meta_format using
> V4L2_BUF_TYPE_META_OUTPUT.
> > + *
> > + * struct ipu3_uapi_params as defined below contains a lot of parameters
> and
> > + * ipu3_uapi_flags selects which parameters to apply.
> > + */
> > +struct ipu3_uapi_params {
> > + /* Flags which of the settings below are to be applied */
> > + struct ipu3_uapi_flags use __attribute__((aligned(32)));
> > +
> > + /* Accelerator cluster parameters */
> > + struct ipu3_uapi_acc_param acc_param;
> > +
> > + /* ISP vector address space parameters */
> > + struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
> > + struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
> > + struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
> > +
> > + /* ISP data memory (DMEM) parameters */
> > + struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
> > + struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
> > +
> > + /* Optical black level compensation */
> > + struct ipu3_uapi_obgrid_param obgrid_param;
> > +} __packed;
> > +#endif
> >
>
> This looks very good. I'm sure you've been less than happy with us for having
> to document all this, but the end result is very impressive.
>
> Based on my comment and those of others I would expect that v8 will be the
> final version.
>
> Regards,
>
> Hans
Thanks a lot, we really appreciate your encouraging words and time spent on this series. Til v8 then.
Yong
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 04/16] intel-ipu3: abi: Add register definitions and enum
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (2 preceding siblings ...)
2018-10-29 22:22 ` [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI Yong Zhi
@ 2018-10-29 22:22 ` Yong Zhi
2018-10-29 22:22 ` [PATCH v7 05/16] intel-ipu3: abi: Add structs Yong Zhi
` (13 subsequent siblings)
17 siblings, 0 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:22 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
Add macros and enums used for IPU3 firmware interface.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-abi.h | 661 ++++++++++++++++++++++++++++++++
1 file changed, 661 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-abi.h b/drivers/media/pci/intel/ipu3/ipu3-abi.h
new file mode 100644
index 0000000..ac08ad3
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-abi.h
@@ -0,0 +1,661 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+
+#ifndef __IPU3_ABI_H
+#define __IPU3_ABI_H
+
+#include <uapi/linux/intel-ipu3.h>
+
+/******************* IMGU Hardware information *******************/
+
+typedef u32 imgu_addr_t;
+
+#define IMGU_ISP_VMEM_ALIGN 128
+#define IMGU_DVS_BLOCK_W 64
+#define IMGU_DVS_BLOCK_H 32
+#define IMGU_GDC_BUF_X (2 * IMGU_DVS_BLOCK_W)
+#define IMGU_GDC_BUF_Y IMGU_DVS_BLOCK_H
+/* n = 0..1 */
+#define IMGU_SP_PMEM_BASE(n) (0x20000 + (n) * 0x4000)
+#define IMGU_MAX_BQ_GRID_WIDTH 80
+#define IMGU_MAX_BQ_GRID_HEIGHT 60
+#define IMGU_OBGRID_TILE_SIZE 16
+#define IMGU_PIXELS_PER_WORD 50
+#define IMGU_BYTES_PER_WORD 64
+#define IMGU_STRIPE_FIXED_HALF_OVERLAP 2
+#define IMGU_SHD_SETS 3
+#define IMGU_BDS_MIN_CLIP_VAL 0
+#define IMGU_BDS_MAX_CLIP_VAL 2
+
+#define IMGU_ABI_AWB_MAX_CELLS_PER_SET 160
+#define IMGU_ABI_AF_MAX_CELLS_PER_SET 32
+#define IMGU_ABI_AWB_FR_MAX_CELLS_PER_SET 32
+
+#define IMGU_ABI_ACC_OP_IDLE 0
+#define IMGU_ABI_ACC_OP_END_OF_ACK 1
+#define IMGU_ABI_ACC_OP_END_OF_OPS 2
+#define IMGU_ABI_ACC_OP_NO_OPS 3
+
+#define IMGU_ABI_ACC_OPTYPE_PROCESS_LINES 0
+#define IMGU_ABI_ACC_OPTYPE_TRANSFER_DATA 1
+
+/* Register definitions */
+
+/* PM_CTRL_0_5_0_IMGHMMADR */
+#define IMGU_REG_PM_CTRL 0x0
+#define IMGU_PM_CTRL_START BIT(0)
+#define IMGU_PM_CTRL_CFG_DONE BIT(1)
+#define IMGU_PM_CTRL_RACE_TO_HALT BIT(2)
+#define IMGU_PM_CTRL_NACK_ALL BIT(3)
+#define IMGU_PM_CTRL_CSS_PWRDN BIT(4)
+#define IMGU_PM_CTRL_RST_AT_EOF BIT(5)
+#define IMGU_PM_CTRL_FORCE_HALT BIT(6)
+#define IMGU_PM_CTRL_FORCE_UNHALT BIT(7)
+#define IMGU_PM_CTRL_FORCE_PWRDN BIT(8)
+#define IMGU_PM_CTRL_FORCE_RESET BIT(9)
+
+/* SYSTEM_REQ_0_5_0_IMGHMMADR */
+#define IMGU_REG_SYSTEM_REQ 0x18
+#define IMGU_SYSTEM_REQ_FREQ_MASK 0x3f
+#define IMGU_SYSTEM_REQ_FREQ_DIVIDER 25
+#define IMGU_REG_INT_STATUS 0x30
+#define IMGU_REG_INT_ENABLE 0x34
+#define IMGU_REG_INT_CSS_IRQ BIT(31)
+/* STATE_0_5_0_IMGHMMADR */
+#define IMGU_REG_STATE 0x130
+#define IMGU_STATE_HALT_STS BIT(0)
+#define IMGU_STATE_IDLE_STS BIT(1)
+#define IMGU_STATE_POWER_UP BIT(2)
+#define IMGU_STATE_POWER_DOWN BIT(3)
+#define IMGU_STATE_CSS_BUSY_MASK 0xc0
+#define IMGU_STATE_PM_FSM_MASK 0x180
+#define IMGU_STATE_PWRDNM_FSM_MASK 0x1E00000
+/* PM_STS_0_5_0_IMGHMMADR */
+#define IMGU_REG_PM_STS 0x140
+
+#define IMGU_REG_BASE 0x4000
+
+#define IMGU_REG_ISP_CTRL (IMGU_REG_BASE + 0x00)
+#define IMGU_CTRL_RST BIT(0)
+#define IMGU_CTRL_START BIT(1)
+#define IMGU_CTRL_BREAK BIT(2)
+#define IMGU_CTRL_RUN BIT(3)
+#define IMGU_CTRL_BROKEN BIT(4)
+#define IMGU_CTRL_IDLE BIT(5)
+#define IMGU_CTRL_SLEEPING BIT(6)
+#define IMGU_CTRL_STALLING BIT(7)
+#define IMGU_CTRL_IRQ_CLEAR BIT(8)
+#define IMGU_CTRL_IRQ_READY BIT(10)
+#define IMGU_CTRL_IRQ_SLEEPING BIT(11)
+#define IMGU_CTRL_ICACHE_INV BIT(12)
+#define IMGU_CTRL_IPREFETCH_EN BIT(13)
+#define IMGU_REG_ISP_START_ADDR (IMGU_REG_BASE + 0x04)
+#define IMGU_REG_ISP_ICACHE_ADDR (IMGU_REG_BASE + 0x10)
+#define IMGU_REG_ISP_PC (IMGU_REG_BASE + 0x1c)
+
+/* SP Registers, sp = 0:SP0; 1:SP1 */
+#define IMGU_REG_SP_CTRL(sp) (IMGU_REG_BASE + (sp) * 0x100 + 0x100)
+ /* For bits in IMGU_REG_SP_CTRL, see IMGU_CTRL_* */
+#define IMGU_REG_SP_START_ADDR(sp) (IMGU_REG_BASE + (sp) * 0x100 + 0x104)
+#define IMGU_REG_SP_ICACHE_ADDR(sp) (IMGU_REG_BASE + (sp) * 0x100 + 0x11c)
+#define IMGU_REG_SP_CTRL_SINK(sp) (IMGU_REG_BASE + (sp) * 0x100 + 0x130)
+#define IMGU_REG_SP_PC(sp) (IMGU_REG_BASE + (sp) * 0x100 + 0x134)
+
+#define IMGU_REG_TLB_INVALIDATE (IMGU_REG_BASE + 0x300)
+#define IMGU_TLB_INVALIDATE 1
+#define IMGU_REG_L1_PHYS (IMGU_REG_BASE + 0x304) /* 27-bit pfn */
+
+#define IMGU_REG_CIO_GATE_BURST_STATE (IMGU_REG_BASE + 0x404)
+#define IMGU_CIO_GATE_BURST_MASK 0x80
+
+#define IMGU_REG_GP_BUSY (IMGU_REG_BASE + 0x500)
+#define IMGU_REG_GP_STARVING (IMGU_REG_BASE + 0x504)
+#define IMGU_REG_GP_WORKLOAD (IMGU_REG_BASE + 0x508)
+#define IMGU_REG_GP_IRQ(n) (IMGU_REG_BASE + (n) * 4 + 0x50c) /* n = 0..4 */
+#define IMGU_REG_GP_SP1_STRMON_STAT (IMGU_REG_BASE + 0x520)
+#define IMGU_REG_GP_SP2_STRMON_STAT (IMGU_REG_BASE + 0x524)
+#define IMGU_REG_GP_ISP_STRMON_STAT (IMGU_REG_BASE + 0x528)
+#define IMGU_REG_GP_MOD_STRMON_STAT (IMGU_REG_BASE + 0x52c)
+
+/* Port definitions for the streaming monitors. */
+/* For each definition there is signal pair : valid [bit 0]- accept [bit 1] */
+#define IMGU_GP_STRMON_STAT_SP1_PORT_SP12DMA BIT(0)
+#define IMGU_GP_STRMON_STAT_SP1_PORT_DMA2SP1 BIT(2)
+#define IMGU_GP_STRMON_STAT_SP1_PORT_SP12SP2 BIT(4)
+#define IMGU_GP_STRMON_STAT_SP1_PORT_SP22SP1 BIT(6)
+#define IMGU_GP_STRMON_STAT_SP1_PORT_SP12ISP BIT(8)
+#define IMGU_GP_STRMON_STAT_SP1_PORT_ISP2SP1 BIT(10)
+
+#define IMGU_GP_STRMON_STAT_SP2_PORT_SP22DMA BIT(0)
+#define IMGU_GP_STRMON_STAT_SP2_PORT_DMA2SP2 BIT(2)
+#define IMGU_GP_STRMON_STAT_SP2_PORT_SP22SP1 BIT(4)
+#define IMGU_GP_STRMON_STAT_SP2_PORT_SP12SP2 BIT(6)
+
+#define IMGU_GP_STRMON_STAT_ISP_PORT_ISP2DMA BIT(0)
+#define IMGU_GP_STRMON_STAT_ISP_PORT_DMA2ISP BIT(2)
+#define IMGU_GP_STRMON_STAT_ISP_PORT_ISP2SP1 BIT(4)
+#define IMGU_GP_STRMON_STAT_ISP_PORT_SP12ISP BIT(6)
+
+/* Between the devices and the fifo */
+#define IMGU_GP_STRMON_STAT_MOD_PORT_SP12DMA BIT(0)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_DMA2SP1 BIT(2)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_SP22DMA BIT(4)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_DMA2SP2 BIT(6)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_ISP2DMA BIT(8)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_DMA2ISP BIT(10)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_CELLS2GDC BIT(12)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_GDC2CELLS BIT(14)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_CELLS2DECOMP BIT(16)
+#define IMGU_GP_STRMON_STAT_MOD_PORT_DECOMP2CELLS BIT(18)
+/* n = 1..6 */
+#define IMGU_GP_STRMON_STAT_MOD_PORT_S2V(n) (1 << (((n) - 1) * 2 + 20))
+
+/* n = 1..15 */
+#define IMGU_GP_STRMON_STAT_ACCS_PORT_ACC(n) (1 << (((n) - 1) * 2))
+
+/* After FIFO and demux before SP1, n = 1..15 */
+#define IMGU_GP_STRMON_STAT_ACCS2SP1_MON_PORT_ACC(n) (1 << (((n) - 1) * 2))
+
+/* After FIFO and demux before SP2, n = 1..15 */
+#define IMGU_GP_STRMON_STAT_ACCS2SP2_MON_PORT_ACC(n) (1 << (((n) - 1) * 2))
+
+#define IMGU_REG_GP_HALT (IMGU_REG_BASE + 0x5dc)
+
+ /* n = 0..2 (main ctrl, SP0, SP1) */
+#define IMGU_REG_IRQCTRL_BASE(n) (IMGU_REG_BASE + (n) * 0x100 + 0x700)
+#define IMGU_IRQCTRL_MAIN 0
+#define IMGU_IRQCTRL_SP0 1
+#define IMGU_IRQCTRL_SP1 2
+#define IMGU_IRQCTRL_NUM 3
+#define IMGU_IRQCTRL_IRQ_SP1 BIT(0)
+#define IMGU_IRQCTRL_IRQ_SP2 BIT(1)
+#define IMGU_IRQCTRL_IRQ_ISP BIT(2)
+#define IMGU_IRQCTRL_IRQ_SP1_STREAM_MON BIT(3)
+#define IMGU_IRQCTRL_IRQ_SP2_STREAM_MON BIT(4)
+#define IMGU_IRQCTRL_IRQ_ISP_STREAM_MON BIT(5)
+#define IMGU_IRQCTRL_IRQ_MOD_STREAM_MON BIT(6)
+#define IMGU_IRQCTRL_IRQ_MOD_ISP_STREAM_MON BIT(7)
+#define IMGU_IRQCTRL_IRQ_ACCS_STREAM_MON BIT(8)
+#define IMGU_IRQCTRL_IRQ_ACCS_SP1_STREAM_MON BIT(9)
+#define IMGU_IRQCTRL_IRQ_ACCS_SP2_STREAM_MON BIT(10)
+#define IMGU_IRQCTRL_IRQ_ISP_PMEM_ERROR BIT(11)
+#define IMGU_IRQCTRL_IRQ_ISP_BAMEM_ERROR BIT(12)
+#define IMGU_IRQCTRL_IRQ_ISP_VMEM_ERROR BIT(13)
+#define IMGU_IRQCTRL_IRQ_ISP_DMEM_ERROR BIT(14)
+#define IMGU_IRQCTRL_IRQ_SP1_ICACHE_MEM_ERROR BIT(15)
+#define IMGU_IRQCTRL_IRQ_SP1_DMEM_ERROR BIT(16)
+#define IMGU_IRQCTRL_IRQ_SP2_ICACHE_MEM_ERROR BIT(17)
+#define IMGU_IRQCTRL_IRQ_SP2_DMEM_ERROR BIT(18)
+#define IMGU_IRQCTRL_IRQ_ACCS_SCRATCH_MEM_ERROR BIT(19)
+#define IMGU_IRQCTRL_IRQ_GP_TIMER(n) BIT(20 + (n)) /* n=0..1 */
+#define IMGU_IRQCTRL_IRQ_DMA BIT(22)
+#define IMGU_IRQCTRL_IRQ_SW_PIN(n) BIT(23 + (n)) /* n=0..4 */
+#define IMGU_IRQCTRL_IRQ_ACC_SYS BIT(28)
+#define IMGU_IRQCTRL_IRQ_OUT_FORM_IRQ_CTRL BIT(29)
+#define IMGU_IRQCTRL_IRQ_SP1_IRQ_CTRL BIT(30)
+#define IMGU_IRQCTRL_IRQ_SP2_IRQ_CTRL BIT(31)
+#define IMGU_REG_IRQCTRL_EDGE(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x00)
+#define IMGU_REG_IRQCTRL_MASK(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x04)
+#define IMGU_REG_IRQCTRL_STATUS(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x08)
+#define IMGU_REG_IRQCTRL_CLEAR(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x0c)
+#define IMGU_REG_IRQCTRL_ENABLE(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x10)
+#define IMGU_REG_IRQCTRL_EDGE_NOT_PULSE(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x14)
+#define IMGU_REG_IRQCTRL_STR_OUT_ENABLE(n) (IMGU_REG_IRQCTRL_BASE(n) + 0x18)
+
+#define IMGU_REG_GP_TIMER (IMGU_REG_BASE + 0xa34)
+
+#define IMGU_REG_SP_DMEM_BASE(n) (IMGU_REG_BASE + (n) * 0x4000 + 0x4000)
+#define IMGU_REG_ISP_DMEM_BASE (IMGU_REG_BASE + 0xc000)
+
+#define IMGU_REG_GDC_BASE (IMGU_REG_BASE + 0x18000)
+#define IMGU_REG_GDC_LUT_BASE (IMGU_REG_GDC_BASE + 0x140)
+#define IMGU_GDC_LUT_MASK ((1 << 12) - 1) /* Range -1024..+1024 */
+
+#define IMGU_SCALER_PHASES 32
+#define IMGU_SCALER_COEFF_BITS 24
+#define IMGU_SCALER_PHASE_COUNTER_PREC_REF 6
+#define IMGU_SCALER_MAX_EXPONENT_SHIFT 3
+#define IMGU_SCALER_FILTER_TAPS 4
+#define IMGU_SCALER_TAPS_Y IMGU_SCALER_FILTER_TAPS
+#define IMGU_SCALER_TAPS_UV (IMGU_SCALER_FILTER_TAPS / 2)
+#define IMGU_SCALER_FIR_PHASES \
+ (IMGU_SCALER_PHASES << IMGU_SCALER_PHASE_COUNTER_PREC_REF)
+
+/******************* imgu_abi_acc_param *******************/
+
+#define IMGU_ABI_SHD_MAX_PROCESS_LINES 31
+#define IMGU_ABI_SHD_MAX_TRANSFERS 31
+#define IMGU_ABI_SHD_MAX_OPERATIONS \
+ (IMGU_ABI_SHD_MAX_PROCESS_LINES + IMGU_ABI_SHD_MAX_TRANSFERS)
+#define IMGU_ABI_SHD_MAX_CELLS_PER_SET 146
+/* largest grid is 73x56 */
+#define IMGU_ABI_SHD_MAX_CFG_SETS (2 * 28)
+
+#define IMGU_ABI_DVS_STAT_MAX_OPERATIONS 100
+#define IMGU_ABI_DVS_STAT_MAX_PROCESS_LINES 52
+#define IMGU_ABI_DVS_STAT_MAX_TRANSFERS 52
+
+#define IMGU_ABI_BDS_SAMPLE_PATTERN_ARRAY_SIZE 8
+#define IMGU_ABI_BDS_PHASE_COEFFS_ARRAY_SIZE 32
+
+#define IMGU_ABI_AWB_FR_MAX_TRANSFERS 30
+#define IMGU_ABI_AWB_FR_MAX_PROCESS_LINES 30
+#define IMGU_ABI_AWB_FR_MAX_OPERATIONS \
+ (IMGU_ABI_AWB_FR_MAX_TRANSFERS + IMGU_ABI_AWB_FR_MAX_PROCESS_LINES)
+
+#define IMGU_ABI_AF_MAX_TRANSFERS 30
+#define IMGU_ABI_AF_MAX_PROCESS_LINES 30
+#define IMGU_ABI_AF_MAX_OPERATIONS \
+ (IMGU_ABI_AF_MAX_TRANSFERS + IMGU_ABI_AF_MAX_PROCESS_LINES)
+
+#define IMGU_ABI_AWB_MAX_PROCESS_LINES 68
+#define IMGU_ABI_AWB_MAX_TRANSFERS 68
+#define IMGU_ABI_AWB_MAX_OPERATIONS \
+ (IMGU_ABI_AWB_MAX_PROCESS_LINES + IMGU_ABI_AWB_MAX_TRANSFERS)
+
+#define IMGU_ABI_OSYS_PIN_VF 0
+#define IMGU_ABI_OSYS_PIN_OUT 1
+#define IMGU_ABI_OSYS_PINS 2
+
+#define IMGU_ABI_DVS_STAT_LEVELS 3
+#define IMGU_ABI_YUVP2_YTM_LUT_ENTRIES 256
+#define IMGU_ABI_GDC_FRAC_BITS 8
+#define IMGU_ABI_BINARY_MAX_OUTPUT_PORTS 2
+#define IMGU_ABI_MAX_BINARY_NAME 64
+#define IMGU_ABI_ISP_DDR_WORD_BITS 256
+#define IMGU_ABI_ISP_DDR_WORD_BYTES (IMGU_ABI_ISP_DDR_WORD_BITS / 8)
+#define IMGU_ABI_MAX_STAGES 3
+#define IMGU_ABI_MAX_IF_CONFIGS 3
+#define IMGU_ABI_PIPE_CONFIG_ACQUIRE_ISP BIT(31)
+#define IMGU_ABI_PORT_CONFIG_TYPE_INPUT_HOST BIT(0)
+#define IMGU_ABI_PORT_CONFIG_TYPE_OUTPUT_HOST BIT(4)
+#define IMGU_ABI_MAX_SP_THREADS 4
+#define IMGU_ABI_FRAMES_REF 3
+#define IMGU_ABI_FRAMES_TNR 4
+#define IMGU_ABI_BUF_SETS_TNR 1
+
+#define IMGU_ABI_EVENT_BUFFER_ENQUEUED(thread, queue) \
+ (0 << 24 | (thread) << 16 | (queue) << 8)
+#define IMGU_ABI_EVENT_BUFFER_DEQUEUED(queue) (1 << 24 | (queue) << 8)
+#define IMGU_ABI_EVENT_EVENT_DEQUEUED (2 << 24)
+#define IMGU_ABI_EVENT_START_STREAM (3 << 24)
+#define IMGU_ABI_EVENT_STOP_STREAM (4 << 24)
+#define IMGU_ABI_EVENT_MIPI_BUFFERS_READY (5 << 24)
+#define IMGU_ABI_EVENT_UNLOCK_RAW_BUFFER (6 << 24)
+#define IMGU_ABI_EVENT_STAGE_ENABLE_DISABLE (7 << 24)
+
+#define IMGU_ABI_HOST2SP_BUFQ_SIZE 3
+#define IMGU_ABI_SP2HOST_BUFQ_SIZE (2 * IMGU_ABI_MAX_SP_THREADS)
+#define IMGU_ABI_HOST2SP_EVTQ_SIZE (IMGU_ABI_QUEUE_NUM * \
+ IMGU_ABI_MAX_SP_THREADS * 2 + IMGU_ABI_MAX_SP_THREADS * 4)
+#define IMGU_ABI_SP2HOST_EVTQ_SIZE (6 * IMGU_ABI_MAX_SP_THREADS)
+
+#define IMGU_ABI_EVTTYPE_EVENT_SHIFT 0
+#define IMGU_ABI_EVTTYPE_EVENT_MASK (0xff << IMGU_ABI_EVTTYPE_EVENT_SHIFT)
+#define IMGU_ABI_EVTTYPE_PIPE_SHIFT 8
+#define IMGU_ABI_EVTTYPE_PIPE_MASK (0xff << IMGU_ABI_EVTTYPE_PIPE_SHIFT)
+#define IMGU_ABI_EVTTYPE_PIPEID_SHIFT 16
+#define IMGU_ABI_EVTTYPE_PIPEID_MASK (0xff << IMGU_ABI_EVTTYPE_PIPEID_SHIFT)
+#define IMGU_ABI_EVTTYPE_MODULEID_SHIFT 8
+#define IMGU_ABI_EVTTYPE_MODULEID_MASK (0xff << IMGU_ABI_EVTTYPE_MODULEID_SHIFT)
+#define IMGU_ABI_EVTTYPE_LINENO_SHIFT 16
+#define IMGU_ABI_EVTTYPE_LINENO_MASK (0xffff << IMGU_ABI_EVTTYPE_LINENO_SHIFT)
+
+/* Output frame ready */
+#define IMGU_ABI_EVTTYPE_OUT_FRAME_DONE 0
+/* Second output frame ready */
+#define IMGU_ABI_EVTTYPE_2ND_OUT_FRAME_DONE 1
+/* Viewfinder Output frame ready */
+#define IMGU_ABI_EVTTYPE_VF_OUT_FRAME_DONE 2
+/* Second viewfinder Output frame ready */
+#define IMGU_ABI_EVTTYPE_2ND_VF_OUT_FRAME_DONE 3
+/* Indication that 3A statistics are available */
+#define IMGU_ABI_EVTTYPE_3A_STATS_DONE 4
+/* Indication that DIS statistics are available */
+#define IMGU_ABI_EVTTYPE_DIS_STATS_DONE 5
+/* Pipeline Done event, sent after last pipeline stage */
+#define IMGU_ABI_EVTTYPE_PIPELINE_DONE 6
+/* Frame tagged */
+#define IMGU_ABI_EVTTYPE_FRAME_TAGGED 7
+/* Input frame ready */
+#define IMGU_ABI_EVTTYPE_INPUT_FRAME_DONE 8
+/* Metadata ready */
+#define IMGU_ABI_EVTTYPE_METADATA_DONE 9
+/* Indication that LACE statistics are available */
+#define IMGU_ABI_EVTTYPE_LACE_STATS_DONE 10
+/* Extension stage executed */
+#define IMGU_ABI_EVTTYPE_ACC_STAGE_COMPLETE 11
+/* Timing measurement data */
+#define IMGU_ABI_EVTTYPE_TIMER 12
+/* End Of Frame event, sent when in buffered sensor mode */
+#define IMGU_ABI_EVTTYPE_PORT_EOF 13
+/* Performance warning encountered by FW */
+#define IMGU_ABI_EVTTYPE_FW_WARNING 14
+/* Assertion hit by FW */
+#define IMGU_ABI_EVTTYPE_FW_ASSERT 15
+
+#define IMGU_ABI_NUM_CONTINUOUS_FRAMES 10
+#define IMGU_ABI_SP_COMM_COMMAND 0x00
+
+/*
+ * The host2sp_cmd_ready command is the only command written by the SP
+ * It acknowledges that is previous command has been received.
+ * (this does not mean that the command has been executed)
+ * It also indicates that a new command can be send (it is a queue
+ * with depth 1).
+ */
+#define IMGU_ABI_SP_COMM_COMMAND_READY 1
+/* Command written by the Host */
+#define IMGU_ABI_SP_COMM_COMMAND_DUMMY 2 /* No action */
+#define IMGU_ABI_SP_COMM_COMMAND_START_FLASH 3 /* Start the flash */
+#define IMGU_ABI_SP_COMM_COMMAND_TERMINATE 4 /* Terminate */
+
+/* n = 0..IPU3_CSS_PIPE_ID_NUM-1 */
+#define IMGU_ABI_SP_COMM_EVENT_IRQ_MASK(n) ((n) * 4 + 0x60)
+#define IMGU_ABI_SP_COMM_EVENT_IRQ_MASK_OR_SHIFT 0
+#define IMGU_ABI_SP_COMM_EVENT_IRQ_MASK_AND_SHIFT 16
+
+#define IMGU_ABI_BL_DMACMD_TYPE_SP_PMEM 1 /* sp_pmem */
+
+/***** For parameter computation *****/
+
+#define IMGU_HIVE_OF_SYS_SCALER_TO_FA_OFFSET 0xC
+#define IMGU_HIVE_OF_SYS_OF_TO_FA_OFFSET 0x8
+#define IMGU_HIVE_OF_SYS_OF_SYSTEM_NWAYS 32
+
+#define IMGU_SCALER_ELEMS_PER_VEC 0x10
+#define IMGU_SCALER_FILTER_TAPS_Y 0x4
+#define IMGU_SCALER_OUT_BPP 0x8
+
+#define IMGU_SCALER_MS_TO_OUTFORMACC_SL_ADDR 0x400
+#define IMGU_SCALER_TO_OF_ACK_FA_ADDR \
+ (0xC00 + IMGU_HIVE_OF_SYS_SCALER_TO_FA_OFFSET)
+#define IMGU_OF_TO_ACK_FA_ADDR (0xC00 + IMGU_HIVE_OF_SYS_OF_TO_FA_OFFSET)
+#define IMGU_OUTFORMACC_MS_TO_SCALER_SL_ADDR 0
+#define IMGU_SCALER_INTR_BPP 10
+
+#define IMGU_PS_SNR_PRESERVE_BITS 3
+#define IMGU_CNTX_BPP 11
+#define IMGU_SCALER_FILTER_TAPS_UV (IMGU_SCALER_FILTER_TAPS_Y / 2)
+
+#define IMGU_VMEM2_ELEMS_PER_VEC (IMGU_SCALER_ELEMS_PER_VEC)
+#define IMGU_STRIDE_Y (IMGU_SCALER_FILTER_TAPS_Y + 1)
+#define IMGU_MAX_FRAME_WIDTH 3840
+#define IMGU_VMEM3_ELEMS_PER_VEC (IMGU_SCALER_ELEMS_PER_VEC)
+
+#define IMGU_VER_CNTX_WORDS DIV_ROUND_UP((IMGU_SCALER_OUT_BPP + \
+ IMGU_PS_SNR_PRESERVE_BITS), IMGU_CNTX_BPP) /* 1 */
+#define IMGU_MAX_INPUT_BLOCK_HEIGHT 64
+#define IMGU_HOR_CNTX_WORDS DIV_ROUND_UP((IMGU_SCALER_INTR_BPP + \
+ IMGU_PS_SNR_PRESERVE_BITS), IMGU_CNTX_BPP) /* 2 */
+#define IMGU_MAX_OUTPUT_BLOCK_WIDTH 128
+#define IMGU_CNTX_STRIDE_UV (IMGU_SCALER_FILTER_TAPS_UV + 1)
+
+#define IMGU_OSYS_DMA_CROP_W_LIMIT 64
+#define IMGU_OSYS_DMA_CROP_H_LIMIT 4
+#define IMGU_OSYS_BLOCK_WIDTH (2 * IPU3_UAPI_ISP_VEC_ELEMS)
+#define IMGU_OSYS_BLOCK_HEIGHT 32
+#define IMGU_OSYS_PHASES 0x20
+#define IMGU_OSYS_FILTER_TAPS 0x4
+#define IMGU_OSYS_PHASE_COUNTER_PREC_REF 6
+#define IMGU_OSYS_NUM_INPUT_BUFFERS 2
+#define IMGU_OSYS_FIR_PHASES \
+ (IMGU_OSYS_PHASES << IMGU_OSYS_PHASE_COUNTER_PREC_REF)
+#define IMGU_OSYS_TAPS_UV (IMGU_OSYS_FILTER_TAPS / 2)
+#define IMGU_OSYS_TAPS_Y (IMGU_OSYS_FILTER_TAPS)
+#define IMGU_OSYS_NUM_INTERM_BUFFERS 2
+
+#define IMGU_VMEM1_Y_SIZE \
+ (IMGU_OSYS_BLOCK_HEIGHT * IMGU_VMEM1_Y_STRIDE)
+#define IMGU_VMEM1_UV_SIZE (IMGU_VMEM1_Y_SIZE / 4)
+#define IMGU_VMEM1_OUT_BUF_ADDR (IMGU_VMEM1_INP_BUF_ADDR + \
+ (IMGU_OSYS_NUM_INPUT_BUFFERS * IMGU_VMEM1_BUF_SIZE))
+#define IMGU_OSYS_NUM_OUTPUT_BUFFERS 2
+
+/* transpose of input height */
+#define IMGU_VMEM2_VECS_PER_LINE \
+ (DIV_ROUND_UP(IMGU_OSYS_BLOCK_HEIGHT, IMGU_VMEM2_ELEMS_PER_VEC))
+/* size in words (vectors) */
+#define IMGU_VMEM2_BUF_SIZE \
+ (IMGU_VMEM2_VECS_PER_LINE * IMGU_VMEM2_LINES_PER_BLOCK)
+#define IMGU_VMEM3_VER_Y_SIZE \
+ ((IMGU_STRIDE_Y * IMGU_MAX_FRAME_WIDTH \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_VER_CNTX_WORDS)
+#define IMGU_VMEM3_HOR_Y_SIZE \
+ ((IMGU_STRIDE_Y * IMGU_MAX_INPUT_BLOCK_HEIGHT \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_HOR_CNTX_WORDS)
+#define IMGU_VMEM3_VER_Y_EXTRA \
+ ((IMGU_STRIDE_Y * IMGU_MAX_OUTPUT_BLOCK_WIDTH \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_VER_CNTX_WORDS)
+#define IMGU_VMEM3_VER_U_SIZE \
+ (((IMGU_CNTX_STRIDE_UV * IMGU_MAX_FRAME_WIDTH \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_VER_CNTX_WORDS) / 2)
+#define IMGU_VMEM3_HOR_U_SIZE \
+ (((IMGU_STRIDE_Y * IMGU_MAX_INPUT_BLOCK_HEIGHT \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_HOR_CNTX_WORDS) / 2)
+#define IMGU_VMEM3_VER_U_EXTRA \
+ (((IMGU_CNTX_STRIDE_UV * IMGU_MAX_OUTPUT_BLOCK_WIDTH \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_VER_CNTX_WORDS) / 2)
+#define IMGU_VMEM3_VER_V_SIZE \
+ (((IMGU_CNTX_STRIDE_UV * IMGU_MAX_FRAME_WIDTH \
+ / IMGU_VMEM3_ELEMS_PER_VEC) * IMGU_VER_CNTX_WORDS) / 2)
+
+#define IMGU_ISP_VEC_NELEMS 64
+#define IMGU_LUMA_TO_CHROMA_RATIO 2
+#define IMGU_INPUT_BLOCK_WIDTH (128)
+#define IMGU_FIFO_ADDR_SCALER_TO_FMT \
+ (IMGU_SCALER_MS_TO_OUTFORMACC_SL_ADDR >> 2)
+#define IMGU_FIFO_ADDR_SCALER_TO_SP (IMGU_SCALER_TO_OF_ACK_FA_ADDR >> 2)
+#define IMGU_VMEM1_INP_BUF_ADDR 0
+#define IMGU_VMEM1_Y_STRIDE \
+ (IMGU_OSYS_BLOCK_WIDTH / IMGU_VMEM1_ELEMS_PER_VEC)
+#define IMGU_VMEM1_BUF_SIZE (IMGU_VMEM1_V_OFFSET + IMGU_VMEM1_UV_SIZE)
+
+#define IMGU_VMEM1_U_OFFSET (IMGU_VMEM1_Y_SIZE)
+#define IMGU_VMEM1_V_OFFSET (IMGU_VMEM1_U_OFFSET + IMGU_VMEM1_UV_SIZE)
+#define IMGU_VMEM1_UV_STRIDE (IMGU_VMEM1_Y_STRIDE / 2)
+#define IMGU_VMEM1_INT_BUF_ADDR (IMGU_VMEM1_OUT_BUF_ADDR + \
+ (IMGU_OSYS_NUM_OUTPUT_BUFFERS * IMGU_VMEM1_BUF_SIZE))
+
+#define IMGU_VMEM1_ELEMS_PER_VEC (IMGU_HIVE_OF_SYS_OF_SYSTEM_NWAYS)
+#define IMGU_VMEM2_BUF_Y_ADDR 0
+#define IMGU_VMEM2_BUF_Y_STRIDE (IMGU_VMEM2_VECS_PER_LINE)
+#define IMGU_VMEM2_BUF_U_ADDR \
+ (IMGU_VMEM2_BUF_Y_ADDR + IMGU_VMEM2_BUF_SIZE)
+#define IMGU_VMEM2_BUF_V_ADDR \
+ (IMGU_VMEM2_BUF_U_ADDR + IMGU_VMEM2_BUF_SIZE / 4)
+#define IMGU_VMEM2_BUF_UV_STRIDE (IMGU_VMEM2_VECS_PER_LINE / 2)
+/* 1.5 x depth of intermediate buffer */
+#define IMGU_VMEM2_LINES_PER_BLOCK 192
+#define IMGU_VMEM3_HOR_Y_ADDR \
+ (IMGU_VMEM3_VER_Y_ADDR + IMGU_VMEM3_VER_Y_SIZE)
+#define IMGU_VMEM3_HOR_U_ADDR \
+ (IMGU_VMEM3_VER_U_ADDR + IMGU_VMEM3_VER_U_SIZE)
+#define IMGU_VMEM3_HOR_V_ADDR \
+ (IMGU_VMEM3_VER_V_ADDR + IMGU_VMEM3_VER_V_SIZE)
+#define IMGU_VMEM3_VER_Y_ADDR 0
+#define IMGU_VMEM3_VER_U_ADDR \
+ (IMGU_VMEM3_VER_Y_ADDR + IMGU_VMEM3_VER_Y_SIZE + \
+ max(IMGU_VMEM3_HOR_Y_SIZE, IMGU_VMEM3_VER_Y_EXTRA))
+#define IMGU_VMEM3_VER_V_ADDR \
+ (IMGU_VMEM3_VER_U_ADDR + IMGU_VMEM3_VER_U_SIZE + \
+ max(IMGU_VMEM3_HOR_U_SIZE, IMGU_VMEM3_VER_U_EXTRA))
+#define IMGU_FIFO_ADDR_FMT_TO_SP (IMGU_OF_TO_ACK_FA_ADDR >> 2)
+#define IMGU_FIFO_ADDR_FMT_TO_SCALER (IMGU_OUTFORMACC_MS_TO_SCALER_SL_ADDR >> 2)
+#define IMGU_VMEM1_HST_BUF_ADDR (IMGU_VMEM1_INT_BUF_ADDR + \
+ (IMGU_OSYS_NUM_INTERM_BUFFERS * IMGU_VMEM1_BUF_SIZE))
+#define IMGU_VMEM1_HST_BUF_STRIDE 120
+#define IMGU_VMEM1_HST_BUF_NLINES 3
+
+enum imgu_abi_frame_format {
+ IMGU_ABI_FRAME_FORMAT_NV11, /* 12 bit YUV 411, Y, UV plane */
+ IMGU_ABI_FRAME_FORMAT_NV12, /* 12 bit YUV 420, Y, UV plane */
+ IMGU_ABI_FRAME_FORMAT_NV12_16, /* 16 bit YUV 420, Y, UV plane */
+ IMGU_ABI_FRAME_FORMAT_NV12_TILEY,/* 12 bit YUV 420,Intel tiled format */
+ IMGU_ABI_FRAME_FORMAT_NV16, /* 16 bit YUV 422, Y, UV plane */
+ IMGU_ABI_FRAME_FORMAT_NV21, /* 12 bit YUV 420, Y, VU plane */
+ IMGU_ABI_FRAME_FORMAT_NV61, /* 16 bit YUV 422, Y, VU plane */
+ IMGU_ABI_FRAME_FORMAT_YV12, /* 12 bit YUV 420, Y, V, U plane */
+ IMGU_ABI_FRAME_FORMAT_YV16, /* 16 bit YUV 422, Y, V, U plane */
+ IMGU_ABI_FRAME_FORMAT_YUV420, /* 12 bit YUV 420, Y, U, V plane */
+ IMGU_ABI_FRAME_FORMAT_YUV420_16,/* yuv420, 16 bits per subpixel */
+ IMGU_ABI_FRAME_FORMAT_YUV422, /* 16 bit YUV 422, Y, U, V plane */
+ IMGU_ABI_FRAME_FORMAT_YUV422_16,/* yuv422, 16 bits per subpixel */
+ IMGU_ABI_FRAME_FORMAT_UYVY, /* 16 bit YUV 422, UYVY interleaved */
+ IMGU_ABI_FRAME_FORMAT_YUYV, /* 16 bit YUV 422, YUYV interleaved */
+ IMGU_ABI_FRAME_FORMAT_YUV444, /* 24 bit YUV 444, Y, U, V plane */
+ IMGU_ABI_FRAME_FORMAT_YUV_LINE, /* Internal format, 2 y lines */
+ /* followed by a uv-interleaved line */
+ IMGU_ABI_FRAME_FORMAT_RAW, /* RAW, 1 plane */
+ IMGU_ABI_FRAME_FORMAT_RGB565, /* 16 bit RGB, 1 plane. Each 3 sub
+ * pixels are packed into one 16 bit
+ * value, 5 bits for R, 6 bits for G
+ * and 5 bits for B.
+ */
+ IMGU_ABI_FRAME_FORMAT_PLANAR_RGB888, /* 24 bit RGB, 3 planes */
+ IMGU_ABI_FRAME_FORMAT_RGBA888, /* 32 bit RGBA, 1 plane, A=Alpha
+ * (alpha is unused)
+ */
+ IMGU_ABI_FRAME_FORMAT_QPLANE6, /* Internal, for advanced ISP */
+ IMGU_ABI_FRAME_FORMAT_BINARY_8, /* byte stream, used for jpeg. For
+ * frames of this type, we set the
+ * height to 1 and the width to the
+ * number of allocated bytes.
+ */
+ IMGU_ABI_FRAME_FORMAT_MIPI, /* MIPI frame, 1 plane */
+ IMGU_ABI_FRAME_FORMAT_RAW_PACKED, /* RAW, 1 plane, packed */
+ IMGU_ABI_FRAME_FORMAT_CSI_MIPI_YUV420_8, /* 8 bit per Y/U/V. Y odd line
+ * UYVY interleaved even line
+ */
+ IMGU_ABI_FRAME_FORMAT_CSI_MIPI_LEGACY_YUV420_8, /* Legacy YUV420.
+ * UY odd line;
+ * VY even line
+ */
+ IMGU_ABI_FRAME_FORMAT_CSI_MIPI_YUV420_10,/* 10 bit per Y/U/V. Y odd
+ * line; UYVY interleaved
+ * even line
+ */
+ IMGU_ABI_FRAME_FORMAT_YCGCO444_16, /* Internal format for ISP2.7,
+ * 16 bits per plane YUV 444,
+ * Y, U, V plane
+ */
+ IMGU_ABI_FRAME_FORMAT_NUM
+};
+
+enum imgu_abi_bayer_order {
+ IMGU_ABI_BAYER_ORDER_GRBG,
+ IMGU_ABI_BAYER_ORDER_RGGB,
+ IMGU_ABI_BAYER_ORDER_BGGR,
+ IMGU_ABI_BAYER_ORDER_GBRG
+};
+
+enum imgu_abi_osys_format {
+ IMGU_ABI_OSYS_FORMAT_YUV420,
+ IMGU_ABI_OSYS_FORMAT_YV12,
+ IMGU_ABI_OSYS_FORMAT_NV12,
+ IMGU_ABI_OSYS_FORMAT_NV21,
+ IMGU_ABI_OSYS_FORMAT_YUV_LINE,
+ IMGU_ABI_OSYS_FORMAT_YUY2, /* = IMGU_ABI_OSYS_FORMAT_YUYV */
+ IMGU_ABI_OSYS_FORMAT_NV16,
+ IMGU_ABI_OSYS_FORMAT_RGBA,
+ IMGU_ABI_OSYS_FORMAT_BGRA
+};
+
+enum imgu_abi_osys_tiling {
+ IMGU_ABI_OSYS_TILING_NONE,
+ IMGU_ABI_OSYS_TILING_Y,
+ IMGU_ABI_OSYS_TILING_YF,
+};
+
+enum imgu_abi_osys_procmode {
+ IMGU_ABI_OSYS_PROCMODE_BYPASS,
+ IMGU_ABI_OSYS_PROCMODE_UPSCALE,
+ IMGU_ABI_OSYS_PROCMODE_DOWNSCALE,
+};
+
+enum imgu_abi_queue_id {
+ IMGU_ABI_QUEUE_EVENT_ID = -1,
+ IMGU_ABI_QUEUE_A_ID = 0,
+ IMGU_ABI_QUEUE_B_ID,
+ IMGU_ABI_QUEUE_C_ID,
+ IMGU_ABI_QUEUE_D_ID,
+ IMGU_ABI_QUEUE_E_ID,
+ IMGU_ABI_QUEUE_F_ID,
+ IMGU_ABI_QUEUE_G_ID,
+ IMGU_ABI_QUEUE_H_ID, /* input frame queue for skycam */
+ IMGU_ABI_QUEUE_NUM
+};
+
+enum imgu_abi_buffer_type {
+ IMGU_ABI_BUFFER_TYPE_INVALID = -1,
+ IMGU_ABI_BUFFER_TYPE_3A_STATISTICS = 0,
+ IMGU_ABI_BUFFER_TYPE_DIS_STATISTICS,
+ IMGU_ABI_BUFFER_TYPE_LACE_STATISTICS,
+ IMGU_ABI_BUFFER_TYPE_INPUT_FRAME,
+ IMGU_ABI_BUFFER_TYPE_OUTPUT_FRAME,
+ IMGU_ABI_BUFFER_TYPE_SEC_OUTPUT_FRAME,
+ IMGU_ABI_BUFFER_TYPE_VF_OUTPUT_FRAME,
+ IMGU_ABI_BUFFER_TYPE_SEC_VF_OUTPUT_FRAME,
+ IMGU_ABI_BUFFER_TYPE_RAW_OUTPUT_FRAME,
+ IMGU_ABI_BUFFER_TYPE_CUSTOM_INPUT,
+ IMGU_ABI_BUFFER_TYPE_CUSTOM_OUTPUT,
+ IMGU_ABI_BUFFER_TYPE_METADATA,
+ IMGU_ABI_BUFFER_TYPE_PARAMETER_SET,
+ IMGU_ABI_BUFFER_TYPE_PER_FRAME_PARAMETER_SET,
+ IMGU_ABI_NUM_DYNAMIC_BUFFER_TYPE,
+ IMGU_ABI_NUM_BUFFER_TYPE
+};
+
+enum imgu_abi_raw_type {
+ IMGU_ABI_RAW_TYPE_BAYER,
+ IMGU_ABI_RAW_TYPE_IR_ON_GR,
+ IMGU_ABI_RAW_TYPE_IR_ON_GB
+};
+
+enum imgu_abi_memories {
+ IMGU_ABI_MEM_ISP_PMEM0 = 0,
+ IMGU_ABI_MEM_ISP_DMEM0,
+ IMGU_ABI_MEM_ISP_VMEM0,
+ IMGU_ABI_MEM_ISP_VAMEM0,
+ IMGU_ABI_MEM_ISP_VAMEM1,
+ IMGU_ABI_MEM_ISP_VAMEM2,
+ IMGU_ABI_MEM_ISP_HMEM0,
+ IMGU_ABI_MEM_SP0_DMEM0,
+ IMGU_ABI_MEM_SP1_DMEM0,
+ IMGU_ABI_MEM_DDR,
+ IMGU_ABI_NUM_MEMORIES
+};
+
+enum imgu_abi_param_class {
+ IMGU_ABI_PARAM_CLASS_PARAM, /* Late binding parameters, like 3A */
+ IMGU_ABI_PARAM_CLASS_CONFIG, /* Pipe config time parameters */
+ IMGU_ABI_PARAM_CLASS_STATE, /* State parameters, eg. buffer index */
+ IMGU_ABI_PARAM_CLASS_NUM
+};
+
+enum imgu_abi_bin_input_src {
+ IMGU_ABI_BINARY_INPUT_SOURCE_SENSOR,
+ IMGU_ABI_BINARY_INPUT_SOURCE_MEMORY,
+ IMGU_ABI_BINARY_INPUT_SOURCE_VARIABLE,
+};
+
+enum imgu_abi_sp_swstate {
+ IMGU_ABI_SP_SWSTATE_TERMINATED,
+ IMGU_ABI_SP_SWSTATE_INITIALIZED,
+ IMGU_ABI_SP_SWSTATE_CONNECTED,
+ IMGU_ABI_SP_SWSTATE_RUNNING,
+};
+
+enum imgu_abi_bl_swstate {
+ IMGU_ABI_BL_SWSTATE_OK = 0x100,
+ IMGU_ABI_BL_SWSTATE_BUSY,
+ IMGU_ABI_BL_SWSTATE_ERR,
+};
+
+/* The type of pipe stage */
+enum imgu_abi_stage_type {
+ IMGU_ABI_STAGE_TYPE_SP,
+ IMGU_ABI_STAGE_TYPE_ISP,
+};
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* [PATCH v7 05/16] intel-ipu3: abi: Add structs
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (3 preceding siblings ...)
2018-10-29 22:22 ` [PATCH v7 04/16] intel-ipu3: abi: Add register definitions and enum Yong Zhi
@ 2018-10-29 22:22 ` Yong Zhi
2018-11-05 8:27 ` Sakari Ailus
2018-10-29 22:23 ` [PATCH v7 06/16] intel-ipu3: mmu: Implement driver Yong Zhi
` (12 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:22 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
This add all the structs of IPU3 firmware ABI.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-abi.h | 1350 +++++++++++++++++++++++++++++++
1 file changed, 1350 insertions(+)
diff --git a/drivers/media/pci/intel/ipu3/ipu3-abi.h b/drivers/media/pci/intel/ipu3/ipu3-abi.h
index ac08ad3..21703da 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-abi.h
+++ b/drivers/media/pci/intel/ipu3/ipu3-abi.h
@@ -658,4 +658,1354 @@ enum imgu_abi_stage_type {
IMGU_ABI_STAGE_TYPE_ISP,
};
+struct imgu_abi_acc_operation {
+ /*
+ * zero means on init,
+ * others mean upon receiving an ack signal from the BC acc.
+ */
+ u8 op_indicator;
+ u8 op_type;
+} __packed;
+
+struct imgu_abi_acc_process_lines_cmd_data {
+ u16 lines;
+ u8 cfg_set;
+ u8 __reserved; /* Align to 4 bytes */
+} __packed;
+
+/* Bayer shading definitions */
+
+struct imgu_abi_shd_transfer_luts_set_data {
+ u8 set_number;
+ u8 padding[3];
+ imgu_addr_t rg_lut_ddr_addr;
+ imgu_addr_t bg_lut_ddr_addr;
+ u32 align_dummy;
+} __packed;
+
+struct imgu_abi_shd_grid_config {
+ /* reg 0 */
+ u32 grid_width:8;
+ u32 grid_height:8;
+ u32 block_width:3;
+ u32 __reserved0:1;
+ u32 block_height:3;
+ u32 __reserved1:1;
+ u32 grid_height_per_slice:8;
+ /* reg 1 */
+ s32 x_start:13;
+ s32 __reserved2:3;
+ s32 y_start:13;
+ s32 __reserved3:3;
+} __packed;
+
+struct imgu_abi_shd_general_config {
+ u32 init_set_vrt_offst_ul:8;
+ u32 shd_enable:1;
+ /* aka 'gf' */
+ u32 gain_factor:2;
+ u32 __reserved:21;
+} __packed;
+
+struct imgu_abi_shd_black_level_config {
+ /* reg 0 */
+ s32 bl_r:12;
+ s32 __reserved0:4;
+ s32 bl_gr:12;
+ u32 __reserved1:1;
+ /* aka 'nf' */
+ u32 normalization_shift:3;
+ /* reg 1 */
+ s32 bl_gb:12;
+ s32 __reserved2:4;
+ s32 bl_b:12;
+ s32 __reserved3:4;
+} __packed;
+
+struct imgu_abi_shd_intra_frame_operations_data {
+ struct imgu_abi_acc_operation
+ operation_list[IMGU_ABI_SHD_MAX_OPERATIONS] __attribute__((aligned(32)));
+ struct imgu_abi_acc_process_lines_cmd_data
+ process_lines_data[IMGU_ABI_SHD_MAX_PROCESS_LINES] __attribute__((aligned(32)));
+ struct imgu_abi_shd_transfer_luts_set_data
+ transfer_data[IMGU_ABI_SHD_MAX_TRANSFERS] __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_shd_config {
+ struct ipu3_uapi_shd_config_static shd __attribute__((aligned(32)));
+ struct imgu_abi_shd_intra_frame_operations_data shd_ops __attribute__((aligned(32)));
+ struct ipu3_uapi_shd_lut shd_lut __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_stripe_input_frame_resolution {
+ u16 width;
+ u16 height;
+ u32 bayer_order; /* enum ipu3_uapi_bayer_order */
+ u32 raw_bit_depth;
+} __packed;
+
+/* Stripe-based processing */
+
+struct imgu_abi_stripes {
+ /* offset from start of frame - measured in pixels */
+ u16 offset;
+ /* stripe width - measured in pixels */
+ u16 width;
+ /* stripe width - measured in pixels */
+ u16 height;
+} __packed;
+
+struct imgu_abi_stripe_data {
+ /*
+ * number of stripes for current processing source
+ * - VLIW binary parameter we currently support 1 or 2 stripes
+ */
+ u16 num_of_stripes;
+
+ u8 padding[2];
+
+ /*
+ * the following data is derived from resolution-related
+ * pipe config and from num_of_stripes
+ */
+
+ /*
+ *'input-stripes' - before input cropping
+ * used by input feeder
+ */
+ struct imgu_abi_stripe_input_frame_resolution input_frame;
+
+ /*'effective-stripes' - after input cropping used dpc, bds */
+ struct imgu_abi_stripes effective_stripes[IPU3_UAPI_MAX_STRIPES];
+
+ /* 'down-scaled-stripes' - after down-scaling ONLY. used by BDS */
+ struct imgu_abi_stripes down_scaled_stripes[IPU3_UAPI_MAX_STRIPES];
+
+ /*
+ *'bds-out-stripes' - after bayer down-scaling and padding.
+ * used by all algos starting with norm up to the ref-frame for GDC
+ * (currently up to the output kernel)
+ */
+ struct imgu_abi_stripes bds_out_stripes[IPU3_UAPI_MAX_STRIPES];
+
+ /* 'bds-out-stripes (no overlap)' - used for ref kernel */
+ struct imgu_abi_stripes
+ bds_out_stripes_no_overlap[IPU3_UAPI_MAX_STRIPES];
+
+ /*
+ * input resolution for output system (equal to bds_out - envelope)
+ * output-system input frame width as configured by user
+ */
+ u16 output_system_in_frame_width;
+ /* output-system input frame height as configured by user */
+ u16 output_system_in_frame_height;
+
+ /*
+ * 'output-stripes' - accounts for stiching on the output (no overlap)
+ * used by the output kernel
+ */
+ struct imgu_abi_stripes output_stripes[IPU3_UAPI_MAX_STRIPES];
+
+ /*
+ * 'block-stripes' - accounts for stiching by the output system
+ * (1 or more blocks overlap)
+ * used by DVS, TNR and the output system kernel
+ */
+ struct imgu_abi_stripes block_stripes[IPU3_UAPI_MAX_STRIPES];
+
+ u16 effective_frame_width; /* Needed for vertical cropping */
+ u16 bds_frame_width;
+ u16 out_frame_width; /* Output frame width as configured by user */
+ u16 out_frame_height; /* Output frame height as configured by user */
+
+ /* GDC in buffer (A.K.A delay frame,ref buffer) info */
+ u16 gdc_in_buffer_width; /* GDC in buffer width */
+ u16 gdc_in_buffer_height; /* GDC in buffer height */
+ /* GDC in buffer first valid pixel x offset */
+ u16 gdc_in_buffer_offset_x;
+ /* GDC in buffer first valid pixel y offset */
+ u16 gdc_in_buffer_offset_y;
+
+ /* Display frame width as configured by user */
+ u16 display_frame_width;
+ /* Display frame height as configured by user */
+ u16 display_frame_height;
+ u16 bds_aligned_frame_width;
+ /* Number of vectors to left-crop when writing stripes (not stripe 0) */
+ u16 half_overlap_vectors;
+ /* Decimate ISP and fixed func resolutions after BDS (ir_extraction) */
+ u16 ir_ext_decimation;
+ u8 padding1[2];
+} __packed;
+
+/* Input feeder related structs */
+
+struct imgu_abi_input_feeder_data {
+ u32 row_stride; /* row stride */
+ u32 start_row_address; /* start row address */
+ u32 start_pixel; /* start pixel */
+} __packed;
+
+struct imgu_abi_input_feeder_data_aligned {
+ struct imgu_abi_input_feeder_data data __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_input_feeder_data_per_stripe {
+ struct imgu_abi_input_feeder_data_aligned
+ input_feeder_data[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+struct imgu_abi_input_feeder_config {
+ struct imgu_abi_input_feeder_data data;
+ struct imgu_abi_input_feeder_data_per_stripe data_per_stripe
+ __attribute__((aligned(32)));
+} __packed;
+
+/* DVS related definitions */
+
+struct imgu_abi_dvs_stat_grd_config {
+ u8 grid_width;
+ u8 grid_height;
+ u8 block_width;
+ u8 block_height;
+ u16 x_start;
+ u16 y_start;
+ u16 enable;
+ u16 x_end;
+ u16 y_end;
+} __packed;
+
+struct imgu_abi_dvs_stat_cfg {
+ u8 __reserved0[4];
+ struct imgu_abi_dvs_stat_grd_config
+ grd_config[IMGU_ABI_DVS_STAT_LEVELS];
+ u8 __reserved1[18];
+} __packed;
+
+struct imgu_abi_dvs_stat_transfer_op_data {
+ u8 set_number;
+} __packed;
+
+struct imgu_abi_dvs_stat_intra_frame_operations_data {
+ struct imgu_abi_acc_operation
+ ops[IMGU_ABI_DVS_STAT_MAX_OPERATIONS] __attribute__((aligned(32)));
+ struct imgu_abi_acc_process_lines_cmd_data
+ process_lines_data[IMGU_ABI_DVS_STAT_MAX_PROCESS_LINES]
+ __attribute__((aligned(32)));
+ struct imgu_abi_dvs_stat_transfer_op_data
+ transfer_data[IMGU_ABI_DVS_STAT_MAX_TRANSFERS] __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_dvs_stat_config {
+ struct imgu_abi_dvs_stat_cfg cfg __attribute__((aligned(32)));
+ u8 __reserved0[128];
+ struct imgu_abi_dvs_stat_intra_frame_operations_data operations_data;
+ u8 __reserved1[64];
+} __packed;
+
+/* Y-tone Mapping */
+
+struct imgu_abi_yuvp2_y_tm_lut_static_config {
+ u16 entries[IMGU_ABI_YUVP2_YTM_LUT_ENTRIES];
+ u32 enable;
+} __packed;
+
+/* Output formatter related structs */
+
+struct imgu_abi_osys_formatter_params {
+ u32 format;
+ u32 flip;
+ u32 mirror;
+ u32 tiling;
+ u32 reduce_range;
+ u32 alpha_blending;
+ u32 release_inp_addr;
+ u32 release_inp_en;
+ u32 process_out_buf_addr;
+ u32 image_width_vecs;
+ u32 image_height_lines;
+ u32 inp_buff_y_st_addr;
+ u32 inp_buff_y_line_stride;
+ u32 inp_buff_y_buffer_stride;
+ u32 int_buff_u_st_addr;
+ u32 int_buff_v_st_addr;
+ u32 inp_buff_uv_line_stride;
+ u32 inp_buff_uv_buffer_stride;
+ u32 out_buff_level;
+ u32 out_buff_nr_y_lines;
+ u32 out_buff_u_st_offset;
+ u32 out_buff_v_st_offset;
+ u32 out_buff_y_line_stride;
+ u32 out_buff_uv_line_stride;
+ u32 hist_buff_st_addr;
+ u32 hist_buff_line_stride;
+ u32 hist_buff_nr_lines;
+} __packed;
+
+struct imgu_abi_osys_formatter {
+ struct imgu_abi_osys_formatter_params param __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_osys_scaler_params {
+ u32 inp_buf_y_st_addr;
+ u32 inp_buf_y_line_stride;
+ u32 inp_buf_y_buffer_stride;
+ u32 inp_buf_u_st_addr;
+ u32 inp_buf_v_st_addr;
+ u32 inp_buf_uv_line_stride;
+ u32 inp_buf_uv_buffer_stride;
+ u32 inp_buf_chunk_width;
+ u32 inp_buf_nr_buffers;
+ /* Output buffers */
+ u32 out_buf_y_st_addr;
+ u32 out_buf_y_line_stride;
+ u32 out_buf_y_buffer_stride;
+ u32 out_buf_u_st_addr;
+ u32 out_buf_v_st_addr;
+ u32 out_buf_uv_line_stride;
+ u32 out_buf_uv_buffer_stride;
+ u32 out_buf_nr_buffers;
+ /* Intermediate buffers */
+ u32 int_buf_y_st_addr;
+ u32 int_buf_y_line_stride;
+ u32 int_buf_u_st_addr;
+ u32 int_buf_v_st_addr;
+ u32 int_buf_uv_line_stride;
+ u32 int_buf_height;
+ u32 int_buf_chunk_width;
+ u32 int_buf_chunk_height;
+ /* Context buffers */
+ u32 ctx_buf_hor_y_st_addr;
+ u32 ctx_buf_hor_u_st_addr;
+ u32 ctx_buf_hor_v_st_addr;
+ u32 ctx_buf_ver_y_st_addr;
+ u32 ctx_buf_ver_u_st_addr;
+ u32 ctx_buf_ver_v_st_addr;
+ /* Addresses for release-input and process-output tokens */
+ u32 release_inp_buf_addr;
+ u32 release_inp_buf_en;
+ u32 release_out_buf_en;
+ u32 process_out_buf_addr;
+ /* Settings dimensions, padding, cropping */
+ u32 input_image_y_width;
+ u32 input_image_y_height;
+ u32 input_image_y_start_column;
+ u32 input_image_uv_start_column;
+ u32 input_image_y_left_pad;
+ u32 input_image_uv_left_pad;
+ u32 input_image_y_right_pad;
+ u32 input_image_uv_right_pad;
+ u32 input_image_y_top_pad;
+ u32 input_image_uv_top_pad;
+ u32 input_image_y_bottom_pad;
+ u32 input_image_uv_bottom_pad;
+ u32 processing_mode; /* enum imgu_abi_osys_procmode */
+ u32 scaling_ratio;
+ u32 y_left_phase_init;
+ u32 uv_left_phase_init;
+ u32 y_top_phase_init;
+ u32 uv_top_phase_init;
+ u32 coeffs_exp_shift;
+ u32 out_y_left_crop;
+ u32 out_uv_left_crop;
+ u32 out_y_top_crop;
+ u32 out_uv_top_crop;
+} __packed;
+
+struct imgu_abi_osys_scaler {
+ struct imgu_abi_osys_scaler_params param __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_osys_frame_params {
+ /* Output pins */
+ u32 enable;
+ u32 format; /* enum imgu_abi_osys_format */
+ u32 flip;
+ u32 mirror;
+ u32 tiling; /* enum imgu_abi_osys_tiling */
+ u32 width;
+ u32 height;
+ u32 stride;
+ u32 scaled;
+} __packed;
+
+struct imgu_abi_osys_frame {
+ struct imgu_abi_osys_frame_params param __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_osys_stripe {
+ /* Input resolution */
+ u32 input_width;
+ u32 input_height;
+ /* Output Stripe */
+ u32 output_width[IMGU_ABI_OSYS_PINS];
+ u32 output_height[IMGU_ABI_OSYS_PINS];
+ u32 output_offset[IMGU_ABI_OSYS_PINS];
+ u32 buf_stride[IMGU_ABI_OSYS_PINS];
+ /* Scaler params */
+ u32 block_width;
+ u32 block_height;
+ /* Output Crop factor */
+ u32 crop_top[IMGU_ABI_OSYS_PINS];
+ u32 crop_left[IMGU_ABI_OSYS_PINS];
+} __packed;
+
+struct imgu_abi_osys_config {
+ struct imgu_abi_osys_formatter
+ formatter[IPU3_UAPI_MAX_STRIPES][IMGU_ABI_OSYS_PINS];
+ struct imgu_abi_osys_scaler scaler[IPU3_UAPI_MAX_STRIPES];
+ struct imgu_abi_osys_frame frame[IMGU_ABI_OSYS_PINS];
+ struct imgu_abi_osys_stripe stripe[IPU3_UAPI_MAX_STRIPES];
+ /* 32 packed coefficients for luma and chroma */
+ s8 scaler_coeffs_chroma[128];
+ s8 scaler_coeffs_luma[128];
+} __packed;
+
+/* BDS */
+
+struct imgu_abi_bds_hor_ctrl0 {
+ u32 sample_patrn_length:9;
+ u32 __reserved0:3;
+ u32 hor_ds_en:1;
+ u32 min_clip_val:1;
+ u32 max_clip_val:2;
+ u32 out_frame_width:13;
+ u32 __reserved1:3;
+} __packed;
+
+struct imgu_abi_bds_ptrn_arr {
+ u32 elems[IMGU_ABI_BDS_SAMPLE_PATTERN_ARRAY_SIZE];
+} __packed;
+
+struct imgu_abi_bds_phase_entry {
+ s8 coeff_min2;
+ s8 coeff_min1;
+ s8 coeff_0;
+ s8 nf;
+ s8 coeff_pls1;
+ s8 coeff_pls2;
+ s8 coeff_pls3;
+ u8 __reserved;
+} __packed;
+
+struct imgu_abi_bds_phase_arr {
+ struct imgu_abi_bds_phase_entry
+ even[IMGU_ABI_BDS_PHASE_COEFFS_ARRAY_SIZE];
+ struct imgu_abi_bds_phase_entry
+ odd[IMGU_ABI_BDS_PHASE_COEFFS_ARRAY_SIZE];
+} __packed;
+
+struct imgu_abi_bds_hor_ctrl1 {
+ u32 hor_crop_start:13;
+ u32 __reserved0:3;
+ u32 hor_crop_end:13;
+ u32 __reserved1:1;
+ u32 hor_crop_en:1;
+ u32 __reserved2:1;
+} __packed;
+
+struct imgu_abi_bds_hor_ctrl2 {
+ u32 input_frame_height:13;
+ u32 __reserved0:19;
+} __packed;
+
+struct imgu_abi_bds_hor {
+ struct imgu_abi_bds_hor_ctrl0 hor_ctrl0;
+ struct imgu_abi_bds_ptrn_arr hor_ptrn_arr;
+ struct imgu_abi_bds_phase_arr hor_phase_arr;
+ struct imgu_abi_bds_hor_ctrl1 hor_ctrl1;
+ struct imgu_abi_bds_hor_ctrl2 hor_ctrl2;
+} __packed;
+
+struct imgu_abi_bds_ver_ctrl0 {
+ u32 sample_patrn_length:9;
+ u32 __reserved0:3;
+ u32 ver_ds_en:1;
+ u32 min_clip_val:1;
+ u32 max_clip_val:2;
+ u32 __reserved1:16;
+} __packed;
+
+struct imgu_abi_bds_ver_ctrl1 {
+ u32 out_frame_width:13;
+ u32 __reserved0:3;
+ u32 out_frame_height:13;
+ u32 __reserved1:3;
+} __packed;
+
+struct imgu_abi_bds_ver {
+ struct imgu_abi_bds_ver_ctrl0 ver_ctrl0;
+ struct imgu_abi_bds_ptrn_arr ver_ptrn_arr;
+ struct imgu_abi_bds_phase_arr ver_phase_arr;
+ struct imgu_abi_bds_ver_ctrl1 ver_ctrl1;
+} __packed;
+
+struct imgu_abi_bds_per_stripe_data {
+ struct imgu_abi_bds_hor_ctrl0 hor_ctrl0;
+ struct imgu_abi_bds_ver_ctrl1 ver_ctrl1;
+ struct imgu_abi_bds_hor_ctrl1 crop;
+} __packed;
+
+struct imgu_abi_bds_per_stripe_data_aligned {
+ struct imgu_abi_bds_per_stripe_data data __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_bds_per_stripe {
+ struct imgu_abi_bds_per_stripe_data_aligned
+ aligned_data[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+struct imgu_abi_bds_config {
+ struct imgu_abi_bds_hor hor __attribute__((aligned(32)));
+ struct imgu_abi_bds_ver ver __attribute__((aligned(32)));
+ struct imgu_abi_bds_per_stripe per_stripe __attribute__((aligned(32)));
+ u32 enabled;
+} __packed;
+
+/* ANR */
+
+struct imgu_abi_anr_search_config {
+ u32 enable;
+ u16 frame_width;
+ u16 frame_height;
+} __packed;
+
+struct imgu_abi_anr_stitch_config {
+ u32 anr_stitch_en;
+ u16 frame_width;
+ u16 frame_height;
+ u8 __reserved[40];
+ struct ipu3_uapi_anr_stitch_pyramid pyramid[IPU3_UAPI_ANR_PYRAMID_SIZE];
+} __packed;
+
+struct imgu_abi_anr_tile2strm_config {
+ u32 enable;
+ u16 frame_width;
+ u16 frame_height;
+} __packed;
+
+struct imgu_abi_anr_config {
+ struct imgu_abi_anr_search_config search __attribute__((aligned(32)));
+ struct ipu3_uapi_anr_transform_config transform __attribute__((aligned(32)));
+ struct imgu_abi_anr_stitch_config stitch __attribute__((aligned(32)));
+ struct imgu_abi_anr_tile2strm_config tile2strm __attribute__((aligned(32)));
+} __packed;
+
+/* AF */
+
+struct imgu_abi_af_frame_size {
+ u16 width;
+ u16 height;
+} __packed;
+
+struct imgu_abi_af_config_s {
+ struct ipu3_uapi_af_filter_config filter_config __attribute__((aligned(32)));
+ struct imgu_abi_af_frame_size frame_size;
+ struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_af_intra_frame_operations_data {
+ struct imgu_abi_acc_operation ops[IMGU_ABI_AF_MAX_OPERATIONS]
+ __attribute__((aligned(32)));
+ struct imgu_abi_acc_process_lines_cmd_data
+ process_lines_data[IMGU_ABI_AF_MAX_PROCESS_LINES] __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_af_stripe_config {
+ struct imgu_abi_af_frame_size frame_size __attribute__((aligned(32)));
+ struct ipu3_uapi_grid_config grid_cfg __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_af_config {
+ struct imgu_abi_af_config_s config;
+ struct imgu_abi_af_intra_frame_operations_data operations_data;
+ struct imgu_abi_af_stripe_config stripes[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+/* AE */
+
+struct imgu_abi_ae_config {
+ struct ipu3_uapi_ae_grid_config grid_cfg __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_weight_elem weights[IPU3_UAPI_AE_WEIGHTS]
+ __attribute__((aligned(32)));
+ struct ipu3_uapi_ae_ccm ae_ccm __attribute__((aligned(32)));
+ struct {
+ struct ipu3_uapi_ae_grid_config grid __attribute__((aligned(32)));
+ } stripes[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+/* AWB_FR */
+
+struct imgu_abi_awb_fr_intra_frame_operations_data {
+ struct imgu_abi_acc_operation ops[IMGU_ABI_AWB_FR_MAX_OPERATIONS]
+ __attribute__((aligned(32)));
+ struct imgu_abi_acc_process_lines_cmd_data
+ process_lines_data[IMGU_ABI_AWB_FR_MAX_PROCESS_LINES] __attribute__((aligned(32)));
+} __packed;
+
+struct imgu_abi_awb_fr_config {
+ struct ipu3_uapi_awb_fr_config_s config;
+ struct imgu_abi_awb_fr_intra_frame_operations_data operations_data;
+ struct ipu3_uapi_awb_fr_config_s stripes[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+struct imgu_abi_acc_transfer_op_data {
+ u8 set_number;
+} __packed;
+
+struct imgu_abi_awb_intra_frame_operations_data {
+ struct imgu_abi_acc_operation ops[IMGU_ABI_AWB_MAX_OPERATIONS]
+ __attribute__((aligned(32)));
+ struct imgu_abi_acc_process_lines_cmd_data
+ process_lines_data[IMGU_ABI_AWB_MAX_PROCESS_LINES] __attribute__((aligned(32)));
+ struct imgu_abi_acc_transfer_op_data
+ transfer_data[IMGU_ABI_AWB_MAX_TRANSFERS] __attribute__((aligned(32)));
+} __attribute__((aligned(32))) __packed;
+
+struct imgu_abi_awb_config {
+ struct ipu3_uapi_awb_config_s config __attribute__((aligned(32)));
+ struct imgu_abi_awb_intra_frame_operations_data operations_data;
+ struct ipu3_uapi_awb_config_s stripes[IPU3_UAPI_MAX_STRIPES];
+} __packed;
+
+struct imgu_abi_acc_param {
+ struct imgu_abi_stripe_data stripe;
+ u8 padding[8];
+ struct imgu_abi_input_feeder_config input_feeder;
+ struct ipu3_uapi_bnr_static_config bnr;
+ struct ipu3_uapi_bnr_static_config_green_disparity green_disparity
+ __attribute__((aligned(32)));
+ struct ipu3_uapi_dm_config dm __attribute__((aligned(32)));
+ struct ipu3_uapi_ccm_mat_config ccm __attribute__((aligned(32)));
+ struct ipu3_uapi_gamma_config gamma __attribute__((aligned(32)));
+ struct ipu3_uapi_csc_mat_config csc __attribute__((aligned(32)));
+ struct ipu3_uapi_cds_params cds __attribute__((aligned(32)));
+ struct imgu_abi_shd_config shd __attribute__((aligned(32)));
+ struct imgu_abi_dvs_stat_config dvs_stat;
+ u8 padding1[224]; /* reserved for lace_stat */
+ struct ipu3_uapi_yuvp1_iefd_config iefd __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_yds_config yds_c0 __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_chnr_config chnr_c0 __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_y_ee_nr_config y_ee_nr __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_yds_config yds __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_chnr_config chnr __attribute__((aligned(32)));
+ struct imgu_abi_yuvp2_y_tm_lut_static_config ytm __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp1_yds_config yds2 __attribute__((aligned(32)));
+ struct ipu3_uapi_yuvp2_tcc_static_config tcc __attribute__((aligned(32)));
+ /* reserved for defect pixel correction */
+ u8 dpc[240832] __attribute__((aligned(32)));
+ struct imgu_abi_bds_config bds;
+ struct imgu_abi_anr_config anr;
+ struct imgu_abi_awb_fr_config awb_fr;
+ struct imgu_abi_ae_config ae;
+ struct imgu_abi_af_config af;
+ struct imgu_abi_awb_config awb;
+ struct imgu_abi_osys_config osys;
+} __packed;
+
+/***** Morphing table entry *****/
+
+struct imgu_abi_gdc_warp_param {
+ u32 origin_x;
+ u32 origin_y;
+ u32 in_addr_offset;
+ u32 in_block_width;
+ u32 in_block_height;
+ u32 p0_x;
+ u32 p0_y;
+ u32 p1_x;
+ u32 p1_y;
+ u32 p2_x;
+ u32 p2_y;
+ u32 p3_x;
+ u32 p3_y;
+ u32 in_block_width_a;
+ u32 in_block_width_b;
+ u32 padding; /* struct size multiple of DDR word */
+} __packed;
+
+/******************* Firmware ABI definitions *******************/
+
+/***** struct imgu_abi_sp_stage *****/
+
+struct imgu_abi_crop_pos {
+ u16 x;
+ u16 y;
+} __packed;
+
+struct imgu_abi_sp_resolution {
+ u16 width; /* Width of valid data in pixels */
+ u16 height; /* Height of valid data in lines */
+} __packed;
+
+/*
+ * Frame info struct. This describes the contents of an image frame buffer.
+ */
+struct imgu_abi_frame_sp_info {
+ struct imgu_abi_sp_resolution res;
+ u16 padded_width; /* stride of line in memory
+ * (in pixels)
+ */
+ u8 format; /* format of the frame data */
+ u8 raw_bit_depth; /* number of valid bits per pixel,
+ * only valid for RAW bayer frames
+ */
+ u8 raw_bayer_order; /* bayer order, only valid
+ * for RAW bayer frames
+ */
+ u8 raw_type; /* To choose the proper raw frame type. for
+ * Legacy SKC pipes/Default is set to
+ * IMGU_ABI_RAW_TYPE_BAYER. For RGB IR sensor -
+ * driver should set it to:
+ * IronGr case - IMGU_ABI_RAW_TYPE_IR_ON_GR
+ * IronGb case - IMGU_ABI_RAW_TYPE_IR_ON_GB
+ */
+ u8 padding[2]; /* Extend to 32 bit multiple */
+} __packed;
+
+struct imgu_abi_buffer_sp {
+ union {
+ imgu_addr_t xmem_addr;
+ s32 queue_id; /* enum imgu_abi_queue_id */
+ } buf_src;
+ s32 buf_type; /* enum imgu_abi_buffer_type */
+} __packed;
+
+struct imgu_abi_frame_sp_plane {
+ u32 offset; /* offset in bytes to start of frame data */
+ /* offset is wrt data in imgu_abi_sp_sp_frame */
+} __packed;
+
+struct imgu_abi_frame_sp_rgb_planes {
+ struct imgu_abi_frame_sp_plane r;
+ struct imgu_abi_frame_sp_plane g;
+ struct imgu_abi_frame_sp_plane b;
+} __packed;
+
+struct imgu_abi_frame_sp_yuv_planes {
+ struct imgu_abi_frame_sp_plane y;
+ struct imgu_abi_frame_sp_plane u;
+ struct imgu_abi_frame_sp_plane v;
+} __packed;
+
+struct imgu_abi_frame_sp_nv_planes {
+ struct imgu_abi_frame_sp_plane y;
+ struct imgu_abi_frame_sp_plane uv;
+} __packed;
+
+struct imgu_abi_frame_sp_plane6 {
+ struct imgu_abi_frame_sp_plane r;
+ struct imgu_abi_frame_sp_plane r_at_b;
+ struct imgu_abi_frame_sp_plane gr;
+ struct imgu_abi_frame_sp_plane gb;
+ struct imgu_abi_frame_sp_plane b;
+ struct imgu_abi_frame_sp_plane b_at_r;
+} __packed;
+
+struct imgu_abi_frame_sp_binary_plane {
+ u32 size;
+ struct imgu_abi_frame_sp_plane data;
+} __packed;
+
+struct imgu_abi_frame_sp {
+ struct imgu_abi_frame_sp_info info;
+ struct imgu_abi_buffer_sp buf_attr;
+ union {
+ struct imgu_abi_frame_sp_plane raw;
+ struct imgu_abi_frame_sp_plane rgb;
+ struct imgu_abi_frame_sp_rgb_planes planar_rgb;
+ struct imgu_abi_frame_sp_plane yuyv;
+ struct imgu_abi_frame_sp_yuv_planes yuv;
+ struct imgu_abi_frame_sp_nv_planes nv;
+ struct imgu_abi_frame_sp_plane6 plane6;
+ struct imgu_abi_frame_sp_binary_plane binary;
+ } planes;
+} __packed;
+
+struct imgu_abi_resolution {
+ u32 width;
+ u32 height;
+} __packed;
+
+struct imgu_abi_frames_sp {
+ struct imgu_abi_frame_sp in;
+ struct imgu_abi_frame_sp out[IMGU_ABI_BINARY_MAX_OUTPUT_PORTS];
+ struct imgu_abi_resolution effective_in_res;
+ struct imgu_abi_frame_sp out_vf;
+ struct imgu_abi_frame_sp_info internal_frame_info;
+ struct imgu_abi_buffer_sp s3a_buf;
+ struct imgu_abi_buffer_sp dvs_buf;
+ struct imgu_abi_buffer_sp lace_buf;
+} __packed;
+
+struct imgu_abi_uds_info {
+ u16 curr_dx;
+ u16 curr_dy;
+ u16 xc;
+ u16 yc;
+} __packed;
+
+/* Information for a single pipeline stage */
+struct imgu_abi_sp_stage {
+ /* Multiple boolean flags can be stored in an integer */
+ u8 num; /* Stage number */
+ u8 isp_online;
+ u8 isp_copy_vf;
+ u8 isp_copy_output;
+ u8 sp_enable_xnr;
+ u8 isp_deci_log_factor;
+ u8 isp_vf_downscale_bits;
+ u8 deinterleaved;
+ /*
+ * NOTE: Programming the input circuit can only be done at the
+ * start of a session. It is illegal to program it during execution
+ * The input circuit defines the connectivity
+ */
+ u8 program_input_circuit;
+ u8 func;
+ u8 stage_type; /* enum imgu_abi_stage_type */
+ u8 num_stripes;
+ u8 isp_pipe_version;
+ struct {
+ u8 vf_output;
+ u8 s3a;
+ u8 sdis;
+ u8 dvs_stats;
+ u8 lace_stats;
+ } enable;
+
+ struct imgu_abi_crop_pos sp_out_crop_pos;
+ u8 padding[2];
+ struct imgu_abi_frames_sp frames;
+ struct imgu_abi_resolution dvs_envelope;
+ struct imgu_abi_uds_info uds;
+ imgu_addr_t isp_stage_addr;
+ imgu_addr_t xmem_bin_addr;
+ imgu_addr_t xmem_map_addr;
+
+ u16 top_cropping;
+ u16 row_stripes_height;
+ u16 row_stripes_overlap_lines;
+ u8 if_config_index; /* Which should be applied by this stage. */
+ u8 padding2;
+} __packed;
+
+/***** struct imgu_abi_isp_stage *****/
+
+struct imgu_abi_isp_param_memory_offsets {
+ u32 offsets[IMGU_ABI_PARAM_CLASS_NUM]; /* offset wrt hdr in bytes */
+} __packed;
+
+/*
+ * Blob descriptor.
+ * This structure describes an SP or ISP blob.
+ * It describes the test, data and bss sections as well as position in a
+ * firmware file.
+ * For convenience, it contains dynamic data after loading.
+ */
+struct imgu_abi_blob_info {
+ /* Static blob data */
+ u32 offset; /* Blob offset in fw file */
+ struct imgu_abi_isp_param_memory_offsets memory_offsets;
+ /* offset wrt hdr in bytes */
+ u32 prog_name_offset; /* offset wrt hdr in bytes */
+ u32 size; /* Size of blob */
+ u32 padding_size; /* total cummulative of bytes added
+ * due to section alignment
+ */
+ u32 icache_source; /* Position of icache in blob */
+ u32 icache_size; /* Size of icache section */
+ u32 icache_padding; /* added due to icache section alignment */
+ u32 text_source; /* Position of text in blob */
+ u32 text_size; /* Size of text section */
+ u32 text_padding; /* bytes added due to text section alignment */
+ u32 data_source; /* Position of data in blob */
+ u32 data_target; /* Start of data in SP dmem */
+ u32 data_size; /* Size of text section */
+ u32 data_padding; /* bytes added due to data section alignment */
+ u32 bss_target; /* Start position of bss in SP dmem */
+ u32 bss_size; /* Size of bss section
+ * Dynamic data filled by loader
+ */
+ u64 code __attribute__((aligned(8))); /* Code section absolute pointer */
+ /* within fw, code = icache + text */
+ u64 data __attribute__((aligned(8))); /* Data section absolute pointer */
+ /* within fw, data = data + bss */
+} __packed;
+
+struct imgu_abi_binary_pipeline_info {
+ u32 mode;
+ u32 isp_pipe_version;
+ u32 pipelining;
+ u32 c_subsampling;
+ u32 top_cropping;
+ u32 left_cropping;
+ u32 variable_resolution;
+} __packed;
+
+struct imgu_abi_binary_input_info {
+ u32 min_width;
+ u32 min_height;
+ u32 max_width;
+ u32 max_height;
+ u32 source; /* enum imgu_abi_bin_input_src */
+} __packed;
+
+struct imgu_abi_binary_output_info {
+ u32 min_width;
+ u32 min_height;
+ u32 max_width;
+ u32 max_height;
+ u32 num_chunks;
+ u32 variable_format;
+} __packed;
+
+struct imgu_abi_binary_internal_info {
+ u32 max_width;
+ u32 max_height;
+} __packed;
+
+struct imgu_abi_binary_bds_info {
+ u32 supported_bds_factors;
+} __packed;
+
+struct imgu_abi_binary_dvs_info {
+ u32 max_envelope_width;
+ u32 max_envelope_height;
+} __packed;
+
+struct imgu_abi_binary_vf_dec_info {
+ u32 is_variable;
+ u32 max_log_downscale;
+} __packed;
+
+struct imgu_abi_binary_s3a_info {
+ u32 s3atbl_use_dmem;
+ u32 fixed_s3a_deci_log;
+} __packed;
+
+struct imgu_abi_binary_dpc_info {
+ u32 bnr_lite; /* bnr lite enable flag */
+} __packed;
+
+struct imgu_abi_binary_iterator_info {
+ u32 num_stripes;
+ u32 row_stripes_height;
+ u32 row_stripes_overlap_lines;
+} __packed;
+
+struct imgu_abi_binary_address_info {
+ u32 isp_addresses; /* Address in ISP dmem */
+ u32 main_entry; /* Address of entry fct */
+ u32 in_frame; /* Address in ISP dmem */
+ u32 out_frame; /* Address in ISP dmem */
+ u32 in_data; /* Address in ISP dmem */
+ u32 out_data; /* Address in ISP dmem */
+ u32 sh_dma_cmd_ptr; /* In ISP dmem */
+} __packed;
+
+struct imgu_abi_binary_uds_info {
+ u16 bpp;
+ u16 use_bci;
+ u16 use_str;
+ u16 woix;
+ u16 woiy;
+ u16 extra_out_vecs;
+ u16 vectors_per_line_in;
+ u16 vectors_per_line_out;
+ u16 vectors_c_per_line_in;
+ u16 vectors_c_per_line_out;
+ u16 vmem_gdc_in_block_height_y;
+ u16 vmem_gdc_in_block_height_c;
+} __packed;
+
+struct imgu_abi_binary_block_info {
+ u32 block_width;
+ u32 block_height;
+ u32 output_block_height;
+} __packed;
+
+struct imgu_abi_isp_data {
+ imgu_addr_t address; /* ISP address */
+ u32 size; /* Disabled if 0 */
+} __packed;
+
+struct imgu_abi_isp_param_segments {
+ struct imgu_abi_isp_data
+ params[IMGU_ABI_PARAM_CLASS_NUM][IMGU_ABI_NUM_MEMORIES];
+} __packed;
+
+struct imgu_abi_binary_info {
+ u32 id __attribute__((aligned(8))); /* IMGU_ABI_BINARY_ID_* */
+ struct imgu_abi_binary_pipeline_info pipeline;
+ struct imgu_abi_binary_input_info input;
+ struct imgu_abi_binary_output_info output;
+ struct imgu_abi_binary_internal_info internal;
+ struct imgu_abi_binary_bds_info bds;
+ struct imgu_abi_binary_dvs_info dvs;
+ struct imgu_abi_binary_vf_dec_info vf_dec;
+ struct imgu_abi_binary_s3a_info s3a;
+ struct imgu_abi_binary_dpc_info dpc_bnr; /* DPC related binary info */
+ struct imgu_abi_binary_iterator_info iterator;
+ struct imgu_abi_binary_address_info addresses;
+ struct imgu_abi_binary_uds_info uds;
+ struct imgu_abi_binary_block_info block;
+ struct imgu_abi_isp_param_segments mem_initializers;
+ struct {
+ u8 input_feeder;
+ u8 output_system;
+ u8 obgrid;
+ u8 lin;
+ u8 dpc_acc;
+ u8 bds_acc;
+ u8 shd_acc;
+ u8 shd_ff;
+ u8 stats_3a_raw_buffer;
+ u8 acc_bayer_denoise;
+ u8 bnr_ff;
+ u8 awb_acc;
+ u8 awb_fr_acc;
+ u8 anr_acc;
+ u8 rgbpp_acc;
+ u8 rgbpp_ff;
+ u8 demosaic_acc;
+ u8 demosaic_ff;
+ u8 dvs_stats;
+ u8 lace_stats;
+ u8 yuvp1_b0_acc;
+ u8 yuvp1_c0_acc;
+ u8 yuvp2_acc;
+ u8 ae;
+ u8 af;
+ u8 dergb;
+ u8 rgb2yuv;
+ u8 high_quality;
+ u8 kerneltest;
+ u8 routing_shd_to_bnr; /* connect SHD with BNR ACCs */
+ u8 routing_bnr_to_anr; /* connect BNR with ANR ACCs */
+ u8 routing_anr_to_de; /* connect ANR with DE ACCs */
+ u8 routing_rgb_to_yuvp1; /* connect RGB with YUVP1 */
+ u8 routing_yuvp1_to_yuvp2; /* connect YUVP1 with YUVP2 */
+ u8 luma_only;
+ u8 input_yuv;
+ u8 input_raw;
+ u8 reduced_pipe;
+ u8 vf_veceven;
+ u8 dis;
+ u8 dvs_envelope;
+ u8 uds;
+ u8 dvs_6axis;
+ u8 block_output;
+ u8 streaming_dma;
+ u8 ds;
+ u8 bayer_fir_6db;
+ u8 raw_binning;
+ u8 continuous;
+ u8 s3a;
+ u8 fpnr;
+ u8 sc;
+ u8 macc;
+ u8 output;
+ u8 ref_frame;
+ u8 tnr;
+ u8 xnr;
+ u8 params;
+ u8 ca_gdc;
+ u8 isp_addresses;
+ u8 in_frame;
+ u8 out_frame;
+ u8 high_speed;
+ u8 dpc;
+ u8 padding[2];
+ u8 rgbir;
+ } enable;
+ struct {
+ u8 ref_y_channel;
+ u8 ref_c_channel;
+ u8 tnr_channel;
+ u8 tnr_out_channel;
+ u8 dvs_coords_channel;
+ u8 output_channel;
+ u8 c_channel;
+ u8 vfout_channel;
+ u8 vfout_c_channel;
+ u8 vfdec_bits_per_pixel;
+ u8 claimed_by_isp;
+ u8 padding[2];
+ } dma;
+} __packed;
+
+struct imgu_abi_isp_stage {
+ struct imgu_abi_blob_info blob_info;
+ struct imgu_abi_binary_info binary_info;
+ char binary_name[IMGU_ABI_MAX_BINARY_NAME];
+ struct imgu_abi_isp_param_segments mem_initializers;
+} __packed;
+
+/***** struct imgu_abi_ddr_address_map and parameter set *****/
+
+/* xmem address map allocation */
+struct imgu_abi_ddr_address_map {
+ imgu_addr_t isp_mem_param[IMGU_ABI_MAX_STAGES][IMGU_ABI_NUM_MEMORIES];
+ imgu_addr_t obgrid_tbl[IPU3_UAPI_MAX_STRIPES];
+ imgu_addr_t acc_cluster_params_for_sp;
+ imgu_addr_t dvs_6axis_params_y;
+} __packed;
+
+struct imgu_abi_parameter_set_info {
+ /* Pointers to Parameters in ISP format IMPT */
+ struct imgu_abi_ddr_address_map mem_map;
+ /* Unique ID to track per-frame configurations */
+ u32 isp_parameters_id;
+ /* Output frame to which this config has to be applied (optional) */
+ imgu_addr_t output_frame_ptr;
+} __packed;
+
+/***** struct imgu_abi_sp_group *****/
+
+/* SP configuration information */
+struct imgu_abi_sp_config {
+ u8 no_isp_sync; /* Signal host immediately after start */
+ u8 enable_raw_pool_locking; /* Enable Raw Buffer Locking for HALv3 */
+ u8 lock_all;
+ u8 disable_cont_vf;
+ u8 disable_preview_on_capture;
+ u8 padding[3];
+} __packed;
+
+/* Information for a pipeline */
+struct imgu_abi_sp_pipeline {
+ u32 pipe_id; /* the pipe ID */
+ u32 pipe_num; /* the dynamic pipe number */
+ u32 thread_id; /* the sp thread ID */
+ u32 pipe_config; /* the pipe config */
+ u32 pipe_qos_config; /* Bitmap of multiple QOS extension fw
+ * state, 0xffffffff indicates non
+ * QOS pipe.
+ */
+ u32 inout_port_config;
+ u32 required_bds_factor;
+ u32 dvs_frame_delay;
+ u32 num_stages; /* the pipe config */
+ u32 running; /* needed for pipe termination */
+ imgu_addr_t sp_stage_addr[IMGU_ABI_MAX_STAGES];
+ imgu_addr_t scaler_pp_lut; /* Early bound LUT */
+ u32 stage; /* stage ptr is only used on sp */
+ s32 num_execs; /* number of times to run if this is
+ * an acceleration pipe.
+ */
+ union {
+ struct {
+ u32 bytes_available;
+ } bin;
+ struct {
+ u32 height;
+ u32 width;
+ u32 padded_width;
+ u32 max_input_width;
+ u32 raw_bit_depth;
+ } raw;
+ } copy;
+
+ /* Parameters passed to Shading Correction kernel. */
+ struct {
+ /* Origin X (bqs) of internal frame on shading table */
+ u32 internal_frame_origin_x_bqs_on_sctbl;
+ /* Origin Y (bqs) of internal frame on shading table */
+ u32 internal_frame_origin_y_bqs_on_sctbl;
+ } shading;
+} __packed;
+
+struct imgu_abi_sp_debug_command {
+ /*
+ * The DMA software-mask,
+ * Bit 31...24: unused.
+ * Bit 23...16: unused.
+ * Bit 15...08: reading-request enabling bits for DMA channel 7..0
+ * Bit 07...00: writing-request enabling bits for DMA channel 7..0
+ *
+ * For example, "0...0 0...0 11111011 11111101" indicates that the
+ * writing request through DMA Channel 1 and the reading request
+ * through DMA channel 2 are both disabled. The others are enabled.
+ */
+ u32 dma_sw_reg;
+} __packed;
+
+/*
+ * Group all host initialized SP variables into this struct.
+ * This is initialized every stage through dma.
+ * The stage part itself is transferred through imgu_abi_sp_stage.
+ */
+struct imgu_abi_sp_group {
+ struct imgu_abi_sp_config config;
+ struct imgu_abi_sp_pipeline pipe[IMGU_ABI_MAX_SP_THREADS];
+ struct imgu_abi_sp_debug_command debug;
+} __packed;
+
+/***** parameter and state class binary configurations *****/
+
+struct imgu_abi_isp_iterator_config {
+ struct imgu_abi_frame_sp_info input_info;
+ struct imgu_abi_frame_sp_info internal_info;
+ struct imgu_abi_frame_sp_info output_info;
+ struct imgu_abi_frame_sp_info vf_info;
+ struct imgu_abi_sp_resolution dvs_envelope;
+} __packed;
+
+struct imgu_abi_dma_port_config {
+ u8 crop, elems;
+ u16 width;
+ u32 stride;
+} __packed;
+
+struct imgu_abi_isp_ref_config {
+ u32 width_a_over_b;
+ struct imgu_abi_dma_port_config port_b;
+ u32 ref_frame_addr_y[IMGU_ABI_FRAMES_REF];
+ u32 ref_frame_addr_c[IMGU_ABI_FRAMES_REF];
+ u32 dvs_frame_delay;
+} __packed;
+
+struct imgu_abi_isp_ref_dmem_state {
+ u32 ref_in_buf_idx;
+ u32 ref_out_buf_idx;
+} __packed;
+
+struct imgu_abi_isp_dvs_config {
+ u32 num_horizontal_blocks;
+ u32 num_vertical_blocks;
+} __packed;
+
+struct imgu_abi_isp_tnr3_config {
+ u32 width_a_over_b;
+ u32 frame_height;
+ struct imgu_abi_dma_port_config port_b;
+ u32 delay_frame;
+ u32 frame_addr[IMGU_ABI_FRAMES_TNR];
+} __packed;
+
+struct imgu_abi_isp_tnr3_dmem_state {
+ u32 in_bufidx;
+ u32 out_bufidx;
+ u32 total_frame_counter;
+ u32 buffer_frame_counter[IMGU_ABI_BUF_SETS_TNR];
+ u32 bypass_filter;
+} __packed;
+
+/***** Queues *****/
+
+struct imgu_abi_queue_info {
+ u8 size; /* the maximum number of elements*/
+ u8 step; /* number of bytes per element */
+ u8 start; /* index of the oldest element */
+ u8 end; /* index at which to write the new element */
+} __packed;
+
+struct imgu_abi_queues {
+ /*
+ * Queues for the dynamic frame information,
+ * i.e. the "in_frame" buffer, the "out_frame"
+ * buffer and the "vf_out_frame" buffer.
+ */
+ struct imgu_abi_queue_info host2sp_bufq_info
+ [IMGU_ABI_MAX_SP_THREADS][IMGU_ABI_QUEUE_NUM];
+ u32 host2sp_bufq[IMGU_ABI_MAX_SP_THREADS][IMGU_ABI_QUEUE_NUM]
+ [IMGU_ABI_HOST2SP_BUFQ_SIZE];
+ struct imgu_abi_queue_info sp2host_bufq_info[IMGU_ABI_QUEUE_NUM];
+ u32 sp2host_bufq[IMGU_ABI_QUEUE_NUM][IMGU_ABI_SP2HOST_BUFQ_SIZE];
+
+ /*
+ * The queues for the events.
+ */
+ struct imgu_abi_queue_info host2sp_evtq_info;
+ u32 host2sp_evtq[IMGU_ABI_HOST2SP_EVTQ_SIZE];
+ struct imgu_abi_queue_info sp2host_evtq_info;
+ u32 sp2host_evtq[IMGU_ABI_SP2HOST_EVTQ_SIZE];
+} __packed;
+
+/***** Buffer descriptor *****/
+
+struct imgu_abi_metadata_info {
+ struct imgu_abi_resolution resolution; /* Resolution */
+ u32 stride; /* Stride in bytes */
+ u32 size; /* Total size in bytes */
+} __packed;
+
+struct imgu_abi_isp_3a_statistics {
+ union {
+ struct {
+ imgu_addr_t s3a_tbl;
+ } dmem;
+ struct {
+ imgu_addr_t s3a_tbl_hi;
+ imgu_addr_t s3a_tbl_lo;
+ } vmem;
+ } data;
+ struct {
+ imgu_addr_t rgby_tbl;
+ } data_hmem;
+ u32 exp_id; /* exposure id, to match statistics to a frame, */
+ u32 isp_config_id; /* Tracks per-frame configs */
+ imgu_addr_t data_ptr; /* pointer to base of all data */
+ u32 size; /* total size of all data */
+ u32 dmem_size;
+ u32 vmem_size; /* both lo and hi have this size */
+ u32 hmem_size;
+} __packed;
+
+struct imgu_abi_metadata {
+ struct imgu_abi_metadata_info info; /* Layout info */
+ imgu_addr_t address; /* CSS virtual address */
+ u32 exp_id; /* Exposure ID */
+} __packed;
+
+struct imgu_abi_time_meas {
+ u32 start_timer_value; /* measured time in ticks */
+ u32 end_timer_value; /* measured time in ticks */
+} __packed;
+
+struct imgu_abi_buffer {
+ union {
+ struct imgu_abi_isp_3a_statistics s3a;
+ u8 __reserved[28];
+ imgu_addr_t skc_dvs_statistics;
+ imgu_addr_t lace_stat;
+ struct imgu_abi_metadata metadata;
+ struct {
+ imgu_addr_t frame_data;
+ u32 flashed;
+ u32 exp_id;
+ u32 isp_parameters_id; /* Tracks per-frame configs */
+ u32 padded_width;
+ } frame;
+ imgu_addr_t ddr_ptrs;
+ } payload;
+ /*
+ * kernel_ptr is present for host administration purposes only.
+ * type is uint64_t in order to be 64-bit host compatible.
+ * uint64_t does not exist on SP/ISP.
+ * Size of the struct is checked by sp.hive.c.
+ */
+ u64 cookie_ptr __attribute__((aligned(8)));
+ u64 kernel_ptr;
+ struct imgu_abi_time_meas timing_data;
+ u32 isys_eof_clock_tick;
+} __packed;
+
+struct imgu_abi_bl_dma_cmd_entry {
+ u32 src_addr; /* virtual DDR address */
+ u32 size; /* number of bytes to transferred */
+ u32 dst_type;
+ u32 dst_addr; /* hmm address of xMEM or MMIO */
+} __packed;
+
+struct imgu_abi_sp_init_dmem_cfg {
+ u32 ddr_data_addr; /* data segment address in ddr */
+ u32 dmem_data_addr; /* data segment address in dmem */
+ u32 dmem_bss_addr; /* bss segment address in dmem */
+ u32 data_size; /* data segment size */
+ u32 bss_size; /* bss segment size */
+ u32 sp_id; /* sp id */
+} __packed;
+
#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 05/16] intel-ipu3: abi: Add structs
2018-10-29 22:22 ` [PATCH v7 05/16] intel-ipu3: abi: Add structs Yong Zhi
@ 2018-11-05 8:27 ` Sakari Ailus
2018-11-05 19:05 ` Mani, Rajmohan
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-05 8:27 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:22:59PM -0700, Yong Zhi wrote:
> This add all the structs of IPU3 firmware ABI.
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
...
> +struct imgu_abi_shd_intra_frame_operations_data {
> + struct imgu_abi_acc_operation
> + operation_list[IMGU_ABI_SHD_MAX_OPERATIONS] __attribute__((aligned(32)));
> + struct imgu_abi_acc_process_lines_cmd_data
> + process_lines_data[IMGU_ABI_SHD_MAX_PROCESS_LINES] __attribute__((aligned(32)));
> + struct imgu_abi_shd_transfer_luts_set_data
> + transfer_data[IMGU_ABI_SHD_MAX_TRANSFERS] __attribute__((aligned(32)));
Could you replace this wth __aligned(32), please? The same for the rest of
the header.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 05/16] intel-ipu3: abi: Add structs
2018-11-05 8:27 ` Sakari Ailus
@ 2018-11-05 19:05 ` Mani, Rajmohan
2018-11-06 8:04 ` Sakari Ailus
0 siblings, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2018-11-05 19:05 UTC (permalink / raw)
To: Sakari Ailus, Zhi, Yong
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu,
Cao, Bingbu
Hi Sakari,
> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> Sent: Monday, November 05, 2018 12:28 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org; mchehab@kernel.org;
> hans.verkuil@cisco.com; laurent.pinchart@ideasonboard.com; Mani,
> Rajmohan <rajmohan.mani@intel.com>; Zheng, Jian Xu
> <jian.xu.zheng@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>; Toivonen,
> Tuukka <tuukka.toivonen@intel.com>; Qiu, Tian Shu
> <tian.shu.qiu@intel.com>; Cao, Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 05/16] intel-ipu3: abi: Add structs
>
> Hi Yong,
>
> On Mon, Oct 29, 2018 at 03:22:59PM -0700, Yong Zhi wrote:
> > This add all the structs of IPU3 firmware ABI.
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
>
> ...
>
> > +struct imgu_abi_shd_intra_frame_operations_data {
> > + struct imgu_abi_acc_operation
> > + operation_list[IMGU_ABI_SHD_MAX_OPERATIONS]
> __attribute__((aligned(32)));
> > + struct imgu_abi_acc_process_lines_cmd_data
> > + process_lines_data[IMGU_ABI_SHD_MAX_PROCESS_LINES]
> __attribute__((aligned(32)));
> > + struct imgu_abi_shd_transfer_luts_set_data
> > + transfer_data[IMGU_ABI_SHD_MAX_TRANSFERS]
> > +__attribute__((aligned(32)));
>
> Could you replace this wth __aligned(32), please? The same for the rest of the
> header.
>
Using __aligned(32) in the uAPI header resulted in compilation errors in
user space / camera HAL code.
e.g
../../../../../../../../usr/include/linux/intel-ipu3.h:464:57: error: expected ';'
at end of declaration list
__u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE] __aligned(32);
So we ended up using __attribute__((aligned(32))) format in uAPI header and
to be consistent, we followed the same format in ABI header as well.
Let us know if it's okay to deviate between uAPI and ABI header for this
alignment qualifier.
> --
> Regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 05/16] intel-ipu3: abi: Add structs
2018-11-05 19:05 ` Mani, Rajmohan
@ 2018-11-06 8:04 ` Sakari Ailus
2018-11-06 23:31 ` Mani, Rajmohan
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-06 8:04 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Zhi, Yong, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi Raj,
On Mon, Nov 05, 2018 at 07:05:53PM +0000, Mani, Rajmohan wrote:
> Hi Sakari,
>
> > -----Original Message-----
> > From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> > Sent: Monday, November 05, 2018 12:28 AM
> > To: Zhi, Yong <yong.zhi@intel.com>
> > Cc: linux-media@vger.kernel.org; tfiga@chromium.org; mchehab@kernel.org;
> > hans.verkuil@cisco.com; laurent.pinchart@ideasonboard.com; Mani,
> > Rajmohan <rajmohan.mani@intel.com>; Zheng, Jian Xu
> > <jian.xu.zheng@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>; Toivonen,
> > Tuukka <tuukka.toivonen@intel.com>; Qiu, Tian Shu
> > <tian.shu.qiu@intel.com>; Cao, Bingbu <bingbu.cao@intel.com>
> > Subject: Re: [PATCH v7 05/16] intel-ipu3: abi: Add structs
> >
> > Hi Yong,
> >
> > On Mon, Oct 29, 2018 at 03:22:59PM -0700, Yong Zhi wrote:
> > > This add all the structs of IPU3 firmware ABI.
> > >
> > > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> >
> > ...
> >
> > > +struct imgu_abi_shd_intra_frame_operations_data {
> > > + struct imgu_abi_acc_operation
> > > + operation_list[IMGU_ABI_SHD_MAX_OPERATIONS]
> > __attribute__((aligned(32)));
> > > + struct imgu_abi_acc_process_lines_cmd_data
> > > + process_lines_data[IMGU_ABI_SHD_MAX_PROCESS_LINES]
> > __attribute__((aligned(32)));
> > > + struct imgu_abi_shd_transfer_luts_set_data
> > > + transfer_data[IMGU_ABI_SHD_MAX_TRANSFERS]
> > > +__attribute__((aligned(32)));
> >
> > Could you replace this wth __aligned(32), please? The same for the rest of the
> > header.
> >
>
> Using __aligned(32) in the uAPI header resulted in compilation errors in
> user space / camera HAL code.
>
> e.g
> ../../../../../../../../usr/include/linux/intel-ipu3.h:464:57: error: expected ';'
> at end of declaration list
> __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE] __aligned(32);
>
> So we ended up using __attribute__((aligned(32))) format in uAPI header and
> to be consistent, we followed the same format in ABI header as well.
>
> Let us know if it's okay to deviate between uAPI and ABI header for this
> alignment qualifier.
There's a reason for using __attribute__((aligned(32))) in the uAPI header,
but not in the in-kernel headers where __aligned(32) is preferred.
I have a patch for addressing this for the uAPI headers as well so
__aligned(32) could be used there, too; I'll submit it soon. Let's see...
there are kerneldoc issues still in this area.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 05/16] intel-ipu3: abi: Add structs
2018-11-06 8:04 ` Sakari Ailus
@ 2018-11-06 23:31 ` Mani, Rajmohan
0 siblings, 0 replies; 123+ messages in thread
From: Mani, Rajmohan @ 2018-11-06 23:31 UTC (permalink / raw)
To: Sakari Ailus
Cc: Zhi, Yong, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi Sakari,
> Subject: Re: [PATCH v7 05/16] intel-ipu3: abi: Add structs
>
> Hi Raj,
>
> On Mon, Nov 05, 2018 at 07:05:53PM +0000, Mani, Rajmohan wrote:
> > Hi Sakari,
> >
> > > Subject: Re: [PATCH v7 05/16] intel-ipu3: abi: Add structs
> > >
> > > Hi Yong,
> > >
> > > On Mon, Oct 29, 2018 at 03:22:59PM -0700, Yong Zhi wrote:
> > > > This add all the structs of IPU3 firmware ABI.
> > > >
> > > > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > > > Signed-off-by: Rajmohan Mani <rajmohan.mani@intel.com>
> > >
> > > ...
> > >
> > > > +struct imgu_abi_shd_intra_frame_operations_data {
> > > > + struct imgu_abi_acc_operation
> > > > + operation_list[IMGU_ABI_SHD_MAX_OPERATIONS]
> > > __attribute__((aligned(32)));
> > > > + struct imgu_abi_acc_process_lines_cmd_data
> > > > + process_lines_data[IMGU_ABI_SHD_MAX_PROCESS_LINES]
> > > __attribute__((aligned(32)));
> > > > + struct imgu_abi_shd_transfer_luts_set_data
> > > > + transfer_data[IMGU_ABI_SHD_MAX_TRANSFERS]
> > > > +__attribute__((aligned(32)));
> > >
> > > Could you replace this wth __aligned(32), please? The same for the
> > > rest of the header.
> > >
> >
> > Using __aligned(32) in the uAPI header resulted in compilation errors
> > in user space / camera HAL code.
> >
> > e.g
> > ../../../../../../../../usr/include/linux/intel-ipu3.h:464:57: error: expected ';'
> > at end of declaration list
> > __u8 bayer_table[IPU3_UAPI_AWB_FR_BAYER_TABLE_MAX_SIZE]
> > __aligned(32);
> >
> > So we ended up using __attribute__((aligned(32))) format in uAPI
> > header and to be consistent, we followed the same format in ABI header as
> well.
> >
> > Let us know if it's okay to deviate between uAPI and ABI header for
> > this alignment qualifier.
>
> There's a reason for using __attribute__((aligned(32))) in the uAPI header, but
> not in the in-kernel headers where __aligned(32) is preferred.
>
> I have a patch for addressing this for the uAPI headers as well so
> __aligned(32) could be used there, too; I'll submit it soon. Let's see...
> there are kerneldoc issues still in this area.
>
Great. Thanks for the patch to address this need.
> --
> Regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (4 preceding siblings ...)
2018-10-29 22:22 ` [PATCH v7 05/16] intel-ipu3: abi: Add structs Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-05 11:55 ` Sakari Ailus
2018-10-29 22:23 ` [PATCH v7 07/16] intel-ipu3: Implement DMA mapping functions Yong Zhi
` (11 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
From: Tomasz Figa <tfiga@chromium.org>
This driver translates IO virtual address to physical
address based on two levels page tables.
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-mmu.c | 560 ++++++++++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-mmu.h | 35 ++
2 files changed, 595 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.c b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
new file mode 100644
index 0000000..b66734a
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
@@ -0,0 +1,560 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018 Intel Corporation.
+ * Copyright 2018 Google LLC.
+ *
+ * Author: Tuukka Toivonen <tuukka.toivonen@intel.com>
+ * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
+ * Author: Samu Onkalo <samu.onkalo@intel.com>
+ * Author: Tomasz Figa <tfiga@chromium.org>
+ *
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/iopoll.h>
+#include <linux/pm_runtime.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+
+#include <asm/set_memory.h>
+
+#include "ipu3-mmu.h"
+
+#define IPU3_PAGE_SHIFT 12
+#define IPU3_PAGE_SIZE (1UL << IPU3_PAGE_SHIFT)
+
+#define IPU3_PT_BITS 10
+#define IPU3_PT_PTES (1UL << IPU3_PT_BITS)
+#define IPU3_PT_SIZE (IPU3_PT_PTES << 2)
+#define IPU3_PT_ORDER (IPU3_PT_SIZE >> PAGE_SHIFT)
+
+#define IPU3_ADDR2PTE(addr) ((addr) >> IPU3_PAGE_SHIFT)
+#define IPU3_PTE2ADDR(pte) ((phys_addr_t)(pte) << IPU3_PAGE_SHIFT)
+
+#define IPU3_L2PT_SHIFT IPU3_PT_BITS
+#define IPU3_L2PT_MASK ((1UL << IPU3_L2PT_SHIFT) - 1)
+
+#define IPU3_L1PT_SHIFT IPU3_PT_BITS
+#define IPU3_L1PT_MASK ((1UL << IPU3_L1PT_SHIFT) - 1)
+
+#define IPU3_MMU_ADDRESS_BITS (IPU3_PAGE_SHIFT + \
+ IPU3_L2PT_SHIFT + \
+ IPU3_L1PT_SHIFT)
+
+#define IMGU_REG_BASE 0x4000
+#define REG_TLB_INVALIDATE (IMGU_REG_BASE + 0x300)
+#define TLB_INVALIDATE 1
+#define REG_L1_PHYS (IMGU_REG_BASE + 0x304) /* 27-bit pfn */
+#define REG_GP_HALT (IMGU_REG_BASE + 0x5dc)
+#define REG_GP_HALTED (IMGU_REG_BASE + 0x5e0)
+
+struct ipu3_mmu {
+ struct device *dev;
+ void __iomem *base;
+ /* protect access to l2pts, l1pt */
+ spinlock_t lock;
+
+ void *dummy_page;
+ u32 dummy_page_pteval;
+
+ u32 *dummy_l2pt;
+ u32 dummy_l2pt_pteval;
+
+ u32 **l2pts;
+ u32 *l1pt;
+
+ struct ipu3_mmu_info geometry;
+};
+
+static inline struct ipu3_mmu *to_ipu3_mmu(struct ipu3_mmu_info *info)
+{
+ return container_of(info, struct ipu3_mmu, geometry);
+}
+
+/**
+ * ipu3_mmu_tlb_invalidate - invalidate translation look-aside buffer
+ * @mmu: MMU to perform the invalidate operation on
+ *
+ * This function invalidates the whole TLB. Must be called when the hardware
+ * is powered on.
+ */
+static void ipu3_mmu_tlb_invalidate(struct ipu3_mmu *mmu)
+{
+ writel(TLB_INVALIDATE, mmu->base + REG_TLB_INVALIDATE);
+}
+
+static void call_if_ipu3_is_powered(struct ipu3_mmu *mmu,
+ void (*func)(struct ipu3_mmu *mmu))
+{
+ pm_runtime_get_noresume(mmu->dev);
+ if (pm_runtime_active(mmu->dev))
+ func(mmu);
+ pm_runtime_put(mmu->dev);
+}
+
+/**
+ * ipu3_mmu_set_halt - set CIO gate halt bit
+ * @mmu: MMU to set the CIO gate bit in.
+ * @halt: Desired state of the gate bit.
+ *
+ * This function sets the CIO gate bit that controls whether external memory
+ * accesses are allowed. Must be called when the hardware is powered on.
+ */
+static void ipu3_mmu_set_halt(struct ipu3_mmu *mmu, bool halt)
+{
+ int ret;
+ u32 val;
+
+ writel(halt, mmu->base + REG_GP_HALT);
+ ret = readl_poll_timeout(mmu->base + REG_GP_HALTED,
+ val, (val & 1) == halt, 1000, 100000);
+
+ if (ret)
+ dev_err(mmu->dev, "failed to %s CIO gate halt\n",
+ halt ? "set" : "clear");
+}
+
+/**
+ * ipu3_mmu_alloc_page_table - allocate a pre-filled page table
+ * @pteval: Value to initialize for page table entries with.
+ *
+ * Return: Pointer to allocated page table or NULL on failure.
+ */
+static u32 *ipu3_mmu_alloc_page_table(u32 pteval)
+{
+ u32 *pt;
+ int pte;
+
+ pt = (u32 *)__get_free_page(GFP_KERNEL);
+ if (!pt)
+ return NULL;
+
+ for (pte = 0; pte < IPU3_PT_PTES; pte++)
+ pt[pte] = pteval;
+
+ set_memory_uc((unsigned long int)pt, IPU3_PT_ORDER);
+
+ return pt;
+}
+
+/**
+ * ipu3_mmu_free_page_table - free page table
+ * @pt: Page table to free.
+ */
+static void ipu3_mmu_free_page_table(u32 *pt)
+{
+ set_memory_wb((unsigned long int)pt, IPU3_PT_ORDER);
+ free_page((unsigned long)pt);
+}
+
+/**
+ * address_to_pte_idx - split IOVA into L1 and L2 page table indices
+ * @iova: IOVA to split.
+ * @l1pt_idx: Output for the L1 page table index.
+ * @l2pt_idx: Output for the L2 page index.
+ */
+static inline void address_to_pte_idx(unsigned long iova, u32 *l1pt_idx,
+ u32 *l2pt_idx)
+{
+ iova >>= IPU3_PAGE_SHIFT;
+
+ if (l2pt_idx)
+ *l2pt_idx = iova & IPU3_L2PT_MASK;
+
+ iova >>= IPU3_L2PT_SHIFT;
+
+ if (l1pt_idx)
+ *l1pt_idx = iova & IPU3_L1PT_MASK;
+}
+
+static u32 *ipu3_mmu_get_l2pt(struct ipu3_mmu *mmu, u32 l1pt_idx)
+{
+ unsigned long flags;
+ u32 *l2pt, *new_l2pt;
+ u32 pteval;
+
+ spin_lock_irqsave(&mmu->lock, flags);
+
+ l2pt = mmu->l2pts[l1pt_idx];
+ if (l2pt)
+ goto done;
+
+ spin_unlock_irqrestore(&mmu->lock, flags);
+
+ new_l2pt = ipu3_mmu_alloc_page_table(mmu->dummy_page_pteval);
+ if (!new_l2pt)
+ return NULL;
+
+ spin_lock_irqsave(&mmu->lock, flags);
+
+ dev_dbg(mmu->dev, "allocated page table %p for l1pt_idx %u\n",
+ new_l2pt, l1pt_idx);
+
+ l2pt = mmu->l2pts[l1pt_idx];
+ if (l2pt) {
+ ipu3_mmu_free_page_table(new_l2pt);
+ goto done;
+ }
+
+ l2pt = new_l2pt;
+ mmu->l2pts[l1pt_idx] = new_l2pt;
+
+ pteval = IPU3_ADDR2PTE(virt_to_phys(new_l2pt));
+ mmu->l1pt[l1pt_idx] = pteval;
+
+done:
+ spin_unlock_irqrestore(&mmu->lock, flags);
+ return l2pt;
+}
+
+static int __ipu3_mmu_map(struct ipu3_mmu *mmu, unsigned long iova,
+ phys_addr_t paddr)
+{
+ u32 l1pt_idx, l2pt_idx;
+ unsigned long flags;
+ u32 *l2pt;
+
+ if (!mmu)
+ return -ENODEV;
+
+ address_to_pte_idx(iova, &l1pt_idx, &l2pt_idx);
+
+ l2pt = ipu3_mmu_get_l2pt(mmu, l1pt_idx);
+ if (!l2pt)
+ return -ENOMEM;
+
+ spin_lock_irqsave(&mmu->lock, flags);
+
+ if (l2pt[l2pt_idx] != mmu->dummy_page_pteval) {
+ spin_unlock_irqrestore(&mmu->lock, flags);
+ return -EBUSY;
+ }
+
+ l2pt[l2pt_idx] = IPU3_ADDR2PTE(paddr);
+
+ spin_unlock_irqrestore(&mmu->lock, flags);
+
+ return 0;
+}
+
+/**
+ * The following four functions are implemented based on iommu.c
+ * drivers/iommu/iommu.c/iommu_pgsize().
+ */
+static size_t ipu3_mmu_pgsize(unsigned long pgsize_bitmap,
+ unsigned long addr_merge, size_t size)
+{
+ unsigned int pgsize_idx;
+ size_t pgsize;
+
+ /* Max page size that still fits into 'size' */
+ pgsize_idx = __fls(size);
+
+ /* need to consider alignment requirements ? */
+ if (likely(addr_merge)) {
+ /* Max page size allowed by address */
+ unsigned int align_pgsize_idx = __ffs(addr_merge);
+
+ pgsize_idx = min(pgsize_idx, align_pgsize_idx);
+ }
+
+ /* build a mask of acceptable page sizes */
+ pgsize = (1UL << (pgsize_idx + 1)) - 1;
+
+ /* throw away page sizes not supported by the hardware */
+ pgsize &= pgsize_bitmap;
+
+ /* make sure we're still sane */
+ WARN_ON(!pgsize);
+
+ /* pick the biggest page */
+ pgsize_idx = __fls(pgsize);
+ pgsize = 1UL << pgsize_idx;
+
+ return pgsize;
+}
+
+/* drivers/iommu/iommu.c/iommu_map() */
+int ipu3_mmu_map(struct ipu3_mmu_info *info, unsigned long iova,
+ phys_addr_t paddr, size_t size)
+{
+ struct ipu3_mmu *mmu = to_ipu3_mmu(info);
+ unsigned int min_pagesz;
+ int ret = 0;
+
+ /* find out the minimum page size supported */
+ min_pagesz = 1 << __ffs(mmu->geometry.pgsize_bitmap);
+
+ /*
+ * both the virtual address and the physical one, as well as
+ * the size of the mapping, must be aligned (at least) to the
+ * size of the smallest page supported by the hardware
+ */
+ if (!IS_ALIGNED(iova | paddr | size, min_pagesz)) {
+ dev_err(mmu->dev, "unaligned: iova 0x%lx pa %pa size 0x%zx min_pagesz 0x%x\n",
+ iova, &paddr, size, min_pagesz);
+ return -EINVAL;
+ }
+
+ dev_dbg(mmu->dev, "map: iova 0x%lx pa %pa size 0x%zx\n",
+ iova, &paddr, size);
+
+ while (size) {
+ size_t pgsize = ipu3_mmu_pgsize(mmu->geometry.pgsize_bitmap,
+ iova | paddr, size);
+
+ dev_dbg(mmu->dev, "mapping: iova 0x%lx pa %pa pgsize 0x%zx\n",
+ iova, &paddr, pgsize);
+
+ ret = __ipu3_mmu_map(mmu, iova, paddr);
+ if (ret)
+ break;
+
+ iova += pgsize;
+ paddr += pgsize;
+ size -= pgsize;
+ }
+
+ call_if_ipu3_is_powered(mmu, ipu3_mmu_tlb_invalidate);
+
+ return ret;
+}
+
+/* drivers/iommu/iommu.c/default_iommu_map_sg() */
+size_t ipu3_mmu_map_sg(struct ipu3_mmu_info *info, unsigned long iova,
+ struct scatterlist *sg, unsigned int nents)
+{
+ struct ipu3_mmu *mmu = to_ipu3_mmu(info);
+ struct scatterlist *s;
+ size_t s_length, mapped = 0;
+ unsigned int i, min_pagesz;
+ int ret;
+
+ min_pagesz = 1 << __ffs(mmu->geometry.pgsize_bitmap);
+
+ for_each_sg(sg, s, nents, i) {
+ phys_addr_t phys = page_to_phys(sg_page(s)) + s->offset;
+
+ s_length = s->length;
+
+ if (!IS_ALIGNED(s->offset, min_pagesz))
+ goto out_err;
+
+ /* must be min_pagesz aligned to be mapped singlely */
+ if (i == nents - 1 && !IS_ALIGNED(s->length, min_pagesz))
+ s_length = PAGE_ALIGN(s->length);
+
+ ret = ipu3_mmu_map(info, iova + mapped, phys, s_length);
+ if (ret)
+ goto out_err;
+
+ mapped += s_length;
+ }
+
+ call_if_ipu3_is_powered(mmu, ipu3_mmu_tlb_invalidate);
+
+ return mapped;
+
+out_err:
+ /* undo mappings already done */
+ ipu3_mmu_unmap(info, iova, mapped);
+
+ return 0;
+}
+
+static size_t __ipu3_mmu_unmap(struct ipu3_mmu *mmu,
+ unsigned long iova, size_t size)
+{
+ u32 l1pt_idx, l2pt_idx;
+ unsigned long flags;
+ size_t unmap = size;
+ u32 *l2pt;
+
+ if (!mmu)
+ return 0;
+
+ address_to_pte_idx(iova, &l1pt_idx, &l2pt_idx);
+
+ spin_lock_irqsave(&mmu->lock, flags);
+
+ l2pt = mmu->l2pts[l1pt_idx];
+ if (!l2pt) {
+ spin_unlock_irqrestore(&mmu->lock, flags);
+ return 0;
+ }
+
+ if (l2pt[l2pt_idx] == mmu->dummy_page_pteval)
+ unmap = 0;
+
+ l2pt[l2pt_idx] = mmu->dummy_page_pteval;
+
+ spin_unlock_irqrestore(&mmu->lock, flags);
+
+ return unmap;
+}
+
+/* drivers/iommu/iommu.c/iommu_unmap() */
+size_t ipu3_mmu_unmap(struct ipu3_mmu_info *info, unsigned long iova,
+ size_t size)
+{
+ struct ipu3_mmu *mmu = to_ipu3_mmu(info);
+ size_t unmapped_page, unmapped = 0;
+ unsigned int min_pagesz;
+
+ /* find out the minimum page size supported */
+ min_pagesz = 1 << __ffs(mmu->geometry.pgsize_bitmap);
+
+ /*
+ * The virtual address, as well as the size of the mapping, must be
+ * aligned (at least) to the size of the smallest page supported
+ * by the hardware
+ */
+ if (!IS_ALIGNED(iova | size, min_pagesz)) {
+ dev_err(mmu->dev, "unaligned: iova 0x%lx size 0x%zx min_pagesz 0x%x\n",
+ iova, size, min_pagesz);
+ return -EINVAL;
+ }
+
+ dev_dbg(mmu->dev, "unmap this: iova 0x%lx size 0x%zx\n", iova, size);
+
+ /*
+ * Keep iterating until we either unmap 'size' bytes (or more)
+ * or we hit an area that isn't mapped.
+ */
+ while (unmapped < size) {
+ size_t pgsize = ipu3_mmu_pgsize(mmu->geometry.pgsize_bitmap,
+ iova, size - unmapped);
+
+ unmapped_page = __ipu3_mmu_unmap(mmu, iova, pgsize);
+ if (!unmapped_page)
+ break;
+
+ dev_dbg(mmu->dev, "unmapped: iova 0x%lx size 0x%zx\n",
+ iova, unmapped_page);
+
+ iova += unmapped_page;
+ unmapped += unmapped_page;
+ }
+
+ call_if_ipu3_is_powered(mmu, ipu3_mmu_tlb_invalidate);
+
+ return unmapped;
+}
+
+/**
+ * ipu3_mmu_init() - initialize IPU3 MMU block
+ * @base: IOMEM base of hardware registers.
+ *
+ * Return: Pointer to IPU3 MMU private data pointer or ERR_PTR() on error.
+ */
+struct ipu3_mmu_info *ipu3_mmu_init(struct device *parent, void __iomem *base)
+{
+ struct ipu3_mmu *mmu;
+ u32 pteval;
+
+ mmu = kzalloc(sizeof(*mmu), GFP_KERNEL);
+ if (!mmu)
+ return ERR_PTR(-ENOMEM);
+
+ mmu->dev = parent;
+ mmu->base = base;
+ spin_lock_init(&mmu->lock);
+
+ /* Disallow external memory access when having no valid page tables. */
+ ipu3_mmu_set_halt(mmu, true);
+
+ /*
+ * The MMU does not have a "valid" bit, so we have to use a dummy
+ * page for invalid entries.
+ */
+ mmu->dummy_page = (void *)__get_free_page(GFP_KERNEL);
+ if (!mmu->dummy_page)
+ goto fail_group;
+ pteval = IPU3_ADDR2PTE(virt_to_phys(mmu->dummy_page));
+ mmu->dummy_page_pteval = pteval;
+
+ /*
+ * Allocate a dummy L2 page table with all entries pointing to
+ * the dummy page.
+ */
+ mmu->dummy_l2pt = ipu3_mmu_alloc_page_table(pteval);
+ if (!mmu->dummy_l2pt)
+ goto fail_dummy_page;
+ pteval = IPU3_ADDR2PTE(virt_to_phys(mmu->dummy_l2pt));
+ mmu->dummy_l2pt_pteval = pteval;
+
+ /*
+ * Allocate the array of L2PT CPU pointers, initialized to zero,
+ * which means the dummy L2PT allocated above.
+ */
+ mmu->l2pts = vzalloc(IPU3_PT_PTES * sizeof(*mmu->l2pts));
+ if (!mmu->l2pts)
+ goto fail_l2pt;
+
+ /* Allocate the L1 page table. */
+ mmu->l1pt = ipu3_mmu_alloc_page_table(mmu->dummy_l2pt_pteval);
+ if (!mmu->l1pt)
+ goto fail_l2pts;
+
+ pteval = IPU3_ADDR2PTE(virt_to_phys(mmu->l1pt));
+ writel(pteval, mmu->base + REG_L1_PHYS);
+ ipu3_mmu_tlb_invalidate(mmu);
+ ipu3_mmu_set_halt(mmu, false);
+
+ mmu->geometry.aperture_start = 0;
+ mmu->geometry.aperture_end = DMA_BIT_MASK(IPU3_MMU_ADDRESS_BITS);
+ mmu->geometry.pgsize_bitmap = IPU3_PAGE_SIZE;
+
+ return &mmu->geometry;
+
+fail_l2pts:
+ vfree(mmu->l2pts);
+fail_l2pt:
+ ipu3_mmu_free_page_table(mmu->dummy_l2pt);
+fail_dummy_page:
+ free_page((unsigned long)mmu->dummy_page);
+fail_group:
+ kfree(mmu);
+
+ return ERR_PTR(-ENOMEM);
+}
+
+/**
+ * ipu3_mmu_exit() - clean up IPU3 MMU block
+ * @mmu: IPU3 MMU private data
+ */
+void ipu3_mmu_exit(struct ipu3_mmu_info *info)
+{
+ struct ipu3_mmu *mmu = to_ipu3_mmu(info);
+
+ /* We are going to free our page tables, no more memory access. */
+ ipu3_mmu_set_halt(mmu, true);
+ ipu3_mmu_tlb_invalidate(mmu);
+
+ ipu3_mmu_free_page_table(mmu->l1pt);
+ vfree(mmu->l2pts);
+ ipu3_mmu_free_page_table(mmu->dummy_l2pt);
+ free_page((unsigned long)mmu->dummy_page);
+ kfree(mmu);
+}
+
+void ipu3_mmu_suspend(struct ipu3_mmu_info *info)
+{
+ struct ipu3_mmu *mmu = to_ipu3_mmu(info);
+
+ ipu3_mmu_set_halt(mmu, true);
+}
+
+void ipu3_mmu_resume(struct ipu3_mmu_info *info)
+{
+ struct ipu3_mmu *mmu = to_ipu3_mmu(info);
+ u32 pteval;
+
+ ipu3_mmu_set_halt(mmu, true);
+
+ pteval = IPU3_ADDR2PTE(virt_to_phys(mmu->l1pt));
+ writel(pteval, mmu->base + REG_L1_PHYS);
+
+ ipu3_mmu_tlb_invalidate(mmu);
+ ipu3_mmu_set_halt(mmu, false);
+}
diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.h b/drivers/media/pci/intel/ipu3/ipu3-mmu.h
new file mode 100644
index 0000000..8fe63b4
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+/* Copyright 2018 Google LLC. */
+
+#ifndef __IPU3_MMU_H
+#define __IPU3_MMU_H
+
+/**
+ * struct ipu3_mmu_info - Describes mmu geometry
+ *
+ * @aperture_start: First address that can be mapped
+ * @aperture_end: Last address that can be mapped
+ * @pgsize_bitmap: Bitmap of page sizes in use
+ */
+struct ipu3_mmu_info {
+ dma_addr_t aperture_start;
+ dma_addr_t aperture_end;
+ unsigned long pgsize_bitmap;
+};
+
+struct device;
+struct scatterlist;
+
+struct ipu3_mmu_info *ipu3_mmu_init(struct device *parent, void __iomem *base);
+void ipu3_mmu_exit(struct ipu3_mmu_info *info);
+void ipu3_mmu_suspend(struct ipu3_mmu_info *info);
+void ipu3_mmu_resume(struct ipu3_mmu_info *info);
+
+int ipu3_mmu_map(struct ipu3_mmu_info *info, unsigned long iova,
+ phys_addr_t paddr, size_t size);
+size_t ipu3_mmu_unmap(struct ipu3_mmu_info *info, unsigned long iova,
+ size_t size);
+size_t ipu3_mmu_map_sg(struct ipu3_mmu_info *info, unsigned long iova,
+ struct scatterlist *sg, unsigned int nents);
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
2018-10-29 22:23 ` [PATCH v7 06/16] intel-ipu3: mmu: Implement driver Yong Zhi
@ 2018-11-05 11:55 ` Sakari Ailus
2018-11-06 5:50 ` Zhi, Yong
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-05 11:55 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:23:00PM -0700, Yong Zhi wrote:
> From: Tomasz Figa <tfiga@chromium.org>
>
> This driver translates IO virtual address to physical
> address based on two levels page tables.
>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> ---
...
> +static void call_if_ipu3_is_powered(struct ipu3_mmu *mmu,
> + void (*func)(struct ipu3_mmu *mmu))
> +{
> + pm_runtime_get_noresume(mmu->dev);
> + if (pm_runtime_active(mmu->dev))
> + func(mmu);
> + pm_runtime_put(mmu->dev);
How about:
if (!pm_runtime_get_if_in_use(mmu->dev))
return;
func(mmu);
pm_runtime_put(mmu->dev);
> +}
--
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
2018-11-05 11:55 ` Sakari Ailus
@ 2018-11-06 5:50 ` Zhi, Yong
2018-11-06 5:56 ` Tomasz Figa
0 siblings, 1 reply; 123+ messages in thread
From: Zhi, Yong @ 2018-11-06 5:50 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi, Sakari,
Thanks for the feedback.
> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> Sent: Monday, November 5, 2018 3:55 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
>
> Hi Yong,
>
> On Mon, Oct 29, 2018 at 03:23:00PM -0700, Yong Zhi wrote:
> > From: Tomasz Figa <tfiga@chromium.org>
> >
> > This driver translates IO virtual address to physical address based on
> > two levels page tables.
> >
> > Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > ---
>
> ...
>
> > +static void call_if_ipu3_is_powered(struct ipu3_mmu *mmu,
> > + void (*func)(struct ipu3_mmu *mmu)) {
> > + pm_runtime_get_noresume(mmu->dev);
> > + if (pm_runtime_active(mmu->dev))
> > + func(mmu);
> > + pm_runtime_put(mmu->dev);
>
> How about:
>
> if (!pm_runtime_get_if_in_use(mmu->dev))
> return;
>
> func(mmu);
> pm_runtime_put(mmu->dev);
>
Ack, unless Tomasz has different opinion.
>
> > +}
>
> --
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
2018-11-06 5:50 ` Zhi, Yong
@ 2018-11-06 5:56 ` Tomasz Figa
0 siblings, 0 replies; 123+ messages in thread
From: Tomasz Figa @ 2018-11-06 5:56 UTC (permalink / raw)
To: Yong Zhi, Sakari Ailus
Cc: Linux Media Mailing List, Mauro Carvalho Chehab, Hans Verkuil,
Laurent Pinchart, Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao Bing Bu
On Tue, Nov 6, 2018 at 2:50 PM Zhi, Yong <yong.zhi@intel.com> wrote:
>
> Hi, Sakari,
>
> Thanks for the feedback.
>
> > -----Original Message-----
> > From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> > Sent: Monday, November 5, 2018 3:55 AM
> > To: Zhi, Yong <yong.zhi@intel.com>
> > Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> > mchehab@kernel.org; hans.verkuil@cisco.com;
> > laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> > <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> > Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> > <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> > Bingbu <bingbu.cao@intel.com>
> > Subject: Re: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
> >
> > Hi Yong,
> >
> > On Mon, Oct 29, 2018 at 03:23:00PM -0700, Yong Zhi wrote:
> > > From: Tomasz Figa <tfiga@chromium.org>
> > >
> > > This driver translates IO virtual address to physical address based on
> > > two levels page tables.
> > >
> > > Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > > ---
> >
> > ...
> >
> > > +static void call_if_ipu3_is_powered(struct ipu3_mmu *mmu,
> > > + void (*func)(struct ipu3_mmu *mmu)) {
> > > + pm_runtime_get_noresume(mmu->dev);
> > > + if (pm_runtime_active(mmu->dev))
> > > + func(mmu);
> > > + pm_runtime_put(mmu->dev);
> >
> > How about:
> >
> > if (!pm_runtime_get_if_in_use(mmu->dev))
> > return;
> >
> > func(mmu);
> > pm_runtime_put(mmu->dev);
> >
>
> Ack, unless Tomasz has different opinion.
It's actually the proper way of doing it. Thanks for the suggestion.
Best regards,
Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 07/16] intel-ipu3: Implement DMA mapping functions
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (5 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 06/16] intel-ipu3: mmu: Implement driver Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-10-29 22:23 ` [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions Yong Zhi
` (10 subsequent siblings)
17 siblings, 0 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
From: Tomasz Figa <tfiga@chromium.org>
This driver uses IOVA space for buffer mapping through IPU3 MMU
to transfer data between imaging pipelines and system DDR.
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 270 +++++++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 22 +++
2 files changed, 292 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-dmamap.c b/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
new file mode 100644
index 0000000..93a393d
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
@@ -0,0 +1,270 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018 Intel Corporation
+ * Copyright 2018 Google LLC.
+ *
+ * Author: Tomasz Figa <tfiga@chromium.org>
+ * Author: Yong Zhi <yong.zhi@intel.com>
+ */
+
+#include <linux/vmalloc.h>
+
+#include "ipu3.h"
+#include "ipu3-css-pool.h"
+#include "ipu3-mmu.h"
+
+/*
+ * Free a buffer allocated by ipu3_dmamap_alloc_buffer()
+ */
+static void ipu3_dmamap_free_buffer(struct page **pages,
+ size_t size)
+{
+ int count = size >> PAGE_SHIFT;
+
+ while (count--)
+ __free_page(pages[count]);
+ kvfree(pages);
+}
+
+/*
+ * Based on the implementation of __iommu_dma_alloc_pages()
+ * defined in drivers/iommu/dma-iommu.c
+ */
+static struct page **ipu3_dmamap_alloc_buffer(size_t size,
+ unsigned long order_mask,
+ gfp_t gfp)
+{
+ struct page **pages;
+ unsigned int i = 0, count = size >> PAGE_SHIFT;
+ const gfp_t high_order_gfp = __GFP_NOWARN | __GFP_NORETRY;
+
+ /* Allocate mem for array of page ptrs */
+ pages = kvmalloc_array(count, sizeof(*pages), GFP_KERNEL);
+
+ if (!pages)
+ return NULL;
+
+ order_mask &= (2U << MAX_ORDER) - 1;
+ if (!order_mask)
+ return NULL;
+
+ gfp |= __GFP_HIGHMEM | __GFP_ZERO;
+
+ while (count) {
+ struct page *page = NULL;
+ unsigned int order_size;
+
+ for (order_mask &= (2U << __fls(count)) - 1;
+ order_mask; order_mask &= ~order_size) {
+ unsigned int order = __fls(order_mask);
+
+ order_size = 1U << order;
+ page = alloc_pages((order_mask - order_size) ?
+ gfp | high_order_gfp : gfp, order);
+ if (!page)
+ continue;
+ if (!order)
+ break;
+ if (!PageCompound(page)) {
+ split_page(page, order);
+ break;
+ }
+
+ __free_pages(page, order);
+ }
+ if (!page) {
+ ipu3_dmamap_free_buffer(pages, i << PAGE_SHIFT);
+ return NULL;
+ }
+ count -= order_size;
+ while (order_size--)
+ pages[i++] = page++;
+ }
+
+ return pages;
+}
+
+/**
+ * ipu3_dmamap_alloc - allocate and map a buffer into KVA
+ * @imgu: struct device pointer
+ * @map: struct to store mapping variables
+ * @len: size required
+ *
+ * Returns:
+ * KVA on success
+ * %NULL on failure
+ */
+void *ipu3_dmamap_alloc(struct imgu_device *imgu, struct ipu3_css_map *map,
+ size_t len)
+{
+ unsigned long shift = iova_shift(&imgu->iova_domain);
+ unsigned int alloc_sizes = imgu->mmu->pgsize_bitmap;
+ struct device *dev = &imgu->pci_dev->dev;
+ size_t size = PAGE_ALIGN(len);
+ struct page **pages;
+ dma_addr_t iovaddr;
+ struct iova *iova;
+ int i, rval;
+
+ dev_dbg(dev, "%s: allocating %zu\n", __func__, size);
+
+ iova = alloc_iova(&imgu->iova_domain, size >> shift,
+ imgu->mmu->aperture_end >> shift, 0);
+ if (!iova)
+ return NULL;
+
+ pages = ipu3_dmamap_alloc_buffer(size, alloc_sizes >> PAGE_SHIFT,
+ GFP_KERNEL);
+ if (!pages)
+ goto out_free_iova;
+
+ /* Call IOMMU driver to setup pgt */
+ iovaddr = iova_dma_addr(&imgu->iova_domain, iova);
+ for (i = 0; i < size / PAGE_SIZE; ++i) {
+ rval = ipu3_mmu_map(imgu->mmu, iovaddr,
+ page_to_phys(pages[i]), PAGE_SIZE);
+ if (rval)
+ goto out_unmap;
+
+ iovaddr += PAGE_SIZE;
+ }
+
+ /* Now grab a virtual region */
+ map->vma = __get_vm_area(size, VM_USERMAP, VMALLOC_START, VMALLOC_END);
+ if (!map->vma)
+ goto out_unmap;
+
+ map->vma->pages = pages;
+ /* And map it in KVA */
+ if (map_vm_area(map->vma, PAGE_KERNEL, pages))
+ goto out_vunmap;
+
+ map->size = size;
+ map->daddr = iova_dma_addr(&imgu->iova_domain, iova);
+ map->vaddr = map->vma->addr;
+
+ dev_dbg(dev, "%s: allocated %zu @ IOVA %pad @ VA %p\n", __func__,
+ size, &map->daddr, map->vma->addr);
+
+ return map->vma->addr;
+
+out_vunmap:
+ vunmap(map->vma->addr);
+
+out_unmap:
+ ipu3_dmamap_free_buffer(pages, size);
+ ipu3_mmu_unmap(imgu->mmu, iova_dma_addr(&imgu->iova_domain, iova),
+ i * PAGE_SIZE);
+ map->vma = NULL;
+
+out_free_iova:
+ __free_iova(&imgu->iova_domain, iova);
+
+ return NULL;
+}
+
+void ipu3_dmamap_unmap(struct imgu_device *imgu, struct ipu3_css_map *map)
+{
+ struct iova *iova;
+
+ iova = find_iova(&imgu->iova_domain,
+ iova_pfn(&imgu->iova_domain, map->daddr));
+ if (WARN_ON(!iova))
+ return;
+
+ ipu3_mmu_unmap(imgu->mmu, iova_dma_addr(&imgu->iova_domain, iova),
+ iova_size(iova) << iova_shift(&imgu->iova_domain));
+
+ __free_iova(&imgu->iova_domain, iova);
+}
+
+/*
+ * Counterpart of ipu3_dmamap_alloc
+ */
+void ipu3_dmamap_free(struct imgu_device *imgu, struct ipu3_css_map *map)
+{
+ struct vm_struct *area = map->vma;
+
+ dev_dbg(&imgu->pci_dev->dev, "%s: freeing %zu @ IOVA %pad @ VA %p\n",
+ __func__, map->size, &map->daddr, map->vaddr);
+
+ if (!map->vaddr)
+ return;
+
+ ipu3_dmamap_unmap(imgu, map);
+
+ if (WARN_ON(!area) || WARN_ON(!area->pages))
+ return;
+
+ ipu3_dmamap_free_buffer(area->pages, map->size);
+ vunmap(map->vaddr);
+ map->vaddr = NULL;
+}
+
+int ipu3_dmamap_map_sg(struct imgu_device *imgu, struct scatterlist *sglist,
+ int nents, struct ipu3_css_map *map)
+{
+ unsigned long shift = iova_shift(&imgu->iova_domain);
+ struct scatterlist *sg;
+ struct iova *iova;
+ size_t size = 0;
+ int i;
+
+ for_each_sg(sglist, sg, nents, i) {
+ if (sg->offset)
+ return -EINVAL;
+
+ if (i != nents - 1 && !PAGE_ALIGNED(sg->length))
+ return -EINVAL;
+
+ size += sg->length;
+ }
+
+ size = iova_align(&imgu->iova_domain, size);
+ dev_dbg(&imgu->pci_dev->dev, "dmamap: mapping sg %d entries, %zu pages\n",
+ nents, size >> shift);
+
+ iova = alloc_iova(&imgu->iova_domain, size >> shift,
+ imgu->mmu->aperture_end >> shift, 0);
+ if (!iova)
+ return -ENOMEM;
+
+ dev_dbg(&imgu->pci_dev->dev, "dmamap: iova low pfn %lu, high pfn %lu\n",
+ iova->pfn_lo, iova->pfn_hi);
+
+ if (ipu3_mmu_map_sg(imgu->mmu, iova_dma_addr(&imgu->iova_domain, iova),
+ sglist, nents) < size)
+ goto out_fail;
+
+ memset(map, 0, sizeof(*map));
+ map->daddr = iova_dma_addr(&imgu->iova_domain, iova);
+ map->size = size;
+
+ return 0;
+
+out_fail:
+ __free_iova(&imgu->iova_domain, iova);
+
+ return -EFAULT;
+}
+
+int ipu3_dmamap_init(struct imgu_device *imgu)
+{
+ unsigned long order, base_pfn;
+ int ret = iova_cache_get();
+
+ if (ret)
+ return ret;
+
+ order = __ffs(imgu->mmu->pgsize_bitmap);
+ base_pfn = max_t(unsigned long, 1, imgu->mmu->aperture_start >> order);
+ init_iova_domain(&imgu->iova_domain, 1UL << order, base_pfn);
+
+ return 0;
+}
+
+void ipu3_dmamap_exit(struct imgu_device *imgu)
+{
+ put_iova_domain(&imgu->iova_domain);
+ iova_cache_put();
+}
diff --git a/drivers/media/pci/intel/ipu3/ipu3-dmamap.h b/drivers/media/pci/intel/ipu3/ipu3-dmamap.h
new file mode 100644
index 0000000..b9d224a
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-dmamap.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+/* Copyright 2018 Google LLC. */
+
+#ifndef __IPU3_DMAMAP_H
+#define __IPU3_DMAMAP_H
+
+struct imgu_device;
+struct scatterlist;
+
+void *ipu3_dmamap_alloc(struct imgu_device *imgu, struct ipu3_css_map *map,
+ size_t len);
+void ipu3_dmamap_free(struct imgu_device *imgu, struct ipu3_css_map *map);
+
+int ipu3_dmamap_map_sg(struct imgu_device *imgu, struct scatterlist *sglist,
+ int nents, struct ipu3_css_map *map);
+void ipu3_dmamap_unmap(struct imgu_device *imgu, struct ipu3_css_map *map);
+
+int ipu3_dmamap_init(struct imgu_device *imgu);
+void ipu3_dmamap_exit(struct imgu_device *imgu);
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (6 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 07/16] intel-ipu3: Implement DMA mapping functions Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-08 15:36 ` Sakari Ailus
2018-10-29 22:23 ` [PATCH v7 09/16] intel-ipu3: css: Add support for firmware management Yong Zhi
` (9 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
The pools are used to store previous parameters set by
user with the parameter queue. Due to pipelining,
there needs to be multiple sets (up to four)
of parameters which are queued in a host-to-sp queue.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 136 +++++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 56 +++++++++++
2 files changed, 192 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.c b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
new file mode 100644
index 0000000..eab41c3
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
@@ -0,0 +1,136 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Intel Corporation
+
+#include <linux/device.h>
+
+#include "ipu3.h"
+#include "ipu3-css-pool.h"
+#include "ipu3-dmamap.h"
+
+int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
+ struct ipu3_css_map *map, size_t size)
+{
+ if (map->size < size && map->vaddr) {
+ dev_warn(&imgu->pci_dev->dev, "dma buf resized from %zu to %zu",
+ map->size, size);
+
+ ipu3_dmamap_free(imgu, map);
+ if (!ipu3_dmamap_alloc(imgu, map, size))
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+void ipu3_css_pool_cleanup(struct imgu_device *imgu, struct ipu3_css_pool *pool)
+{
+ unsigned int i;
+
+ for (i = 0; i < IPU3_CSS_POOL_SIZE; i++)
+ ipu3_dmamap_free(imgu, &pool->entry[i].param);
+}
+
+int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool *pool,
+ size_t size)
+{
+ unsigned int i;
+
+ for (i = 0; i < IPU3_CSS_POOL_SIZE; i++) {
+ /*
+ * entry[i].framenum is initialized to INT_MIN so that
+ * ipu3_css_pool_check() can treat it as usesable slot.
+ */
+ pool->entry[i].framenum = INT_MIN;
+
+ if (size == 0) {
+ pool->entry[i].param.vaddr = NULL;
+ continue;
+ }
+
+ if (!ipu3_dmamap_alloc(imgu, &pool->entry[i].param, size))
+ goto fail;
+ }
+
+ pool->last = IPU3_CSS_POOL_SIZE;
+
+ return 0;
+
+fail:
+ ipu3_css_pool_cleanup(imgu, pool);
+ return -ENOMEM;
+}
+
+/*
+ * Check that the following call to pool_get succeeds.
+ * Return negative on error.
+ */
+static int ipu3_css_pool_check(struct ipu3_css_pool *pool, long framenum)
+{
+ /* Get the oldest entry */
+ int n = (pool->last + 1) % IPU3_CSS_POOL_SIZE;
+ long diff = framenum - pool->entry[n].framenum;
+
+ /* if framenum wraps around and becomes smaller than entry n */
+ if (diff < 0)
+ diff += LONG_MAX;
+
+ /*
+ * pool->entry[n].framenum stores the frame number where that
+ * entry was allocated. If that was allocated more than POOL_SIZE
+ * frames back, it is old enough that we know it is no more in
+ * use by firmware.
+ */
+ if (diff > IPU3_CSS_POOL_SIZE)
+ return n;
+
+ return -ENOSPC;
+}
+
+/*
+ * Allocate a new parameter from pool at frame number `framenum'.
+ * Release the oldest entry in the pool to make space for the new entry.
+ * Return negative on error.
+ */
+int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum)
+{
+ int n = ipu3_css_pool_check(pool, framenum);
+
+ if (n < 0)
+ return n;
+
+ pool->entry[n].framenum = framenum;
+ pool->last = n;
+
+ return n;
+}
+
+/*
+ * Undo, for all practical purposes, the effect of pool_get().
+ */
+void ipu3_css_pool_put(struct ipu3_css_pool *pool)
+{
+ pool->entry[pool->last].framenum = INT_MIN;
+ pool->last = (pool->last + IPU3_CSS_POOL_SIZE - 1) % IPU3_CSS_POOL_SIZE;
+}
+
+/**
+ * ipu3_css_pool_last - Retrieve the nth pool entry from last
+ *
+ * @pool: a pointer to &struct ipu3_css_pool.
+ * @n: the distance to the last index.
+ *
+ * Return: The nth entry from last or null map to indicate no frame stored.
+ */
+const struct ipu3_css_map *
+ipu3_css_pool_last(struct ipu3_css_pool *pool, unsigned int n)
+{
+ static const struct ipu3_css_map null_map = { 0 };
+ int i = (pool->last + IPU3_CSS_POOL_SIZE - n) % IPU3_CSS_POOL_SIZE;
+
+ WARN_ON(n >= IPU3_CSS_POOL_SIZE);
+
+ if (pool->entry[i].framenum < 0)
+ return &null_map;
+
+ return &pool->entry[i].param;
+}
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.h b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
new file mode 100644
index 0000000..71e48d1
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+
+#ifndef __IPU3_UTIL_H
+#define __IPU3_UTIL_H
+
+struct device;
+struct imgu_device;
+
+#define IPU3_CSS_POOL_SIZE 4
+
+/**
+ * ipu3_css_map - store DMA mapping info for buffer
+ *
+ * @size: size of the buffer in bytes.
+ * @vaddr: kernel virtual address.
+ * @daddr: iova dma address to access IPU3.
+ * @vma: private, a pointer to &struct vm_struct,
+ * used for ipu3_dmamap_free.
+ */
+struct ipu3_css_map {
+ size_t size;
+ void *vaddr;
+ dma_addr_t daddr;
+ struct vm_struct *vma;
+};
+
+/**
+ * ipu3_css_pool - circular buffer pool definition
+ *
+ * @entry: array with IPU3_CSS_POOL_SIZE elements.
+ * @entry.param: a &struct ipu3_css_map for storing the mem mapping.
+ * @entry.framenum: the css frame number, used to determine if the entry
+ * is old enough to be recycled.
+ * @last: write pointer, initialized to IPU3_CSS_POOL_SIZE.
+ */
+struct ipu3_css_pool {
+ struct {
+ struct ipu3_css_map param;
+ long framenum;
+ } entry[IPU3_CSS_POOL_SIZE];
+ unsigned int last; /* Latest entry */
+};
+
+int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
+ struct ipu3_css_map *map, size_t size);
+void ipu3_css_pool_cleanup(struct imgu_device *imgu,
+ struct ipu3_css_pool *pool);
+int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool *pool,
+ size_t size);
+int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum);
+void ipu3_css_pool_put(struct ipu3_css_pool *pool);
+const struct ipu3_css_map *ipu3_css_pool_last(struct ipu3_css_pool *pool,
+ unsigned int last);
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions
2018-10-29 22:23 ` [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions Yong Zhi
@ 2018-11-08 15:36 ` Sakari Ailus
2018-11-09 23:16 ` Zhi, Yong
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-08 15:36 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:23:02PM -0700, Yong Zhi wrote:
> The pools are used to store previous parameters set by
> user with the parameter queue. Due to pipelining,
> there needs to be multiple sets (up to four)
> of parameters which are queued in a host-to-sp queue.
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> ---
> drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 136 +++++++++++++++++++++++++++
> drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 56 +++++++++++
> 2 files changed, 192 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
>
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.c b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> new file mode 100644
> index 0000000..eab41c3
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> @@ -0,0 +1,136 @@
> +// SPDX-License-Identifier: GPL-2.0
> +// Copyright (C) 2018 Intel Corporation
> +
> +#include <linux/device.h>
> +
> +#include "ipu3.h"
> +#include "ipu3-css-pool.h"
> +#include "ipu3-dmamap.h"
> +
> +int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
> + struct ipu3_css_map *map, size_t size)
> +{
> + if (map->size < size && map->vaddr) {
> + dev_warn(&imgu->pci_dev->dev, "dma buf resized from %zu to %zu",
> + map->size, size);
> +
> + ipu3_dmamap_free(imgu, map);
> + if (!ipu3_dmamap_alloc(imgu, map, size))
> + return -ENOMEM;
> + }
> +
> + return 0;
> +}
> +
> +void ipu3_css_pool_cleanup(struct imgu_device *imgu, struct ipu3_css_pool *pool)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < IPU3_CSS_POOL_SIZE; i++)
> + ipu3_dmamap_free(imgu, &pool->entry[i].param);
> +}
> +
> +int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool *pool,
> + size_t size)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < IPU3_CSS_POOL_SIZE; i++) {
> + /*
> + * entry[i].framenum is initialized to INT_MIN so that
> + * ipu3_css_pool_check() can treat it as usesable slot.
> + */
> + pool->entry[i].framenum = INT_MIN;
> +
> + if (size == 0) {
> + pool->entry[i].param.vaddr = NULL;
> + continue;
> + }
> +
> + if (!ipu3_dmamap_alloc(imgu, &pool->entry[i].param, size))
> + goto fail;
> + }
> +
> + pool->last = IPU3_CSS_POOL_SIZE;
> +
> + return 0;
> +
> +fail:
> + ipu3_css_pool_cleanup(imgu, pool);
> + return -ENOMEM;
> +}
> +
> +/*
> + * Check that the following call to pool_get succeeds.
> + * Return negative on error.
> + */
> +static int ipu3_css_pool_check(struct ipu3_css_pool *pool, long framenum)
> +{
> + /* Get the oldest entry */
> + int n = (pool->last + 1) % IPU3_CSS_POOL_SIZE;
> + long diff = framenum - pool->entry[n].framenum;
> +
> + /* if framenum wraps around and becomes smaller than entry n */
> + if (diff < 0)
> + diff += LONG_MAX;
Have you tested the wrap-around? As a result, the value of the diff is
between -1 and LONG_MAX - 1 (without considering more than just the two
lines above). Is that intended?
You seem to be using different types for the frame number; sometimes int,
sometimes long. Could you align that, preferrably to an unsigned type? u32
would probably be a sound choice.
The entry (index to pool->entry array) should be unsigned as well.
> +
> + /*
> + * pool->entry[n].framenum stores the frame number where that
> + * entry was allocated. If that was allocated more than POOL_SIZE
> + * frames back, it is old enough that we know it is no more in
> + * use by firmware.
> + */
> + if (diff > IPU3_CSS_POOL_SIZE)
> + return n;
> +
> + return -ENOSPC;
> +}
> +
> +/*
> + * Allocate a new parameter from pool at frame number `framenum'.
> + * Release the oldest entry in the pool to make space for the new entry.
> + * Return negative on error.
> + */
> +int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum)
> +{
> + int n = ipu3_css_pool_check(pool, framenum);
> +
> + if (n < 0)
> + return n;
> +
> + pool->entry[n].framenum = framenum;
> + pool->last = n;
> +
> + return n;
> +}
> +
> +/*
> + * Undo, for all practical purposes, the effect of pool_get().
> + */
> +void ipu3_css_pool_put(struct ipu3_css_pool *pool)
> +{
> + pool->entry[pool->last].framenum = INT_MIN;
> + pool->last = (pool->last + IPU3_CSS_POOL_SIZE - 1) % IPU3_CSS_POOL_SIZE;
> +}
> +
> +/**
> + * ipu3_css_pool_last - Retrieve the nth pool entry from last
> + *
> + * @pool: a pointer to &struct ipu3_css_pool.
> + * @n: the distance to the last index.
> + *
> + * Return: The nth entry from last or null map to indicate no frame stored.
> + */
> +const struct ipu3_css_map *
> +ipu3_css_pool_last(struct ipu3_css_pool *pool, unsigned int n)
> +{
> + static const struct ipu3_css_map null_map = { 0 };
> + int i = (pool->last + IPU3_CSS_POOL_SIZE - n) % IPU3_CSS_POOL_SIZE;
> +
> + WARN_ON(n >= IPU3_CSS_POOL_SIZE);
> +
> + if (pool->entry[i].framenum < 0)
> + return &null_map;
> +
> + return &pool->entry[i].param;
> +}
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.h b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> new file mode 100644
> index 0000000..71e48d1
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> @@ -0,0 +1,56 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2018 Intel Corporation */
> +
> +#ifndef __IPU3_UTIL_H
> +#define __IPU3_UTIL_H
> +
> +struct device;
> +struct imgu_device;
> +
> +#define IPU3_CSS_POOL_SIZE 4
> +
> +/**
> + * ipu3_css_map - store DMA mapping info for buffer
> + *
> + * @size: size of the buffer in bytes.
> + * @vaddr: kernel virtual address.
> + * @daddr: iova dma address to access IPU3.
> + * @vma: private, a pointer to &struct vm_struct,
> + * used for ipu3_dmamap_free.
> + */
> +struct ipu3_css_map {
> + size_t size;
> + void *vaddr;
> + dma_addr_t daddr;
> + struct vm_struct *vma;
> +};
> +
> +/**
> + * ipu3_css_pool - circular buffer pool definition
> + *
> + * @entry: array with IPU3_CSS_POOL_SIZE elements.
> + * @entry.param: a &struct ipu3_css_map for storing the mem mapping.
> + * @entry.framenum: the css frame number, used to determine if the entry
> + * is old enough to be recycled.
> + * @last: write pointer, initialized to IPU3_CSS_POOL_SIZE.
> + */
> +struct ipu3_css_pool {
> + struct {
> + struct ipu3_css_map param;
> + long framenum;
> + } entry[IPU3_CSS_POOL_SIZE];
> + unsigned int last; /* Latest entry */
> +};
> +
> +int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
> + struct ipu3_css_map *map, size_t size);
> +void ipu3_css_pool_cleanup(struct imgu_device *imgu,
> + struct ipu3_css_pool *pool);
> +int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool *pool,
> + size_t size);
> +int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum);
> +void ipu3_css_pool_put(struct ipu3_css_pool *pool);
> +const struct ipu3_css_map *ipu3_css_pool_last(struct ipu3_css_pool *pool,
> + unsigned int last);
> +
> +#endif
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions
2018-11-08 15:36 ` Sakari Ailus
@ 2018-11-09 23:16 ` Zhi, Yong
2018-11-12 9:21 ` Sakari Ailus
0 siblings, 1 reply; 123+ messages in thread
From: Zhi, Yong @ 2018-11-09 23:16 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi, Sakari,
> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> Sent: Thursday, November 8, 2018 9:36 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility
> functions
>
> Hi Yong,
>
> On Mon, Oct 29, 2018 at 03:23:02PM -0700, Yong Zhi wrote:
> > The pools are used to store previous parameters set by user with the
> > parameter queue. Due to pipelining, there needs to be multiple sets
> > (up to four) of parameters which are queued in a host-to-sp queue.
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > ---
> > drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 136
> > +++++++++++++++++++++++++++
> > drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 56 +++++++++++
> > 2 files changed, 192 insertions(+)
> > create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> >
> > diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > new file mode 100644
> > index 0000000..eab41c3
> > --- /dev/null
> > +++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > @@ -0,0 +1,136 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +// Copyright (C) 2018 Intel Corporation
> > +
> > +#include <linux/device.h>
> > +
> > +#include "ipu3.h"
> > +#include "ipu3-css-pool.h"
> > +#include "ipu3-dmamap.h"
> > +
> > +int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
> > + struct ipu3_css_map *map, size_t size) {
> > + if (map->size < size && map->vaddr) {
> > + dev_warn(&imgu->pci_dev->dev, "dma buf resized from %zu
> to %zu",
> > + map->size, size);
> > +
> > + ipu3_dmamap_free(imgu, map);
> > + if (!ipu3_dmamap_alloc(imgu, map, size))
> > + return -ENOMEM;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +void ipu3_css_pool_cleanup(struct imgu_device *imgu, struct
> > +ipu3_css_pool *pool) {
> > + unsigned int i;
> > +
> > + for (i = 0; i < IPU3_CSS_POOL_SIZE; i++)
> > + ipu3_dmamap_free(imgu, &pool->entry[i].param); }
> > +
> > +int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool
> *pool,
> > + size_t size)
> > +{
> > + unsigned int i;
> > +
> > + for (i = 0; i < IPU3_CSS_POOL_SIZE; i++) {
> > + /*
> > + * entry[i].framenum is initialized to INT_MIN so that
> > + * ipu3_css_pool_check() can treat it as usesable slot.
> > + */
> > + pool->entry[i].framenum = INT_MIN;
> > +
> > + if (size == 0) {
> > + pool->entry[i].param.vaddr = NULL;
> > + continue;
> > + }
> > +
> > + if (!ipu3_dmamap_alloc(imgu, &pool->entry[i].param, size))
> > + goto fail;
> > + }
> > +
> > + pool->last = IPU3_CSS_POOL_SIZE;
> > +
> > + return 0;
> > +
> > +fail:
> > + ipu3_css_pool_cleanup(imgu, pool);
> > + return -ENOMEM;
> > +}
> > +
> > +/*
> > + * Check that the following call to pool_get succeeds.
> > + * Return negative on error.
> > + */
> > +static int ipu3_css_pool_check(struct ipu3_css_pool *pool, long
> > +framenum) {
> > + /* Get the oldest entry */
> > + int n = (pool->last + 1) % IPU3_CSS_POOL_SIZE;
> > + long diff = framenum - pool->entry[n].framenum;
> > +
> > + /* if framenum wraps around and becomes smaller than entry n */
> > + if (diff < 0)
> > + diff += LONG_MAX;
>
> Have you tested the wrap-around? As a result, the value of the diff is
> between -1 and LONG_MAX - 1 (without considering more than just the two
> lines above). Is that intended?
>
Yes, I simulated wrap-around using a smaller limit in v5.
> You seem to be using different types for the frame number; sometimes int,
> sometimes long. Could you align that, preferrably to an unsigned type? u32
> would probably be a sound choice.
>
Will use u32 at places except entry.framenum, which is initialized to INT_MIN. This is because the frame is counted from 0 at stream start, and the entry.framenum must be smaller enough for the ipu3_css_pool_check() to not return -ENOSPC.
> The entry (index to pool->entry array) should be unsigned as well.
Ack, will refactor the code a little to make this function cleaner.
Thanks,
Yong
>
> > +
> > + /*
> > + * pool->entry[n].framenum stores the frame number where that
> > + * entry was allocated. If that was allocated more than POOL_SIZE
> > + * frames back, it is old enough that we know it is no more in
> > + * use by firmware.
> > + */
> > + if (diff > IPU3_CSS_POOL_SIZE)
> > + return n;
> > +
> > + return -ENOSPC;
> > +}
> > +
> > +/*
> > + * Allocate a new parameter from pool at frame number `framenum'.
> > + * Release the oldest entry in the pool to make space for the new entry.
> > + * Return negative on error.
> > + */
> > +int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum) {
> > + int n = ipu3_css_pool_check(pool, framenum);
> > +
> > + if (n < 0)
> > + return n;
> > +
> > + pool->entry[n].framenum = framenum;
> > + pool->last = n;
> > +
> > + return n;
> > +}
> > +
> > +/*
> > + * Undo, for all practical purposes, the effect of pool_get().
> > + */
> > +void ipu3_css_pool_put(struct ipu3_css_pool *pool) {
> > + pool->entry[pool->last].framenum = INT_MIN;
> > + pool->last = (pool->last + IPU3_CSS_POOL_SIZE - 1) %
> > +IPU3_CSS_POOL_SIZE; }
> > +
> > +/**
> > + * ipu3_css_pool_last - Retrieve the nth pool entry from last
> > + *
> > + * @pool: a pointer to &struct ipu3_css_pool.
> > + * @n: the distance to the last index.
> > + *
> > + * Return: The nth entry from last or null map to indicate no frame stored.
> > + */
> > +const struct ipu3_css_map *
> > +ipu3_css_pool_last(struct ipu3_css_pool *pool, unsigned int n) {
> > + static const struct ipu3_css_map null_map = { 0 };
> > + int i = (pool->last + IPU3_CSS_POOL_SIZE - n) % IPU3_CSS_POOL_SIZE;
> > +
> > + WARN_ON(n >= IPU3_CSS_POOL_SIZE);
> > +
> > + if (pool->entry[i].framenum < 0)
> > + return &null_map;
> > +
> > + return &pool->entry[i].param;
> > +}
> > diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> > b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> > new file mode 100644
> > index 0000000..71e48d1
> > --- /dev/null
> > +++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> > @@ -0,0 +1,56 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2018 Intel Corporation */
> > +
> > +#ifndef __IPU3_UTIL_H
> > +#define __IPU3_UTIL_H
> > +
> > +struct device;
> > +struct imgu_device;
> > +
> > +#define IPU3_CSS_POOL_SIZE 4
> > +
> > +/**
> > + * ipu3_css_map - store DMA mapping info for buffer
> > + *
> > + * @size: size of the buffer in bytes.
> > + * @vaddr: kernel virtual address.
> > + * @daddr: iova dma address to access IPU3.
> > + * @vma: private, a pointer to &struct vm_struct,
> > + * used for ipu3_dmamap_free.
> > + */
> > +struct ipu3_css_map {
> > + size_t size;
> > + void *vaddr;
> > + dma_addr_t daddr;
> > + struct vm_struct *vma;
> > +};
> > +
> > +/**
> > + * ipu3_css_pool - circular buffer pool definition
> > + *
> > + * @entry: array with IPU3_CSS_POOL_SIZE elements.
> > + * @entry.param: a &struct ipu3_css_map for storing the mem
> mapping.
> > + * @entry.framenum: the css frame number, used to determine if
> the entry
> > + * is old enough to be recycled.
> > + * @last: write pointer, initialized to IPU3_CSS_POOL_SIZE.
> > + */
> > +struct ipu3_css_pool {
> > + struct {
> > + struct ipu3_css_map param;
> > + long framenum;
> > + } entry[IPU3_CSS_POOL_SIZE];
> > + unsigned int last; /* Latest entry */ };
> > +
> > +int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
> > + struct ipu3_css_map *map, size_t size); void
> > +ipu3_css_pool_cleanup(struct imgu_device *imgu,
> > + struct ipu3_css_pool *pool);
> > +int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool
> *pool,
> > + size_t size);
> > +int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum);
> > +void ipu3_css_pool_put(struct ipu3_css_pool *pool); const struct
> > +ipu3_css_map *ipu3_css_pool_last(struct ipu3_css_pool *pool,
> > + unsigned int last);
> > +
> > +#endif
>
> --
> Regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions
2018-11-09 23:16 ` Zhi, Yong
@ 2018-11-12 9:21 ` Sakari Ailus
0 siblings, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-11-12 9:21 UTC (permalink / raw)
To: Zhi, Yong
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi Yong,
On Fri, Nov 09, 2018 at 11:16:44PM +0000, Zhi, Yong wrote:
> Hi, Sakari,
>
> > -----Original Message-----
> > From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> > Sent: Thursday, November 8, 2018 9:36 AM
> > To: Zhi, Yong <yong.zhi@intel.com>
> > Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> > mchehab@kernel.org; hans.verkuil@cisco.com;
> > laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> > <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> > Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> > <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> > Bingbu <bingbu.cao@intel.com>
> > Subject: Re: [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility
> > functions
> >
> > Hi Yong,
> >
> > On Mon, Oct 29, 2018 at 03:23:02PM -0700, Yong Zhi wrote:
> > > The pools are used to store previous parameters set by user with the
> > > parameter queue. Due to pipelining, there needs to be multiple sets
> > > (up to four) of parameters which are queued in a host-to-sp queue.
> > >
> > > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > > ---
> > > drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 136
> > > +++++++++++++++++++++++++++
> > > drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 56 +++++++++++
> > > 2 files changed, 192 insertions(+)
> > > create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > > create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> > >
> > > diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > > b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > > new file mode 100644
> > > index 0000000..eab41c3
> > > --- /dev/null
> > > +++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> > > @@ -0,0 +1,136 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +// Copyright (C) 2018 Intel Corporation
> > > +
> > > +#include <linux/device.h>
> > > +
> > > +#include "ipu3.h"
> > > +#include "ipu3-css-pool.h"
> > > +#include "ipu3-dmamap.h"
> > > +
> > > +int ipu3_css_dma_buffer_resize(struct imgu_device *imgu,
> > > + struct ipu3_css_map *map, size_t size) {
> > > + if (map->size < size && map->vaddr) {
> > > + dev_warn(&imgu->pci_dev->dev, "dma buf resized from %zu
> > to %zu",
> > > + map->size, size);
> > > +
> > > + ipu3_dmamap_free(imgu, map);
> > > + if (!ipu3_dmamap_alloc(imgu, map, size))
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +void ipu3_css_pool_cleanup(struct imgu_device *imgu, struct
> > > +ipu3_css_pool *pool) {
> > > + unsigned int i;
> > > +
> > > + for (i = 0; i < IPU3_CSS_POOL_SIZE; i++)
> > > + ipu3_dmamap_free(imgu, &pool->entry[i].param); }
> > > +
> > > +int ipu3_css_pool_init(struct imgu_device *imgu, struct ipu3_css_pool
> > *pool,
> > > + size_t size)
> > > +{
> > > + unsigned int i;
> > > +
> > > + for (i = 0; i < IPU3_CSS_POOL_SIZE; i++) {
> > > + /*
> > > + * entry[i].framenum is initialized to INT_MIN so that
> > > + * ipu3_css_pool_check() can treat it as usesable slot.
> > > + */
> > > + pool->entry[i].framenum = INT_MIN;
> > > +
> > > + if (size == 0) {
> > > + pool->entry[i].param.vaddr = NULL;
> > > + continue;
> > > + }
> > > +
> > > + if (!ipu3_dmamap_alloc(imgu, &pool->entry[i].param, size))
> > > + goto fail;
> > > + }
> > > +
> > > + pool->last = IPU3_CSS_POOL_SIZE;
> > > +
> > > + return 0;
> > > +
> > > +fail:
> > > + ipu3_css_pool_cleanup(imgu, pool);
> > > + return -ENOMEM;
> > > +}
> > > +
> > > +/*
> > > + * Check that the following call to pool_get succeeds.
> > > + * Return negative on error.
> > > + */
> > > +static int ipu3_css_pool_check(struct ipu3_css_pool *pool, long
> > > +framenum) {
> > > + /* Get the oldest entry */
> > > + int n = (pool->last + 1) % IPU3_CSS_POOL_SIZE;
> > > + long diff = framenum - pool->entry[n].framenum;
> > > +
> > > + /* if framenum wraps around and becomes smaller than entry n */
> > > + if (diff < 0)
> > > + diff += LONG_MAX;
> >
> > Have you tested the wrap-around? As a result, the value of the diff is
> > between -1 and LONG_MAX - 1 (without considering more than just the two
> > lines above). Is that intended?
> >
>
> Yes, I simulated wrap-around using a smaller limit in v5.
>
> > You seem to be using different types for the frame number; sometimes int,
> > sometimes long. Could you align that, preferrably to an unsigned type? u32
> > would probably be a sound choice.
> >
>
> Will use u32 at places except entry.framenum, which is initialized to
> INT_MIN. This is because the frame is counted from 0 at stream start, and
> the entry.framenum must be smaller enough for the ipu3_css_pool_check()
> to not return -ENOSPC.
You could use another field to tell whether an entry is valid or not.
That'd simplify the code, as well as remove the need to cap the frame
number to an arbitrary value.
--
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 09/16] intel-ipu3: css: Add support for firmware management
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (7 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 08/16] intel-ipu3: css: Add dma buff pool utility functions Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-28 22:22 ` Sakari Ailus
2018-10-29 22:23 ` [PATCH v7 11/16] intel-ipu3: css: Compute and program ccs Yong Zhi
` (8 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
Introduce functions to load and install ImgU FW blobs.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 264 +++++++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 188 ++++++++++++++++++++
2 files changed, 452 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-fw.c b/drivers/media/pci/intel/ipu3/ipu3-css-fw.c
new file mode 100644
index 0000000..ba459e9
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-fw.c
@@ -0,0 +1,264 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Intel Corporation
+
+#include <linux/device.h>
+#include <linux/firmware.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include "ipu3-css.h"
+#include "ipu3-css-fw.h"
+#include "ipu3-dmamap.h"
+
+static void ipu3_css_fw_show_binary(struct device *dev, struct imgu_fw_info *bi,
+ const char *name)
+{
+ unsigned int i;
+
+ dev_dbg(dev, "found firmware binary type %i size %i name %s\n",
+ bi->type, bi->blob.size, name);
+ if (bi->type != IMGU_FW_ISP_FIRMWARE)
+ return;
+
+ dev_dbg(dev, " id %i mode %i bds 0x%x veceven %i/%i out_pins %i\n",
+ bi->info.isp.sp.id, bi->info.isp.sp.pipeline.mode,
+ bi->info.isp.sp.bds.supported_bds_factors,
+ bi->info.isp.sp.enable.vf_veceven,
+ bi->info.isp.sp.vf_dec.is_variable,
+ bi->info.isp.num_output_pins);
+
+ dev_dbg(dev, " input (%i,%i)-(%i,%i) formats %s%s%s\n",
+ bi->info.isp.sp.input.min_width,
+ bi->info.isp.sp.input.min_height,
+ bi->info.isp.sp.input.max_width,
+ bi->info.isp.sp.input.max_height,
+ bi->info.isp.sp.enable.input_yuv ? "yuv420 " : "",
+ bi->info.isp.sp.enable.input_feeder ||
+ bi->info.isp.sp.enable.input_raw ? "raw8 raw10 " : "",
+ bi->info.isp.sp.enable.input_raw ? "raw12" : "");
+
+ dev_dbg(dev, " internal (%i,%i)\n",
+ bi->info.isp.sp.internal.max_width,
+ bi->info.isp.sp.internal.max_height);
+
+ dev_dbg(dev, " output (%i,%i)-(%i,%i) formats",
+ bi->info.isp.sp.output.min_width,
+ bi->info.isp.sp.output.min_height,
+ bi->info.isp.sp.output.max_width,
+ bi->info.isp.sp.output.max_height);
+ for (i = 0; i < bi->info.isp.num_output_formats; i++)
+ dev_dbg(dev, " %i", bi->info.isp.output_formats[i]);
+ dev_dbg(dev, " vf");
+ for (i = 0; i < bi->info.isp.num_vf_formats; i++)
+ dev_dbg(dev, " %i", bi->info.isp.vf_formats[i]);
+ dev_dbg(dev, "\n");
+}
+
+unsigned int ipu3_css_fw_obgrid_size(const struct imgu_fw_info *bi)
+{
+ unsigned int width = DIV_ROUND_UP(bi->info.isp.sp.internal.max_width,
+ IMGU_OBGRID_TILE_SIZE * 2) + 1;
+ unsigned int height = DIV_ROUND_UP(bi->info.isp.sp.internal.max_height,
+ IMGU_OBGRID_TILE_SIZE * 2) + 1;
+ unsigned int obgrid_size;
+
+ width = ALIGN(width, IPU3_UAPI_ISP_VEC_ELEMS / 4);
+ obgrid_size = PAGE_ALIGN(width * height *
+ sizeof(struct ipu3_uapi_obgrid_param)) *
+ bi->info.isp.sp.iterator.num_stripes;
+ return obgrid_size;
+}
+
+void *ipu3_css_fw_pipeline_params(struct ipu3_css *css,
+ enum imgu_abi_param_class c,
+ enum imgu_abi_memories m,
+ struct imgu_fw_isp_parameter *par,
+ size_t par_size, void *binary_params)
+{
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+
+ if (par->offset + par->size >
+ bi->info.isp.sp.mem_initializers.params[c][m].size)
+ return NULL;
+
+ if (par->size != par_size)
+ pr_warn("parameter size doesn't match defined size\n");
+
+ if (par->size < par_size)
+ return NULL;
+
+ return binary_params + par->offset;
+}
+
+void ipu3_css_fw_cleanup(struct ipu3_css *css)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+
+ if (css->binary) {
+ unsigned int i;
+
+ for (i = 0; i < css->fwp->file_header.binary_nr; i++)
+ ipu3_dmamap_free(imgu, &css->binary[i]);
+ kfree(css->binary);
+ }
+ if (css->fw)
+ release_firmware(css->fw);
+
+ css->binary = NULL;
+ css->fw = NULL;
+}
+
+int ipu3_css_fw_init(struct ipu3_css *css)
+{
+ static const u32 BLOCK_MAX = 65536;
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ struct device *dev = css->dev;
+ unsigned int i, j, binary_nr;
+ int r;
+
+ r = request_firmware(&css->fw, IMGU_FW_NAME, css->dev);
+ if (r)
+ return r;
+
+ /* Check and display fw header info */
+
+ css->fwp = (struct imgu_fw_header *)css->fw->data;
+ if (css->fw->size < sizeof(struct imgu_fw_header *) ||
+ css->fwp->file_header.h_size != sizeof(struct imgu_fw_bi_file_h))
+ goto bad_fw;
+ if (sizeof(struct imgu_fw_bi_file_h) +
+ css->fwp->file_header.binary_nr * sizeof(struct imgu_fw_info) >
+ css->fw->size)
+ goto bad_fw;
+
+ dev_info(dev, "loaded firmware version %.64s, %u binaries, %zu bytes\n",
+ css->fwp->file_header.version, css->fwp->file_header.binary_nr,
+ css->fw->size);
+
+ /* Validate and display info on fw binaries */
+
+ binary_nr = css->fwp->file_header.binary_nr;
+
+ css->fw_bl = -1;
+ css->fw_sp[0] = -1;
+ css->fw_sp[1] = -1;
+
+ for (i = 0; i < binary_nr; i++) {
+ struct imgu_fw_info *bi = &css->fwp->binary_header[i];
+ const char *name = (void *)css->fwp + bi->blob.prog_name_offset;
+ size_t len;
+
+ if (bi->blob.prog_name_offset >= css->fw->size)
+ goto bad_fw;
+ len = strnlen(name, css->fw->size - bi->blob.prog_name_offset);
+ if (len + 1 > css->fw->size - bi->blob.prog_name_offset ||
+ len + 1 >= IMGU_ABI_MAX_BINARY_NAME)
+ goto bad_fw;
+
+ if (bi->blob.size != bi->blob.text_size + bi->blob.icache_size
+ + bi->blob.data_size + bi->blob.padding_size)
+ goto bad_fw;
+ if (bi->blob.offset + bi->blob.size > css->fw->size)
+ goto bad_fw;
+
+ if (bi->type == IMGU_FW_BOOTLOADER_FIRMWARE) {
+ css->fw_bl = i;
+ if (bi->info.bl.sw_state >= css->iomem_length ||
+ bi->info.bl.num_dma_cmds >= css->iomem_length ||
+ bi->info.bl.dma_cmd_list >= css->iomem_length)
+ goto bad_fw;
+ }
+ if (bi->type == IMGU_FW_SP_FIRMWARE ||
+ bi->type == IMGU_FW_SP1_FIRMWARE) {
+ css->fw_sp[bi->type == IMGU_FW_SP_FIRMWARE ? 0 : 1] = i;
+ if (bi->info.sp.per_frame_data >= css->iomem_length ||
+ bi->info.sp.init_dmem_data >= css->iomem_length ||
+ bi->info.sp.host_sp_queue >= css->iomem_length ||
+ bi->info.sp.isp_started >= css->iomem_length ||
+ bi->info.sp.sw_state >= css->iomem_length ||
+ bi->info.sp.sleep_mode >= css->iomem_length ||
+ bi->info.sp.invalidate_tlb >= css->iomem_length ||
+ bi->info.sp.host_sp_com >= css->iomem_length ||
+ bi->info.sp.output + 12 >= css->iomem_length ||
+ bi->info.sp.host_sp_queues_initialized >=
+ css->iomem_length)
+ goto bad_fw;
+ }
+ if (bi->type != IMGU_FW_ISP_FIRMWARE)
+ continue;
+
+ if (bi->info.isp.sp.pipeline.mode >= IPU3_CSS_PIPE_ID_NUM)
+ goto bad_fw;
+
+ if (bi->info.isp.sp.iterator.num_stripes >
+ IPU3_UAPI_MAX_STRIPES)
+ goto bad_fw;
+
+ if (bi->info.isp.num_vf_formats > IMGU_ABI_FRAME_FORMAT_NUM ||
+ bi->info.isp.num_output_formats > IMGU_ABI_FRAME_FORMAT_NUM)
+ goto bad_fw;
+
+ for (j = 0; j < bi->info.isp.num_output_formats; j++)
+ if (bi->info.isp.output_formats[j] < 0 ||
+ bi->info.isp.output_formats[j] >=
+ IMGU_ABI_FRAME_FORMAT_NUM)
+ goto bad_fw;
+ for (j = 0; j < bi->info.isp.num_vf_formats; j++)
+ if (bi->info.isp.vf_formats[j] < 0 ||
+ bi->info.isp.vf_formats[j] >=
+ IMGU_ABI_FRAME_FORMAT_NUM)
+ goto bad_fw;
+
+ if (bi->info.isp.sp.block.block_width <= 0 ||
+ bi->info.isp.sp.block.block_width > BLOCK_MAX ||
+ bi->info.isp.sp.block.output_block_height <= 0 ||
+ bi->info.isp.sp.block.output_block_height > BLOCK_MAX)
+ goto bad_fw;
+
+ if (bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_PARAM]
+ + sizeof(struct imgu_fw_param_memory_offsets) >
+ css->fw->size ||
+ bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_CONFIG]
+ + sizeof(struct imgu_fw_config_memory_offsets) >
+ css->fw->size ||
+ bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_STATE]
+ + sizeof(struct imgu_fw_state_memory_offsets) >
+ css->fw->size)
+ goto bad_fw;
+
+ ipu3_css_fw_show_binary(dev, bi, name);
+ }
+
+ if (css->fw_bl == -1 || css->fw_sp[0] == -1 || css->fw_sp[1] == -1)
+ goto bad_fw;
+
+ /* Allocate and map fw binaries into IMGU */
+
+ css->binary = kcalloc(binary_nr, sizeof(*css->binary), GFP_KERNEL);
+ if (!css->binary) {
+ r = -ENOMEM;
+ goto error_out;
+ }
+
+ for (i = 0; i < css->fwp->file_header.binary_nr; i++) {
+ struct imgu_fw_info *bi = &css->fwp->binary_header[i];
+ void *blob = (void *)css->fwp + bi->blob.offset;
+ size_t size = bi->blob.size;
+
+ if (!ipu3_dmamap_alloc(imgu, &css->binary[i], size)) {
+ r = -ENOMEM;
+ goto error_out;
+ }
+ memcpy(css->binary[i].vaddr, blob, size);
+ }
+
+ return 0;
+
+bad_fw:
+ dev_err(dev, "invalid firmware binary, size %u\n", (int)css->fw->size);
+ r = -ENODEV;
+
+error_out:
+ ipu3_css_fw_cleanup(css);
+ return r;
+}
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-fw.h b/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
new file mode 100644
index 0000000..954bb31
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
@@ -0,0 +1,188 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+
+#ifndef __IPU3_CSS_FW_H
+#define __IPU3_CSS_FW_H
+
+/******************* Firmware file definitions *******************/
+
+#define IMGU_FW_NAME "ipu3-fw.bin"
+
+typedef u32 imgu_fw_ptr;
+
+enum imgu_fw_type {
+ IMGU_FW_SP_FIRMWARE, /* Firmware for the SP */
+ IMGU_FW_SP1_FIRMWARE, /* Firmware for the SP1 */
+ IMGU_FW_ISP_FIRMWARE, /* Firmware for the ISP */
+ IMGU_FW_BOOTLOADER_FIRMWARE, /* Firmware for the BootLoader */
+ IMGU_FW_ACC_FIRMWARE /* Firmware for accelerations */
+};
+
+enum imgu_fw_acc_type {
+ IMGU_FW_ACC_NONE, /* Normal binary */
+ IMGU_FW_ACC_OUTPUT, /* Accelerator stage on output frame */
+ IMGU_FW_ACC_VIEWFINDER, /* Accelerator stage on viewfinder frame */
+ IMGU_FW_ACC_STANDALONE, /* Stand-alone acceleration */
+};
+
+struct imgu_fw_isp_parameter {
+ u32 offset; /* Offset in isp_<mem> config, params, etc. */
+ u32 size; /* Disabled if 0 */
+};
+
+struct imgu_fw_param_memory_offsets {
+ struct {
+ struct imgu_fw_isp_parameter lin; /* lin_vmem_params */
+ struct imgu_fw_isp_parameter tnr3; /* tnr3_vmem_params */
+ struct imgu_fw_isp_parameter xnr3; /* xnr3_vmem_params */
+ } vmem;
+ struct {
+ struct imgu_fw_isp_parameter tnr;
+ struct imgu_fw_isp_parameter tnr3; /* tnr3_params */
+ struct imgu_fw_isp_parameter xnr3; /* xnr3_params */
+ struct imgu_fw_isp_parameter plane_io_config; /* 192 bytes */
+ struct imgu_fw_isp_parameter rgbir; /* rgbir_params */
+ } dmem;
+};
+
+struct imgu_fw_config_memory_offsets {
+ struct {
+ struct imgu_fw_isp_parameter iterator;
+ struct imgu_fw_isp_parameter dvs;
+ struct imgu_fw_isp_parameter output;
+ struct imgu_fw_isp_parameter raw;
+ struct imgu_fw_isp_parameter input_yuv;
+ struct imgu_fw_isp_parameter tnr;
+ struct imgu_fw_isp_parameter tnr3;
+ struct imgu_fw_isp_parameter ref;
+ } dmem;
+};
+
+struct imgu_fw_state_memory_offsets {
+ struct {
+ struct imgu_fw_isp_parameter tnr;
+ struct imgu_fw_isp_parameter tnr3;
+ struct imgu_fw_isp_parameter ref;
+ } dmem;
+};
+
+union imgu_fw_all_memory_offsets {
+ struct {
+ u64 imgu_fw_mem_offsets[3]; /* params, config, state */
+ } offsets;
+ struct {
+ u64 ptr;
+ } array[IMGU_ABI_PARAM_CLASS_NUM];
+};
+
+struct imgu_fw_binary_xinfo {
+ /* Part that is of interest to the SP. */
+ struct imgu_abi_binary_info sp;
+
+ /* Rest of the binary info, only interesting to the host. */
+ u32 type; /* enum imgu_fw_acc_type */
+
+ u32 num_output_formats __aligned(8);
+ u32 output_formats[IMGU_ABI_FRAME_FORMAT_NUM]; /* enum frame_format */
+
+ /* number of supported vf formats */
+ u32 num_vf_formats __aligned(8);
+ /* types of supported vf formats */
+ u32 vf_formats[IMGU_ABI_FRAME_FORMAT_NUM]; /* enum frame_format */
+ u8 num_output_pins;
+ imgu_fw_ptr xmem_addr;
+
+ u64 imgu_fw_blob_descr_ptr __aligned(8);
+ u32 blob_index __aligned(8);
+ union imgu_fw_all_memory_offsets mem_offsets __aligned(8);
+ struct imgu_fw_binary_xinfo *next __aligned(8);
+};
+
+struct imgu_fw_sp_info {
+ u32 init_dmem_data; /* data sect config, stored to dmem */
+ u32 per_frame_data; /* Per frame data, stored to dmem */
+ u32 group; /* Per pipeline data, loaded by dma */
+ u32 output; /* SP output data, loaded by dmem */
+ u32 host_sp_queue; /* Host <-> SP queues */
+ u32 host_sp_com; /* Host <-> SP commands */
+ u32 isp_started; /* P'ed from sensor thread, csim only */
+ u32 sw_state; /* Polled from css, enum imgu_abi_sp_swstate */
+ u32 host_sp_queues_initialized; /* Polled from the SP */
+ u32 sleep_mode; /* different mode to halt SP */
+ u32 invalidate_tlb; /* inform SP to invalidate mmu TLB */
+ u32 debug_buffer_ddr_address; /* the addr of DDR debug queue */
+
+ /* input system perf count array */
+ u32 perf_counter_input_system_error;
+ u32 threads_stack; /* sp thread's stack pointers */
+ u32 threads_stack_size; /* sp thread's stack sizes */
+ u32 curr_binary_id; /* current binary id */
+ u32 raw_copy_line_count; /* raw copy line counter */
+ u32 ddr_parameter_address; /* acc param ddrptr, sp dmem */
+ u32 ddr_parameter_size; /* acc param size, sp dmem */
+ /* Entry functions */
+ u32 sp_entry; /* The SP entry function */
+ u32 tagger_frames_addr; /* Base address of tagger state */
+};
+
+struct imgu_fw_bl_info {
+ u32 num_dma_cmds; /* Number of cmds sent by CSS */
+ u32 dma_cmd_list; /* Dma command list sent by CSS */
+ u32 sw_state; /* Polled from css, enum imgu_abi_bl_swstate */
+ /* Entry functions */
+ u32 bl_entry; /* The SP entry function */
+};
+
+struct imgu_fw_acc_info {
+ u32 per_frame_data; /* Dummy for now */
+};
+
+union imgu_fw_union {
+ struct imgu_fw_binary_xinfo isp; /* ISP info */
+ struct imgu_fw_sp_info sp; /* SP info */
+ struct imgu_fw_sp_info sp1; /* SP1 info */
+ struct imgu_fw_bl_info bl; /* Bootloader info */
+ struct imgu_fw_acc_info acc; /* Accelerator info */
+};
+
+struct imgu_fw_info {
+ size_t header_size; /* size of fw header */
+ u32 type __aligned(8); /* enum imgu_fw_type */
+ union imgu_fw_union info; /* Binary info */
+ struct imgu_abi_blob_info blob; /* Blob info */
+ /* Dynamic part */
+ u64 next;
+
+ u32 loaded __aligned(8); /* Firmware has been loaded */
+ const u64 isp_code __aligned(8); /* ISP pointer to code */
+ /* Firmware handle between user space and kernel */
+ u32 handle __aligned(8);
+ /* Sections to copy from/to ISP */
+ struct imgu_abi_isp_param_segments mem_initializers;
+ /* Initializer for local ISP memories */
+};
+
+struct imgu_fw_bi_file_h {
+ char version[64]; /* branch tag + week day + time */
+ int binary_nr; /* Number of binaries */
+ unsigned int h_size; /* sizeof(struct imgu_fw_bi_file_h) */
+};
+
+struct imgu_fw_header {
+ struct imgu_fw_bi_file_h file_header;
+ struct imgu_fw_info binary_header[1]; /* binary_nr items */
+};
+
+/******************* Firmware functions *******************/
+
+int ipu3_css_fw_init(struct ipu3_css *css);
+void ipu3_css_fw_cleanup(struct ipu3_css *css);
+
+unsigned int ipu3_css_fw_obgrid_size(const struct imgu_fw_info *bi);
+void *ipu3_css_fw_pipeline_params(struct ipu3_css *css,
+ enum imgu_abi_param_class c,
+ enum imgu_abi_memories m,
+ struct imgu_fw_isp_parameter *par,
+ size_t par_size, void *binary_params);
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 09/16] intel-ipu3: css: Add support for firmware management
2018-10-29 22:23 ` [PATCH v7 09/16] intel-ipu3: css: Add support for firmware management Yong Zhi
@ 2018-11-28 22:22 ` Sakari Ailus
0 siblings, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-11-28 22:22 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:23:03PM -0700, Yong Zhi wrote:
...
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-fw.h b/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
> new file mode 100644
> index 0000000..954bb31
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
> @@ -0,0 +1,188 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2018 Intel Corporation */
> +
> +#ifndef __IPU3_CSS_FW_H
> +#define __IPU3_CSS_FW_H
> +
> +/******************* Firmware file definitions *******************/
> +
> +#define IMGU_FW_NAME "ipu3-fw.bin"
Shouldn't this be "intel/ipu3-fw.bin"?
--
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 11/16] intel-ipu3: css: Compute and program ccs
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (8 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 09/16] intel-ipu3: css: Add support for firmware management Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-10-29 22:23 ` [PATCH v7 12/16] intel-ipu3: css: Initialize css hardware Yong Zhi
` (7 subsequent siblings)
17 siblings, 0 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
A collection of routines that are mainly used
to calculate the parameters for accelerator cluster.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-css-params.c | 2907 ++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-css-params.h | 25 +
2 files changed, 2932 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-params.c b/drivers/media/pci/intel/ipu3/ipu3-css-params.c
new file mode 100644
index 0000000..add2be4
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-params.c
@@ -0,0 +1,2907 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Intel Corporation
+
+#include <linux/device.h>
+
+#include "ipu3-css.h"
+#include "ipu3-css-fw.h"
+#include "ipu3-tables.h"
+
+#define DIV_ROUND_CLOSEST_DOWN(a, b) (((a) + ((b) / 2) - 1) / (b))
+#define roundclosest_down(a, b) (DIV_ROUND_CLOSEST_DOWN(a, b) * (b))
+
+#define IPU3_UAPI_ANR_MAX_RESET ((1 << 12) - 1)
+#define IPU3_UAPI_ANR_MIN_RESET (((-1) << 12) + 1)
+
+struct ipu3_css_scaler_info {
+ unsigned int phase_step; /* Same for luma/chroma */
+ int exp_shift;
+
+ unsigned int phase_init; /* luma/chroma dependent */
+ int pad_left;
+ int pad_right;
+ int crop_left;
+ int crop_top;
+};
+
+static unsigned int ipu3_css_scaler_get_exp(unsigned int counter,
+ unsigned int divider)
+{
+ int i = fls(divider) - fls(counter);
+
+ if (i <= 0)
+ return 0;
+
+ if (divider >> i < counter)
+ i = i - 1;
+
+ return i;
+}
+
+/* Set up the CSS scaler look up table */
+static void
+ipu3_css_scaler_setup_lut(unsigned int taps, unsigned int input_width,
+ unsigned int output_width, int phase_step_correction,
+ const int *coeffs, unsigned int coeffs_size,
+ s8 coeff_lut[], struct ipu3_css_scaler_info *info)
+{
+ int tap, phase, phase_sum_left, phase_sum_right;
+ int exponent = ipu3_css_scaler_get_exp(output_width, input_width);
+ int mantissa = (1 << exponent) * output_width;
+ unsigned int phase_step;
+
+ if (input_width == output_width) {
+ for (phase = 0; phase < IMGU_SCALER_PHASES; phase++) {
+ for (tap = 0; tap < taps; tap++) {
+ coeff_lut[phase * IMGU_SCALER_FILTER_TAPS + tap]
+ = 0;
+ }
+ }
+
+ info->phase_step = IMGU_SCALER_PHASES *
+ (1 << IMGU_SCALER_PHASE_COUNTER_PREC_REF);
+ info->exp_shift = 0;
+ info->pad_left = 0;
+ info->pad_right = 0;
+ info->phase_init = 0;
+ info->crop_left = 0;
+ info->crop_top = 0;
+ return;
+ }
+
+ for (phase = 0; phase < IMGU_SCALER_PHASES; phase++) {
+ for (tap = 0; tap < taps; tap++) {
+ /* flip table to for convolution reverse indexing */
+ s64 coeff = coeffs[coeffs_size -
+ ((tap * (coeffs_size / taps)) + phase) - 1];
+ coeff *= mantissa;
+ coeff = div64_long(coeff, input_width);
+
+ /* Add +"0.5" */
+ coeff += 1 << (IMGU_SCALER_COEFF_BITS - 1);
+ coeff >>= IMGU_SCALER_COEFF_BITS;
+
+ coeff_lut[phase * IMGU_SCALER_FILTER_TAPS + tap] =
+ coeff;
+ }
+ }
+
+ phase_step = IMGU_SCALER_PHASES *
+ (1 << IMGU_SCALER_PHASE_COUNTER_PREC_REF) *
+ output_width / input_width;
+ phase_step += phase_step_correction;
+ phase_sum_left = (taps / 2 * IMGU_SCALER_PHASES *
+ (1 << IMGU_SCALER_PHASE_COUNTER_PREC_REF)) -
+ (1 << (IMGU_SCALER_PHASE_COUNTER_PREC_REF - 1));
+ phase_sum_right = (taps / 2 * IMGU_SCALER_PHASES *
+ (1 << IMGU_SCALER_PHASE_COUNTER_PREC_REF)) +
+ (1 << (IMGU_SCALER_PHASE_COUNTER_PREC_REF - 1));
+
+ info->exp_shift = IMGU_SCALER_MAX_EXPONENT_SHIFT - exponent;
+ info->pad_left = (phase_sum_left % phase_step == 0) ?
+ phase_sum_left / phase_step - 1 : phase_sum_left / phase_step;
+ info->pad_right = (phase_sum_right % phase_step == 0) ?
+ phase_sum_right / phase_step - 1 : phase_sum_right / phase_step;
+ info->phase_init = phase_sum_left - phase_step * info->pad_left;
+ info->phase_step = phase_step;
+ info->crop_left = taps - 1;
+ info->crop_top = taps - 1;
+}
+
+/*
+ * Calculates the exact output image width/height, based on phase_step setting
+ * (must be perfectly aligned with hardware).
+ */
+static unsigned int
+ipu3_css_scaler_calc_scaled_output(unsigned int input,
+ struct ipu3_css_scaler_info *info)
+{
+ unsigned int arg1 = input * info->phase_step +
+ (1 - IMGU_SCALER_TAPS_Y / 2) * IMGU_SCALER_FIR_PHASES -
+ IMGU_SCALER_FIR_PHASES / (2 * IMGU_SCALER_PHASES);
+ unsigned int arg2 = ((IMGU_SCALER_TAPS_Y / 2) * IMGU_SCALER_FIR_PHASES +
+ IMGU_SCALER_FIR_PHASES / (2 * IMGU_SCALER_PHASES)) *
+ IMGU_SCALER_FIR_PHASES + info->phase_step / 2;
+
+ return ((arg1 + (arg2 - IMGU_SCALER_FIR_PHASES * info->phase_step) /
+ IMGU_SCALER_FIR_PHASES) / (2 * IMGU_SCALER_FIR_PHASES)) * 2;
+}
+
+/*
+ * Calculate the output width and height, given the luma
+ * and chroma details of a scaler
+ */
+static void
+ipu3_css_scaler_calc(u32 input_width, u32 input_height, u32 target_width,
+ u32 target_height, struct imgu_abi_osys_config *cfg,
+ struct ipu3_css_scaler_info *info_luma,
+ struct ipu3_css_scaler_info *info_chroma,
+ unsigned int *output_width, unsigned int *output_height,
+ unsigned int *procmode)
+{
+ u32 out_width = target_width;
+ u32 out_height = target_height;
+ const unsigned int height_alignment = 2;
+ int phase_step_correction = -1;
+
+ /*
+ * Calculate scaled output width. If the horizontal and vertical scaling
+ * factor is different, then choose the biggest and crop off excess
+ * lines or columns after formatting.
+ */
+ if (target_height * input_width > target_width * input_height)
+ target_width = DIV_ROUND_UP(target_height * input_width,
+ input_height);
+
+ if (input_width == target_width)
+ *procmode = IMGU_ABI_OSYS_PROCMODE_BYPASS;
+ else
+ *procmode = IMGU_ABI_OSYS_PROCMODE_DOWNSCALE;
+
+ memset(&cfg->scaler_coeffs_chroma, 0,
+ sizeof(cfg->scaler_coeffs_chroma));
+ memset(&cfg->scaler_coeffs_luma, 0, sizeof(*cfg->scaler_coeffs_luma));
+ do {
+ phase_step_correction++;
+
+ ipu3_css_scaler_setup_lut(IMGU_SCALER_TAPS_Y,
+ input_width, target_width,
+ phase_step_correction,
+ ipu3_css_downscale_4taps,
+ IMGU_SCALER_DOWNSCALE_4TAPS_LEN,
+ cfg->scaler_coeffs_luma, info_luma);
+
+ ipu3_css_scaler_setup_lut(IMGU_SCALER_TAPS_UV,
+ input_width, target_width,
+ phase_step_correction,
+ ipu3_css_downscale_2taps,
+ IMGU_SCALER_DOWNSCALE_2TAPS_LEN,
+ cfg->scaler_coeffs_chroma,
+ info_chroma);
+
+ out_width = ipu3_css_scaler_calc_scaled_output(input_width,
+ info_luma);
+ out_height = ipu3_css_scaler_calc_scaled_output(input_height,
+ info_luma);
+ } while ((out_width < target_width || out_height < target_height ||
+ !IS_ALIGNED(out_height, height_alignment)) &&
+ phase_step_correction <= 5);
+
+ *output_width = out_width;
+ *output_height = out_height;
+}
+
+/********************** Osys routines for scaler****************************/
+
+static void ipu3_css_osys_set_format(enum imgu_abi_frame_format host_format,
+ unsigned int *osys_format,
+ unsigned int *osys_tiling)
+{
+ *osys_format = IMGU_ABI_OSYS_FORMAT_YUV420;
+ *osys_tiling = IMGU_ABI_OSYS_TILING_NONE;
+
+ switch (host_format) {
+ case IMGU_ABI_FRAME_FORMAT_YUV420:
+ *osys_format = IMGU_ABI_OSYS_FORMAT_YUV420;
+ break;
+ case IMGU_ABI_FRAME_FORMAT_YV12:
+ *osys_format = IMGU_ABI_OSYS_FORMAT_YV12;
+ break;
+ case IMGU_ABI_FRAME_FORMAT_NV12:
+ *osys_format = IMGU_ABI_OSYS_FORMAT_NV12;
+ break;
+ case IMGU_ABI_FRAME_FORMAT_NV16:
+ *osys_format = IMGU_ABI_OSYS_FORMAT_NV16;
+ break;
+ case IMGU_ABI_FRAME_FORMAT_NV21:
+ *osys_format = IMGU_ABI_OSYS_FORMAT_NV21;
+ break;
+ case IMGU_ABI_FRAME_FORMAT_NV12_TILEY:
+ *osys_format = IMGU_ABI_OSYS_FORMAT_NV12;
+ *osys_tiling = IMGU_ABI_OSYS_TILING_Y;
+ break;
+ default:
+ /* For now, assume use default values */
+ break;
+ }
+}
+
+/*
+ * Function calculates input frame stripe offset, based
+ * on output frame stripe offset and filter parameters.
+ */
+static int ipu3_css_osys_calc_stripe_offset(int stripe_offset_out,
+ int fir_phases, int phase_init,
+ int phase_step, int pad_left)
+{
+ int stripe_offset_inp = stripe_offset_out * fir_phases -
+ pad_left * phase_step;
+
+ return DIV_ROUND_UP(stripe_offset_inp - phase_init, phase_step);
+}
+
+/*
+ * Calculate input frame phase, given the output frame
+ * stripe offset and filter parameters
+ */
+static int ipu3_css_osys_calc_stripe_phase_init(int stripe_offset_out,
+ int fir_phases, int phase_init,
+ int phase_step, int pad_left)
+{
+ int stripe_offset_inp =
+ ipu3_css_osys_calc_stripe_offset(stripe_offset_out,
+ fir_phases, phase_init,
+ phase_step, pad_left);
+
+ return phase_init + ((pad_left + stripe_offset_inp) * phase_step) -
+ stripe_offset_out * fir_phases;
+}
+
+/*
+ * This function calculates input frame stripe width,
+ * based on output frame stripe offset and filter parameters
+ */
+static int ipu3_css_osys_calc_inp_stripe_width(int stripe_width_out,
+ int fir_phases, int phase_init,
+ int phase_step, int fir_taps,
+ int pad_left, int pad_right)
+{
+ int stripe_width_inp = (stripe_width_out + fir_taps - 1) * fir_phases;
+
+ stripe_width_inp = DIV_ROUND_UP(stripe_width_inp - phase_init,
+ phase_step);
+
+ return stripe_width_inp - pad_left - pad_right;
+}
+
+/*
+ * This function calculates output frame stripe width, basedi
+ * on output frame stripe offset and filter parameters
+ */
+static int ipu3_css_osys_out_stripe_width(int stripe_width_inp, int fir_phases,
+ int phase_init, int phase_step,
+ int fir_taps, int pad_left,
+ int pad_right, int column_offset)
+{
+ int stripe_width_out = (pad_left + stripe_width_inp +
+ pad_right - column_offset) * phase_step;
+
+ stripe_width_out = (stripe_width_out + phase_init) / fir_phases;
+
+ return stripe_width_out - (fir_taps - 1);
+}
+
+struct ipu3_css_reso {
+ unsigned int input_width;
+ unsigned int input_height;
+ enum imgu_abi_frame_format input_format;
+ unsigned int pin_width[IMGU_ABI_OSYS_PINS];
+ unsigned int pin_height[IMGU_ABI_OSYS_PINS];
+ unsigned int pin_stride[IMGU_ABI_OSYS_PINS];
+ enum imgu_abi_frame_format pin_format[IMGU_ABI_OSYS_PINS];
+ int chunk_width;
+ int chunk_height;
+ int block_height;
+ int block_width;
+};
+
+struct ipu3_css_frame_params {
+ /* Output pins */
+ unsigned int enable;
+ unsigned int format;
+ unsigned int flip;
+ unsigned int mirror;
+ unsigned int tiling;
+ unsigned int reduce_range;
+ unsigned int width;
+ unsigned int height;
+ unsigned int stride;
+ unsigned int scaled;
+ unsigned int crop_left;
+ unsigned int crop_top;
+};
+
+struct ipu3_css_stripe_params {
+ unsigned int processing_mode;
+ unsigned int phase_step;
+ unsigned int exp_shift;
+ unsigned int phase_init_left_y;
+ unsigned int phase_init_left_uv;
+ unsigned int phase_init_top_y;
+ unsigned int phase_init_top_uv;
+ unsigned int pad_left_y;
+ unsigned int pad_left_uv;
+ unsigned int pad_right_y;
+ unsigned int pad_right_uv;
+ unsigned int pad_top_y;
+ unsigned int pad_top_uv;
+ unsigned int pad_bottom_y;
+ unsigned int pad_bottom_uv;
+ unsigned int crop_left_y;
+ unsigned int crop_top_y;
+ unsigned int crop_left_uv;
+ unsigned int crop_top_uv;
+ unsigned int start_column_y;
+ unsigned int start_column_uv;
+ unsigned int chunk_width;
+ unsigned int chunk_height;
+ unsigned int block_width;
+ unsigned int block_height;
+ unsigned int input_width;
+ unsigned int input_height;
+ int output_width[IMGU_ABI_OSYS_PINS];
+ int output_height[IMGU_ABI_OSYS_PINS];
+ int output_offset[IMGU_ABI_OSYS_PINS];
+};
+
+/*
+ * frame_params - size IMGU_ABI_OSYS_PINS
+ * stripe_params - size IPU3_UAPI_MAX_STRIPES
+ */
+static int ipu3_css_osys_calc_frame_and_stripe_params(
+ struct ipu3_css *css, unsigned int stripes,
+ struct imgu_abi_osys_config *osys,
+ struct ipu3_css_scaler_info *scaler_luma,
+ struct ipu3_css_scaler_info *scaler_chroma,
+ struct ipu3_css_frame_params frame_params[],
+ struct ipu3_css_stripe_params stripe_params[])
+{
+ u32 input_width = css->rect[IPU3_CSS_RECT_GDC].width;
+ u32 input_height = css->rect[IPU3_CSS_RECT_GDC].height;
+ u32 target_width = css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ u32 target_height = css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ unsigned int procmode = 0;
+ struct ipu3_css_reso reso;
+ unsigned int output_width, pin, s;
+
+ /* Frame parameters */
+
+ /* Input width for Output System is output width of DVS (with GDC) */
+ reso.input_width = css->rect[IPU3_CSS_RECT_GDC].width;
+
+ /* Input height for Output System is output height of DVS (with GDC) */
+ reso.input_height = css->rect[IPU3_CSS_RECT_GDC].height;
+
+ reso.input_format =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+
+ reso.pin_width[IMGU_ABI_OSYS_PIN_OUT] =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ reso.pin_height[IMGU_ABI_OSYS_PIN_OUT] =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ reso.pin_stride[IMGU_ABI_OSYS_PIN_OUT] =
+ css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ reso.pin_format[IMGU_ABI_OSYS_PIN_OUT] =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+
+ reso.pin_width[IMGU_ABI_OSYS_PIN_VF] =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ reso.pin_height[IMGU_ABI_OSYS_PIN_VF] =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ reso.pin_stride[IMGU_ABI_OSYS_PIN_VF] =
+ css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ reso.pin_format[IMGU_ABI_OSYS_PIN_VF] =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+
+ /* Configure the frame parameters for all output pins */
+
+ frame_params[IMGU_ABI_OSYS_PIN_OUT].width =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ frame_params[IMGU_ABI_OSYS_PIN_OUT].height =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ frame_params[IMGU_ABI_OSYS_PIN_VF].width =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ frame_params[IMGU_ABI_OSYS_PIN_VF].height =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ frame_params[IMGU_ABI_OSYS_PIN_VF].crop_top = 0;
+ frame_params[IMGU_ABI_OSYS_PIN_VF].crop_left = 0;
+
+ for (pin = 0; pin < IMGU_ABI_OSYS_PINS; pin++) {
+ int enable = 0;
+ int scaled = 0;
+ unsigned int format = 0;
+ unsigned int tiling = 0;
+
+ frame_params[pin].flip = 0;
+ frame_params[pin].mirror = 0;
+ frame_params[pin].reduce_range = 0;
+ if (reso.pin_width[pin] != 0 && reso.pin_height[pin] != 0) {
+ enable = 1;
+ if (pin == IMGU_ABI_OSYS_PIN_OUT) {
+ if (reso.input_width < reso.pin_width[pin] ||
+ reso.input_height < reso.pin_height[pin])
+ return -EINVAL;
+ /*
+ * When input and output resolution is
+ * different instead of scaling, cropping
+ * should happen. Determine the crop factor
+ * to do the symmetric cropping
+ */
+ frame_params[pin].crop_left = roundclosest_down(
+ (reso.input_width -
+ reso.pin_width[pin]) / 2,
+ IMGU_OSYS_DMA_CROP_W_LIMIT);
+ frame_params[pin].crop_top = roundclosest_down(
+ (reso.input_height -
+ reso.pin_height[pin]) / 2,
+ IMGU_OSYS_DMA_CROP_H_LIMIT);
+ } else {
+ if (reso.pin_width[pin] != reso.input_width ||
+ reso.pin_height[pin] != reso.input_height) {
+ /*
+ * If resolution is different at input
+ * and output of OSYS, scaling is
+ * considered except when pin is MAIN.
+ * Later it will be decide whether
+ * scaler factor is 1 or other
+ * and cropping has to be done or not.
+ */
+ scaled = 1;
+ }
+ }
+ ipu3_css_osys_set_format(reso.pin_format[pin], &format,
+ &tiling);
+ } else {
+ enable = 0;
+ }
+ frame_params[pin].enable = enable;
+ frame_params[pin].format = format;
+ frame_params[pin].tiling = tiling;
+ frame_params[pin].stride = reso.pin_stride[pin];
+ frame_params[pin].scaled = scaled;
+ }
+
+ ipu3_css_scaler_calc(input_width, input_height, target_width,
+ target_height, osys, scaler_luma, scaler_chroma,
+ &reso.pin_width[IMGU_ABI_OSYS_PIN_VF],
+ &reso.pin_height[IMGU_ABI_OSYS_PIN_VF], &procmode);
+ dev_dbg(css->dev, "osys scaler procmode is %u", procmode);
+ output_width = reso.pin_width[IMGU_ABI_OSYS_PIN_VF];
+
+ if (output_width < reso.input_width / 2) {
+ /* Scaling factor <= 0.5 */
+ reso.chunk_width = IMGU_OSYS_BLOCK_WIDTH;
+ reso.block_width = IMGU_OSYS_BLOCK_WIDTH;
+ } else { /* 0.5 <= Scaling factor <= 1.0 */
+ reso.chunk_width = IMGU_OSYS_BLOCK_WIDTH / 2;
+ reso.block_width = IMGU_OSYS_BLOCK_WIDTH;
+ }
+
+ if (output_width <= reso.input_width * 7 / 8) {
+ /* Scaling factor <= 0.875 */
+ reso.chunk_height = IMGU_OSYS_BLOCK_HEIGHT;
+ reso.block_height = IMGU_OSYS_BLOCK_HEIGHT;
+ } else { /* 1.0 <= Scaling factor <= 1.75 */
+ reso.chunk_height = IMGU_OSYS_BLOCK_HEIGHT / 2;
+ reso.block_height = IMGU_OSYS_BLOCK_HEIGHT;
+ }
+
+ /*
+ * Calculate scaler configuration parameters based on input and output
+ * resolution.
+ */
+
+ if (frame_params[IMGU_ABI_OSYS_PIN_VF].enable) {
+ /*
+ * When aspect ratio is different between target resolution and
+ * required resolution, determine the crop factor to do
+ * symmetric cropping
+ */
+ u32 w = reso.pin_width[IMGU_ABI_OSYS_PIN_VF] -
+ frame_params[IMGU_ABI_OSYS_PIN_VF].width;
+ u32 h = reso.pin_height[IMGU_ABI_OSYS_PIN_VF] -
+ frame_params[IMGU_ABI_OSYS_PIN_VF].height;
+
+ frame_params[IMGU_ABI_OSYS_PIN_VF].crop_left =
+ roundclosest_down(w / 2, IMGU_OSYS_DMA_CROP_W_LIMIT);
+ frame_params[IMGU_ABI_OSYS_PIN_VF].crop_top =
+ roundclosest_down(h / 2, IMGU_OSYS_DMA_CROP_H_LIMIT);
+
+ if (reso.input_height % 4 || reso.input_width % 8) {
+ dev_err(css->dev, "OSYS input width is not multiple of 8 or\n");
+ dev_err(css->dev, "height is not multiple of 4\n");
+ return -EINVAL;
+ }
+ }
+
+ /* Stripe parameters */
+
+ if (frame_params[IMGU_ABI_OSYS_PIN_VF].enable) {
+ output_width = reso.pin_width[IMGU_ABI_OSYS_PIN_VF];
+ } else {
+ /*
+ * in case scaler output is not enabled
+ * take output width as input width since
+ * there is no scaling at main pin.
+ * Due to the fact that main pin can be different
+ * from input resolution to osys in the case of cropping,
+ * main pin resolution is not taken.
+ */
+ output_width = reso.input_width;
+ }
+
+ for (s = 0; s < stripes; s++) {
+ int stripe_offset_inp_y = 0;
+ int stripe_offset_inp_uv = 0;
+ int stripe_offset_out_y = 0;
+ int stripe_offset_out_uv = 0;
+ int stripe_phase_init_y = scaler_luma->phase_init;
+ int stripe_phase_init_uv = scaler_chroma->phase_init;
+ int stripe_offset_blk_y = 0;
+ int stripe_offset_blk_uv = 0;
+ int stripe_offset_col_y = 0;
+ int stripe_offset_col_uv = 0;
+ int stripe_pad_left_y = scaler_luma->pad_left;
+ int stripe_pad_left_uv = scaler_chroma->pad_left;
+ int stripe_pad_right_y = scaler_luma->pad_right;
+ int stripe_pad_right_uv = scaler_chroma->pad_right;
+ int stripe_crop_left_y = scaler_luma->crop_left;
+ int stripe_crop_left_uv = scaler_chroma->crop_left;
+ int stripe_input_width_y = reso.input_width;
+ int stripe_input_width_uv = 0;
+ int stripe_output_width_y = output_width;
+ int stripe_output_width_uv = 0;
+ int chunk_floor_y = 0;
+ int chunk_floor_uv = 0;
+ int chunk_ceil_uv = 0;
+
+ if (stripes > 1) {
+ if (s > 0) {
+ /* Calculate stripe offsets */
+ stripe_offset_out_y =
+ output_width * s / stripes;
+ stripe_offset_out_y =
+ rounddown(stripe_offset_out_y,
+ IPU3_UAPI_ISP_VEC_ELEMS);
+ stripe_offset_out_uv = stripe_offset_out_y /
+ IMGU_LUMA_TO_CHROMA_RATIO;
+ stripe_offset_inp_y =
+ ipu3_css_osys_calc_stripe_offset(
+ stripe_offset_out_y,
+ IMGU_OSYS_FIR_PHASES,
+ scaler_luma->phase_init,
+ scaler_luma->phase_step,
+ scaler_luma->pad_left);
+ stripe_offset_inp_uv =
+ ipu3_css_osys_calc_stripe_offset(
+ stripe_offset_out_uv,
+ IMGU_OSYS_FIR_PHASES,
+ scaler_chroma->phase_init,
+ scaler_chroma->phase_step,
+ scaler_chroma->pad_left);
+
+ /* Calculate stripe phase init */
+ stripe_phase_init_y =
+ ipu3_css_osys_calc_stripe_phase_init(
+ stripe_offset_out_y,
+ IMGU_OSYS_FIR_PHASES,
+ scaler_luma->phase_init,
+ scaler_luma->phase_step,
+ scaler_luma->pad_left);
+ stripe_phase_init_uv =
+ ipu3_css_osys_calc_stripe_phase_init(
+ stripe_offset_out_uv,
+ IMGU_OSYS_FIR_PHASES,
+ scaler_chroma->phase_init,
+ scaler_chroma->phase_step,
+ scaler_chroma->pad_left);
+
+ /*
+ * Chunk boundary corner case - luma and chroma
+ * start from different input chunks.
+ */
+ chunk_floor_y = rounddown(stripe_offset_inp_y,
+ reso.chunk_width);
+ chunk_floor_uv =
+ rounddown(stripe_offset_inp_uv,
+ reso.chunk_width /
+ IMGU_LUMA_TO_CHROMA_RATIO);
+
+ if (chunk_floor_y != chunk_floor_uv *
+ IMGU_LUMA_TO_CHROMA_RATIO) {
+ /*
+ * Match starting luma/chroma chunks.
+ * Decrease offset for UV and add output
+ * cropping.
+ */
+ stripe_offset_inp_uv -= 1;
+ stripe_crop_left_uv += 1;
+ stripe_phase_init_uv -=
+ scaler_luma->phase_step;
+ if (stripe_phase_init_uv < 0)
+ stripe_phase_init_uv =
+ stripe_phase_init_uv +
+ IMGU_OSYS_FIR_PHASES;
+ }
+ /*
+ * FW workaround for a HW bug: if the first
+ * chroma pixel is generated exactly at the end
+ * of chunck scaler HW may not output the pixel
+ * for downscale factors smaller than 1.5
+ * (timing issue).
+ */
+ chunk_ceil_uv =
+ roundup(stripe_offset_inp_uv,
+ reso.chunk_width /
+ IMGU_LUMA_TO_CHROMA_RATIO);
+
+ if (stripe_offset_inp_uv ==
+ chunk_ceil_uv - IMGU_OSYS_TAPS_UV) {
+ /*
+ * Decrease input offset and add
+ * output cropping
+ */
+ stripe_offset_inp_uv -= 1;
+ stripe_phase_init_uv -=
+ scaler_luma->phase_step;
+ if (stripe_phase_init_uv < 0) {
+ stripe_phase_init_uv +=
+ IMGU_OSYS_FIR_PHASES;
+ stripe_crop_left_uv += 1;
+ }
+ }
+
+ /*
+ * Calculate block and column offsets for the
+ * input stripe
+ */
+ stripe_offset_blk_y =
+ rounddown(stripe_offset_inp_y,
+ IMGU_INPUT_BLOCK_WIDTH);
+ stripe_offset_blk_uv =
+ rounddown(stripe_offset_inp_uv,
+ IMGU_INPUT_BLOCK_WIDTH /
+ IMGU_LUMA_TO_CHROMA_RATIO);
+ stripe_offset_col_y = stripe_offset_inp_y -
+ stripe_offset_blk_y;
+ stripe_offset_col_uv = stripe_offset_inp_uv -
+ stripe_offset_blk_uv;
+
+ /* Left padding is only for the first stripe */
+ stripe_pad_left_y = 0;
+ stripe_pad_left_uv = 0;
+ }
+
+ /* Right padding is only for the last stripe */
+ if (s < stripes - 1) {
+ int next_offset;
+
+ stripe_pad_right_y = 0;
+ stripe_pad_right_uv = 0;
+
+ next_offset = output_width * (s + 1) / stripes;
+ next_offset = rounddown(next_offset, 64);
+ stripe_output_width_y = next_offset -
+ stripe_offset_out_y;
+ } else {
+ stripe_output_width_y = output_width -
+ stripe_offset_out_y;
+ }
+
+ /* Calculate target output stripe width */
+ stripe_output_width_uv = stripe_output_width_y /
+ IMGU_LUMA_TO_CHROMA_RATIO;
+ /* Calculate input stripe width */
+ stripe_input_width_y = stripe_offset_col_y +
+ ipu3_css_osys_calc_inp_stripe_width(
+ stripe_output_width_y,
+ IMGU_OSYS_FIR_PHASES,
+ stripe_phase_init_y,
+ scaler_luma->phase_step,
+ IMGU_OSYS_TAPS_Y,
+ stripe_pad_left_y,
+ stripe_pad_right_y);
+
+ stripe_input_width_uv = stripe_offset_col_uv +
+ ipu3_css_osys_calc_inp_stripe_width(
+ stripe_output_width_uv,
+ IMGU_OSYS_FIR_PHASES,
+ stripe_phase_init_uv,
+ scaler_chroma->phase_step,
+ IMGU_OSYS_TAPS_UV,
+ stripe_pad_left_uv,
+ stripe_pad_right_uv);
+
+ stripe_input_width_uv = max(DIV_ROUND_UP(
+ stripe_input_width_y,
+ IMGU_LUMA_TO_CHROMA_RATIO),
+ stripe_input_width_uv);
+
+ stripe_input_width_y = stripe_input_width_uv *
+ IMGU_LUMA_TO_CHROMA_RATIO;
+
+ if (s >= stripes - 1) {
+ stripe_input_width_y = reso.input_width -
+ stripe_offset_blk_y;
+ /*
+ * The scaler requires that the last stripe
+ * spans at least two input blocks.
+ */
+ }
+
+ /*
+ * Spec: input stripe width must be a multiple of 8.
+ * Increase the input width and recalculate the output
+ * width. This may produce an extra column of junk
+ * blocks which will be overwritten by the
+ * next stripe.
+ */
+ stripe_input_width_y = ALIGN(stripe_input_width_y, 8);
+ stripe_output_width_y =
+ ipu3_css_osys_out_stripe_width(
+ stripe_input_width_y,
+ IMGU_OSYS_FIR_PHASES,
+ stripe_phase_init_y,
+ scaler_luma->phase_step,
+ IMGU_OSYS_TAPS_Y,
+ stripe_pad_left_y,
+ stripe_pad_right_y,
+ stripe_offset_col_y);
+
+ stripe_output_width_y =
+ rounddown(stripe_output_width_y,
+ IMGU_LUMA_TO_CHROMA_RATIO);
+ }
+ /*
+ * Following section executes and process parameters
+ * for both cases - Striping or No Striping.
+ */
+ {
+ unsigned int i;
+ int pin_scale = 0;
+ /*Input resolution */
+
+ stripe_params[s].input_width = stripe_input_width_y;
+ stripe_params[s].input_height = reso.input_height;
+
+ for (i = 0; i < IMGU_ABI_OSYS_PINS; i++) {
+ if (frame_params[i].scaled) {
+ /*
+ * Output stripe resolution and offset
+ * as produced by the scaler; actual
+ * output resolution may be slightly
+ * smaller.
+ */
+ stripe_params[s].output_width[i] =
+ stripe_output_width_y;
+ stripe_params[s].output_height[i] =
+ reso.pin_height[i];
+ stripe_params[s].output_offset[i] =
+ stripe_offset_out_y;
+
+ pin_scale += frame_params[i].scaled;
+ } else {
+ /* Unscaled pin */
+ stripe_params[s].output_width[i] =
+ stripe_params[s].input_width;
+ stripe_params[s].output_height[i] =
+ stripe_params[s].input_height;
+ stripe_params[s].output_offset[i] =
+ stripe_offset_blk_y;
+ }
+ }
+
+ /* If no pin use scale, we use BYPASS mode */
+ stripe_params[s].processing_mode = procmode;
+ stripe_params[s].phase_step = scaler_luma->phase_step;
+ stripe_params[s].exp_shift = scaler_luma->exp_shift;
+ stripe_params[s].phase_init_left_y =
+ stripe_phase_init_y;
+ stripe_params[s].phase_init_left_uv =
+ stripe_phase_init_uv;
+ stripe_params[s].phase_init_top_y =
+ scaler_luma->phase_init;
+ stripe_params[s].phase_init_top_uv =
+ scaler_chroma->phase_init;
+ stripe_params[s].pad_left_y = stripe_pad_left_y;
+ stripe_params[s].pad_left_uv = stripe_pad_left_uv;
+ stripe_params[s].pad_right_y = stripe_pad_right_y;
+ stripe_params[s].pad_right_uv = stripe_pad_right_uv;
+ stripe_params[s].pad_top_y = scaler_luma->pad_left;
+ stripe_params[s].pad_top_uv = scaler_chroma->pad_left;
+ stripe_params[s].pad_bottom_y = scaler_luma->pad_right;
+ stripe_params[s].pad_bottom_uv =
+ scaler_chroma->pad_right;
+ stripe_params[s].crop_left_y = stripe_crop_left_y;
+ stripe_params[s].crop_top_y = scaler_luma->crop_top;
+ stripe_params[s].crop_left_uv = stripe_crop_left_uv;
+ stripe_params[s].crop_top_uv = scaler_chroma->crop_top;
+ stripe_params[s].start_column_y = stripe_offset_col_y;
+ stripe_params[s].start_column_uv = stripe_offset_col_uv;
+ stripe_params[s].chunk_width = reso.chunk_width;
+ stripe_params[s].chunk_height = reso.chunk_height;
+ stripe_params[s].block_width = reso.block_width;
+ stripe_params[s].block_height = reso.block_height;
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * This function configures the Output Formatter System, given the number of
+ * stripes, scaler luma and chrome parameters
+ */
+static void ipu3_css_osys_calc(struct ipu3_css *css, unsigned int stripes,
+ struct imgu_abi_osys_config *osys,
+ struct ipu3_css_scaler_info *scaler_luma,
+ struct ipu3_css_scaler_info *scaler_chroma,
+ struct imgu_abi_stripes block_stripes[])
+{
+ struct ipu3_css_frame_params frame_params[IMGU_ABI_OSYS_PINS];
+ struct ipu3_css_stripe_params stripe_params[IPU3_UAPI_MAX_STRIPES];
+ struct imgu_abi_osys_formatter_params *param;
+ unsigned int pin, s;
+
+ memset(osys, 0, sizeof(*osys));
+
+ /* Compute the frame and stripe params */
+ ipu3_css_osys_calc_frame_and_stripe_params(css, stripes, osys,
+ scaler_luma, scaler_chroma,
+ frame_params, stripe_params);
+
+ /* Output formatter system parameters */
+
+ for (s = 0; s < stripes; s++) {
+ struct imgu_abi_osys_scaler_params *scaler =
+ &osys->scaler[s].param;
+ int fifo_addr_fmt = IMGU_FIFO_ADDR_SCALER_TO_FMT;
+ int fifo_addr_ack = IMGU_FIFO_ADDR_SCALER_TO_SP;
+
+ /* OUTPUT 0 / PIN 0 is only Scaler output */
+ scaler->inp_buf_y_st_addr = IMGU_VMEM1_INP_BUF_ADDR;
+
+ /*
+ * = (IMGU_OSYS_BLOCK_WIDTH / IMGU_VMEM1_ELEMS_PER_VEC)
+ * = (2 * IPU3_UAPI_ISP_VEC_ELEMS) /
+ * (IMGU_HIVE_OF_SYS_OF_SYSTEM_NWAYS)
+ * = 2 * 64 / 32 = 4
+ */
+ scaler->inp_buf_y_line_stride = IMGU_VMEM1_Y_STRIDE;
+ /*
+ * = (IMGU_VMEM1_V_OFFSET + VMEM1_uv_size)
+ * = (IMGU_VMEM1_U_OFFSET + VMEM1_uv_size) +
+ * (VMEM1_y_size / 4)
+ * = (VMEM1_y_size) + (VMEM1_y_size / 4) +
+ * (IMGU_OSYS_BLOCK_HEIGHT * IMGU_VMEM1_Y_STRIDE)/4
+ * = (IMGU_OSYS_BLOCK_HEIGHT * IMGU_VMEM1_Y_STRIDE)
+ */
+ scaler->inp_buf_y_buffer_stride = IMGU_VMEM1_BUF_SIZE;
+ scaler->inp_buf_u_st_addr = IMGU_VMEM1_INP_BUF_ADDR +
+ IMGU_VMEM1_U_OFFSET;
+ scaler->inp_buf_v_st_addr = IMGU_VMEM1_INP_BUF_ADDR +
+ IMGU_VMEM1_V_OFFSET;
+ scaler->inp_buf_uv_line_stride = IMGU_VMEM1_UV_STRIDE;
+ scaler->inp_buf_uv_buffer_stride = IMGU_VMEM1_BUF_SIZE;
+ scaler->inp_buf_chunk_width = stripe_params[s].chunk_width;
+ scaler->inp_buf_nr_buffers = IMGU_OSYS_NUM_INPUT_BUFFERS;
+
+ /* Output buffers */
+ scaler->out_buf_y_st_addr = IMGU_VMEM1_INT_BUF_ADDR;
+ scaler->out_buf_y_line_stride = stripe_params[s].block_width /
+ IMGU_VMEM1_ELEMS_PER_VEC;
+ scaler->out_buf_y_buffer_stride = IMGU_VMEM1_BUF_SIZE;
+ scaler->out_buf_u_st_addr = IMGU_VMEM1_INT_BUF_ADDR +
+ IMGU_VMEM1_U_OFFSET;
+ scaler->out_buf_v_st_addr = IMGU_VMEM1_INT_BUF_ADDR +
+ IMGU_VMEM1_V_OFFSET;
+ scaler->out_buf_uv_line_stride = stripe_params[s].block_width /
+ IMGU_VMEM1_ELEMS_PER_VEC / 2;
+ scaler->out_buf_uv_buffer_stride = IMGU_VMEM1_BUF_SIZE;
+ scaler->out_buf_nr_buffers = IMGU_OSYS_NUM_INTERM_BUFFERS;
+
+ /* Intermediate buffers */
+ scaler->int_buf_y_st_addr = IMGU_VMEM2_BUF_Y_ADDR;
+ scaler->int_buf_y_line_stride = IMGU_VMEM2_BUF_Y_STRIDE;
+ scaler->int_buf_u_st_addr = IMGU_VMEM2_BUF_U_ADDR;
+ scaler->int_buf_v_st_addr = IMGU_VMEM2_BUF_V_ADDR;
+ scaler->int_buf_uv_line_stride = IMGU_VMEM2_BUF_UV_STRIDE;
+ scaler->int_buf_height = IMGU_VMEM2_LINES_PER_BLOCK;
+ scaler->int_buf_chunk_width = stripe_params[s].chunk_height;
+ scaler->int_buf_chunk_height = stripe_params[s].block_width;
+
+ /* Context buffers */
+ scaler->ctx_buf_hor_y_st_addr = IMGU_VMEM3_HOR_Y_ADDR;
+ scaler->ctx_buf_hor_u_st_addr = IMGU_VMEM3_HOR_U_ADDR;
+ scaler->ctx_buf_hor_v_st_addr = IMGU_VMEM3_HOR_V_ADDR;
+ scaler->ctx_buf_ver_y_st_addr = IMGU_VMEM3_VER_Y_ADDR;
+ scaler->ctx_buf_ver_u_st_addr = IMGU_VMEM3_VER_U_ADDR;
+ scaler->ctx_buf_ver_v_st_addr = IMGU_VMEM3_VER_V_ADDR;
+
+ /* Addresses for release-input and process-output tokens */
+ scaler->release_inp_buf_addr = fifo_addr_ack;
+ scaler->release_inp_buf_en = 1;
+ scaler->release_out_buf_en = 1;
+ scaler->process_out_buf_addr = fifo_addr_fmt;
+
+ /* Settings dimensions, padding, cropping */
+ scaler->input_image_y_width = stripe_params[s].input_width;
+ scaler->input_image_y_height = stripe_params[s].input_height;
+ scaler->input_image_y_start_column =
+ stripe_params[s].start_column_y;
+ scaler->input_image_uv_start_column =
+ stripe_params[s].start_column_uv;
+ scaler->input_image_y_left_pad = stripe_params[s].pad_left_y;
+ scaler->input_image_uv_left_pad = stripe_params[s].pad_left_uv;
+ scaler->input_image_y_right_pad = stripe_params[s].pad_right_y;
+ scaler->input_image_uv_right_pad =
+ stripe_params[s].pad_right_uv;
+ scaler->input_image_y_top_pad = stripe_params[s].pad_top_y;
+ scaler->input_image_uv_top_pad = stripe_params[s].pad_top_uv;
+ scaler->input_image_y_bottom_pad =
+ stripe_params[s].pad_bottom_y;
+ scaler->input_image_uv_bottom_pad =
+ stripe_params[s].pad_bottom_uv;
+ scaler->processing_mode = stripe_params[s].processing_mode;
+ scaler->scaling_ratio = stripe_params[s].phase_step;
+ scaler->y_left_phase_init = stripe_params[s].phase_init_left_y;
+ scaler->uv_left_phase_init =
+ stripe_params[s].phase_init_left_uv;
+ scaler->y_top_phase_init = stripe_params[s].phase_init_top_y;
+ scaler->uv_top_phase_init = stripe_params[s].phase_init_top_uv;
+ scaler->coeffs_exp_shift = stripe_params[s].exp_shift;
+ scaler->out_y_left_crop = stripe_params[s].crop_left_y;
+ scaler->out_uv_left_crop = stripe_params[s].crop_left_uv;
+ scaler->out_y_top_crop = stripe_params[s].crop_top_y;
+ scaler->out_uv_top_crop = stripe_params[s].crop_top_uv;
+
+ for (pin = 0; pin < IMGU_ABI_OSYS_PINS; pin++) {
+ int in_fifo_addr;
+ int out_fifo_addr;
+ int block_width_vecs;
+ int input_width_s;
+ int input_width_vecs;
+ int input_buf_y_st_addr;
+ int input_buf_u_st_addr;
+ int input_buf_v_st_addr;
+ int input_buf_y_line_stride;
+ int input_buf_uv_line_stride;
+ int output_buf_y_line_stride;
+ int output_buf_uv_line_stride;
+ int output_buf_nr_y_lines;
+ int block_height;
+ int block_width;
+ struct imgu_abi_osys_frame_params *fr_pr;
+
+ fr_pr = &osys->frame[pin].param;
+
+ /* Frame parameters */
+ fr_pr->enable = frame_params[pin].enable;
+ fr_pr->format = frame_params[pin].format;
+ fr_pr->mirror = frame_params[pin].mirror;
+ fr_pr->flip = frame_params[pin].flip;
+ fr_pr->tiling = frame_params[pin].tiling;
+ fr_pr->width = frame_params[pin].width;
+ fr_pr->height = frame_params[pin].height;
+ fr_pr->stride = frame_params[pin].stride;
+ fr_pr->scaled = frame_params[pin].scaled;
+
+ /* Stripe parameters */
+ osys->stripe[s].crop_top[pin] =
+ frame_params[pin].crop_top;
+ osys->stripe[s].input_width =
+ stripe_params[s].input_width;
+ osys->stripe[s].input_height =
+ stripe_params[s].input_height;
+ osys->stripe[s].block_height =
+ stripe_params[s].block_height;
+ osys->stripe[s].block_width =
+ stripe_params[s].block_width;
+ osys->stripe[s].output_width[pin] =
+ stripe_params[s].output_width[pin];
+ osys->stripe[s].output_height[pin] =
+ stripe_params[s].output_height[pin];
+
+ if (s == 0) {
+ /* Only first stripe should do left cropping */
+ osys->stripe[s].crop_left[pin] =
+ frame_params[pin].crop_left;
+ osys->stripe[s].output_offset[pin] =
+ stripe_params[s].output_offset[pin];
+ } else {
+ /*
+ * Stripe offset for other strips should be
+ * adjusted according to the cropping done
+ * at the first strip
+ */
+ osys->stripe[s].crop_left[pin] = 0;
+ osys->stripe[s].output_offset[pin] =
+ (stripe_params[s].output_offset[pin] -
+ osys->stripe[0].crop_left[pin]);
+ }
+
+ if (!frame_params[pin].enable)
+ continue;
+
+ /* Formatter: configurations */
+
+ /*
+ * Get the dimensions of the input blocks of the
+ * formatter, which is the same as the output
+ * blocks of the scaler.
+ */
+ if (frame_params[pin].scaled) {
+ block_height = stripe_params[s].block_height;
+ block_width = stripe_params[s].block_width;
+ } else {
+ block_height = IMGU_OSYS_BLOCK_HEIGHT;
+ block_width = IMGU_OSYS_BLOCK_WIDTH;
+ }
+ block_width_vecs =
+ block_width / IMGU_VMEM1_ELEMS_PER_VEC;
+ /*
+ * The input/output line stride depends on the
+ * block size.
+ */
+ input_buf_y_line_stride = block_width_vecs;
+ input_buf_uv_line_stride = block_width_vecs / 2;
+ output_buf_y_line_stride = block_width_vecs;
+ output_buf_uv_line_stride = block_width_vecs / 2;
+ output_buf_nr_y_lines = block_height;
+ if (frame_params[pin].format ==
+ IMGU_ABI_OSYS_FORMAT_NV12 ||
+ frame_params[pin].format ==
+ IMGU_ABI_OSYS_FORMAT_NV21)
+ output_buf_uv_line_stride =
+ output_buf_y_line_stride;
+
+ /*
+ * Tiled outputs use a different output buffer
+ * configuration. The input (= scaler output) block
+ * width translates to a tile height, and the block
+ * height to the tile width. The default block size of
+ * 128x32 maps exactly onto a 4kB tile (512x8) for Y.
+ * For UV, the tile width is always half.
+ */
+ if (frame_params[pin].tiling) {
+ output_buf_nr_y_lines = 8;
+ output_buf_y_line_stride = 512 /
+ IMGU_VMEM1_ELEMS_PER_VEC;
+ output_buf_uv_line_stride = 256 /
+ IMGU_VMEM1_ELEMS_PER_VEC;
+ }
+
+ /*
+ * Store the output buffer line stride. Will be
+ * used to compute buffer offsets in boundary
+ * conditions when output blocks are partially
+ * outside the image.
+ */
+ osys->stripe[s].buf_stride[pin] =
+ output_buf_y_line_stride *
+ IMGU_HIVE_OF_SYS_OF_SYSTEM_NWAYS;
+ if (frame_params[pin].scaled) {
+ /*
+ * The input buffs are the intermediate
+ * buffers (scalers' output)
+ */
+ input_buf_y_st_addr = IMGU_VMEM1_INT_BUF_ADDR;
+ input_buf_u_st_addr = IMGU_VMEM1_INT_BUF_ADDR +
+ IMGU_VMEM1_U_OFFSET;
+ input_buf_v_st_addr = IMGU_VMEM1_INT_BUF_ADDR +
+ IMGU_VMEM1_V_OFFSET;
+ } else {
+ /*
+ * The input bufferss are the buffers
+ * filled by the SP
+ */
+ input_buf_y_st_addr = IMGU_VMEM1_INP_BUF_ADDR;
+ input_buf_u_st_addr = IMGU_VMEM1_INP_BUF_ADDR +
+ IMGU_VMEM1_U_OFFSET;
+ input_buf_v_st_addr = IMGU_VMEM1_INP_BUF_ADDR +
+ IMGU_VMEM1_V_OFFSET;
+ }
+
+ /*
+ * The formatter input width must be rounded to
+ * the block width. Otherwise the formatter will
+ * not recognize the end of the line, resulting
+ * in incorrect tiling (system may hang!) and
+ * possibly other problems.
+ */
+ input_width_s =
+ roundup(stripe_params[s].output_width[pin],
+ block_width);
+ input_width_vecs = input_width_s /
+ IMGU_VMEM1_ELEMS_PER_VEC;
+ out_fifo_addr = IMGU_FIFO_ADDR_FMT_TO_SP;
+ /*
+ * Process-output tokens must be sent to the SP.
+ * When scaling, the release-input tokens can be
+ * sent directly to the scaler, otherwise the
+ * formatter should send them to the SP.
+ */
+ if (frame_params[pin].scaled)
+ in_fifo_addr = IMGU_FIFO_ADDR_FMT_TO_SCALER;
+ else
+ in_fifo_addr = IMGU_FIFO_ADDR_FMT_TO_SP;
+
+ /* Formatter */
+ param = &osys->formatter[s][pin].param;
+
+ param->format = frame_params[pin].format;
+ param->flip = frame_params[pin].flip;
+ param->mirror = frame_params[pin].mirror;
+ param->tiling = frame_params[pin].tiling;
+ param->reduce_range = frame_params[pin].reduce_range;
+ param->alpha_blending = 0;
+ param->release_inp_addr = in_fifo_addr;
+ param->release_inp_en = 1;
+ param->process_out_buf_addr = out_fifo_addr;
+ param->image_width_vecs = input_width_vecs;
+ param->image_height_lines =
+ stripe_params[s].output_height[pin];
+ param->inp_buff_y_st_addr = input_buf_y_st_addr;
+ param->inp_buff_y_line_stride = input_buf_y_line_stride;
+ param->inp_buff_y_buffer_stride = IMGU_VMEM1_BUF_SIZE;
+ param->int_buff_u_st_addr = input_buf_u_st_addr;
+ param->int_buff_v_st_addr = input_buf_v_st_addr;
+ param->inp_buff_uv_line_stride =
+ input_buf_uv_line_stride;
+ param->inp_buff_uv_buffer_stride = IMGU_VMEM1_BUF_SIZE;
+ param->out_buff_level = 0;
+ param->out_buff_nr_y_lines = output_buf_nr_y_lines;
+ param->out_buff_u_st_offset = IMGU_VMEM1_U_OFFSET;
+ param->out_buff_v_st_offset = IMGU_VMEM1_V_OFFSET;
+ param->out_buff_y_line_stride =
+ output_buf_y_line_stride;
+ param->out_buff_uv_line_stride =
+ output_buf_uv_line_stride;
+ param->hist_buff_st_addr = IMGU_VMEM1_HST_BUF_ADDR;
+ param->hist_buff_line_stride =
+ IMGU_VMEM1_HST_BUF_STRIDE;
+ param->hist_buff_nr_lines = IMGU_VMEM1_HST_BUF_NLINES;
+ }
+ }
+
+ block_stripes[0].offset = 0;
+ if (stripes <= 1) {
+ block_stripes[0].width = stripe_params[0].input_width;
+ block_stripes[0].height = stripe_params[0].input_height;
+ } else {
+ struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->current_binary];
+ unsigned int sp_block_width = IPU3_UAPI_ISP_VEC_ELEMS *
+ bi->info.isp.sp.block.block_width;
+
+ block_stripes[0].width = roundup(stripe_params[0].input_width,
+ sp_block_width);
+ block_stripes[1].offset =
+ rounddown(css->rect[IPU3_CSS_RECT_GDC].width -
+ stripe_params[1].input_width, sp_block_width);
+ block_stripes[1].width =
+ roundup(css->rect[IPU3_CSS_RECT_GDC].width -
+ block_stripes[1].offset, sp_block_width);
+ block_stripes[0].height = css->rect[IPU3_CSS_RECT_GDC].height;
+ block_stripes[1].height = block_stripes[0].height;
+ }
+}
+
+/*********************** Mostly 3A operations ******************************/
+
+/*
+ * This function creates a "TO-DO list" (operations) for the sp code.
+ *
+ * There are 2 types of operations:
+ * 1. Transfer: Issue DMA transfer request for copying grid cells from DDR to
+ * accelerator space (NOTE that this space is limited) associated data:
+ * DDR address + accelerator's config set index(acc's address).
+ *
+ * 2. Issue "Process Lines Command" to shd accelerator
+ * associated data: #lines + which config set to use (actually, accelerator
+ * will use x AND (x+1)%num_of_sets - NOTE that this implies the restriction
+ * of not touching config sets x & (x+1)%num_of_sets when process_lines(x)
+ * is active).
+ *
+ * Basically there are 2 types of operations "chunks":
+ * 1. "initial chunk": Initially, we do as much transfers as we can (and need)
+ * [0 - max sets(3) ] followed by 1 or 2 "process lines" operations.
+ *
+ * 2. "regular chunk" - 1 transfer followed by 1 process line operation.
+ * (in some cases we might need additional transfer ate the last chunk).
+ *
+ * for some case:
+ * --> init
+ * tr (0)
+ * tr (1)
+ * tr (2)
+ * pl (0)
+ * pl (1)
+ * --> ack (0)
+ * tr (3)
+ * pl (2)
+ * --> ack (1)
+ * pl (3)
+ * --> ack (2)
+ * do nothing
+ * --> ack (3)
+ * do nothing
+ */
+
+static int
+ipu3_css_shd_ops_calc(struct imgu_abi_shd_intra_frame_operations_data *ops,
+ const struct ipu3_uapi_shd_grid_config *grid,
+ unsigned int image_height)
+{
+ unsigned int block_height = 1 << grid->block_height_log2;
+ unsigned int grid_height_per_slice = grid->grid_height_per_slice;
+ unsigned int set_height = grid_height_per_slice * block_height;
+
+ /* We currently support only abs(y_start) > grid_height_per_slice */
+ unsigned int positive_y_start = (unsigned int)-grid->y_start;
+ unsigned int first_process_lines =
+ set_height - (positive_y_start % set_height);
+ unsigned int last_set_height;
+ unsigned int num_of_sets;
+
+ struct imgu_abi_acc_operation *p_op;
+ struct imgu_abi_acc_process_lines_cmd_data *p_pl;
+ struct imgu_abi_shd_transfer_luts_set_data *p_tr;
+
+ unsigned int op_idx, pl_idx, tr_idx;
+ unsigned char tr_set_num, pl_cfg_set;
+
+ /*
+ * When the number of lines for the last process lines command
+ * is equal to a set height, we need another line of grid cell -
+ * additional transfer is required.
+ */
+ unsigned char last_tr = 0;
+
+ /* Add "process lines" command to the list of operations */
+ bool add_pl;
+ /* Add DMA xfer (config set) command to the list of ops */
+ bool add_tr;
+
+ /*
+ * Available partial grid (the part that fits into #IMGU_SHD_SETS sets)
+ * doesn't cover whole frame - need to process in chunks
+ */
+ if (image_height > first_process_lines) {
+ last_set_height =
+ (image_height - first_process_lines) % set_height;
+ num_of_sets = last_set_height > 0 ?
+ (image_height - first_process_lines) / set_height + 2 :
+ (image_height - first_process_lines) / set_height + 1;
+ last_tr = (set_height - last_set_height <= block_height ||
+ last_set_height == 0) ? 1 : 0;
+ } else { /* partial grid covers whole frame */
+ last_set_height = 0;
+ num_of_sets = 1;
+ first_process_lines = image_height;
+ last_tr = set_height - image_height <= block_height ? 1 : 0;
+ }
+
+ /* Init operations lists and counters */
+ p_op = ops->operation_list;
+ op_idx = 0;
+ p_pl = ops->process_lines_data;
+ pl_idx = 0;
+ p_tr = ops->transfer_data;
+ tr_idx = 0;
+
+ memset(ops, 0, sizeof(*ops));
+
+ /* Cyclic counters that holds config set number [0,IMGU_SHD_SETS) */
+ tr_set_num = 0;
+ pl_cfg_set = 0;
+
+ /*
+ * Always start with a transfer - process lines command must be
+ * initiated only after appropriate config sets are in place
+ * (2 configuration sets per process line command, except for last one).
+ */
+ add_pl = false;
+ add_tr = true;
+
+ while (add_pl || add_tr) {
+ /* Transfer ops */
+ if (add_tr) {
+ if (op_idx >= IMGU_ABI_SHD_MAX_OPERATIONS ||
+ tr_idx >= IMGU_ABI_SHD_MAX_TRANSFERS)
+ return -EINVAL;
+ p_op[op_idx].op_type =
+ IMGU_ABI_ACC_OPTYPE_TRANSFER_DATA;
+ p_op[op_idx].op_indicator = IMGU_ABI_ACC_OP_IDLE;
+ op_idx++;
+ p_tr[tr_idx].set_number = tr_set_num;
+ tr_idx++;
+ tr_set_num = (tr_set_num + 1) % IMGU_SHD_SETS;
+ }
+
+ /* Process-lines ops */
+ if (add_pl) {
+ if (op_idx >= IMGU_ABI_SHD_MAX_OPERATIONS ||
+ pl_idx >= IMGU_ABI_SHD_MAX_PROCESS_LINES)
+ return -EINVAL;
+ p_op[op_idx].op_type =
+ IMGU_ABI_ACC_OPTYPE_PROCESS_LINES;
+
+ /*
+ * In case we have 2 process lines commands -
+ * don't stop after the first one
+ */
+ if (pl_idx == 0 && num_of_sets != 1)
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_IDLE;
+ /*
+ * Initiate last process lines command -
+ * end of operation list.
+ */
+ else if (pl_idx == num_of_sets - 1)
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_END_OF_OPS;
+ /*
+ * Intermediate process line command - end of operation
+ * "chunk" (meaning few "transfers" followed by few
+ * "process lines" commands).
+ */
+ else
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_END_OF_ACK;
+
+ op_idx++;
+
+ /* first process line operation */
+ if (pl_idx == 0)
+ p_pl[pl_idx].lines = first_process_lines;
+ /* Last process line operation */
+ else if (pl_idx == num_of_sets - 1 &&
+ last_set_height > 0)
+ p_pl[pl_idx].lines = last_set_height;
+ else /* "regular" process lines operation */
+ p_pl[pl_idx].lines = set_height;
+
+ p_pl[pl_idx].cfg_set = pl_cfg_set;
+ pl_idx++;
+ pl_cfg_set = (pl_cfg_set + 1) % IMGU_SHD_SETS;
+ }
+
+ /*
+ * Initially, we always transfer
+ * min(IMGU_SHD_SETS, num_of_sets) - after that we fill in the
+ * corresponding process lines commands.
+ */
+ if (tr_idx == IMGU_SHD_SETS ||
+ tr_idx == num_of_sets + last_tr) {
+ add_tr = false;
+ add_pl = true;
+ }
+
+ /*
+ * We have finished the "initial" operations chunk -
+ * be ready to get more chunks.
+ */
+ if (pl_idx == 2) {
+ add_tr = true;
+ add_pl = true;
+ }
+
+ /* Stop conditions for each operation type */
+ if (tr_idx == num_of_sets + last_tr)
+ add_tr = false;
+ if (pl_idx == num_of_sets)
+ add_pl = false;
+ }
+
+ return 0;
+}
+
+/*
+ * The follow handshake procotol is the same for AF, AWB and AWB FR.
+ *
+ * for n sets of meta-data, the flow is:
+ * --> init
+ * process-lines (0)
+ * process-lines (1) eoc
+ * --> ack (0)
+ * read-meta-data (0)
+ * process-lines (2) eoc
+ * --> ack (1)
+ * read-meta-data (1)
+ * process-lines (3) eoc
+ * ...
+ *
+ * --> ack (n-3)
+ * read-meta-data (n-3)
+ * process-lines (n-1) eoc
+ * --> ack (n-2)
+ * read-meta-data (n-2) eoc
+ * --> ack (n-1)
+ * read-meta-data (n-1) eof
+ *
+ * for 2 sets we get:
+ * --> init
+ * pl (0)
+ * pl (1) eoc
+ * --> ack (0)
+ * pl (2) - rest of image, if applicable)
+ * rmd (0) eoc
+ * --> ack (1)
+ * rmd (1) eof
+ * --> (ack (2))
+ * do nothing
+ *
+ * for only one set:
+ *
+ * --> init
+ * pl(0) eoc
+ * --> ack (0)
+ * rmd (0) eof
+ *
+ * grid smaller than image case
+ * for example 128x128 grid (block size 8x8, 16x16 num of blocks)
+ * start at (0,0)
+ * 1st set holds 160 cells - 10 blocks vertical, 16 horizontal
+ * => 1st process lines = 80
+ * we're left with 128-80=48 lines (6 blocks vertical)
+ * => 2nd process lines = 48
+ * last process lines to cover the image - image_height - 128
+ *
+ * --> init
+ * pl (0) first
+ * pl (1) last-in-grid
+ * --> ack (0)
+ * rmd (0)
+ * pl (2) after-grid
+ * --> ack (1)
+ * rmd (1) eof
+ * --> ack (2)
+ * do nothing
+ */
+struct process_lines {
+ unsigned int image_height;
+ unsigned short grid_height;
+ unsigned short block_height;
+ unsigned short y_start;
+ unsigned char grid_height_per_slice;
+
+ unsigned short max_op; /* max operation */
+ unsigned short max_tr; /* max transaction */
+ unsigned char acc_enable;
+};
+
+/* Helper to config intra_frame_operations_data. */
+static int
+ipu3_css_acc_process_lines(const struct process_lines *pl,
+ struct imgu_abi_acc_operation *p_op,
+ struct imgu_abi_acc_process_lines_cmd_data *p_pl,
+ struct imgu_abi_acc_transfer_op_data *p_tr)
+{
+ unsigned short op_idx = 0, pl_idx = 0, tr_idx = 0;
+ unsigned char tr_set_num = 0, pl_cfg_set = 0;
+ const unsigned short grid_last_line =
+ pl->y_start + pl->grid_height * pl->block_height;
+ const unsigned short process_lines =
+ pl->grid_height_per_slice * pl->block_height;
+
+ unsigned int process_lines_after_grid;
+ unsigned short first_process_lines;
+ unsigned short last_process_lines_in_grid;
+
+ unsigned short num_of_process_lines;
+ unsigned short num_of_sets;
+
+ if (pl->grid_height_per_slice == 0)
+ return -EINVAL;
+
+ if (pl->acc_enable && grid_last_line > pl->image_height)
+ return -EINVAL;
+
+ num_of_sets = pl->grid_height / pl->grid_height_per_slice;
+ if (num_of_sets * pl->grid_height_per_slice < pl->grid_height)
+ num_of_sets++;
+
+ /* Account for two line delay inside the FF */
+ if (pl->max_op == IMGU_ABI_AF_MAX_OPERATIONS) {
+ first_process_lines = process_lines + pl->y_start + 2;
+ last_process_lines_in_grid =
+ (grid_last_line - first_process_lines) -
+ ((num_of_sets - 2) * process_lines) + 4;
+ process_lines_after_grid =
+ pl->image_height - grid_last_line - 4;
+ } else {
+ first_process_lines = process_lines + pl->y_start;
+ last_process_lines_in_grid =
+ (grid_last_line - first_process_lines) -
+ ((num_of_sets - 2) * process_lines);
+ process_lines_after_grid = pl->image_height - grid_last_line;
+ }
+
+ num_of_process_lines = num_of_sets;
+ if (process_lines_after_grid > 0)
+ num_of_process_lines++;
+
+ while (tr_idx < num_of_sets || pl_idx < num_of_process_lines) {
+ /* Read meta-data */
+ if (pl_idx >= 2 || (pl_idx == 1 && num_of_sets == 1)) {
+ if (op_idx >= pl->max_op || tr_idx >= pl->max_tr)
+ return -EINVAL;
+
+ p_op[op_idx].op_type =
+ IMGU_ABI_ACC_OPTYPE_TRANSFER_DATA;
+
+ if (tr_idx == num_of_sets - 1)
+ /* The last operation is always a tr */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_END_OF_OPS;
+ else if (tr_idx == num_of_sets - 2)
+ if (process_lines_after_grid == 0)
+ /*
+ * No additional pl op left -
+ * this op is left as lats of cycle
+ */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_END_OF_ACK;
+ else
+ /*
+ * We still have to process-lines after
+ * the grid so have one more pl op
+ */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_IDLE;
+ else
+ /* Default - usually there's a pl after a tr */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_IDLE;
+
+ op_idx++;
+ if (p_tr) {
+ p_tr[tr_idx].set_number = tr_set_num;
+ tr_set_num = 1 - tr_set_num;
+ }
+ tr_idx++;
+ }
+
+ /* process_lines */
+ if (pl_idx < num_of_process_lines) {
+ if (op_idx >= pl->max_op || pl_idx >= pl->max_tr)
+ return -EINVAL;
+
+ p_op[op_idx].op_type =
+ IMGU_ABI_ACC_OPTYPE_PROCESS_LINES;
+ if (pl_idx == 0)
+ if (num_of_process_lines == 1)
+ /* Only one pl op */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_END_OF_ACK;
+ else
+ /* On init - do two pl ops */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_IDLE;
+ else
+ /* Usually pl is the end of the ack cycle */
+ p_op[op_idx].op_indicator =
+ IMGU_ABI_ACC_OP_END_OF_ACK;
+
+ op_idx++;
+
+ if (pl_idx == 0)
+ /* First process line */
+ p_pl[pl_idx].lines = first_process_lines;
+ else if (pl_idx == num_of_sets - 1)
+ /* Last in grid */
+ p_pl[pl_idx].lines = last_process_lines_in_grid;
+ else if (pl_idx == num_of_process_lines - 1)
+ /* After the grid */
+ p_pl[pl_idx].lines = process_lines_after_grid;
+ else
+ /* Inside the grid */
+ p_pl[pl_idx].lines = process_lines;
+
+ if (p_tr) {
+ p_pl[pl_idx].cfg_set = pl_cfg_set;
+ pl_cfg_set = 1 - pl_cfg_set;
+ }
+ pl_idx++;
+ }
+ }
+
+ return 0;
+}
+
+static int ipu3_css_af_ops_calc(struct ipu3_css *css,
+ struct imgu_abi_af_config *af_config)
+{
+ struct imgu_abi_af_intra_frame_operations_data *to =
+ &af_config->operations_data;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+
+ struct process_lines pl = {
+ .image_height = css->rect[IPU3_CSS_RECT_BDS].height,
+ .grid_height = af_config->config.grid_cfg.height,
+ .block_height =
+ 1 << af_config->config.grid_cfg.block_height_log2,
+ .y_start = af_config->config.grid_cfg.y_start &
+ IPU3_UAPI_GRID_START_MASK,
+ .grid_height_per_slice =
+ af_config->stripes[0].grid_cfg.height_per_slice,
+ .max_op = IMGU_ABI_AF_MAX_OPERATIONS,
+ .max_tr = IMGU_ABI_AF_MAX_TRANSFERS,
+ .acc_enable = bi->info.isp.sp.enable.af,
+ };
+
+ return ipu3_css_acc_process_lines(&pl, to->ops, to->process_lines_data,
+ NULL);
+}
+
+static int
+ipu3_css_awb_fr_ops_calc(struct ipu3_css *css,
+ struct imgu_abi_awb_fr_config *awb_fr_config)
+{
+ struct imgu_abi_awb_fr_intra_frame_operations_data *to =
+ &awb_fr_config->operations_data;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+ struct process_lines pl = {
+ .image_height = css->rect[IPU3_CSS_RECT_BDS].height,
+ .grid_height = awb_fr_config->config.grid_cfg.height,
+ .block_height =
+ 1 << awb_fr_config->config.grid_cfg.block_height_log2,
+ .y_start = awb_fr_config->config.grid_cfg.y_start &
+ IPU3_UAPI_GRID_START_MASK,
+ .grid_height_per_slice =
+ awb_fr_config->stripes[0].grid_cfg.height_per_slice,
+ .max_op = IMGU_ABI_AWB_FR_MAX_OPERATIONS,
+ .max_tr = IMGU_ABI_AWB_FR_MAX_PROCESS_LINES,
+ .acc_enable = bi->info.isp.sp.enable.awb_fr_acc,
+ };
+
+ return ipu3_css_acc_process_lines(&pl, to->ops, to->process_lines_data,
+ NULL);
+}
+
+static int ipu3_css_awb_ops_calc(struct ipu3_css *css,
+ struct imgu_abi_awb_config *awb_config)
+{
+ struct imgu_abi_awb_intra_frame_operations_data *to =
+ &awb_config->operations_data;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+
+ struct process_lines pl = {
+ .image_height = css->rect[IPU3_CSS_RECT_BDS].height,
+ .grid_height = awb_config->config.grid.height,
+ .block_height =
+ 1 << awb_config->config.grid.block_height_log2,
+ .y_start = awb_config->config.grid.y_start,
+ .grid_height_per_slice =
+ awb_config->stripes[0].grid.height_per_slice,
+ .max_op = IMGU_ABI_AWB_MAX_OPERATIONS,
+ .max_tr = IMGU_ABI_AWB_MAX_TRANSFERS,
+ .acc_enable = bi->info.isp.sp.enable.awb_acc,
+ };
+
+ return ipu3_css_acc_process_lines(&pl, to->ops, to->process_lines_data,
+ to->transfer_data);
+}
+
+static u16 ipu3_css_grid_end(u16 start, u8 width, u8 block_width_log2)
+{
+ return (start & IPU3_UAPI_GRID_START_MASK) +
+ (width << block_width_log2) - 1;
+}
+
+static void ipu3_css_grid_end_calc(struct ipu3_uapi_grid_config *grid_cfg)
+{
+ grid_cfg->x_end = ipu3_css_grid_end(grid_cfg->x_start, grid_cfg->width,
+ grid_cfg->block_width_log2);
+ grid_cfg->y_end = ipu3_css_grid_end(grid_cfg->y_start, grid_cfg->height,
+ grid_cfg->block_height_log2);
+}
+
+/****************** config computation *****************************/
+
+static void ipu3_css_cfg_acc_stripe(struct ipu3_css *css,
+ struct imgu_abi_acc_param *acc)
+{
+ const struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->current_binary];
+ const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
+ const unsigned int F = IPU3_UAPI_ISP_VEC_ELEMS * 2;
+ struct ipu3_css_scaler_info scaler_luma, scaler_chroma;
+ unsigned int bds_ds, i;
+
+ memset(acc, 0, sizeof(*acc));
+
+ /* acc_param: osys_config */
+
+ ipu3_css_osys_calc(css, stripes, &acc->osys, &scaler_luma,
+ &scaler_chroma, acc->stripe.block_stripes);
+
+ /* acc_param: stripe data */
+
+ /*
+ * For the striped case the approach is as follows:
+ * 1. down-scaled stripes are calculated - with 128 overlap
+ * (this is the main limiter therefore it's first)
+ * 2. input stripes are derived by up-scaling the down-scaled stripes
+ * (there are no alignment requirements on input stripes)
+ * 3. output stripes are derived from down-scaled stripes too
+ */
+
+ acc->stripe.num_of_stripes = stripes;
+ acc->stripe.input_frame.width =
+ css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
+ acc->stripe.input_frame.height =
+ css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
+ acc->stripe.input_frame.bayer_order =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
+
+ for (i = 0; i < stripes; i++)
+ acc->stripe.bds_out_stripes[i].height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->stripe.bds_out_stripes[0].offset = 0;
+ if (stripes <= 1) {
+ acc->stripe.bds_out_stripes[0].width =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, F);
+ } else {
+ /* Image processing is divided into two stripes */
+ acc->stripe.bds_out_stripes[0].width =
+ acc->stripe.bds_out_stripes[1].width =
+ (css->rect[IPU3_CSS_RECT_BDS].width / 2 & ~(F - 1)) + F;
+ /*
+ * Sum of width of the two stripes should not be smaller
+ * than output width and must be even times of overlapping
+ * unit f.
+ */
+ if ((css->rect[IPU3_CSS_RECT_BDS].width / F & 1) !=
+ !!(css->rect[IPU3_CSS_RECT_BDS].width & (F - 1)))
+ acc->stripe.bds_out_stripes[0].width += F;
+ if ((css->rect[IPU3_CSS_RECT_BDS].width / F & 1) &&
+ (css->rect[IPU3_CSS_RECT_BDS].width & (F - 1))) {
+ acc->stripe.bds_out_stripes[0].width += F;
+ acc->stripe.bds_out_stripes[1].width += F;
+ }
+ /* Overlap between stripes is IPU3_UAPI_ISP_VEC_ELEMS * 4 */
+ acc->stripe.bds_out_stripes[1].offset =
+ acc->stripe.bds_out_stripes[0].width - 2 * F;
+ }
+
+ acc->stripe.effective_stripes[0].height =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ acc->stripe.effective_stripes[0].offset = 0;
+ acc->stripe.bds_out_stripes_no_overlap[0].height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->stripe.bds_out_stripes_no_overlap[0].offset = 0;
+ acc->stripe.output_stripes[0].height =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ acc->stripe.output_stripes[0].offset = 0;
+ if (stripes <= 1) {
+ acc->stripe.down_scaled_stripes[0].width =
+ css->rect[IPU3_CSS_RECT_BDS].width;
+ acc->stripe.down_scaled_stripes[0].height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->stripe.down_scaled_stripes[0].offset = 0;
+
+ acc->stripe.effective_stripes[0].width =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].width;
+ acc->stripe.bds_out_stripes_no_overlap[0].width =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, F);
+
+ acc->stripe.output_stripes[0].width =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ } else { /* Two stripes */
+ bds_ds = css->rect[IPU3_CSS_RECT_EFFECTIVE].width *
+ IMGU_BDS_GRANULARITY /
+ css->rect[IPU3_CSS_RECT_BDS].width;
+
+ acc->stripe.down_scaled_stripes[0] =
+ acc->stripe.bds_out_stripes[0];
+ acc->stripe.down_scaled_stripes[1] =
+ acc->stripe.bds_out_stripes[1];
+ if (!IS_ALIGNED(css->rect[IPU3_CSS_RECT_BDS].width, F))
+ acc->stripe.down_scaled_stripes[1].width += -F +
+ (css->rect[IPU3_CSS_RECT_BDS].width & (F - 1));
+
+ acc->stripe.effective_stripes[0].width = bds_ds *
+ acc->stripe.down_scaled_stripes[0].width /
+ IMGU_BDS_GRANULARITY;
+ acc->stripe.effective_stripes[1].width = bds_ds *
+ acc->stripe.down_scaled_stripes[1].width /
+ IMGU_BDS_GRANULARITY;
+ acc->stripe.effective_stripes[1].height =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ acc->stripe.effective_stripes[1].offset = bds_ds *
+ acc->stripe.down_scaled_stripes[1].offset /
+ IMGU_BDS_GRANULARITY;
+
+ acc->stripe.bds_out_stripes_no_overlap[0].width =
+ acc->stripe.bds_out_stripes_no_overlap[1].offset =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, 2 * F) / 2;
+ acc->stripe.bds_out_stripes_no_overlap[1].width =
+ DIV_ROUND_UP(css->rect[IPU3_CSS_RECT_BDS].width, F) /
+ 2 * F;
+ acc->stripe.bds_out_stripes_no_overlap[1].height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+
+ acc->stripe.output_stripes[0].width =
+ acc->stripe.down_scaled_stripes[0].width - F;
+ acc->stripe.output_stripes[1].width =
+ acc->stripe.down_scaled_stripes[1].width - F;
+ acc->stripe.output_stripes[1].height =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ acc->stripe.output_stripes[1].offset =
+ acc->stripe.output_stripes[0].width;
+ }
+
+ acc->stripe.output_system_in_frame_width =
+ css->rect[IPU3_CSS_RECT_GDC].width;
+ acc->stripe.output_system_in_frame_height =
+ css->rect[IPU3_CSS_RECT_GDC].height;
+
+ acc->stripe.effective_frame_width =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].width;
+ acc->stripe.bds_frame_width = css->rect[IPU3_CSS_RECT_BDS].width;
+ acc->stripe.out_frame_width =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ acc->stripe.out_frame_height =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ acc->stripe.gdc_in_buffer_width =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline /
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel;
+ acc->stripe.gdc_in_buffer_height =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
+ acc->stripe.gdc_in_buffer_offset_x = IMGU_GDC_BUF_X;
+ acc->stripe.gdc_in_buffer_offset_y = IMGU_GDC_BUF_Y;
+ acc->stripe.display_frame_width =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ acc->stripe.display_frame_height =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ acc->stripe.bds_aligned_frame_width =
+ roundup(css->rect[IPU3_CSS_RECT_BDS].width,
+ 2 * IPU3_UAPI_ISP_VEC_ELEMS);
+
+ if (stripes > 1)
+ acc->stripe.half_overlap_vectors =
+ IMGU_STRIPE_FIXED_HALF_OVERLAP;
+ else
+ acc->stripe.half_overlap_vectors = 0;
+}
+
+static void ipu3_css_cfg_acc_dvs(struct ipu3_css *css,
+ struct imgu_abi_acc_param *acc)
+{
+ unsigned int i;
+
+ /* Disable DVS statistics */
+ acc->dvs_stat.operations_data.process_lines_data[0].lines =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->dvs_stat.operations_data.process_lines_data[0].cfg_set = 0;
+ acc->dvs_stat.operations_data.ops[0].op_type =
+ IMGU_ABI_ACC_OPTYPE_PROCESS_LINES;
+ acc->dvs_stat.operations_data.ops[0].op_indicator =
+ IMGU_ABI_ACC_OP_NO_OPS;
+ for (i = 0; i < IMGU_ABI_DVS_STAT_LEVELS; i++)
+ acc->dvs_stat.cfg.grd_config[i].enable = 0;
+}
+
+static void acc_bds_per_stripe_data(struct ipu3_css *css,
+ struct imgu_abi_acc_param *acc,
+ const int i)
+{
+ acc->bds.per_stripe.aligned_data[i].data.crop.hor_crop_en = 0;
+ acc->bds.per_stripe.aligned_data[i].data.crop.hor_crop_start = 0;
+ acc->bds.per_stripe.aligned_data[i].data.crop.hor_crop_end = 0;
+ acc->bds.per_stripe.aligned_data[i].data.hor_ctrl0 =
+ acc->bds.hor.hor_ctrl0;
+ acc->bds.per_stripe.aligned_data[i].data.hor_ctrl0.out_frame_width =
+ acc->stripe.down_scaled_stripes[i].width;
+ acc->bds.per_stripe.aligned_data[i].data.ver_ctrl1.out_frame_width =
+ acc->stripe.down_scaled_stripes[i].width;
+ acc->bds.per_stripe.aligned_data[i].data.ver_ctrl1.out_frame_height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+}
+
+/*
+ * Configure `acc' parameters. `acc_old' contains the old values (or is NULL)
+ * and `acc_user' contains new prospective values. `use' contains flags
+ * telling which fields to take from the old values (or generate if it is NULL)
+ * and which to take from the new user values.
+ */
+int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+ struct imgu_abi_acc_param *acc,
+ struct imgu_abi_acc_param *acc_old,
+ struct ipu3_uapi_acc_param *acc_user)
+{
+ const struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->current_binary];
+ const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
+ const unsigned int tnr_frame_width =
+ acc->stripe.bds_aligned_frame_width;
+ const unsigned int min_overlap = 10;
+ const struct v4l2_pix_format_mplane *pixm =
+ &css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix;
+ const struct ipu3_css_bds_config *cfg_bds;
+ struct imgu_abi_input_feeder_data *feeder_data;
+
+ unsigned int bds_ds, ofs_x, ofs_y, i, width, height;
+ u8 b_w_log2; /* Block width log2 */
+
+ /* Update stripe using chroma and luma */
+
+ ipu3_css_cfg_acc_stripe(css, acc);
+
+ /* acc_param: input_feeder_config */
+
+ ofs_x = ((pixm->width -
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].width) >> 1) & ~1;
+ ofs_x += css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ IMGU_ABI_BAYER_ORDER_RGGB ||
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ IMGU_ABI_BAYER_ORDER_GBRG ? 1 : 0;
+ ofs_y = ((pixm->height -
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].height) >> 1) & ~1;
+ ofs_y += css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ IMGU_ABI_BAYER_ORDER_BGGR ||
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ IMGU_ABI_BAYER_ORDER_GBRG ? 1 : 0;
+ acc->input_feeder.data.row_stride = pixm->plane_fmt[0].bytesperline;
+ acc->input_feeder.data.start_row_address =
+ ofs_x / IMGU_PIXELS_PER_WORD * IMGU_BYTES_PER_WORD +
+ ofs_y * acc->input_feeder.data.row_stride;
+ acc->input_feeder.data.start_pixel = ofs_x % IMGU_PIXELS_PER_WORD;
+
+ acc->input_feeder.data_per_stripe.input_feeder_data[0].data =
+ acc->input_feeder.data;
+
+ ofs_x += acc->stripe.effective_stripes[1].offset;
+
+ feeder_data =
+ &acc->input_feeder.data_per_stripe.input_feeder_data[1].data;
+ feeder_data->row_stride = acc->input_feeder.data.row_stride;
+ feeder_data->start_row_address =
+ ofs_x / IMGU_PIXELS_PER_WORD * IMGU_BYTES_PER_WORD +
+ ofs_y * acc->input_feeder.data.row_stride;
+ feeder_data->start_pixel = ofs_x % IMGU_PIXELS_PER_WORD;
+
+ /* acc_param: bnr_static_config */
+
+ /*
+ * Originate from user or be the original default values if user has
+ * never set them before, when user gives a new set of parameters,
+ * for each chunk in the parameter structure there is a flag use->xxx
+ * whether to use the user-provided parameter or not. If not, the
+ * parameter remains unchanged in the driver:
+ * it's value is taken from acc_old.
+ */
+ if (use && use->acc_bnr) {
+ /* Take values from user */
+ acc->bnr = acc_user->bnr;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->bnr = acc_old->bnr;
+ } else {
+ /* Calculate from scratch */
+ acc->bnr = ipu3_css_bnr_defaults;
+ }
+
+ acc->bnr.column_size = tnr_frame_width;
+
+ /* acc_param: bnr_static_config_green_disparity */
+
+ if (use && use->acc_green_disparity) {
+ /* Take values from user */
+ acc->green_disparity = acc_user->green_disparity;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->green_disparity = acc_old->green_disparity;
+ } else {
+ /* Calculate from scratch */
+ memset(&acc->green_disparity, 0, sizeof(acc->green_disparity));
+ }
+
+ /* acc_param: dm_config */
+
+ if (use && use->acc_dm) {
+ /* Take values from user */
+ acc->dm = acc_user->dm;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->dm = acc_old->dm;
+ } else {
+ /* Calculate from scratch */
+ acc->dm = ipu3_css_dm_defaults;
+ }
+
+ acc->dm.frame_width = tnr_frame_width;
+
+ /* acc_param: ccm_mat_config */
+
+ if (use && use->acc_ccm) {
+ /* Take values from user */
+ acc->ccm = acc_user->ccm;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->ccm = acc_old->ccm;
+ } else {
+ /* Calculate from scratch */
+ acc->ccm = ipu3_css_ccm_defaults;
+ }
+
+ /* acc_param: gamma_config */
+
+ if (use && use->acc_gamma) {
+ /* Take values from user */
+ acc->gamma = acc_user->gamma;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->gamma = acc_old->gamma;
+ } else {
+ /* Calculate from scratch */
+ acc->gamma.gc_ctrl.enable = 1;
+ acc->gamma.gc_lut = ipu3_css_gamma_lut;
+ }
+
+ /* acc_param: csc_mat_config */
+
+ if (use && use->acc_csc) {
+ /* Take values from user */
+ acc->csc = acc_user->csc;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->csc = acc_old->csc;
+ } else {
+ /* Calculate from scratch */
+ acc->csc = ipu3_css_csc_defaults;
+ }
+
+ /* acc_param: cds_params */
+
+ if (use && use->acc_cds) {
+ /* Take values from user */
+ acc->cds = acc_user->cds;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->cds = acc_old->cds;
+ } else {
+ /* Calculate from scratch */
+ acc->cds = ipu3_css_cds_defaults;
+ }
+
+ /* acc_param: shd_config */
+
+ if (use && use->acc_shd) {
+ /* Take values from user */
+ acc->shd.shd = acc_user->shd.shd;
+ acc->shd.shd_lut = acc_user->shd.shd_lut;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->shd.shd = acc_old->shd.shd;
+ acc->shd.shd_lut = acc_old->shd.shd_lut;
+ } else {
+ /* Calculate from scratch */
+ acc->shd.shd = ipu3_css_shd_defaults;
+ memset(&acc->shd.shd_lut, 0, sizeof(acc->shd.shd_lut));
+ }
+
+ if (acc->shd.shd.grid.width <= 0)
+ return -EINVAL;
+
+ acc->shd.shd.grid.grid_height_per_slice =
+ IMGU_ABI_SHD_MAX_CELLS_PER_SET / acc->shd.shd.grid.width;
+
+ if (acc->shd.shd.grid.grid_height_per_slice <= 0)
+ return -EINVAL;
+
+ acc->shd.shd.general.init_set_vrt_offst_ul =
+ (-acc->shd.shd.grid.y_start >>
+ acc->shd.shd.grid.block_height_log2) %
+ acc->shd.shd.grid.grid_height_per_slice;
+
+ if (ipu3_css_shd_ops_calc(&acc->shd.shd_ops, &acc->shd.shd.grid,
+ css->rect[IPU3_CSS_RECT_BDS].height))
+ return -EINVAL;
+
+ /* acc_param: dvs_stat_config */
+ ipu3_css_cfg_acc_dvs(css, acc);
+
+ /* acc_param: yuvp1_iefd_config */
+
+ if (use && use->acc_iefd) {
+ /* Take values from user */
+ acc->iefd = acc_user->iefd;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->iefd = acc_old->iefd;
+ } else {
+ /* Calculate from scratch */
+ acc->iefd = ipu3_css_iefd_defaults;
+ }
+
+ /* acc_param: yuvp1_yds_config yds_c0 */
+
+ if (use && use->acc_yds_c0) {
+ /* Take values from user */
+ acc->yds_c0 = acc_user->yds_c0;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->yds_c0 = acc_old->yds_c0;
+ } else {
+ /* Calculate from scratch */
+ acc->yds_c0 = ipu3_css_yds_defaults;
+ }
+
+ /* acc_param: yuvp1_chnr_config chnr_c0 */
+
+ if (use && use->acc_chnr_c0) {
+ /* Take values from user */
+ acc->chnr_c0 = acc_user->chnr_c0;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->chnr_c0 = acc_old->chnr_c0;
+ } else {
+ /* Calculate from scratch */
+ acc->chnr_c0 = ipu3_css_chnr_defaults;
+ }
+
+ /* acc_param: yuvp1_y_ee_nr_config */
+
+ if (use && use->acc_y_ee_nr) {
+ /* Take values from user */
+ acc->y_ee_nr = acc_user->y_ee_nr;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->y_ee_nr = acc_old->y_ee_nr;
+ } else {
+ /* Calculate from scratch */
+ acc->y_ee_nr = ipu3_css_y_ee_nr_defaults;
+ }
+
+ /* acc_param: yuvp1_yds_config yds */
+
+ if (use && use->acc_yds) {
+ /* Take values from user */
+ acc->yds = acc_user->yds;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->yds = acc_old->yds;
+ } else {
+ /* Calculate from scratch */
+ acc->yds = ipu3_css_yds_defaults;
+ }
+
+ /* acc_param: yuvp1_chnr_config chnr */
+
+ if (use && use->acc_chnr) {
+ /* Take values from user */
+ acc->chnr = acc_user->chnr;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->chnr = acc_old->chnr;
+ } else {
+ /* Calculate from scratch */
+ acc->chnr = ipu3_css_chnr_defaults;
+ }
+
+ /* acc_param: yuvp2_y_tm_lut_static_config */
+
+ for (i = 0; i < IMGU_ABI_YUVP2_YTM_LUT_ENTRIES; i++)
+ acc->ytm.entries[i] = i * 32;
+ acc->ytm.enable = 0; /* Always disabled on IPU3 */
+
+ /* acc_param: yuvp1_yds_config yds2 */
+
+ if (use && use->acc_yds2) {
+ /* Take values from user */
+ acc->yds2 = acc_user->yds2;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->yds2 = acc_old->yds2;
+ } else {
+ /* Calculate from scratch */
+ acc->yds2 = ipu3_css_yds_defaults;
+ }
+
+ /* acc_param: yuvp2_tcc_static_config */
+
+ if (use && use->acc_tcc) {
+ /* Take values from user */
+ acc->tcc = acc_user->tcc;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->tcc = acc_old->tcc;
+ } else {
+ /* Calculate from scratch */
+ memset(&acc->tcc, 0, sizeof(acc->tcc));
+
+ acc->tcc.gen_control.en = 1;
+ acc->tcc.gen_control.blend_shift = 3;
+ acc->tcc.gen_control.gain_according_to_y_only = 1;
+ acc->tcc.gen_control.gamma = 8;
+ acc->tcc.gen_control.delta = 0;
+
+ for (i = 0; i < IPU3_UAPI_YUVP2_TCC_MACC_TABLE_ELEMENTS; i++) {
+ acc->tcc.macc_table.entries[i].a = 1024;
+ acc->tcc.macc_table.entries[i].b = 0;
+ acc->tcc.macc_table.entries[i].c = 0;
+ acc->tcc.macc_table.entries[i].d = 1024;
+ }
+
+ acc->tcc.inv_y_lut.entries[6] = 1023;
+ for (i = 7; i < IPU3_UAPI_YUVP2_TCC_INV_Y_LUT_ELEMENTS; i++)
+ acc->tcc.inv_y_lut.entries[i] = 1024 >> (i - 6);
+
+ acc->tcc.gain_pcwl = ipu3_css_tcc_gain_pcwl_lut;
+ acc->tcc.r_sqr_lut = ipu3_css_tcc_r_sqr_lut;
+ }
+
+ /* acc_param: dpc_config */
+
+ if (use && use->acc_dpc)
+ return -EINVAL; /* Not supported yet */
+
+ /* Just disable by default */
+ memset(&acc->dpc, 0, sizeof(acc->dpc));
+
+ /* acc_param: bds_config */
+
+ bds_ds = (css->rect[IPU3_CSS_RECT_EFFECTIVE].height *
+ IMGU_BDS_GRANULARITY) / css->rect[IPU3_CSS_RECT_BDS].height;
+ if (bds_ds < IMGU_BDS_MIN_SF_INV ||
+ bds_ds - IMGU_BDS_MIN_SF_INV >= ARRAY_SIZE(ipu3_css_bds_configs))
+ return -EINVAL;
+
+ cfg_bds = &ipu3_css_bds_configs[bds_ds - IMGU_BDS_MIN_SF_INV];
+ acc->bds.hor.hor_ctrl1.hor_crop_en = 0;
+ acc->bds.hor.hor_ctrl1.hor_crop_start = 0;
+ acc->bds.hor.hor_ctrl1.hor_crop_end = 0;
+ acc->bds.hor.hor_ctrl0.sample_patrn_length =
+ cfg_bds->sample_patrn_length;
+ acc->bds.hor.hor_ctrl0.hor_ds_en = cfg_bds->hor_ds_en;
+ acc->bds.hor.hor_ctrl0.min_clip_val = IMGU_BDS_MIN_CLIP_VAL;
+ acc->bds.hor.hor_ctrl0.max_clip_val = IMGU_BDS_MAX_CLIP_VAL;
+ acc->bds.hor.hor_ctrl0.out_frame_width =
+ css->rect[IPU3_CSS_RECT_BDS].width;
+ acc->bds.hor.hor_ptrn_arr = cfg_bds->ptrn_arr;
+ acc->bds.hor.hor_phase_arr = cfg_bds->hor_phase_arr;
+ acc->bds.hor.hor_ctrl2.input_frame_height =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ acc->bds.ver.ver_ctrl0.min_clip_val = IMGU_BDS_MIN_CLIP_VAL;
+ acc->bds.ver.ver_ctrl0.max_clip_val = IMGU_BDS_MAX_CLIP_VAL;
+ acc->bds.ver.ver_ctrl0.sample_patrn_length =
+ cfg_bds->sample_patrn_length;
+ acc->bds.ver.ver_ctrl0.ver_ds_en = cfg_bds->ver_ds_en;
+ acc->bds.ver.ver_ptrn_arr = cfg_bds->ptrn_arr;
+ acc->bds.ver.ver_phase_arr = cfg_bds->ver_phase_arr;
+ acc->bds.ver.ver_ctrl1.out_frame_width =
+ css->rect[IPU3_CSS_RECT_BDS].width;
+ acc->bds.ver.ver_ctrl1.out_frame_height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ for (i = 0; i < stripes; i++)
+ acc_bds_per_stripe_data(css, acc, i);
+
+ acc->bds.enabled = cfg_bds->hor_ds_en || cfg_bds->ver_ds_en;
+
+ /* acc_param: anr_config */
+
+ if (use && use->acc_anr) {
+ /* Take values from user */
+ acc->anr.transform = acc_user->anr.transform;
+ acc->anr.stitch.anr_stitch_en =
+ acc_user->anr.stitch.anr_stitch_en;
+ memcpy(acc->anr.stitch.pyramid, acc_user->anr.stitch.pyramid,
+ sizeof(acc->anr.stitch.pyramid));
+ } else if (acc_old) {
+ /* Use old value */
+ acc->anr.transform = acc_old->anr.transform;
+ acc->anr.stitch.anr_stitch_en =
+ acc_old->anr.stitch.anr_stitch_en;
+ memcpy(acc->anr.stitch.pyramid, acc_old->anr.stitch.pyramid,
+ sizeof(acc->anr.stitch.pyramid));
+ } else {
+ /* Calculate from scratch */
+ acc->anr = ipu3_css_anr_defaults;
+ }
+
+ /* Always enabled */
+ acc->anr.search.enable = 1;
+ acc->anr.transform.enable = 1;
+ acc->anr.tile2strm.enable = 1;
+ acc->anr.tile2strm.frame_width =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
+ acc->anr.search.frame_width = acc->anr.tile2strm.frame_width;
+ acc->anr.stitch.frame_width = acc->anr.tile2strm.frame_width;
+ acc->anr.tile2strm.frame_height = css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->anr.search.frame_height = acc->anr.tile2strm.frame_height;
+ acc->anr.stitch.frame_height = acc->anr.tile2strm.frame_height;
+
+ width = ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
+ height = css->rect[IPU3_CSS_RECT_BDS].height;
+
+ if (acc->anr.transform.xreset + width > IPU3_UAPI_ANR_MAX_RESET)
+ acc->anr.transform.xreset = IPU3_UAPI_ANR_MAX_RESET - width;
+ if (acc->anr.transform.xreset < IPU3_UAPI_ANR_MIN_RESET)
+ acc->anr.transform.xreset = IPU3_UAPI_ANR_MIN_RESET;
+
+ if (acc->anr.transform.yreset + height > IPU3_UAPI_ANR_MAX_RESET)
+ acc->anr.transform.yreset = IPU3_UAPI_ANR_MAX_RESET - height;
+ if (acc->anr.transform.yreset < IPU3_UAPI_ANR_MIN_RESET)
+ acc->anr.transform.yreset = IPU3_UAPI_ANR_MIN_RESET;
+
+ /* acc_param: awb_fr_config */
+
+ if (use && use->acc_awb_fr) {
+ /* Take values from user */
+ acc->awb_fr.config = acc_user->awb_fr.config;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->awb_fr.config = acc_old->awb_fr.config;
+ } else {
+ /* Set from scratch */
+ acc->awb_fr.config = ipu3_css_awb_fr_defaults;
+ }
+
+ ipu3_css_grid_end_calc(&acc->awb_fr.config.grid_cfg);
+
+ if (acc->awb_fr.config.grid_cfg.width <= 0)
+ return -EINVAL;
+
+ acc->awb_fr.config.grid_cfg.height_per_slice =
+ IMGU_ABI_AWB_FR_MAX_CELLS_PER_SET /
+ acc->awb_fr.config.grid_cfg.width;
+
+ for (i = 0; i < stripes; i++)
+ acc->awb_fr.stripes[i] = acc->awb_fr.config;
+
+ if (acc->awb_fr.config.grid_cfg.x_start >=
+ acc->stripe.down_scaled_stripes[1].offset + min_overlap) {
+ /* Enable only for rightmost stripe, disable left */
+ acc->awb_fr.stripes[0].grid_cfg.y_start &=
+ ~IPU3_UAPI_GRID_Y_START_EN;
+ } else if (acc->awb_fr.config.grid_cfg.x_end <=
+ acc->stripe.bds_out_stripes[0].width - min_overlap) {
+ /* Enable only for leftmost stripe, disable right */
+ acc->awb_fr.stripes[1].grid_cfg.y_start &=
+ ~IPU3_UAPI_GRID_Y_START_EN;
+ } else {
+ /* Enable for both stripes */
+ u16 end; /* width for grid end */
+
+ acc->awb_fr.stripes[0].grid_cfg.width =
+ (acc->stripe.bds_out_stripes[0].width - min_overlap -
+ acc->awb_fr.config.grid_cfg.x_start + 1) >>
+ acc->awb_fr.config.grid_cfg.block_width_log2;
+ acc->awb_fr.stripes[1].grid_cfg.width =
+ acc->awb_fr.config.grid_cfg.width -
+ acc->awb_fr.stripes[0].grid_cfg.width;
+
+ b_w_log2 = acc->awb_fr.stripes[0].grid_cfg.block_width_log2;
+ end = ipu3_css_grid_end(acc->awb_fr.stripes[0].grid_cfg.x_start,
+ acc->awb_fr.stripes[0].grid_cfg.width,
+ b_w_log2);
+ acc->awb_fr.stripes[0].grid_cfg.x_end = end;
+
+ acc->awb_fr.stripes[1].grid_cfg.x_start =
+ (acc->awb_fr.stripes[0].grid_cfg.x_end + 1 -
+ acc->stripe.down_scaled_stripes[1].offset) &
+ IPU3_UAPI_GRID_START_MASK;
+ b_w_log2 = acc->awb_fr.stripes[1].grid_cfg.block_width_log2;
+ end = ipu3_css_grid_end(acc->awb_fr.stripes[1].grid_cfg.x_start,
+ acc->awb_fr.stripes[1].grid_cfg.width,
+ b_w_log2);
+ acc->awb_fr.stripes[1].grid_cfg.x_end = end;
+
+ /*
+ * To reduce complexity of debubbling and loading
+ * statistics fix grid_height_per_slice to 1 for both
+ * stripes.
+ */
+ for (i = 0; i < stripes; i++)
+ acc->awb_fr.stripes[i].grid_cfg.height_per_slice = 1;
+ }
+
+ if (ipu3_css_awb_fr_ops_calc(css, &acc->awb_fr))
+ return -EINVAL;
+
+ /* acc_param: ae_config */
+
+ if (use && use->acc_ae) {
+ /* Take values from user */
+ acc->ae.grid_cfg = acc_user->ae.grid_cfg;
+ acc->ae.ae_ccm = acc_user->ae.ae_ccm;
+ for (i = 0; i < IPU3_UAPI_AE_WEIGHTS; i++)
+ acc->ae.weights[i] = acc_user->ae.weights[i];
+ } else if (acc_old) {
+ /* Use old value */
+ acc->ae.grid_cfg = acc_old->ae.grid_cfg;
+ acc->ae.ae_ccm = acc_old->ae.ae_ccm;
+ for (i = 0; i < IPU3_UAPI_AE_WEIGHTS; i++)
+ acc->ae.weights[i] = acc_old->ae.weights[i];
+ } else {
+ /* Set from scratch */
+ static const struct ipu3_uapi_ae_weight_elem
+ weight_def = { 1, 1, 1, 1, 1, 1, 1, 1 };
+
+ acc->ae.grid_cfg = ipu3_css_ae_grid_defaults;
+ acc->ae.ae_ccm = ipu3_css_ae_ccm_defaults;
+ for (i = 0; i < IPU3_UAPI_AE_WEIGHTS; i++)
+ acc->ae.weights[i] = weight_def;
+ }
+
+ b_w_log2 = acc->ae.grid_cfg.block_width_log2;
+ acc->ae.grid_cfg.x_end = ipu3_css_grid_end(acc->ae.grid_cfg.x_start,
+ acc->ae.grid_cfg.width,
+ b_w_log2);
+ b_w_log2 = acc->ae.grid_cfg.block_height_log2;
+ acc->ae.grid_cfg.y_end = ipu3_css_grid_end(acc->ae.grid_cfg.y_start,
+ acc->ae.grid_cfg.height,
+ b_w_log2);
+
+ for (i = 0; i < stripes; i++)
+ acc->ae.stripes[i].grid = acc->ae.grid_cfg;
+
+ if (acc->ae.grid_cfg.x_start >=
+ acc->stripe.down_scaled_stripes[1].offset) {
+ /* Enable only for rightmost stripe, disable left */
+ acc->ae.stripes[0].grid.ae_en = 0;
+ } else if (acc->ae.grid_cfg.x_end <=
+ acc->stripe.bds_out_stripes[0].width) {
+ /* Enable only for leftmost stripe, disable right */
+ acc->ae.stripes[1].grid.ae_en = 0;
+ } else {
+ /* Enable for both stripes */
+ u8 b_w_log2;
+
+ acc->ae.stripes[0].grid.width =
+ (acc->stripe.bds_out_stripes[0].width -
+ acc->ae.grid_cfg.x_start + 1) >>
+ acc->ae.grid_cfg.block_width_log2;
+
+ acc->ae.stripes[1].grid.width =
+ acc->ae.grid_cfg.width - acc->ae.stripes[0].grid.width;
+
+ b_w_log2 = acc->ae.stripes[0].grid.block_width_log2;
+ acc->ae.stripes[0].grid.x_end =
+ ipu3_css_grid_end(acc->ae.stripes[0].grid.x_start,
+ acc->ae.stripes[0].grid.width,
+ b_w_log2);
+
+ acc->ae.stripes[1].grid.x_start =
+ (acc->ae.stripes[0].grid.x_end + 1 -
+ acc->stripe.down_scaled_stripes[1].offset) &
+ IPU3_UAPI_GRID_START_MASK;
+ b_w_log2 = acc->ae.stripes[1].grid.block_width_log2;
+ acc->ae.stripes[1].grid.x_end =
+ ipu3_css_grid_end(acc->ae.stripes[1].grid.x_start,
+ acc->ae.stripes[1].grid.width,
+ b_w_log2);
+ }
+
+ /* acc_param: af_config */
+
+ if (use && use->acc_af) {
+ /* Take values from user */
+ acc->af.config.filter_config =
+ acc_user->af.config.filter_config;
+ acc->af.config.grid_cfg = acc_user->af.config.grid_cfg;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->af.config = acc_old->af.config;
+ } else {
+ /* Set from scratch */
+ acc->af.config.filter_config =
+ ipu3_css_af_defaults.filter_config;
+ acc->af.config.grid_cfg = ipu3_css_af_defaults.grid_cfg;
+ }
+
+ ipu3_css_grid_end_calc(&acc->af.config.grid_cfg);
+
+ if (acc->af.config.grid_cfg.width <= 0)
+ return -EINVAL;
+
+ acc->af.config.grid_cfg.height_per_slice =
+ IMGU_ABI_AF_MAX_CELLS_PER_SET / acc->af.config.grid_cfg.width;
+ acc->af.config.frame_size.width =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
+ acc->af.config.frame_size.height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+
+ if (acc->stripe.bds_out_stripes[0].width <= min_overlap)
+ return -EINVAL;
+
+ for (i = 0; i < stripes; i++) {
+ acc->af.stripes[i].grid_cfg = acc->af.config.grid_cfg;
+ acc->af.stripes[i].frame_size.height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->af.stripes[i].frame_size.width =
+ acc->stripe.bds_out_stripes[i].width;
+ }
+
+ if (acc->af.config.grid_cfg.x_start >=
+ acc->stripe.down_scaled_stripes[1].offset + min_overlap) {
+ /* Enable only for rightmost stripe, disable left */
+ acc->af.stripes[0].grid_cfg.y_start &=
+ ~IPU3_UAPI_GRID_Y_START_EN;
+ } else if (acc->af.config.grid_cfg.x_end <=
+ acc->stripe.bds_out_stripes[0].width - min_overlap) {
+ /* Enable only for leftmost stripe, disable right */
+ acc->af.stripes[1].grid_cfg.y_start &=
+ ~IPU3_UAPI_GRID_Y_START_EN;
+ } else {
+ /* Enable for both stripes */
+
+ acc->af.stripes[0].grid_cfg.width =
+ (acc->stripe.bds_out_stripes[0].width - min_overlap -
+ acc->af.config.grid_cfg.x_start + 1) >>
+ acc->af.config.grid_cfg.block_width_log2;
+ acc->af.stripes[1].grid_cfg.width =
+ acc->af.config.grid_cfg.width -
+ acc->af.stripes[0].grid_cfg.width;
+
+ b_w_log2 = acc->af.stripes[0].grid_cfg.block_width_log2;
+ acc->af.stripes[0].grid_cfg.x_end =
+ ipu3_css_grid_end(acc->af.stripes[0].grid_cfg.x_start,
+ acc->af.stripes[0].grid_cfg.width,
+ b_w_log2);
+
+ acc->af.stripes[1].grid_cfg.x_start =
+ (acc->af.stripes[0].grid_cfg.x_end + 1 -
+ acc->stripe.down_scaled_stripes[1].offset) &
+ IPU3_UAPI_GRID_START_MASK;
+
+ b_w_log2 = acc->af.stripes[1].grid_cfg.block_width_log2;
+ acc->af.stripes[1].grid_cfg.x_end =
+ ipu3_css_grid_end(acc->af.stripes[1].grid_cfg.x_start,
+ acc->af.stripes[1].grid_cfg.width,
+ b_w_log2);
+
+ /*
+ * To reduce complexity of debubbling and loading statistics
+ * fix grid_height_per_slice to 1 for both stripes
+ */
+ for (i = 0; i < stripes; i++)
+ acc->af.stripes[i].grid_cfg.height_per_slice = 1;
+ }
+
+ if (ipu3_css_af_ops_calc(css, &acc->af))
+ return -EINVAL;
+
+ /* acc_param: awb_config */
+
+ if (use && use->acc_awb) {
+ /* Take values from user */
+ acc->awb.config = acc_user->awb.config;
+ } else if (acc_old) {
+ /* Use old value */
+ acc->awb.config = acc_old->awb.config;
+ } else {
+ /* Set from scratch */
+ acc->awb.config = ipu3_css_awb_defaults;
+ }
+
+ if (acc->awb.config.grid.width <= 0)
+ return -EINVAL;
+
+ acc->awb.config.grid.height_per_slice =
+ IMGU_ABI_AWB_MAX_CELLS_PER_SET / acc->awb.config.grid.width,
+ ipu3_css_grid_end_calc(&acc->awb.config.grid);
+
+ for (i = 0; i < stripes; i++)
+ acc->awb.stripes[i] = acc->awb.config;
+
+ if (acc->awb.config.grid.x_start >=
+ acc->stripe.down_scaled_stripes[1].offset + min_overlap) {
+ /* Enable only for rightmost stripe, disable left */
+ acc->awb.stripes[0].rgbs_thr_b &= ~IPU3_UAPI_AWB_RGBS_THR_B_EN;
+ } else if (acc->awb.config.grid.x_end <=
+ acc->stripe.bds_out_stripes[0].width - min_overlap) {
+ /* Enable only for leftmost stripe, disable right */
+ acc->awb.stripes[1].rgbs_thr_b &= ~IPU3_UAPI_AWB_RGBS_THR_B_EN;
+ } else {
+ /* Enable for both stripes */
+
+ acc->awb.stripes[0].grid.width =
+ (acc->stripe.bds_out_stripes[0].width -
+ acc->awb.config.grid.x_start + 1) >>
+ acc->awb.config.grid.block_width_log2;
+ acc->awb.stripes[1].grid.width = acc->awb.config.grid.width -
+ acc->awb.stripes[0].grid.width;
+
+ b_w_log2 = acc->awb.stripes[0].grid.block_width_log2;
+ acc->awb.stripes[0].grid.x_end =
+ ipu3_css_grid_end(acc->awb.stripes[0].grid.x_start,
+ acc->awb.stripes[0].grid.width,
+ b_w_log2);
+
+ acc->awb.stripes[1].grid.x_start =
+ (acc->awb.stripes[0].grid.x_end + 1 -
+ acc->stripe.down_scaled_stripes[1].offset) &
+ IPU3_UAPI_GRID_START_MASK;
+
+ b_w_log2 = acc->awb.stripes[1].grid.block_width_log2;
+ acc->awb.stripes[1].grid.x_end =
+ ipu3_css_grid_end(acc->awb.stripes[1].grid.x_start,
+ acc->awb.stripes[1].grid.width,
+ b_w_log2);
+
+ /*
+ * To reduce complexity of debubbling and loading statistics
+ * fix grid_height_per_slice to 1 for both stripes
+ */
+ for (i = 0; i < stripes; i++)
+ acc->awb.stripes[i].grid.height_per_slice = 1;
+ }
+
+ if (ipu3_css_awb_ops_calc(css, &acc->awb))
+ return -EINVAL;
+
+ return 0;
+}
+
+/*
+ * Fill the indicated structure in `new_binary_params' from the possible
+ * sources based on `use_user' flag: if the flag is false, copy from
+ * `old_binary_params', or if the flag is true, copy from `user_setting'
+ * and return NULL (or error pointer on error).
+ * If the flag is false and `old_binary_params' is NULL, return pointer
+ * to the structure inside `new_binary_params'. In that case the caller
+ * should calculate and fill the structure from scratch.
+ */
+static void *ipu3_css_cfg_copy(struct ipu3_css *css, bool use_user,
+ void *user_setting, void *old_binary_params,
+ void *new_binary_params,
+ enum imgu_abi_memories m,
+ struct imgu_fw_isp_parameter *par,
+ size_t par_size)
+{
+ const enum imgu_abi_param_class c = IMGU_ABI_PARAM_CLASS_PARAM;
+ void *new_setting, *old_setting;
+
+ new_setting = ipu3_css_fw_pipeline_params(css, c, m, par, par_size,
+ new_binary_params);
+ if (!new_setting)
+ return ERR_PTR(-EPROTO); /* Corrupted firmware */
+
+ if (use_user) {
+ /* Take new user parameters */
+ memcpy(new_setting, user_setting, par_size);
+ } else if (old_binary_params) {
+ /* Take previous value */
+ old_setting = ipu3_css_fw_pipeline_params(css, c, m, par,
+ par_size,
+ old_binary_params);
+ if (!old_setting)
+ return ERR_PTR(-EPROTO);
+ memcpy(new_setting, old_setting, par_size);
+ } else {
+ return new_setting; /* Need to calculate */
+ }
+
+ return NULL; /* Copied from other value */
+}
+
+/*
+ * Configure VMEM0 parameters (late binding parameters).
+ */
+int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+ void *vmem0, void *vmem0_old,
+ struct ipu3_uapi_params *user)
+{
+ const struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->current_binary];
+ struct imgu_fw_param_memory_offsets *pofs = (void *)css->fwp +
+ bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_PARAM];
+ struct ipu3_uapi_isp_lin_vmem_params *lin_vmem = NULL;
+ struct ipu3_uapi_isp_tnr3_vmem_params *tnr_vmem = NULL;
+ struct ipu3_uapi_isp_xnr3_vmem_params *xnr_vmem = NULL;
+ const enum imgu_abi_param_class c = IMGU_ABI_PARAM_CLASS_PARAM;
+ const enum imgu_abi_memories m = IMGU_ABI_MEM_ISP_VMEM0;
+ unsigned int i;
+
+ /* Configure VMEM0 */
+
+ memset(vmem0, 0, bi->info.isp.sp.mem_initializers.params[c][m].size);
+
+ /* Configure Linearization VMEM0 parameters */
+
+ lin_vmem = ipu3_css_cfg_copy(css, use && use->lin_vmem_params,
+ &user->lin_vmem_params, vmem0_old, vmem0,
+ m, &pofs->vmem.lin, sizeof(*lin_vmem));
+ if (!IS_ERR_OR_NULL(lin_vmem)) {
+ /* Generate parameter from scratch */
+ for (i = 0; i < IPU3_UAPI_LIN_LUT_SIZE; i++) {
+ lin_vmem->lin_lutlow_gr[i] = 32 * i;
+ lin_vmem->lin_lutlow_r[i] = 32 * i;
+ lin_vmem->lin_lutlow_b[i] = 32 * i;
+ lin_vmem->lin_lutlow_gb[i] = 32 * i;
+
+ lin_vmem->lin_lutdif_gr[i] = 32;
+ lin_vmem->lin_lutdif_r[i] = 32;
+ lin_vmem->lin_lutdif_b[i] = 32;
+ lin_vmem->lin_lutdif_gb[i] = 32;
+ }
+ }
+
+ /* Configure TNR3 VMEM parameters */
+ if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ tnr_vmem = ipu3_css_cfg_copy(css, use && use->tnr3_vmem_params,
+ &user->tnr3_vmem_params,
+ vmem0_old, vmem0, m,
+ &pofs->vmem.tnr3,
+ sizeof(*tnr_vmem));
+ if (!IS_ERR_OR_NULL(tnr_vmem)) {
+ /* Generate parameter from scratch */
+ for (i = 0; i < IPU3_UAPI_ISP_TNR3_VMEM_LEN; i++)
+ tnr_vmem->sigma[i] = 256;
+ }
+ }
+ i = IPU3_UAPI_ISP_TNR3_VMEM_LEN;
+
+ /* Configure XNR3 VMEM parameters */
+
+ xnr_vmem = ipu3_css_cfg_copy(css, use && use->xnr3_vmem_params,
+ &user->xnr3_vmem_params, vmem0_old, vmem0,
+ m, &pofs->vmem.xnr3, sizeof(*xnr_vmem));
+ if (!IS_ERR_OR_NULL(xnr_vmem)) {
+ xnr_vmem->x[i] = ipu3_css_xnr3_vmem_defaults.x
+ [i % IMGU_XNR3_VMEM_LUT_LEN];
+ xnr_vmem->a[i] = ipu3_css_xnr3_vmem_defaults.a
+ [i % IMGU_XNR3_VMEM_LUT_LEN];
+ xnr_vmem->b[i] = ipu3_css_xnr3_vmem_defaults.b
+ [i % IMGU_XNR3_VMEM_LUT_LEN];
+ xnr_vmem->c[i] = ipu3_css_xnr3_vmem_defaults.c
+ [i % IMGU_XNR3_VMEM_LUT_LEN];
+ }
+
+ return IS_ERR(lin_vmem) || IS_ERR(tnr_vmem) || IS_ERR(xnr_vmem) ?
+ -EPROTO : 0;
+}
+
+/*
+ * Configure DMEM0 parameters (late binding parameters).
+ */
+int ipu3_css_cfg_dmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+ void *dmem0, void *dmem0_old,
+ struct ipu3_uapi_params *user)
+{
+ const struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->current_binary];
+ struct imgu_fw_param_memory_offsets *pofs = (void *)css->fwp +
+ bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_PARAM];
+
+ struct ipu3_uapi_isp_tnr3_params *tnr_dmem = NULL;
+ struct ipu3_uapi_isp_xnr3_params *xnr_dmem;
+
+ const enum imgu_abi_param_class c = IMGU_ABI_PARAM_CLASS_PARAM;
+ const enum imgu_abi_memories m = IMGU_ABI_MEM_ISP_DMEM0;
+
+ /* Configure DMEM0 */
+
+ memset(dmem0, 0, bi->info.isp.sp.mem_initializers.params[c][m].size);
+
+ /* Configure TNR3 DMEM0 parameters */
+ if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ tnr_dmem = ipu3_css_cfg_copy(css, use && use->tnr3_dmem_params,
+ &user->tnr3_dmem_params, dmem0_old,
+ dmem0, m, &pofs->dmem.tnr3,
+ sizeof(*tnr_dmem));
+ if (!IS_ERR_OR_NULL(tnr_dmem)) {
+ /* Generate parameter from scratch */
+ tnr_dmem->knee_y1 = 768;
+ tnr_dmem->knee_y2 = 1280;
+ }
+ }
+
+ /* Configure XNR3 DMEM0 parameters */
+
+ xnr_dmem = ipu3_css_cfg_copy(css, use && use->xnr3_dmem_params,
+ &user->xnr3_dmem_params, dmem0_old, dmem0,
+ m, &pofs->dmem.xnr3, sizeof(*xnr_dmem));
+ if (!IS_ERR_OR_NULL(xnr_dmem)) {
+ /* Generate parameter from scratch */
+ xnr_dmem->alpha.y0 = 2047;
+ xnr_dmem->alpha.u0 = 2047;
+ xnr_dmem->alpha.v0 = 2047;
+ }
+
+ return IS_ERR(tnr_dmem) || IS_ERR(xnr_dmem) ? -EPROTO : 0;
+}
+
+/* Generate unity morphing table without morphing effect */
+void ipu3_css_cfg_gdc_table(struct imgu_abi_gdc_warp_param *gdc,
+ int frame_in_x, int frame_in_y,
+ int frame_out_x, int frame_out_y,
+ int env_w, int env_h)
+{
+ static const unsigned int FRAC_BITS = IMGU_ABI_GDC_FRAC_BITS;
+ static const unsigned int XMEM_ALIGN = 1 << 4;
+ const unsigned int XMEM_ALIGN_MASK = ~(XMEM_ALIGN - 1);
+ static const unsigned int BCI_ENV = 4;
+ static const unsigned int BYP = 2; /* Bytes per pixel */
+ const unsigned int OFFSET_X = 2 * IMGU_DVS_BLOCK_W + env_w + 1;
+ const unsigned int OFFSET_Y = IMGU_DVS_BLOCK_H + env_h + 1;
+
+ struct imgu_abi_gdc_warp_param gdc_luma, gdc_chroma;
+
+ unsigned int blocks_x = ALIGN(DIV_ROUND_UP(frame_out_x,
+ IMGU_DVS_BLOCK_W), 2);
+ unsigned int blocks_y = DIV_ROUND_UP(frame_out_y, IMGU_DVS_BLOCK_H);
+ unsigned int y0, x0, x1, x, y;
+
+ /* Global luma settings */
+ gdc_luma.origin_x = 0;
+ gdc_luma.origin_y = 0;
+ gdc_luma.p0_x = (OFFSET_X - (OFFSET_X & XMEM_ALIGN_MASK)) << FRAC_BITS;
+ gdc_luma.p0_y = 0;
+ gdc_luma.p1_x = gdc_luma.p0_x + (IMGU_DVS_BLOCK_W << FRAC_BITS);
+ gdc_luma.p1_y = gdc_luma.p0_y;
+ gdc_luma.p2_x = gdc_luma.p0_x;
+ gdc_luma.p2_y = gdc_luma.p0_y + (IMGU_DVS_BLOCK_H << FRAC_BITS);
+ gdc_luma.p3_x = gdc_luma.p1_x;
+ gdc_luma.p3_y = gdc_luma.p2_y;
+
+ gdc_luma.in_block_width = IMGU_DVS_BLOCK_W + BCI_ENV +
+ OFFSET_X - (OFFSET_X & XMEM_ALIGN_MASK);
+ gdc_luma.in_block_width_a = DIV_ROUND_UP(gdc_luma.in_block_width,
+ IPU3_UAPI_ISP_VEC_ELEMS);
+ gdc_luma.in_block_width_b = DIV_ROUND_UP(gdc_luma.in_block_width,
+ IMGU_ABI_ISP_DDR_WORD_BYTES /
+ BYP);
+ gdc_luma.in_block_height = IMGU_DVS_BLOCK_H + BCI_ENV;
+ gdc_luma.padding = 0;
+
+ /* Global chroma settings */
+ gdc_chroma.origin_x = 0;
+ gdc_chroma.origin_y = 0;
+ gdc_chroma.p0_x = (OFFSET_X / 2 - (OFFSET_X / 2 & XMEM_ALIGN_MASK)) <<
+ FRAC_BITS;
+ gdc_chroma.p0_y = 0;
+ gdc_chroma.p1_x = gdc_chroma.p0_x + (IMGU_DVS_BLOCK_W << FRAC_BITS);
+ gdc_chroma.p1_y = gdc_chroma.p0_y;
+ gdc_chroma.p2_x = gdc_chroma.p0_x;
+ gdc_chroma.p2_y = gdc_chroma.p0_y + (IMGU_DVS_BLOCK_H / 2 << FRAC_BITS);
+ gdc_chroma.p3_x = gdc_chroma.p1_x;
+ gdc_chroma.p3_y = gdc_chroma.p2_y;
+
+ gdc_chroma.in_block_width = IMGU_DVS_BLOCK_W + BCI_ENV;
+ gdc_chroma.in_block_width_a = DIV_ROUND_UP(gdc_chroma.in_block_width,
+ IPU3_UAPI_ISP_VEC_ELEMS);
+ gdc_chroma.in_block_width_b = DIV_ROUND_UP(gdc_chroma.in_block_width,
+ IMGU_ABI_ISP_DDR_WORD_BYTES /
+ BYP);
+ gdc_chroma.in_block_height = IMGU_DVS_BLOCK_H / 2 + BCI_ENV;
+ gdc_chroma.padding = 0;
+
+ /* Calculate block offsets for luma and chroma */
+ for (y0 = 0; y0 < blocks_y; y0++) {
+ for (x0 = 0; x0 < blocks_x / 2; x0++) {
+ for (x1 = 0; x1 < 2; x1++) {
+ /* Luma blocks */
+ x = (x0 * 2 + x1) * IMGU_DVS_BLOCK_W + OFFSET_X;
+ x &= XMEM_ALIGN_MASK;
+ y = y0 * IMGU_DVS_BLOCK_H + OFFSET_Y;
+ *gdc = gdc_luma;
+ gdc->in_addr_offset =
+ (y * frame_in_x + x) * BYP;
+ gdc++;
+ }
+
+ /* Chroma block */
+ x = x0 * IMGU_DVS_BLOCK_W + OFFSET_X / 2;
+ x &= XMEM_ALIGN_MASK;
+ y = y0 * (IMGU_DVS_BLOCK_H / 2) + OFFSET_Y / 2;
+ *gdc = gdc_chroma;
+ gdc->in_addr_offset = (y * frame_in_x + x) * BYP;
+ gdc++;
+ }
+ }
+}
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-params.h b/drivers/media/pci/intel/ipu3/ipu3-css-params.h
new file mode 100644
index 0000000..f93ed027
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-params.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+
+#ifndef __IPU3_PARAMS_H
+#define __IPU3_PARAMS_H
+
+int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+ struct imgu_abi_acc_param *acc,
+ struct imgu_abi_acc_param *acc_old,
+ struct ipu3_uapi_acc_param *acc_user);
+
+int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+ void *vmem0, void *vmem0_old,
+ struct ipu3_uapi_params *user);
+
+int ipu3_css_cfg_dmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+ void *dmem0, void *dmem0_old,
+ struct ipu3_uapi_params *user);
+
+void ipu3_css_cfg_gdc_table(struct imgu_abi_gdc_warp_param *gdc,
+ int frame_in_x, int frame_in_y,
+ int frame_out_x, int frame_out_y,
+ int env_w, int env_h);
+
+#endif /*__IPU3_PARAMS_H */
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* [PATCH v7 12/16] intel-ipu3: css: Initialize css hardware
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (9 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 11/16] intel-ipu3: css: Compute and program ccs Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-09 12:06 ` Sakari Ailus
2018-10-29 22:23 ` [PATCH v7 13/16] intel-ipu3: Add css pipeline programming Yong Zhi
` (6 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
This patch implements the functions to initialize
and configure IPU3 h/w such as clock, irq and power.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
---
drivers/media/pci/intel/ipu3/ipu3-css.c | 537 ++++++++++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3-css.h | 203 ++++++++++++
2 files changed, 740 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css.c b/drivers/media/pci/intel/ipu3/ipu3-css.c
new file mode 100644
index 0000000..164830f
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css.c
@@ -0,0 +1,537 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Intel Corporation
+
+#include <linux/device.h>
+#include <linux/iopoll.h>
+
+#include "ipu3-css.h"
+#include "ipu3-css-fw.h"
+#include "ipu3-css-params.h"
+#include "ipu3-dmamap.h"
+#include "ipu3-tables.h"
+
+/* IRQ configuration */
+#define IMGU_IRQCTRL_IRQ_MASK (IMGU_IRQCTRL_IRQ_SP1 | \
+ IMGU_IRQCTRL_IRQ_SP2 | \
+ IMGU_IRQCTRL_IRQ_SW_PIN(0) | \
+ IMGU_IRQCTRL_IRQ_SW_PIN(1))
+
+/******************* css hw *******************/
+
+/* In the style of writesl() defined in include/asm-generic/io.h */
+static inline void writes(const void *mem, ssize_t count, void __iomem *addr)
+{
+ if (count >= 4) {
+ const u32 *buf = mem;
+
+ count /= 4;
+ do {
+ writel(*buf++, addr);
+ addr += 4;
+ } while (--count);
+ }
+}
+
+/* Wait until register `reg', masked with `mask', becomes `cmp' */
+static int ipu3_hw_wait(void __iomem *base, int reg, u32 mask, u32 cmp)
+{
+ u32 val;
+
+ return readl_poll_timeout(base + reg, val, (val & mask) == cmp,
+ 1000, 100 * 1000);
+}
+
+/* Initialize the IPU3 CSS hardware and associated h/w blocks */
+
+int ipu3_css_set_powerup(struct device *dev, void __iomem *base)
+{
+ static const unsigned int freq = 450;
+ u32 pm_ctrl, state, val;
+
+ dev_dbg(dev, "%s\n", __func__);
+ /* Clear the CSS busy signal */
+ readl(base + IMGU_REG_GP_BUSY);
+ writel(0, base + IMGU_REG_GP_BUSY);
+
+ /* Wait for idle signal */
+ if (ipu3_hw_wait(base, IMGU_REG_STATE, IMGU_STATE_IDLE_STS,
+ IMGU_STATE_IDLE_STS)) {
+ dev_err(dev, "failed to set CSS idle\n");
+ goto fail;
+ }
+
+ /* Reset the css */
+ writel(readl(base + IMGU_REG_PM_CTRL) | IMGU_PM_CTRL_FORCE_RESET,
+ base + IMGU_REG_PM_CTRL);
+
+ usleep_range(200, 300);
+
+ /** Prepare CSS */
+
+ pm_ctrl = readl(base + IMGU_REG_PM_CTRL);
+ state = readl(base + IMGU_REG_STATE);
+
+ dev_dbg(dev, "CSS pm_ctrl 0x%x state 0x%x (power %s)\n",
+ pm_ctrl, state, state & IMGU_STATE_POWER_DOWN ? "down" : "up");
+
+ /* Power up CSS using wrapper */
+ if (state & IMGU_STATE_POWER_DOWN) {
+ writel(IMGU_PM_CTRL_RACE_TO_HALT | IMGU_PM_CTRL_START,
+ base + IMGU_REG_PM_CTRL);
+ if (ipu3_hw_wait(base, IMGU_REG_PM_CTRL,
+ IMGU_PM_CTRL_START, 0)) {
+ dev_err(dev, "failed to power up CSS\n");
+ goto fail;
+ }
+ usleep_range(2000, 3000);
+ } else {
+ writel(IMGU_PM_CTRL_RACE_TO_HALT, base + IMGU_REG_PM_CTRL);
+ }
+
+ /* Set the busy bit */
+ writel(readl(base + IMGU_REG_GP_BUSY) | 1, base + IMGU_REG_GP_BUSY);
+
+ /* Set CSS clock frequency */
+ pm_ctrl = readl(base + IMGU_REG_PM_CTRL);
+ val = pm_ctrl & ~(IMGU_PM_CTRL_CSS_PWRDN | IMGU_PM_CTRL_RST_AT_EOF);
+ writel(val, base + IMGU_REG_PM_CTRL);
+ writel(0, base + IMGU_REG_GP_BUSY);
+ if (ipu3_hw_wait(base, IMGU_REG_STATE,
+ IMGU_STATE_PWRDNM_FSM_MASK, 0)) {
+ dev_err(dev, "failed to pwrdn CSS\n");
+ goto fail;
+ }
+ val = (freq / IMGU_SYSTEM_REQ_FREQ_DIVIDER) & IMGU_SYSTEM_REQ_FREQ_MASK;
+ writel(val, base + IMGU_REG_SYSTEM_REQ);
+ writel(1, base + IMGU_REG_GP_BUSY);
+ writel(readl(base + IMGU_REG_PM_CTRL) | IMGU_PM_CTRL_FORCE_HALT,
+ base + IMGU_REG_PM_CTRL);
+ if (ipu3_hw_wait(base, IMGU_REG_STATE, IMGU_STATE_HALT_STS,
+ IMGU_STATE_HALT_STS)) {
+ dev_err(dev, "failed to halt CSS\n");
+ goto fail;
+ }
+
+ writel(readl(base + IMGU_REG_PM_CTRL) | IMGU_PM_CTRL_START,
+ base + IMGU_REG_PM_CTRL);
+ if (ipu3_hw_wait(base, IMGU_REG_PM_CTRL, IMGU_PM_CTRL_START, 0)) {
+ dev_err(dev, "failed to start CSS\n");
+ goto fail;
+ }
+ writel(readl(base + IMGU_REG_PM_CTRL) | IMGU_PM_CTRL_FORCE_UNHALT,
+ base + IMGU_REG_PM_CTRL);
+
+ val = readl(base + IMGU_REG_PM_CTRL); /* get pm_ctrl */
+ val &= ~(IMGU_PM_CTRL_CSS_PWRDN | IMGU_PM_CTRL_RST_AT_EOF);
+ val |= pm_ctrl & (IMGU_PM_CTRL_CSS_PWRDN | IMGU_PM_CTRL_RST_AT_EOF);
+ writel(val, base + IMGU_REG_PM_CTRL);
+
+ return 0;
+
+fail:
+ ipu3_css_set_powerdown(dev, base);
+ return -EIO;
+}
+
+void ipu3_css_set_powerdown(struct device *dev, void __iomem *base)
+{
+ dev_dbg(dev, "%s\n", __func__);
+ /* wait for cio idle signal */
+ if (ipu3_hw_wait(base, IMGU_REG_CIO_GATE_BURST_STATE,
+ IMGU_CIO_GATE_BURST_MASK, 0))
+ dev_warn(dev, "wait cio gate idle timeout");
+
+ /* wait for css idle signal */
+ if (ipu3_hw_wait(base, IMGU_REG_STATE, IMGU_STATE_IDLE_STS,
+ IMGU_STATE_IDLE_STS))
+ dev_warn(dev, "wait css idle timeout\n");
+
+ /* do halt-halted handshake with css */
+ writel(1, base + IMGU_REG_GP_HALT);
+ if (ipu3_hw_wait(base, IMGU_REG_STATE, IMGU_STATE_HALT_STS,
+ IMGU_STATE_HALT_STS))
+ dev_warn(dev, "failed to halt css");
+
+ /* de-assert the busy bit */
+ writel(0, base + IMGU_REG_GP_BUSY);
+}
+
+static void ipu3_css_hw_enable_irq(struct ipu3_css *css)
+{
+ void __iomem *const base = css->base;
+ u32 val, i;
+
+ /* Set up interrupts */
+
+ /*
+ * Enable IRQ on the SP which signals that SP goes to idle
+ * (aka ready state) and set trigger to pulse
+ */
+ val = readl(base + IMGU_REG_SP_CTRL(0)) | IMGU_CTRL_IRQ_READY;
+ writel(val, base + IMGU_REG_SP_CTRL(0));
+ writel(val | IMGU_CTRL_IRQ_CLEAR, base + IMGU_REG_SP_CTRL(0));
+
+ /* Enable IRQs from the IMGU wrapper */
+ writel(IMGU_REG_INT_CSS_IRQ, base + IMGU_REG_INT_ENABLE);
+ /* Clear */
+ writel(IMGU_REG_INT_CSS_IRQ, base + IMGU_REG_INT_STATUS);
+
+ /* Enable IRQs from main IRQ controller */
+ writel(~0, base + IMGU_REG_IRQCTRL_EDGE_NOT_PULSE(IMGU_IRQCTRL_MAIN));
+ writel(0, base + IMGU_REG_IRQCTRL_MASK(IMGU_IRQCTRL_MAIN));
+ writel(IMGU_IRQCTRL_IRQ_MASK,
+ base + IMGU_REG_IRQCTRL_EDGE(IMGU_IRQCTRL_MAIN));
+ writel(IMGU_IRQCTRL_IRQ_MASK,
+ base + IMGU_REG_IRQCTRL_ENABLE(IMGU_IRQCTRL_MAIN));
+ writel(IMGU_IRQCTRL_IRQ_MASK,
+ base + IMGU_REG_IRQCTRL_CLEAR(IMGU_IRQCTRL_MAIN));
+ writel(IMGU_IRQCTRL_IRQ_MASK,
+ base + IMGU_REG_IRQCTRL_MASK(IMGU_IRQCTRL_MAIN));
+ /* Wait for write complete */
+ readl(base + IMGU_REG_IRQCTRL_ENABLE(IMGU_IRQCTRL_MAIN));
+
+ /* Enable IRQs from SP0 and SP1 controllers */
+ for (i = IMGU_IRQCTRL_SP0; i <= IMGU_IRQCTRL_SP1; i++) {
+ writel(~0, base + IMGU_REG_IRQCTRL_EDGE_NOT_PULSE(i));
+ writel(0, base + IMGU_REG_IRQCTRL_MASK(i));
+ writel(IMGU_IRQCTRL_IRQ_MASK, base + IMGU_REG_IRQCTRL_EDGE(i));
+ writel(IMGU_IRQCTRL_IRQ_MASK,
+ base + IMGU_REG_IRQCTRL_ENABLE(i));
+ writel(IMGU_IRQCTRL_IRQ_MASK, base + IMGU_REG_IRQCTRL_CLEAR(i));
+ writel(IMGU_IRQCTRL_IRQ_MASK, base + IMGU_REG_IRQCTRL_MASK(i));
+ /* Wait for write complete */
+ readl(base + IMGU_REG_IRQCTRL_ENABLE(i));
+ }
+}
+
+static int ipu3_css_hw_init(struct ipu3_css *css)
+{
+ /* For checking that streaming monitor statuses are valid */
+ static const struct {
+ u32 reg;
+ u32 mask;
+ const char *name;
+ } stream_monitors[] = {
+ {
+ IMGU_REG_GP_SP1_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_ISP_PORT_SP12ISP,
+ "ISP0 to SP0"
+ }, {
+ IMGU_REG_GP_ISP_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_SP1_PORT_ISP2SP1,
+ "SP0 to ISP0"
+ }, {
+ IMGU_REG_GP_MOD_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_MOD_PORT_ISP2DMA,
+ "ISP0 to DMA0"
+ }, {
+ IMGU_REG_GP_ISP_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_ISP_PORT_DMA2ISP,
+ "DMA0 to ISP0"
+ }, {
+ IMGU_REG_GP_MOD_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_MOD_PORT_CELLS2GDC,
+ "ISP0 to GDC0"
+ }, {
+ IMGU_REG_GP_MOD_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_MOD_PORT_GDC2CELLS,
+ "GDC0 to ISP0"
+ }, {
+ IMGU_REG_GP_MOD_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_MOD_PORT_SP12DMA,
+ "SP0 to DMA0"
+ }, {
+ IMGU_REG_GP_SP1_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_SP1_PORT_DMA2SP1,
+ "DMA0 to SP0"
+ }, {
+ IMGU_REG_GP_MOD_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_MOD_PORT_CELLS2GDC,
+ "SP0 to GDC0"
+ }, {
+ IMGU_REG_GP_MOD_STRMON_STAT,
+ IMGU_GP_STRMON_STAT_MOD_PORT_GDC2CELLS,
+ "GDC0 to SP0"
+ },
+ };
+
+ struct device *dev = css->dev;
+ void __iomem *const base = css->base;
+ u32 val, i;
+
+ /* Set instruction cache address and inv bit for ISP, SP, and SP1 */
+ for (i = 0; i < IMGU_NUM_SP; i++) {
+ struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->fw_sp[i]];
+
+ writel(css->binary[css->fw_sp[i]].daddr,
+ base + IMGU_REG_SP_ICACHE_ADDR(bi->type));
+ writel(readl(base + IMGU_REG_SP_CTRL(bi->type)) |
+ IMGU_CTRL_ICACHE_INV,
+ base + IMGU_REG_SP_CTRL(bi->type));
+ }
+ writel(css->binary[css->fw_bl].daddr, base + IMGU_REG_ISP_ICACHE_ADDR);
+ writel(readl(base + IMGU_REG_ISP_CTRL) | IMGU_CTRL_ICACHE_INV,
+ base + IMGU_REG_ISP_CTRL);
+
+ /* Check that IMGU hardware is ready */
+
+ if (!(readl(base + IMGU_REG_SP_CTRL(0)) & IMGU_CTRL_IDLE)) {
+ dev_err(dev, "SP is not idle\n");
+ return -EIO;
+ }
+ if (!(readl(base + IMGU_REG_ISP_CTRL) & IMGU_CTRL_IDLE)) {
+ dev_err(dev, "ISP is not idle\n");
+ return -EIO;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(stream_monitors); i++) {
+ val = readl(base + stream_monitors[i].reg);
+ if (val & stream_monitors[i].mask) {
+ dev_err(dev, "error: Stream monitor %s is valid\n",
+ stream_monitors[i].name);
+ return -EIO;
+ }
+ }
+
+ /* Initialize GDC with default values */
+
+ for (i = 0; i < ARRAY_SIZE(ipu3_css_gdc_lut[0]); i++) {
+ u32 val0 = ipu3_css_gdc_lut[0][i] & IMGU_GDC_LUT_MASK;
+ u32 val1 = ipu3_css_gdc_lut[1][i] & IMGU_GDC_LUT_MASK;
+ u32 val2 = ipu3_css_gdc_lut[2][i] & IMGU_GDC_LUT_MASK;
+ u32 val3 = ipu3_css_gdc_lut[3][i] & IMGU_GDC_LUT_MASK;
+
+ writel(val0 | (val1 << 16),
+ base + IMGU_REG_GDC_LUT_BASE + i * 8);
+ writel(val2 | (val3 << 16),
+ base + IMGU_REG_GDC_LUT_BASE + i * 8 + 4);
+ }
+
+ return 0;
+}
+
+/* Boot the given IPU3 CSS SP */
+static int ipu3_css_hw_start_sp(struct ipu3_css *css, int sp)
+{
+ void __iomem *const base = css->base;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->fw_sp[sp]];
+ struct imgu_abi_sp_init_dmem_cfg dmem_cfg = {
+ .ddr_data_addr = css->binary[css->fw_sp[sp]].daddr
+ + bi->blob.data_source,
+ .dmem_data_addr = bi->blob.data_target,
+ .dmem_bss_addr = bi->blob.bss_target,
+ .data_size = bi->blob.data_size,
+ .bss_size = bi->blob.bss_size,
+ .sp_id = sp,
+ };
+
+ writes(&dmem_cfg, sizeof(dmem_cfg), base +
+ IMGU_REG_SP_DMEM_BASE(sp) + bi->info.sp.init_dmem_data);
+
+ writel(bi->info.sp.sp_entry, base + IMGU_REG_SP_START_ADDR(sp));
+
+ writel(readl(base + IMGU_REG_SP_CTRL(sp))
+ | IMGU_CTRL_START | IMGU_CTRL_RUN, base + IMGU_REG_SP_CTRL(sp));
+
+ if (ipu3_hw_wait(css->base, IMGU_REG_SP_DMEM_BASE(sp)
+ + bi->info.sp.sw_state,
+ ~0, IMGU_ABI_SP_SWSTATE_INITIALIZED))
+ return -EIO;
+
+ return 0;
+}
+
+/* Start the IPU3 CSS ImgU (Imaging Unit) and all the SPs */
+static int ipu3_css_hw_start(struct ipu3_css *css)
+{
+ static const u32 event_mask =
+ ((1 << IMGU_ABI_EVTTYPE_OUT_FRAME_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_2ND_OUT_FRAME_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_VF_OUT_FRAME_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_2ND_VF_OUT_FRAME_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_3A_STATS_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_DIS_STATS_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_PIPELINE_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_FRAME_TAGGED) |
+ (1 << IMGU_ABI_EVTTYPE_INPUT_FRAME_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_METADATA_DONE) |
+ (1 << IMGU_ABI_EVTTYPE_ACC_STAGE_COMPLETE))
+ << IMGU_ABI_SP_COMM_EVENT_IRQ_MASK_OR_SHIFT;
+
+ void __iomem *const base = css->base;
+ struct imgu_fw_info *bi, *bl = &css->fwp->binary_header[css->fw_bl];
+ unsigned int i;
+
+ writel(IMGU_TLB_INVALIDATE, base + IMGU_REG_TLB_INVALIDATE);
+
+ /* Start bootloader */
+
+ writel(IMGU_ABI_BL_SWSTATE_BUSY,
+ base + IMGU_REG_ISP_DMEM_BASE + bl->info.bl.sw_state);
+ writel(IMGU_NUM_SP,
+ base + IMGU_REG_ISP_DMEM_BASE + bl->info.bl.num_dma_cmds);
+
+ for (i = 0; i < IMGU_NUM_SP; i++) {
+ int j = IMGU_NUM_SP - i - 1; /* load sp1 first, then sp0 */
+ struct imgu_fw_info *sp =
+ &css->fwp->binary_header[css->fw_sp[j]];
+ struct imgu_abi_bl_dma_cmd_entry dma_cmd = {
+ .src_addr = css->binary[css->fw_sp[j]].daddr
+ + sp->blob.text_source,
+ .size = sp->blob.text_size,
+ .dst_type = IMGU_ABI_BL_DMACMD_TYPE_SP_PMEM,
+ .dst_addr = IMGU_SP_PMEM_BASE(j),
+ };
+
+ writes(&dma_cmd, sizeof(dma_cmd),
+ base + IMGU_REG_ISP_DMEM_BASE + i * sizeof(dma_cmd) +
+ bl->info.bl.dma_cmd_list);
+ }
+
+ writel(bl->info.bl.bl_entry, base + IMGU_REG_ISP_START_ADDR);
+
+ writel(readl(base + IMGU_REG_ISP_CTRL)
+ | IMGU_CTRL_START | IMGU_CTRL_RUN, base + IMGU_REG_ISP_CTRL);
+ if (ipu3_hw_wait(css->base, IMGU_REG_ISP_DMEM_BASE
+ + bl->info.bl.sw_state, ~0,
+ IMGU_ABI_BL_SWSTATE_OK)) {
+ dev_err(css->dev, "failed to start bootloader\n");
+ return -EIO;
+ }
+
+ /* Start ISP */
+
+ memset(css->xmem_sp_group_ptrs.vaddr, 0,
+ sizeof(struct imgu_abi_sp_group));
+
+ bi = &css->fwp->binary_header[css->fw_sp[0]];
+
+ writel(css->xmem_sp_group_ptrs.daddr,
+ base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.per_frame_data);
+
+ writel(IMGU_ABI_SP_SWSTATE_TERMINATED,
+ base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.sw_state);
+ writel(1, base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.invalidate_tlb);
+
+ if (ipu3_css_hw_start_sp(css, 0))
+ return -EIO;
+
+ writel(0, base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.isp_started);
+ writel(0, base + IMGU_REG_SP_DMEM_BASE(0) +
+ bi->info.sp.host_sp_queues_initialized);
+ writel(0, base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.sleep_mode);
+ writel(0, base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.invalidate_tlb);
+ writel(IMGU_ABI_SP_COMM_COMMAND_READY, base + IMGU_REG_SP_DMEM_BASE(0)
+ + bi->info.sp.host_sp_com + IMGU_ABI_SP_COMM_COMMAND);
+
+ /* Enable all events for all queues */
+
+ for (i = 0; i < IPU3_CSS_PIPE_ID_NUM; i++)
+ writel(event_mask, base + IMGU_REG_SP_DMEM_BASE(0)
+ + bi->info.sp.host_sp_com
+ + IMGU_ABI_SP_COMM_EVENT_IRQ_MASK(i));
+ writel(1, base + IMGU_REG_SP_DMEM_BASE(0) +
+ bi->info.sp.host_sp_queues_initialized);
+
+ /* Start SP1 */
+
+ bi = &css->fwp->binary_header[css->fw_sp[1]];
+
+ writel(IMGU_ABI_SP_SWSTATE_TERMINATED,
+ base + IMGU_REG_SP_DMEM_BASE(1) + bi->info.sp.sw_state);
+
+ if (ipu3_css_hw_start_sp(css, 1))
+ return -EIO;
+
+ writel(IMGU_ABI_SP_COMM_COMMAND_READY, base + IMGU_REG_SP_DMEM_BASE(1)
+ + bi->info.sp.host_sp_com + IMGU_ABI_SP_COMM_COMMAND);
+
+ return 0;
+}
+
+static void ipu3_css_hw_stop(struct ipu3_css *css)
+{
+ void __iomem *const base = css->base;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->fw_sp[0]];
+
+ /* Stop fw */
+ writel(IMGU_ABI_SP_COMM_COMMAND_TERMINATE,
+ base + IMGU_REG_SP_DMEM_BASE(0) +
+ bi->info.sp.host_sp_com + IMGU_ABI_SP_COMM_COMMAND);
+ if (ipu3_hw_wait(css->base, IMGU_REG_SP_CTRL(0),
+ IMGU_CTRL_IDLE, IMGU_CTRL_IDLE))
+ dev_err(css->dev, "wait sp0 idle timeout.\n");
+ if (readl(base + IMGU_REG_SP_DMEM_BASE(0) + bi->info.sp.sw_state) !=
+ IMGU_ABI_SP_SWSTATE_TERMINATED)
+ dev_err(css->dev, "sp0 is not terminated.\n");
+ if (ipu3_hw_wait(css->base, IMGU_REG_ISP_CTRL,
+ IMGU_CTRL_IDLE, IMGU_CTRL_IDLE))
+ dev_err(css->dev, "wait isp idle timeout\n");
+}
+
+static void ipu3_css_hw_cleanup(struct ipu3_css *css)
+{
+ void __iomem *const base = css->base;
+
+ /** Reset CSS **/
+
+ /* Clear the CSS busy signal */
+ readl(base + IMGU_REG_GP_BUSY);
+ writel(0, base + IMGU_REG_GP_BUSY);
+
+ /* Wait for idle signal */
+ if (ipu3_hw_wait(css->base, IMGU_REG_STATE, IMGU_STATE_IDLE_STS,
+ IMGU_STATE_IDLE_STS))
+ dev_err(css->dev, "failed to shut down hw cleanly\n");
+
+ /* Reset the css */
+ writel(readl(base + IMGU_REG_PM_CTRL) | IMGU_PM_CTRL_FORCE_RESET,
+ base + IMGU_REG_PM_CTRL);
+
+ usleep_range(200, 300);
+}
+
+int ipu3_css_irq_ack(struct ipu3_css *css)
+{
+ static const int NUM_SWIRQS = 3;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->fw_sp[0]];
+ void __iomem *const base = css->base;
+ u32 irq_status[IMGU_IRQCTRL_NUM];
+ int i;
+
+ u32 imgu_status = readl(base + IMGU_REG_INT_STATUS);
+
+ writel(imgu_status, base + IMGU_REG_INT_STATUS);
+ for (i = 0; i < IMGU_IRQCTRL_NUM; i++)
+ irq_status[i] = readl(base + IMGU_REG_IRQCTRL_STATUS(i));
+
+ for (i = 0; i < NUM_SWIRQS; i++) {
+ if (irq_status[IMGU_IRQCTRL_SP0] & IMGU_IRQCTRL_IRQ_SW_PIN(i)) {
+ /* SP SW interrupt */
+ u32 cnt = readl(base + IMGU_REG_SP_DMEM_BASE(0) +
+ bi->info.sp.output);
+ u32 val = readl(base + IMGU_REG_SP_DMEM_BASE(0) +
+ bi->info.sp.output + 4 + 4 * i);
+
+ dev_dbg(css->dev, "%s: swirq %i cnt %i val 0x%x\n",
+ __func__, i, cnt, val);
+ }
+ }
+
+ for (i = IMGU_IRQCTRL_NUM - 1; i >= 0; i--)
+ if (irq_status[i]) {
+ writel(irq_status[i], base + IMGU_REG_IRQCTRL_CLEAR(i));
+ /* Wait for write to complete */
+ readl(base + IMGU_REG_IRQCTRL_ENABLE(i));
+ }
+
+ dev_dbg(css->dev, "%s: imgu 0x%x main 0x%x sp0 0x%x sp1 0x%x\n",
+ __func__, imgu_status, irq_status[IMGU_IRQCTRL_MAIN],
+ irq_status[IMGU_IRQCTRL_SP0], irq_status[IMGU_IRQCTRL_SP1]);
+
+ if (!imgu_status && !irq_status[IMGU_IRQCTRL_MAIN])
+ return -ENOMSG;
+
+ return 0;
+}
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css.h b/drivers/media/pci/intel/ipu3/ipu3-css.h
new file mode 100644
index 0000000..d16d0c4
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-css.h
@@ -0,0 +1,203 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+
+#ifndef __IPU3_CSS_H
+#define __IPU3_CSS_H
+
+#include <linux/videodev2.h>
+#include <linux/types.h>
+
+#include "ipu3-abi.h"
+#include "ipu3-css-pool.h"
+
+/* 2 stages for split isp pipeline, 1 for scaling */
+#define IMGU_NUM_SP 2
+#define IMGU_MAX_PIPELINE_NUM 20
+
+/* For DVS etc., format FRAME_FMT_YUV420_16 */
+#define IPU3_CSS_AUX_FRAME_REF 0
+/* For temporal noise reduction DVS etc., format FRAME_FMT_YUV_LINE */
+#define IPU3_CSS_AUX_FRAME_TNR 1
+#define IPU3_CSS_AUX_FRAME_TYPES 2 /* REF and TNR */
+#define IPU3_CSS_AUX_FRAMES 2 /* 2 for REF and 2 for TNR */
+
+#define IPU3_CSS_QUEUE_IN 0
+#define IPU3_CSS_QUEUE_PARAMS 1
+#define IPU3_CSS_QUEUE_OUT 2
+#define IPU3_CSS_QUEUE_VF 3
+#define IPU3_CSS_QUEUE_STAT_3A 4
+#define IPU3_CSS_QUEUES 5
+
+#define IPU3_CSS_RECT_EFFECTIVE 0 /* Effective resolution */
+#define IPU3_CSS_RECT_BDS 1 /* Resolution after BDS */
+#define IPU3_CSS_RECT_ENVELOPE 2 /* DVS envelope size */
+#define IPU3_CSS_RECT_GDC 3 /* gdc output res */
+#define IPU3_CSS_RECTS 4 /* number of rects */
+
+#define IA_CSS_BINARY_MODE_PRIMARY 2
+#define IA_CSS_BINARY_MODE_VIDEO 3
+#define IPU3_CSS_DEFAULT_BINARY 3 /* default binary index */
+
+/*
+ * The pipe id type, distinguishes the kind of pipes that
+ * can be run in parallel.
+ */
+enum ipu3_css_pipe_id {
+ IPU3_CSS_PIPE_ID_PREVIEW,
+ IPU3_CSS_PIPE_ID_COPY,
+ IPU3_CSS_PIPE_ID_VIDEO,
+ IPU3_CSS_PIPE_ID_CAPTURE,
+ IPU3_CSS_PIPE_ID_YUVPP,
+ IPU3_CSS_PIPE_ID_ACC,
+ IPU3_CSS_PIPE_ID_NUM
+};
+
+struct ipu3_css_resolution {
+ u32 w;
+ u32 h;
+};
+
+enum ipu3_css_vf_status {
+ IPU3_NODE_VF_ENABLED,
+ IPU3_NODE_PV_ENABLED,
+ IPU3_NODE_VF_DISABLED
+};
+
+enum ipu3_css_buffer_state {
+ IPU3_CSS_BUFFER_NEW, /* Not yet queued */
+ IPU3_CSS_BUFFER_QUEUED, /* Queued, waiting to be filled */
+ IPU3_CSS_BUFFER_DONE, /* Finished processing, removed from queue */
+ IPU3_CSS_BUFFER_FAILED, /* Was not processed, removed from queue */
+};
+
+struct ipu3_css_buffer {
+ /* Private fields: user doesn't touch */
+ dma_addr_t daddr;
+ unsigned int queue;
+ enum ipu3_css_buffer_state state;
+ struct list_head list;
+ u8 queue_pos;
+};
+
+struct ipu3_css_format {
+ u32 pixelformat;
+ enum v4l2_colorspace colorspace;
+ enum imgu_abi_frame_format frame_format;
+ enum imgu_abi_bayer_order bayer_order;
+ enum imgu_abi_osys_format osys_format;
+ enum imgu_abi_osys_tiling osys_tiling;
+ u32 bytesperpixel_num; /* Bytes per pixel in first plane * 50 */
+ u8 bit_depth; /* Effective bits per pixel */
+ u8 chroma_decim; /* Chroma plane decimation, 0=no chroma plane */
+ u8 width_align; /* Alignment requirement for width_pad */
+ u8 flags;
+};
+
+struct ipu3_css_queue {
+ union {
+ struct v4l2_pix_format_mplane mpix;
+ struct v4l2_meta_format meta;
+
+ } fmt;
+ const struct ipu3_css_format *css_fmt;
+ unsigned int width_pad; /* bytesperline / byp */
+ struct list_head bufs;
+};
+
+/* IPU3 Camera Sub System structure */
+struct ipu3_css {
+ struct device *dev;
+ void __iomem *base;
+ const struct firmware *fw;
+ struct imgu_fw_header *fwp;
+ int iomem_length;
+ int fw_bl, fw_sp[IMGU_NUM_SP]; /* Indices of bl and SP binaries */
+ struct ipu3_css_map *binary; /* fw binaries mapped to device */
+ unsigned int current_binary; /* Currently selected binary */
+ bool streaming; /* true when streaming is enabled */
+ long frame; /* Latest frame not yet processed */
+ enum ipu3_css_pipe_id pipe_id; /* CSS pipe ID. */
+
+ /* Data structures shared with IMGU and driver, always allocated */
+ struct ipu3_css_map xmem_sp_stage_ptrs[IPU3_CSS_PIPE_ID_NUM]
+ [IMGU_ABI_MAX_STAGES];
+ struct ipu3_css_map xmem_isp_stage_ptrs[IPU3_CSS_PIPE_ID_NUM]
+ [IMGU_ABI_MAX_STAGES];
+ struct ipu3_css_map sp_ddr_ptrs;
+ struct ipu3_css_map xmem_sp_group_ptrs;
+
+ /* Data structures shared with IMGU and driver, binary specific */
+ /* PARAM_CLASS_CONFIG and PARAM_CLASS_STATE parameters */
+ struct ipu3_css_map binary_params_cs[IMGU_ABI_PARAM_CLASS_NUM - 1]
+ [IMGU_ABI_NUM_MEMORIES];
+
+ struct {
+ struct ipu3_css_map mem[IPU3_CSS_AUX_FRAMES];
+ unsigned int width;
+ unsigned int height;
+ unsigned int bytesperline;
+ unsigned int bytesperpixel;
+ } aux_frames[IPU3_CSS_AUX_FRAME_TYPES];
+
+ struct ipu3_css_queue queue[IPU3_CSS_QUEUES];
+ struct v4l2_rect rect[IPU3_CSS_RECTS];
+ struct ipu3_css_map abi_buffers[IPU3_CSS_QUEUES]
+ [IMGU_ABI_HOST2SP_BUFQ_SIZE];
+
+ struct {
+ struct ipu3_css_pool parameter_set_info;
+ struct ipu3_css_pool acc;
+ struct ipu3_css_pool gdc;
+ struct ipu3_css_pool obgrid;
+ /* PARAM_CLASS_PARAM parameters for binding while streaming */
+ struct ipu3_css_pool binary_params_p[IMGU_ABI_NUM_MEMORIES];
+ } pool;
+
+ enum ipu3_css_vf_status vf_output_en;
+ /* Protect access to css->queue[] */
+ spinlock_t qlock;
+};
+
+/******************* css v4l *******************/
+int ipu3_css_init(struct device *dev, struct ipu3_css *css,
+ void __iomem *base, int length);
+void ipu3_css_cleanup(struct ipu3_css *css);
+int ipu3_css_fmt_try(struct ipu3_css *css,
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
+ struct v4l2_rect *rects[IPU3_CSS_RECTS]);
+int ipu3_css_fmt_set(struct ipu3_css *css,
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
+ struct v4l2_rect *rects[IPU3_CSS_RECTS]);
+int ipu3_css_meta_fmt_set(struct v4l2_meta_format *fmt);
+int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b);
+struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css);
+int ipu3_css_start_streaming(struct ipu3_css *css);
+void ipu3_css_stop_streaming(struct ipu3_css *css);
+bool ipu3_css_queue_empty(struct ipu3_css *css);
+bool ipu3_css_is_streaming(struct ipu3_css *css);
+
+/******************* css hw *******************/
+int ipu3_css_set_powerup(struct device *dev, void __iomem *base);
+void ipu3_css_set_powerdown(struct device *dev, void __iomem *base);
+int ipu3_css_irq_ack(struct ipu3_css *css);
+
+/******************* set parameters ************/
+int ipu3_css_set_parameters(struct ipu3_css *css,
+ struct ipu3_uapi_params *set_params);
+
+/******************* css misc *******************/
+static inline enum ipu3_css_buffer_state
+ipu3_css_buf_state(struct ipu3_css_buffer *b)
+{
+ return b->state;
+}
+
+/* Initialize given buffer. May be called several times. */
+static inline void ipu3_css_buf_init(struct ipu3_css_buffer *b,
+ unsigned int queue, dma_addr_t daddr)
+{
+ b->state = IPU3_CSS_BUFFER_NEW;
+ b->queue = queue;
+ b->daddr = daddr;
+}
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 12/16] intel-ipu3: css: Initialize css hardware
2018-10-29 22:23 ` [PATCH v7 12/16] intel-ipu3: css: Initialize css hardware Yong Zhi
@ 2018-11-09 12:06 ` Sakari Ailus
0 siblings, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-11-09 12:06 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:23:06PM -0700, Yong Zhi wrote:
> This patch implements the functions to initialize
> and configure IPU3 h/w such as clock, irq and power.
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> ---
> drivers/media/pci/intel/ipu3/ipu3-css.c | 537 ++++++++++++++++++++++++++++++++
> drivers/media/pci/intel/ipu3/ipu3-css.h | 203 ++++++++++++
> 2 files changed, 740 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
>
...
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-css.h b/drivers/media/pci/intel/ipu3/ipu3-css.h
> new file mode 100644
> index 0000000..d16d0c4
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-css.h
...
> +/* IPU3 Camera Sub System structure */
> +struct ipu3_css {
> + struct device *dev;
> + void __iomem *base;
> + const struct firmware *fw;
> + struct imgu_fw_header *fwp;
> + int iomem_length;
u32? The same for the length parameter in ccs_init().
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 13/16] intel-ipu3: Add css pipeline programming
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (10 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 12/16] intel-ipu3: css: Initialize css hardware Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-10-29 22:23 ` [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework Yong Zhi
` (5 subsequent siblings)
17 siblings, 0 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
This provides helper library to be used by
v4l2 level to program imaging pipelines and
control the streaming.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-css.c | 1760 +++++++++++++++++++++++++++++++
1 file changed, 1760 insertions(+)
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css.c b/drivers/media/pci/intel/ipu3/ipu3-css.c
index 164830f..c63b387 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-css.c
@@ -16,6 +16,173 @@
IMGU_IRQCTRL_IRQ_SW_PIN(0) | \
IMGU_IRQCTRL_IRQ_SW_PIN(1))
+#define IPU3_CSS_FORMAT_BPP_DEN 50 /* Denominator */
+
+/* Some sane limits for resolutions */
+#define IPU3_CSS_MIN_RES 32
+#define IPU3_CSS_MAX_H 3136
+#define IPU3_CSS_MAX_W 4224
+
+/* filter size from graph settings is fixed as 4 */
+#define FILTER_SIZE 4
+#define MIN_ENVELOPE 8
+
+/*
+ * pre-allocated buffer size for CSS ABI, auxiliary frames
+ * after BDS and before GDC. Those values should be tuned
+ * to big enough to avoid buffer re-allocation when
+ * streaming to lower streaming latency.
+ */
+#define CSS_ABI_SIZE 136
+#define CSS_BDS_SIZE (4480 * 3200 * 3)
+#define CSS_GDC_SIZE (4224 * 3200 * 12 / 8)
+
+#define IPU3_CSS_QUEUE_TO_FLAGS(q) (1 << (q))
+#define IPU3_CSS_FORMAT_FL_IN \
+ IPU3_CSS_QUEUE_TO_FLAGS(IPU3_CSS_QUEUE_IN)
+#define IPU3_CSS_FORMAT_FL_OUT \
+ IPU3_CSS_QUEUE_TO_FLAGS(IPU3_CSS_QUEUE_OUT)
+#define IPU3_CSS_FORMAT_FL_VF \
+ IPU3_CSS_QUEUE_TO_FLAGS(IPU3_CSS_QUEUE_VF)
+
+/* Formats supported by IPU3 Camera Sub System */
+static const struct ipu3_css_format ipu3_css_formats[] = {
+ {
+ .pixelformat = V4L2_PIX_FMT_NV12,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .frame_format = IMGU_ABI_FRAME_FORMAT_NV12,
+ .osys_format = IMGU_ABI_OSYS_FORMAT_NV12,
+ .osys_tiling = IMGU_ABI_OSYS_TILING_NONE,
+ .bytesperpixel_num = 1 * IPU3_CSS_FORMAT_BPP_DEN,
+ .chroma_decim = 4,
+ .width_align = IPU3_UAPI_ISP_VEC_ELEMS,
+ .flags = IPU3_CSS_FORMAT_FL_OUT | IPU3_CSS_FORMAT_FL_VF,
+ }, {
+ /* Each 32 bytes contains 25 10-bit pixels */
+ .pixelformat = V4L2_PIX_FMT_IPU3_SBGGR10,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .frame_format = IMGU_ABI_FRAME_FORMAT_RAW_PACKED,
+ .bayer_order = IMGU_ABI_BAYER_ORDER_BGGR,
+ .bit_depth = 10,
+ .bytesperpixel_num = 64,
+ .width_align = 2 * IPU3_UAPI_ISP_VEC_ELEMS,
+ .flags = IPU3_CSS_FORMAT_FL_IN,
+ }, {
+ .pixelformat = V4L2_PIX_FMT_IPU3_SGBRG10,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .frame_format = IMGU_ABI_FRAME_FORMAT_RAW_PACKED,
+ .bayer_order = IMGU_ABI_BAYER_ORDER_GBRG,
+ .bit_depth = 10,
+ .bytesperpixel_num = 64,
+ .width_align = 2 * IPU3_UAPI_ISP_VEC_ELEMS,
+ .flags = IPU3_CSS_FORMAT_FL_IN,
+ }, {
+ .pixelformat = V4L2_PIX_FMT_IPU3_SGRBG10,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .frame_format = IMGU_ABI_FRAME_FORMAT_RAW_PACKED,
+ .bayer_order = IMGU_ABI_BAYER_ORDER_GRBG,
+ .bit_depth = 10,
+ .bytesperpixel_num = 64,
+ .width_align = 2 * IPU3_UAPI_ISP_VEC_ELEMS,
+ .flags = IPU3_CSS_FORMAT_FL_IN,
+ }, {
+ .pixelformat = V4L2_PIX_FMT_IPU3_SRGGB10,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .frame_format = IMGU_ABI_FRAME_FORMAT_RAW_PACKED,
+ .bayer_order = IMGU_ABI_BAYER_ORDER_RGGB,
+ .bit_depth = 10,
+ .bytesperpixel_num = 64,
+ .width_align = 2 * IPU3_UAPI_ISP_VEC_ELEMS,
+ .flags = IPU3_CSS_FORMAT_FL_IN,
+ },
+};
+
+static const struct {
+ enum imgu_abi_queue_id qid;
+ size_t ptr_ofs;
+} ipu3_css_queues[IPU3_CSS_QUEUES] = {
+ [IPU3_CSS_QUEUE_IN] = {
+ IMGU_ABI_QUEUE_C_ID,
+ offsetof(struct imgu_abi_buffer, payload.frame.frame_data)
+ },
+ [IPU3_CSS_QUEUE_OUT] = {
+ IMGU_ABI_QUEUE_D_ID,
+ offsetof(struct imgu_abi_buffer, payload.frame.frame_data)
+ },
+ [IPU3_CSS_QUEUE_VF] = {
+ IMGU_ABI_QUEUE_E_ID,
+ offsetof(struct imgu_abi_buffer, payload.frame.frame_data)
+ },
+ [IPU3_CSS_QUEUE_STAT_3A] = {
+ IMGU_ABI_QUEUE_F_ID,
+ offsetof(struct imgu_abi_buffer, payload.s3a.data_ptr)
+ },
+};
+
+/* Initialize queue based on given format, adjust format as needed */
+static int ipu3_css_queue_init(struct ipu3_css_queue *queue,
+ struct v4l2_pix_format_mplane *fmt, u32 flags)
+{
+ struct v4l2_pix_format_mplane *const f = &queue->fmt.mpix;
+ unsigned int i;
+ u32 sizeimage;
+
+ INIT_LIST_HEAD(&queue->bufs);
+
+ queue->css_fmt = NULL; /* Disable */
+ if (!fmt)
+ return 0;
+
+ for (i = 0; i < ARRAY_SIZE(ipu3_css_formats); i++) {
+ if (!(ipu3_css_formats[i].flags & flags))
+ continue;
+ queue->css_fmt = &ipu3_css_formats[i];
+ if (ipu3_css_formats[i].pixelformat == fmt->pixelformat)
+ break;
+ }
+ if (!queue->css_fmt)
+ return -EINVAL; /* Could not find any suitable format */
+
+ queue->fmt.mpix = *fmt;
+
+ f->width = ALIGN(clamp_t(u32, f->width,
+ IPU3_CSS_MIN_RES, IPU3_CSS_MAX_W), 2);
+ f->height = ALIGN(clamp_t(u32, f->height,
+ IPU3_CSS_MIN_RES, IPU3_CSS_MAX_H), 2);
+ queue->width_pad = ALIGN(f->width, queue->css_fmt->width_align);
+ if (queue->css_fmt->frame_format != IMGU_ABI_FRAME_FORMAT_RAW_PACKED)
+ f->plane_fmt[0].bytesperline = DIV_ROUND_UP(queue->width_pad *
+ queue->css_fmt->bytesperpixel_num,
+ IPU3_CSS_FORMAT_BPP_DEN);
+ else
+ /* For packed raw, alignment for bpl is by 50 to the width */
+ f->plane_fmt[0].bytesperline =
+ DIV_ROUND_UP(f->width,
+ IPU3_CSS_FORMAT_BPP_DEN) *
+ queue->css_fmt->bytesperpixel_num;
+
+ sizeimage = f->height * f->plane_fmt[0].bytesperline;
+ if (queue->css_fmt->chroma_decim)
+ sizeimage += 2 * sizeimage / queue->css_fmt->chroma_decim;
+
+ f->plane_fmt[0].sizeimage = sizeimage;
+ f->field = V4L2_FIELD_NONE;
+ f->num_planes = 1;
+ f->colorspace = queue->css_fmt->colorspace;
+ f->flags = 0;
+ f->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+ f->quantization = V4L2_QUANTIZATION_DEFAULT;
+ f->xfer_func = V4L2_XFER_FUNC_DEFAULT;
+ memset(f->reserved, 0, sizeof(f->reserved));
+
+ return 0;
+}
+
+static bool ipu3_css_queue_enabled(struct ipu3_css_queue *q)
+{
+ return q->css_fmt;
+}
+
/******************* css hw *******************/
/* In the style of writesl() defined in include/asm-generic/io.h */
@@ -492,6 +659,1599 @@ static void ipu3_css_hw_cleanup(struct ipu3_css *css)
usleep_range(200, 300);
}
+static void ipu3_css_pipeline_cleanup(struct ipu3_css *css)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ unsigned int i;
+
+ ipu3_css_pool_cleanup(imgu, &css->pool.parameter_set_info);
+ ipu3_css_pool_cleanup(imgu, &css->pool.acc);
+ ipu3_css_pool_cleanup(imgu, &css->pool.gdc);
+ ipu3_css_pool_cleanup(imgu, &css->pool.obgrid);
+
+ for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
+ ipu3_css_pool_cleanup(imgu, &css->pool.binary_params_p[i]);
+}
+
+/*
+ * This function initializes various stages of the
+ * IPU3 CSS ISP pipeline
+ */
+static int ipu3_css_pipeline_init(struct ipu3_css *css)
+{
+ static const unsigned int PIPE_ID = IPU3_CSS_PIPE_ID_VIDEO;
+ static const int BYPC = 2; /* Bytes per component */
+ static const struct imgu_abi_buffer_sp buffer_sp_init = {
+ .buf_src = {.queue_id = IMGU_ABI_QUEUE_EVENT_ID},
+ .buf_type = IMGU_ABI_BUFFER_TYPE_INVALID,
+ };
+
+ struct imgu_abi_isp_iterator_config *cfg_iter;
+ struct imgu_abi_isp_ref_config *cfg_ref;
+ struct imgu_abi_isp_dvs_config *cfg_dvs;
+ struct imgu_abi_isp_tnr3_config *cfg_tnr;
+ struct imgu_abi_isp_ref_dmem_state *cfg_ref_state;
+ struct imgu_abi_isp_tnr3_dmem_state *cfg_tnr_state;
+
+ const int pipe = 0, stage = 0, thread = 0;
+ unsigned int i, j;
+
+ const struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->current_binary];
+ const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
+
+ struct imgu_fw_config_memory_offsets *cofs = (void *)css->fwp +
+ bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_CONFIG];
+ struct imgu_fw_state_memory_offsets *sofs = (void *)css->fwp +
+ bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_STATE];
+
+ struct imgu_abi_isp_stage *isp_stage;
+ struct imgu_abi_sp_stage *sp_stage;
+ struct imgu_abi_sp_group *sp_group;
+
+ const unsigned int bds_width_pad =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].width,
+ 2 * IPU3_UAPI_ISP_VEC_ELEMS);
+
+ const enum imgu_abi_memories m0 = IMGU_ABI_MEM_ISP_DMEM0;
+ enum imgu_abi_param_class cfg = IMGU_ABI_PARAM_CLASS_CONFIG;
+ void *vaddr = css->binary_params_cs[cfg - 1][m0].vaddr;
+
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+
+ /* Configure iterator */
+
+ cfg_iter = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ &cofs->dmem.iterator,
+ sizeof(*cfg_iter), vaddr);
+ if (!cfg_iter)
+ goto bad_firmware;
+
+ cfg_iter->input_info.res.width =
+ css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
+ cfg_iter->input_info.res.height =
+ css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
+ cfg_iter->input_info.padded_width =
+ css->queue[IPU3_CSS_QUEUE_IN].width_pad;
+ cfg_iter->input_info.format =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->frame_format;
+ cfg_iter->input_info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bit_depth;
+ cfg_iter->input_info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
+ cfg_iter->input_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+
+ cfg_iter->internal_info.res.width = css->rect[IPU3_CSS_RECT_BDS].width;
+ cfg_iter->internal_info.res.height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ cfg_iter->internal_info.padded_width = bds_width_pad;
+ cfg_iter->internal_info.format =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ cfg_iter->internal_info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ cfg_iter->internal_info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ cfg_iter->internal_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+
+ cfg_iter->output_info.res.width =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ cfg_iter->output_info.res.height =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ cfg_iter->output_info.padded_width =
+ css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ cfg_iter->output_info.format =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ cfg_iter->output_info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ cfg_iter->output_info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ cfg_iter->output_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+
+ cfg_iter->vf_info.res.width =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ cfg_iter->vf_info.res.height =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ cfg_iter->vf_info.padded_width =
+ css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ cfg_iter->vf_info.format =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+ cfg_iter->vf_info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bit_depth;
+ cfg_iter->vf_info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bayer_order;
+ cfg_iter->vf_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+
+ cfg_iter->dvs_envelope.width = css->rect[IPU3_CSS_RECT_ENVELOPE].width;
+ cfg_iter->dvs_envelope.height =
+ css->rect[IPU3_CSS_RECT_ENVELOPE].height;
+
+ /* Configure reference (delay) frames */
+
+ cfg_ref = ipu3_css_fw_pipeline_params(css, cfg, m0, &cofs->dmem.ref,
+ sizeof(*cfg_ref), vaddr);
+ if (!cfg_ref)
+ goto bad_firmware;
+
+ cfg_ref->port_b.crop = 0;
+ cfg_ref->port_b.elems = IMGU_ABI_ISP_DDR_WORD_BYTES / BYPC;
+ cfg_ref->port_b.width = css->aux_frames[IPU3_CSS_AUX_FRAME_REF].width;
+ cfg_ref->port_b.stride =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline;
+ cfg_ref->width_a_over_b =
+ IPU3_UAPI_ISP_VEC_ELEMS / cfg_ref->port_b.elems;
+ cfg_ref->dvs_frame_delay = IPU3_CSS_AUX_FRAMES - 1;
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++) {
+ cfg_ref->ref_frame_addr_y[i] =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i].daddr;
+ cfg_ref->ref_frame_addr_c[i] =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i].daddr +
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline *
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
+ }
+ for (; i < IMGU_ABI_FRAMES_REF; i++) {
+ cfg_ref->ref_frame_addr_y[i] = 0;
+ cfg_ref->ref_frame_addr_c[i] = 0;
+ }
+
+ /* Configure DVS (digital video stabilization) */
+
+ cfg_dvs = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ &cofs->dmem.dvs, sizeof(*cfg_dvs),
+ vaddr);
+ if (!cfg_dvs)
+ goto bad_firmware;
+
+ cfg_dvs->num_horizontal_blocks =
+ ALIGN(DIV_ROUND_UP(css->rect[IPU3_CSS_RECT_GDC].width,
+ IMGU_DVS_BLOCK_W), 2);
+ cfg_dvs->num_vertical_blocks =
+ DIV_ROUND_UP(css->rect[IPU3_CSS_RECT_GDC].height,
+ IMGU_DVS_BLOCK_H);
+
+ /* Configure TNR (temporal noise reduction) */
+
+ if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ cfg_tnr = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ &cofs->dmem.tnr3,
+ sizeof(*cfg_tnr),
+ vaddr);
+ if (!cfg_tnr)
+ goto bad_firmware;
+
+ cfg_tnr->port_b.crop = 0;
+ cfg_tnr->port_b.elems = IMGU_ABI_ISP_DDR_WORD_BYTES;
+ cfg_tnr->port_b.width =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width;
+ cfg_tnr->port_b.stride =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperline;
+ cfg_tnr->width_a_over_b =
+ IPU3_UAPI_ISP_VEC_ELEMS / cfg_tnr->port_b.elems;
+ cfg_tnr->frame_height =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height;
+ cfg_tnr->delay_frame = IPU3_CSS_AUX_FRAMES - 1;
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ cfg_tnr->frame_addr[i] =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR]
+ .mem[i].daddr;
+ for (; i < IMGU_ABI_FRAMES_TNR; i++)
+ cfg_tnr->frame_addr[i] = 0;
+ }
+
+ /* Configure ref dmem state parameters */
+
+ cfg = IMGU_ABI_PARAM_CLASS_STATE;
+ vaddr = css->binary_params_cs[cfg - 1][m0].vaddr;
+
+ cfg_ref_state = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ &sofs->dmem.ref,
+ sizeof(*cfg_ref_state),
+ vaddr);
+ if (!cfg_ref_state)
+ goto bad_firmware;
+
+ cfg_ref_state->ref_in_buf_idx = 0;
+ cfg_ref_state->ref_out_buf_idx = 1;
+
+ /* Configure tnr dmem state parameters */
+ if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ cfg_tnr_state =
+ ipu3_css_fw_pipeline_params(css, cfg, m0,
+ &sofs->dmem.tnr3,
+ sizeof(*cfg_tnr_state),
+ vaddr);
+ if (!cfg_tnr_state)
+ goto bad_firmware;
+
+ cfg_tnr_state->in_bufidx = 0;
+ cfg_tnr_state->out_bufidx = 1;
+ cfg_tnr_state->bypass_filter = 0;
+ cfg_tnr_state->total_frame_counter = 0;
+ for (i = 0; i < IMGU_ABI_BUF_SETS_TNR; i++)
+ cfg_tnr_state->buffer_frame_counter[i] = 0;
+ }
+
+ /* Configure ISP stage */
+
+ isp_stage = css->xmem_isp_stage_ptrs[pipe][stage].vaddr;
+ memset(isp_stage, 0, sizeof(*isp_stage));
+ isp_stage->blob_info = bi->blob;
+ isp_stage->binary_info = bi->info.isp.sp;
+ strlcpy(isp_stage->binary_name,
+ (char *)css->fwp + bi->blob.prog_name_offset,
+ sizeof(isp_stage->binary_name));
+ isp_stage->mem_initializers = bi->info.isp.sp.mem_initializers;
+ for (i = IMGU_ABI_PARAM_CLASS_CONFIG; i < IMGU_ABI_PARAM_CLASS_NUM; i++)
+ for (j = 0; j < IMGU_ABI_NUM_MEMORIES; j++)
+ isp_stage->mem_initializers.params[i][j].address =
+ css->binary_params_cs[i - 1][j].daddr;
+
+ /* Configure SP stage */
+
+ sp_stage = css->xmem_sp_stage_ptrs[pipe][stage].vaddr;
+ memset(sp_stage, 0, sizeof(*sp_stage));
+
+ sp_stage->frames.in.buf_attr = buffer_sp_init;
+ for (i = 0; i < IMGU_ABI_BINARY_MAX_OUTPUT_PORTS; i++)
+ sp_stage->frames.out[i].buf_attr = buffer_sp_init;
+ sp_stage->frames.out_vf.buf_attr = buffer_sp_init;
+ sp_stage->frames.s3a_buf = buffer_sp_init;
+ sp_stage->frames.dvs_buf = buffer_sp_init;
+
+ sp_stage->stage_type = IMGU_ABI_STAGE_TYPE_ISP;
+ sp_stage->num = stage;
+ sp_stage->isp_online = 0;
+ sp_stage->isp_copy_vf = 0;
+ sp_stage->isp_copy_output = 0;
+
+ /* Enable VF output only when VF or PV queue requested by user */
+
+ sp_stage->enable.vf_output =
+ (css->vf_output_en != IPU3_NODE_VF_DISABLED);
+
+ sp_stage->frames.effective_in_res.width =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].width;
+ sp_stage->frames.effective_in_res.height =
+ css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ sp_stage->frames.in.info.res.width =
+ css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
+ sp_stage->frames.in.info.res.height =
+ css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
+ sp_stage->frames.in.info.padded_width =
+ css->queue[IPU3_CSS_QUEUE_IN].width_pad;
+ sp_stage->frames.in.info.format =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->frame_format;
+ sp_stage->frames.in.info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bit_depth;
+ sp_stage->frames.in.info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
+ sp_stage->frames.in.info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+ sp_stage->frames.in.buf_attr.buf_src.queue_id = IMGU_ABI_QUEUE_C_ID;
+ sp_stage->frames.in.buf_attr.buf_type =
+ IMGU_ABI_BUFFER_TYPE_INPUT_FRAME;
+
+ sp_stage->frames.out[0].info.res.width =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ sp_stage->frames.out[0].info.res.height =
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ sp_stage->frames.out[0].info.padded_width =
+ css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ sp_stage->frames.out[0].info.format =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ sp_stage->frames.out[0].info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ sp_stage->frames.out[0].info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ sp_stage->frames.out[0].info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+ sp_stage->frames.out[0].planes.nv.uv.offset =
+ css->queue[IPU3_CSS_QUEUE_OUT].width_pad *
+ css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ sp_stage->frames.out[0].buf_attr.buf_src.queue_id = IMGU_ABI_QUEUE_D_ID;
+ sp_stage->frames.out[0].buf_attr.buf_type =
+ IMGU_ABI_BUFFER_TYPE_OUTPUT_FRAME;
+
+ sp_stage->frames.out[1].buf_attr.buf_src.queue_id =
+ IMGU_ABI_QUEUE_EVENT_ID;
+
+ sp_stage->frames.internal_frame_info.res.width =
+ css->rect[IPU3_CSS_RECT_BDS].width;
+ sp_stage->frames.internal_frame_info.res.height =
+ css->rect[IPU3_CSS_RECT_BDS].height;
+ sp_stage->frames.internal_frame_info.padded_width = bds_width_pad;
+
+ sp_stage->frames.internal_frame_info.format =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ sp_stage->frames.internal_frame_info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ sp_stage->frames.internal_frame_info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ sp_stage->frames.internal_frame_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+
+ sp_stage->frames.out_vf.info.res.width =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ sp_stage->frames.out_vf.info.res.height =
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ sp_stage->frames.out_vf.info.padded_width =
+ css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ sp_stage->frames.out_vf.info.format =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+ sp_stage->frames.out_vf.info.raw_bit_depth =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bit_depth;
+ sp_stage->frames.out_vf.info.raw_bayer_order =
+ css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bayer_order;
+ sp_stage->frames.out_vf.info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
+ sp_stage->frames.out_vf.planes.yuv.u.offset =
+ css->queue[IPU3_CSS_QUEUE_VF].width_pad *
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ sp_stage->frames.out_vf.planes.yuv.v.offset =
+ css->queue[IPU3_CSS_QUEUE_VF].width_pad *
+ css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height * 5 / 4;
+ sp_stage->frames.out_vf.buf_attr.buf_src.queue_id = IMGU_ABI_QUEUE_E_ID;
+ sp_stage->frames.out_vf.buf_attr.buf_type =
+ IMGU_ABI_BUFFER_TYPE_VF_OUTPUT_FRAME;
+
+ sp_stage->frames.s3a_buf.buf_src.queue_id = IMGU_ABI_QUEUE_F_ID;
+ sp_stage->frames.s3a_buf.buf_type = IMGU_ABI_BUFFER_TYPE_3A_STATISTICS;
+
+ sp_stage->frames.dvs_buf.buf_src.queue_id = IMGU_ABI_QUEUE_G_ID;
+ sp_stage->frames.dvs_buf.buf_type = IMGU_ABI_BUFFER_TYPE_DIS_STATISTICS;
+
+ sp_stage->dvs_envelope.width = css->rect[IPU3_CSS_RECT_ENVELOPE].width;
+ sp_stage->dvs_envelope.height =
+ css->rect[IPU3_CSS_RECT_ENVELOPE].height;
+
+ sp_stage->isp_pipe_version =
+ bi->info.isp.sp.pipeline.isp_pipe_version;
+ sp_stage->isp_deci_log_factor =
+ clamp(max(fls(css->rect[IPU3_CSS_RECT_BDS].width /
+ IMGU_MAX_BQ_GRID_WIDTH),
+ fls(css->rect[IPU3_CSS_RECT_BDS].height /
+ IMGU_MAX_BQ_GRID_HEIGHT)) - 1, 3, 5);
+ sp_stage->isp_vf_downscale_bits = 0;
+ sp_stage->if_config_index = 255;
+ sp_stage->sp_enable_xnr = 0;
+ sp_stage->num_stripes = stripes;
+ sp_stage->enable.s3a = 1;
+ sp_stage->enable.dvs_stats = 0;
+
+ sp_stage->xmem_bin_addr = css->binary[css->current_binary].daddr;
+ sp_stage->xmem_map_addr = css->sp_ddr_ptrs.daddr;
+ sp_stage->isp_stage_addr = css->xmem_isp_stage_ptrs[pipe][stage].daddr;
+
+ /* Configure SP group */
+
+ sp_group = css->xmem_sp_group_ptrs.vaddr;
+ memset(sp_group, 0, sizeof(*sp_group));
+
+ sp_group->pipe[thread].num_stages = 1;
+ sp_group->pipe[thread].pipe_id = PIPE_ID;
+ sp_group->pipe[thread].thread_id = thread;
+ sp_group->pipe[thread].pipe_num = pipe;
+ sp_group->pipe[thread].num_execs = -1;
+ sp_group->pipe[thread].pipe_qos_config = -1;
+ sp_group->pipe[thread].required_bds_factor = 0;
+ sp_group->pipe[thread].dvs_frame_delay = IPU3_CSS_AUX_FRAMES - 1;
+ sp_group->pipe[thread].inout_port_config =
+ IMGU_ABI_PORT_CONFIG_TYPE_INPUT_HOST |
+ IMGU_ABI_PORT_CONFIG_TYPE_OUTPUT_HOST;
+ sp_group->pipe[thread].scaler_pp_lut = 0;
+ sp_group->pipe[thread].shading.internal_frame_origin_x_bqs_on_sctbl = 0;
+ sp_group->pipe[thread].shading.internal_frame_origin_y_bqs_on_sctbl = 0;
+ sp_group->pipe[thread].sp_stage_addr[stage] =
+ css->xmem_sp_stage_ptrs[pipe][stage].daddr;
+ sp_group->pipe[thread].pipe_config =
+ bi->info.isp.sp.enable.params ? (1 << thread) : 0;
+ sp_group->pipe[thread].pipe_config |= IMGU_ABI_PIPE_CONFIG_ACQUIRE_ISP;
+
+ /* Initialize parameter pools */
+
+ if (ipu3_css_pool_init(imgu, &css->pool.parameter_set_info,
+ sizeof(struct imgu_abi_parameter_set_info)) ||
+ ipu3_css_pool_init(imgu, &css->pool.acc,
+ sizeof(struct imgu_abi_acc_param)) ||
+ ipu3_css_pool_init(imgu, &css->pool.gdc,
+ sizeof(struct imgu_abi_gdc_warp_param) *
+ 3 * cfg_dvs->num_horizontal_blocks / 2 *
+ cfg_dvs->num_vertical_blocks) ||
+ ipu3_css_pool_init(imgu, &css->pool.obgrid,
+ ipu3_css_fw_obgrid_size(
+ &css->fwp->binary_header[css->current_binary])))
+ goto out_of_memory;
+
+ for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
+ if (ipu3_css_pool_init(imgu, &css->pool.binary_params_p[i],
+ bi->info.isp.sp.mem_initializers.params
+ [IMGU_ABI_PARAM_CLASS_PARAM][i].size))
+ goto out_of_memory;
+
+ return 0;
+
+bad_firmware:
+ ipu3_css_pipeline_cleanup(css);
+ return -EPROTO;
+
+out_of_memory:
+ ipu3_css_pipeline_cleanup(css);
+ return -ENOMEM;
+}
+
+static u8 ipu3_css_queue_pos(struct ipu3_css *css, int queue, int thread)
+{
+ static const unsigned int sp;
+ void __iomem *const base = css->base;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->fw_sp[sp]];
+ struct imgu_abi_queues __iomem *q = base + IMGU_REG_SP_DMEM_BASE(sp) +
+ bi->info.sp.host_sp_queue;
+
+ return queue >= 0 ? readb(&q->host2sp_bufq_info[thread][queue].end) :
+ readb(&q->host2sp_evtq_info.end);
+}
+
+/* Sent data to sp using given buffer queue, or if queue < 0, event queue. */
+static int ipu3_css_queue_data(struct ipu3_css *css,
+ int queue, int thread, u32 data)
+{
+ static const unsigned int sp;
+ void __iomem *const base = css->base;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->fw_sp[sp]];
+ struct imgu_abi_queues __iomem *q = base + IMGU_REG_SP_DMEM_BASE(sp) +
+ bi->info.sp.host_sp_queue;
+ u8 size, start, end, end2;
+
+ if (queue >= 0) {
+ size = readb(&q->host2sp_bufq_info[thread][queue].size);
+ start = readb(&q->host2sp_bufq_info[thread][queue].start);
+ end = readb(&q->host2sp_bufq_info[thread][queue].end);
+ } else {
+ size = readb(&q->host2sp_evtq_info.size);
+ start = readb(&q->host2sp_evtq_info.start);
+ end = readb(&q->host2sp_evtq_info.end);
+ }
+
+ if (size == 0)
+ return -EIO;
+
+ end2 = (end + 1) % size;
+ if (end2 == start)
+ return -EBUSY; /* Queue full */
+
+ if (queue >= 0) {
+ writel(data, &q->host2sp_bufq[thread][queue][end]);
+ writeb(end2, &q->host2sp_bufq_info[thread][queue].end);
+ } else {
+ writel(data, &q->host2sp_evtq[end]);
+ writeb(end2, &q->host2sp_evtq_info.end);
+ }
+
+ return 0;
+}
+
+/* Receive data using given buffer queue, or if queue < 0, event queue. */
+static int ipu3_css_dequeue_data(struct ipu3_css *css, int queue, u32 *data)
+{
+ static const unsigned int sp;
+ void __iomem *const base = css->base;
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css->fw_sp[sp]];
+ struct imgu_abi_queues __iomem *q = base + IMGU_REG_SP_DMEM_BASE(sp) +
+ bi->info.sp.host_sp_queue;
+ u8 size, start, end, start2;
+
+ if (queue >= 0) {
+ size = readb(&q->sp2host_bufq_info[queue].size);
+ start = readb(&q->sp2host_bufq_info[queue].start);
+ end = readb(&q->sp2host_bufq_info[queue].end);
+ } else {
+ size = readb(&q->sp2host_evtq_info.size);
+ start = readb(&q->sp2host_evtq_info.start);
+ end = readb(&q->sp2host_evtq_info.end);
+ }
+
+ if (size == 0)
+ return -EIO;
+
+ if (end == start)
+ return -EBUSY; /* Queue empty */
+
+ start2 = (start + 1) % size;
+
+ if (queue >= 0) {
+ *data = readl(&q->sp2host_bufq[queue][start]);
+ writeb(start2, &q->sp2host_bufq_info[queue].start);
+ } else {
+ int r;
+
+ *data = readl(&q->sp2host_evtq[start]);
+ writeb(start2, &q->sp2host_evtq_info.start);
+
+ /* Acknowledge events dequeued from event queue */
+ r = ipu3_css_queue_data(css, queue, 0,
+ IMGU_ABI_EVENT_EVENT_DEQUEUED);
+ if (r < 0)
+ return r;
+ }
+
+ return 0;
+}
+
+/* Free binary-specific resources */
+static void ipu3_css_binary_cleanup(struct ipu3_css *css)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ unsigned int i, j;
+
+ for (j = 0; j < IMGU_ABI_PARAM_CLASS_NUM - 1; j++)
+ for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
+ ipu3_dmamap_free(imgu, &css->binary_params_cs[j][i]);
+
+ j = IPU3_CSS_AUX_FRAME_REF;
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ ipu3_dmamap_free(imgu, &css->aux_frames[j].mem[i]);
+
+ j = IPU3_CSS_AUX_FRAME_TNR;
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ ipu3_dmamap_free(imgu, &css->aux_frames[j].mem[i]);
+}
+
+static int ipu3_css_binary_preallocate(struct ipu3_css *css)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ unsigned int i, j;
+
+ for (j = IMGU_ABI_PARAM_CLASS_CONFIG; j < IMGU_ABI_PARAM_CLASS_NUM; j++)
+ for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++) {
+ if (!ipu3_dmamap_alloc(imgu,
+ &css->binary_params_cs[j - 1][i],
+ CSS_ABI_SIZE))
+ goto out_of_memory;
+ }
+
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ if (!ipu3_dmamap_alloc(imgu,
+ &css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i],
+ CSS_BDS_SIZE))
+ goto out_of_memory;
+
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ if (!ipu3_dmamap_alloc(imgu,
+ &css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].mem[i],
+ CSS_GDC_SIZE))
+ goto out_of_memory;
+
+ return 0;
+
+out_of_memory:
+ ipu3_css_binary_cleanup(css);
+ return -ENOMEM;
+}
+
+/* allocate binary-specific resources */
+static int ipu3_css_binary_setup(struct ipu3_css *css)
+{
+ const struct imgu_abi_binary_info *sp =
+ &css->fwp->binary_header[css->current_binary].info.isp.sp;
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ static const int BYPC = 2; /* Bytes per component */
+ unsigned int w, h, size, i, j;
+
+ /* Allocate parameter memory blocks for this binary */
+
+ for (j = IMGU_ABI_PARAM_CLASS_CONFIG; j < IMGU_ABI_PARAM_CLASS_NUM; j++)
+ for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++) {
+ if (ipu3_css_dma_buffer_resize(
+ imgu, &css->binary_params_cs[j - 1][i],
+ sp->mem_initializers.params[j][i].size))
+ goto out_of_memory;
+ }
+
+ /* Allocate internal frame buffers */
+
+ /* Reference frames for DVS, FRAME_FORMAT_YUV420_16 */
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel = BYPC;
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].width =
+ css->rect[IPU3_CSS_RECT_BDS].width;
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height =
+ ALIGN(css->rect[IPU3_CSS_RECT_BDS].height,
+ IMGU_DVS_BLOCK_H) + 2 * IMGU_GDC_BUF_Y;
+ h = css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
+ w = ALIGN(css->rect[IPU3_CSS_RECT_BDS].width,
+ 2 * IPU3_UAPI_ISP_VEC_ELEMS) + 2 * IMGU_GDC_BUF_X;
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline =
+ css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel * w;
+ size = w * h * BYPC + (w / 2) * (h / 2) * BYPC * 2;
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ if (ipu3_css_dma_buffer_resize(
+ imgu,
+ &css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i], size))
+ goto out_of_memory;
+
+ /* TNR frames for temporal noise reduction, FRAME_FORMAT_YUV_LINE */
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperpixel = 1;
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width =
+ roundup(css->rect[IPU3_CSS_RECT_GDC].width,
+ sp->block.block_width *
+ IPU3_UAPI_ISP_VEC_ELEMS);
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height =
+ roundup(css->rect[IPU3_CSS_RECT_GDC].height,
+ sp->block.output_block_height);
+
+ w = css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width;
+ css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperline = w;
+ h = css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height;
+ size = w * ALIGN(h * 3 / 2 + 3, 2); /* +3 for vf_pp prefetch */
+ for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
+ if (ipu3_css_dma_buffer_resize(
+ imgu,
+ &css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].mem[i], size))
+ goto out_of_memory;
+
+ return 0;
+
+out_of_memory:
+ ipu3_css_binary_cleanup(css);
+ return -ENOMEM;
+}
+
+int ipu3_css_start_streaming(struct ipu3_css *css)
+{
+ u32 data;
+ int r;
+
+ if (css->streaming)
+ return -EPROTO;
+
+ r = ipu3_css_binary_setup(css);
+ if (r < 0)
+ return r;
+
+ r = ipu3_css_hw_init(css);
+ if (r < 0)
+ return r;
+
+ r = ipu3_css_hw_start(css);
+ if (r < 0)
+ goto fail;
+
+ r = ipu3_css_pipeline_init(css);
+ if (r < 0)
+ goto fail;
+
+ css->streaming = true;
+ css->frame = 0;
+
+ ipu3_css_hw_enable_irq(css);
+
+ /* Initialize parameters to default */
+ r = ipu3_css_set_parameters(css, NULL);
+ if (r < 0)
+ goto fail;
+
+ while (!(r = ipu3_css_dequeue_data(css, IMGU_ABI_QUEUE_A_ID, &data)))
+ ;
+ if (r != -EBUSY)
+ goto fail;
+
+ while (!(r = ipu3_css_dequeue_data(css, IMGU_ABI_QUEUE_B_ID, &data)))
+ ;
+ if (r != -EBUSY)
+ goto fail;
+
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0,
+ IMGU_ABI_EVENT_START_STREAM);
+ if (r < 0)
+ goto fail;
+
+ return 0;
+
+fail:
+ css->streaming = false;
+ ipu3_css_hw_cleanup(css);
+ ipu3_css_pipeline_cleanup(css);
+ ipu3_css_binary_cleanup(css);
+
+ return r;
+}
+
+void ipu3_css_stop_streaming(struct ipu3_css *css)
+{
+ struct ipu3_css_buffer *b, *b0;
+ int q, r;
+
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0,
+ IMGU_ABI_EVENT_STOP_STREAM);
+
+ if (r < 0)
+ dev_warn(css->dev, "failed on stop stream event\n");
+
+ if (!css->streaming)
+ return;
+
+ ipu3_css_hw_stop(css);
+
+ ipu3_css_hw_cleanup(css);
+
+ ipu3_css_pipeline_cleanup(css);
+
+ spin_lock(&css->qlock);
+ for (q = 0; q < IPU3_CSS_QUEUES; q++)
+ list_for_each_entry_safe(b, b0, &css->queue[q].bufs, list) {
+ b->state = IPU3_CSS_BUFFER_FAILED;
+ list_del(&b->list);
+ }
+ spin_unlock(&css->qlock);
+
+ css->streaming = false;
+}
+
+bool ipu3_css_queue_empty(struct ipu3_css *css)
+{
+ int q;
+
+ spin_lock(&css->qlock);
+ for (q = 0; q < IPU3_CSS_QUEUES; q++)
+ if (!list_empty(&css->queue[q].bufs))
+ break;
+ spin_unlock(&css->qlock);
+
+ return (q == IPU3_CSS_QUEUES);
+}
+
+bool ipu3_css_is_streaming(struct ipu3_css *css)
+{
+ return css->streaming;
+}
+
+void ipu3_css_cleanup(struct ipu3_css *css)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ unsigned int p, q, i;
+
+ ipu3_css_stop_streaming(css);
+ ipu3_css_binary_cleanup(css);
+
+ for (q = 0; q < IPU3_CSS_QUEUES; q++)
+ for (i = 0; i < ARRAY_SIZE(css->abi_buffers[q]); i++)
+ ipu3_dmamap_free(imgu, &css->abi_buffers[q][i]);
+
+ for (p = 0; p < IPU3_CSS_PIPE_ID_NUM; p++)
+ for (i = 0; i < IMGU_ABI_MAX_STAGES; i++) {
+ ipu3_dmamap_free(imgu, &css->xmem_sp_stage_ptrs[p][i]);
+ ipu3_dmamap_free(imgu, &css->xmem_isp_stage_ptrs[p][i]);
+ }
+
+ ipu3_dmamap_free(imgu, &css->sp_ddr_ptrs);
+ ipu3_dmamap_free(imgu, &css->xmem_sp_group_ptrs);
+
+ ipu3_css_fw_cleanup(css);
+}
+
+int ipu3_css_init(struct device *dev, struct ipu3_css *css,
+ void __iomem *base, int length)
+{
+ struct imgu_device *imgu = dev_get_drvdata(dev);
+ int r, p, q, i;
+
+ /* Initialize main data structure */
+ css->dev = dev;
+ css->base = base;
+ css->iomem_length = length;
+ css->current_binary = IPU3_CSS_DEFAULT_BINARY;
+ css->pipe_id = IPU3_CSS_PIPE_ID_NUM;
+ css->vf_output_en = IPU3_NODE_VF_DISABLED;
+ spin_lock_init(&css->qlock);
+
+ for (q = 0; q < IPU3_CSS_QUEUES; q++) {
+ r = ipu3_css_queue_init(&css->queue[q], NULL, 0);
+ if (r)
+ return r;
+ }
+
+ r = ipu3_css_fw_init(css);
+ if (r)
+ return r;
+
+ /* Allocate and map common structures with imgu hardware */
+
+ for (p = 0; p < IPU3_CSS_PIPE_ID_NUM; p++)
+ for (i = 0; i < IMGU_ABI_MAX_STAGES; i++) {
+ if (!ipu3_dmamap_alloc(imgu,
+ &css->xmem_sp_stage_ptrs[p][i],
+ sizeof(struct imgu_abi_sp_stage)))
+ goto error_no_memory;
+ if (!ipu3_dmamap_alloc(imgu,
+ &css->xmem_isp_stage_ptrs[p][i],
+ sizeof(struct imgu_abi_isp_stage)))
+ goto error_no_memory;
+ }
+
+ if (!ipu3_dmamap_alloc(imgu, &css->sp_ddr_ptrs,
+ ALIGN(sizeof(struct imgu_abi_ddr_address_map),
+ IMGU_ABI_ISP_DDR_WORD_BYTES)))
+ goto error_no_memory;
+
+ if (!ipu3_dmamap_alloc(imgu, &css->xmem_sp_group_ptrs,
+ sizeof(struct imgu_abi_sp_group)))
+ goto error_no_memory;
+
+ for (q = 0; q < IPU3_CSS_QUEUES; q++)
+ for (i = 0; i < ARRAY_SIZE(css->abi_buffers[q]); i++)
+ if (!ipu3_dmamap_alloc(imgu, &css->abi_buffers[q][i],
+ sizeof(struct imgu_abi_buffer)))
+ goto error_no_memory;
+
+ if (ipu3_css_binary_preallocate(css))
+ goto error_binary_setup;
+
+ return 0;
+
+error_binary_setup:
+ ipu3_css_binary_cleanup(css);
+error_no_memory:
+ ipu3_css_cleanup(css);
+
+ return -ENOMEM;
+}
+
+static u32 ipu3_css_adjust(u32 res, u32 align)
+{
+ u32 val = max_t(u32, IPU3_CSS_MIN_RES, res);
+
+ return DIV_ROUND_CLOSEST(val, align) * align;
+}
+
+/* Select a binary matching the required resolutions and formats */
+static int ipu3_css_find_binary(struct ipu3_css *css,
+ struct ipu3_css_queue queue[IPU3_CSS_QUEUES],
+ struct v4l2_rect rects[IPU3_CSS_RECTS])
+{
+ const int binary_nr = css->fwp->file_header.binary_nr;
+ unsigned int binary_mode = (css->pipe_id == IPU3_CSS_PIPE_ID_CAPTURE) ?
+ IA_CSS_BINARY_MODE_PRIMARY : IA_CSS_BINARY_MODE_VIDEO;
+ const struct v4l2_pix_format_mplane *in =
+ &queue[IPU3_CSS_QUEUE_IN].fmt.mpix;
+ const struct v4l2_pix_format_mplane *out =
+ &queue[IPU3_CSS_QUEUE_OUT].fmt.mpix;
+ const struct v4l2_pix_format_mplane *vf =
+ &queue[IPU3_CSS_QUEUE_VF].fmt.mpix;
+ u32 stripe_w = 0, stripe_h = 0;
+ const char *name;
+ int i, j;
+
+ if (!ipu3_css_queue_enabled(&queue[IPU3_CSS_QUEUE_IN]))
+ return -EINVAL;
+
+ /* Find out the strip size boundary */
+ for (i = 0; i < binary_nr; i++) {
+ struct imgu_fw_info *bi = &css->fwp->binary_header[i];
+
+ u32 max_width = bi->info.isp.sp.output.max_width;
+ u32 max_height = bi->info.isp.sp.output.max_height;
+
+ if (bi->info.isp.sp.iterator.num_stripes <= 1) {
+ stripe_w = stripe_w ?
+ min(stripe_w, max_width) : max_width;
+ stripe_h = stripe_h ?
+ min(stripe_h, max_height) : max_height;
+ }
+ }
+
+ for (i = 0; i < binary_nr; i++) {
+ struct imgu_fw_info *bi = &css->fwp->binary_header[i];
+ enum imgu_abi_frame_format q_fmt;
+
+ name = (void *)css->fwp + bi->blob.prog_name_offset;
+
+ /* Check that binary supports memory-to-memory processing */
+ if (bi->info.isp.sp.input.source !=
+ IMGU_ABI_BINARY_INPUT_SOURCE_MEMORY)
+ continue;
+
+ /* Check that binary supports raw10 input */
+ if (!bi->info.isp.sp.enable.input_feeder &&
+ !bi->info.isp.sp.enable.input_raw)
+ continue;
+
+ /* Check binary mode */
+ if (bi->info.isp.sp.pipeline.mode != binary_mode)
+ continue;
+
+ /* Since input is RGGB bayer, need to process colors */
+ if (bi->info.isp.sp.enable.luma_only)
+ continue;
+
+ if (in->width < bi->info.isp.sp.input.min_width ||
+ in->width > bi->info.isp.sp.input.max_width ||
+ in->height < bi->info.isp.sp.input.min_height ||
+ in->height > bi->info.isp.sp.input.max_height)
+ continue;
+
+ if (ipu3_css_queue_enabled(&queue[IPU3_CSS_QUEUE_OUT])) {
+ if (bi->info.isp.num_output_pins <= 0)
+ continue;
+
+ q_fmt = queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ for (j = 0; j < bi->info.isp.num_output_formats; j++)
+ if (bi->info.isp.output_formats[j] == q_fmt)
+ break;
+ if (j >= bi->info.isp.num_output_formats)
+ continue;
+
+ if (out->width < bi->info.isp.sp.output.min_width ||
+ out->width > bi->info.isp.sp.output.max_width ||
+ out->height < bi->info.isp.sp.output.min_height ||
+ out->height > bi->info.isp.sp.output.max_height)
+ continue;
+
+ if (out->width > bi->info.isp.sp.internal.max_width ||
+ out->height > bi->info.isp.sp.internal.max_height)
+ continue;
+ }
+
+ if (ipu3_css_queue_enabled(&queue[IPU3_CSS_QUEUE_VF])) {
+ if (bi->info.isp.num_output_pins <= 1)
+ continue;
+
+ q_fmt = queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+ for (j = 0; j < bi->info.isp.num_output_formats; j++)
+ if (bi->info.isp.output_formats[j] == q_fmt)
+ break;
+ if (j >= bi->info.isp.num_output_formats)
+ continue;
+
+ if (vf->width < bi->info.isp.sp.output.min_width ||
+ vf->width > bi->info.isp.sp.output.max_width ||
+ vf->height < bi->info.isp.sp.output.min_height ||
+ vf->height > bi->info.isp.sp.output.max_height)
+ continue;
+ }
+
+ /* All checks passed, select the binary */
+ dev_dbg(css->dev, "using binary %s\n", name);
+ return i;
+ }
+
+ /* Can not find suitable binary for these parameters */
+ return -EINVAL;
+}
+
+/*
+ * Check that there is a binary matching requirements. Parameters may be
+ * NULL indicating disabled input/output. Return negative if given
+ * parameters can not be supported or on error, zero or positive indicating
+ * found binary number. May modify the given parameters if not exact match
+ * is found.
+ */
+int ipu3_css_fmt_try(struct ipu3_css *css,
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
+ struct v4l2_rect *rects[IPU3_CSS_RECTS])
+{
+ static const u32 EFF_ALIGN_W = 2;
+ static const u32 BDS_ALIGN_W = 4;
+ static const u32 OUT_ALIGN_W = 8;
+ static const u32 OUT_ALIGN_H = 4;
+ static const u32 VF_ALIGN_W = 2;
+ static const char *qnames[IPU3_CSS_QUEUES] = {
+ [IPU3_CSS_QUEUE_IN] = "in",
+ [IPU3_CSS_QUEUE_PARAMS] = "params",
+ [IPU3_CSS_QUEUE_OUT] = "out",
+ [IPU3_CSS_QUEUE_VF] = "vf",
+ [IPU3_CSS_QUEUE_STAT_3A] = "3a",
+ };
+ static const char *rnames[IPU3_CSS_RECTS] = {
+ [IPU3_CSS_RECT_EFFECTIVE] = "effective resolution",
+ [IPU3_CSS_RECT_BDS] = "bayer-domain scaled resolution",
+ [IPU3_CSS_RECT_ENVELOPE] = "DVS envelope size",
+ [IPU3_CSS_RECT_GDC] = "GDC output res",
+ };
+ struct v4l2_rect r[IPU3_CSS_RECTS] = { };
+ struct v4l2_rect *const eff = &r[IPU3_CSS_RECT_EFFECTIVE];
+ struct v4l2_rect *const bds = &r[IPU3_CSS_RECT_BDS];
+ struct v4l2_rect *const env = &r[IPU3_CSS_RECT_ENVELOPE];
+ struct v4l2_rect *const gdc = &r[IPU3_CSS_RECT_GDC];
+ struct ipu3_css_queue q[IPU3_CSS_QUEUES];
+ struct v4l2_pix_format_mplane *const in =
+ &q[IPU3_CSS_QUEUE_IN].fmt.mpix;
+ struct v4l2_pix_format_mplane *const out =
+ &q[IPU3_CSS_QUEUE_OUT].fmt.mpix;
+ struct v4l2_pix_format_mplane *const vf =
+ &q[IPU3_CSS_QUEUE_VF].fmt.mpix;
+ int binary, i, s;
+
+ /* Decide which pipe to use */
+ if (css->vf_output_en == IPU3_NODE_PV_ENABLED)
+ css->pipe_id = IPU3_CSS_PIPE_ID_CAPTURE;
+ else if (css->vf_output_en == IPU3_NODE_VF_ENABLED)
+ css->pipe_id = IPU3_CSS_PIPE_ID_VIDEO;
+
+ /* Adjust all formats, get statistics buffer sizes and formats */
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ if (fmts[i])
+ dev_dbg(css->dev, "%s %s: (%i,%i) fmt 0x%x\n", __func__,
+ qnames[i], fmts[i]->width, fmts[i]->height,
+ fmts[i]->pixelformat);
+ else
+ dev_dbg(css->dev, "%s %s: (not set)\n", __func__,
+ qnames[i]);
+ if (ipu3_css_queue_init(&q[i], fmts[i],
+ IPU3_CSS_QUEUE_TO_FLAGS(i))) {
+ dev_notice(css->dev, "can not initialize queue %s\n",
+ qnames[i]);
+ return -EINVAL;
+ }
+ }
+ for (i = 0; i < IPU3_CSS_RECTS; i++) {
+ if (rects[i]) {
+ dev_dbg(css->dev, "%s %s: (%i,%i)\n", __func__,
+ rnames[i], rects[i]->width, rects[i]->height);
+ r[i].width = rects[i]->width;
+ r[i].height = rects[i]->height;
+ } else {
+ dev_dbg(css->dev, "%s %s: (not set)\n", __func__,
+ rnames[i]);
+ }
+ /* For now, force known good resolutions */
+ r[i].left = 0;
+ r[i].top = 0;
+ }
+
+ /* Always require one input and vf only if out is also enabled */
+ if (!ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_IN]) ||
+ (ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_VF]) &&
+ !ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_OUT]))) {
+ dev_dbg(css->dev, "required queues are disabled\n");
+ return -EINVAL;
+ }
+
+ if (!ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_OUT])) {
+ out->width = in->width;
+ out->height = in->height;
+ }
+ if (eff->width <= 0 || eff->height <= 0) {
+ eff->width = in->width;
+ eff->height = in->height;
+ }
+ if (bds->width <= 0 || bds->height <= 0) {
+ bds->width = out->width;
+ bds->height = out->height;
+ }
+ if (gdc->width <= 0 || gdc->height <= 0) {
+ gdc->width = out->width;
+ gdc->height = out->height;
+ }
+
+ in->width = ipu3_css_adjust(in->width, 1);
+ in->height = ipu3_css_adjust(in->height, 1);
+ eff->width = ipu3_css_adjust(eff->width, EFF_ALIGN_W);
+ eff->height = ipu3_css_adjust(eff->height, 1);
+ bds->width = ipu3_css_adjust(bds->width, BDS_ALIGN_W);
+ bds->height = ipu3_css_adjust(bds->height, 1);
+ gdc->width = ipu3_css_adjust(gdc->width, OUT_ALIGN_W);
+ gdc->height = ipu3_css_adjust(gdc->height, OUT_ALIGN_H);
+ out->width = ipu3_css_adjust(out->width, OUT_ALIGN_W);
+ out->height = ipu3_css_adjust(out->height, OUT_ALIGN_H);
+ vf->width = ipu3_css_adjust(vf->width, VF_ALIGN_W);
+ vf->height = ipu3_css_adjust(vf->height, 1);
+
+ s = (bds->width - gdc->width) / 2 - FILTER_SIZE;
+ env->width = s < MIN_ENVELOPE ? MIN_ENVELOPE : s;
+ s = (bds->height - gdc->height) / 2 - FILTER_SIZE;
+ env->height = s < MIN_ENVELOPE ? MIN_ENVELOPE : s;
+
+ binary = ipu3_css_find_binary(css, q, r);
+ if (binary < 0) {
+ dev_err(css->dev, "failed to find suitable binary\n");
+ return -EINVAL;
+ }
+
+ /* Final adjustment and set back the queried formats */
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ if (fmts[i]) {
+ if (ipu3_css_queue_init(&q[i], &q[i].fmt.mpix,
+ IPU3_CSS_QUEUE_TO_FLAGS(i))) {
+ dev_err(css->dev,
+ "final resolution adjustment failed\n");
+ return -EINVAL;
+ }
+ *fmts[i] = q[i].fmt.mpix;
+ }
+ }
+
+ for (i = 0; i < IPU3_CSS_RECTS; i++)
+ if (rects[i])
+ *rects[i] = r[i];
+
+ dev_dbg(css->dev,
+ "in(%u,%u) if(%u,%u) ds(%u,%u) gdc(%u,%u) out(%u,%u) vf(%u,%u)",
+ in->width, in->height, eff->width, eff->height,
+ bds->width, bds->height, gdc->width, gdc->height,
+ out->width, out->height, vf->width, vf->height);
+
+ return binary;
+}
+
+int ipu3_css_fmt_set(struct ipu3_css *css,
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
+ struct v4l2_rect *rects[IPU3_CSS_RECTS])
+{
+ struct v4l2_rect rect_data[IPU3_CSS_RECTS];
+ struct v4l2_rect *all_rects[IPU3_CSS_RECTS];
+ int i, r;
+
+ for (i = 0; i < IPU3_CSS_RECTS; i++) {
+ if (rects[i])
+ rect_data[i] = *rects[i];
+ else
+ memset(&rect_data[i], 0, sizeof(rect_data[i]));
+ all_rects[i] = &rect_data[i];
+ }
+ r = ipu3_css_fmt_try(css, fmts, all_rects);
+ if (r < 0)
+ return r;
+ css->current_binary = (unsigned int)r;
+
+ for (i = 0; i < IPU3_CSS_QUEUES; i++)
+ if (ipu3_css_queue_init(&css->queue[i], fmts[i],
+ IPU3_CSS_QUEUE_TO_FLAGS(i)))
+ return -EINVAL;
+ for (i = 0; i < IPU3_CSS_RECTS; i++) {
+ css->rect[i] = rect_data[i];
+ if (rects[i])
+ *rects[i] = rect_data[i];
+ }
+
+ return 0;
+}
+
+int ipu3_css_meta_fmt_set(struct v4l2_meta_format *fmt)
+{
+ switch (fmt->dataformat) {
+ case V4L2_META_FMT_IPU3_PARAMS:
+ fmt->buffersize = sizeof(struct ipu3_uapi_params);
+ break;
+ case V4L2_META_FMT_IPU3_STAT_3A:
+ fmt->buffersize = sizeof(struct ipu3_uapi_stats_3a);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+/*
+ * Queue given buffer to CSS. ipu3_css_buf_prepare() must have been first
+ * called for the buffer. May be called from interrupt context.
+ * Returns 0 on success, -EBUSY if the buffer queue is full, or some other
+ * code on error conditions.
+ */
+int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b)
+{
+ static const int thread;
+ struct imgu_abi_buffer *abi_buf;
+ struct imgu_addr_t *buf_addr;
+ u32 data;
+ int r;
+
+ if (!css->streaming)
+ return -EPROTO; /* CSS or buffer in wrong state */
+
+ if (b->queue >= IPU3_CSS_QUEUES || !ipu3_css_queues[b->queue].qid)
+ return -EINVAL;
+
+ b->queue_pos = ipu3_css_queue_pos(css, ipu3_css_queues[b->queue].qid,
+ thread);
+
+ if (b->queue_pos >= ARRAY_SIZE(css->abi_buffers[b->queue]))
+ return -EIO;
+ abi_buf = css->abi_buffers[b->queue][b->queue_pos].vaddr;
+
+ /* Fill struct abi_buffer for firmware */
+ memset(abi_buf, 0, sizeof(*abi_buf));
+
+ buf_addr = (void *)abi_buf + ipu3_css_queues[b->queue].ptr_ofs;
+ *(imgu_addr_t *)buf_addr = b->daddr;
+
+ if (b->queue == IPU3_CSS_QUEUE_STAT_3A)
+ abi_buf->payload.s3a.data.dmem.s3a_tbl = b->daddr;
+
+ if (b->queue == IPU3_CSS_QUEUE_OUT)
+ abi_buf->payload.frame.padded_width =
+ css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+
+ if (b->queue == IPU3_CSS_QUEUE_VF)
+ abi_buf->payload.frame.padded_width =
+ css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+
+ spin_lock(&css->qlock);
+ list_add_tail(&b->list, &css->queue[b->queue].bufs);
+ spin_unlock(&css->qlock);
+ b->state = IPU3_CSS_BUFFER_QUEUED;
+
+ data = css->abi_buffers[b->queue][b->queue_pos].daddr;
+ r = ipu3_css_queue_data(css, ipu3_css_queues[b->queue].qid,
+ thread, data);
+ if (r < 0)
+ goto queueing_failed;
+
+ data = IMGU_ABI_EVENT_BUFFER_ENQUEUED(thread,
+ ipu3_css_queues[b->queue].qid);
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0, data);
+ if (r < 0)
+ goto queueing_failed;
+
+ dev_dbg(css->dev, "queued buffer %p to css queue %i\n", b, b->queue);
+
+ return 0;
+
+queueing_failed:
+ b->state = (r == -EBUSY || r == -EAGAIN) ?
+ IPU3_CSS_BUFFER_NEW : IPU3_CSS_BUFFER_FAILED;
+ list_del(&b->list);
+
+ return r;
+}
+
+/*
+ * Get next ready CSS buffer. Returns -EAGAIN in which case the function
+ * should be called again, or -EBUSY which means that there are no more
+ * buffers available. May be called from interrupt context.
+ */
+struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css)
+{
+ static const int thread;
+ static const unsigned char evtype_to_queue[] = {
+ [IMGU_ABI_EVTTYPE_INPUT_FRAME_DONE] = IPU3_CSS_QUEUE_IN,
+ [IMGU_ABI_EVTTYPE_OUT_FRAME_DONE] = IPU3_CSS_QUEUE_OUT,
+ [IMGU_ABI_EVTTYPE_VF_OUT_FRAME_DONE] = IPU3_CSS_QUEUE_VF,
+ [IMGU_ABI_EVTTYPE_3A_STATS_DONE] = IPU3_CSS_QUEUE_STAT_3A,
+ };
+ struct ipu3_css_buffer *b = ERR_PTR(-EAGAIN);
+ u32 event, daddr;
+ int evtype, pipe, pipeid, queue, qid, r;
+
+ if (!css->streaming)
+ return ERR_PTR(-EPROTO);
+
+ r = ipu3_css_dequeue_data(css, IMGU_ABI_QUEUE_EVENT_ID, &event);
+ if (r < 0)
+ return ERR_PTR(r);
+
+ evtype = (event & IMGU_ABI_EVTTYPE_EVENT_MASK) >>
+ IMGU_ABI_EVTTYPE_EVENT_SHIFT;
+
+ switch (evtype) {
+ case IMGU_ABI_EVTTYPE_OUT_FRAME_DONE:
+ case IMGU_ABI_EVTTYPE_VF_OUT_FRAME_DONE:
+ case IMGU_ABI_EVTTYPE_3A_STATS_DONE:
+ case IMGU_ABI_EVTTYPE_INPUT_FRAME_DONE:
+ pipe = (event & IMGU_ABI_EVTTYPE_PIPE_MASK) >>
+ IMGU_ABI_EVTTYPE_PIPE_SHIFT;
+ pipeid = (event & IMGU_ABI_EVTTYPE_PIPEID_MASK) >>
+ IMGU_ABI_EVTTYPE_PIPEID_SHIFT;
+ queue = evtype_to_queue[evtype];
+ qid = ipu3_css_queues[queue].qid;
+
+ if (qid >= IMGU_ABI_QUEUE_NUM) {
+ dev_err(css->dev, "Invalid qid: %i\n", qid);
+ return ERR_PTR(-EIO);
+ }
+
+ dev_dbg(css->dev,
+ "event: buffer done 0x%x queue %i pipe %i pipeid %i\n",
+ event, queue, pipe, pipeid);
+
+ r = ipu3_css_dequeue_data(css, qid, &daddr);
+ if (r < 0) {
+ dev_err(css->dev, "failed to dequeue buffer\n");
+ /* Force real error, not -EBUSY */
+ return ERR_PTR(-EIO);
+ }
+
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, thread,
+ IMGU_ABI_EVENT_BUFFER_DEQUEUED(qid));
+ if (r < 0) {
+ dev_err(css->dev, "failed to queue event\n");
+ return ERR_PTR(-EIO);
+ }
+
+ spin_lock(&css->qlock);
+ if (list_empty(&css->queue[queue].bufs)) {
+ spin_unlock(&css->qlock);
+ dev_err(css->dev, "event on empty queue\n");
+ return ERR_PTR(-EIO);
+ }
+ b = list_first_entry(&css->queue[queue].bufs,
+ struct ipu3_css_buffer, list);
+ if (queue != b->queue ||
+ daddr != css->abi_buffers[b->queue][b->queue_pos].daddr) {
+ spin_unlock(&css->qlock);
+ dev_err(css->dev, "dequeued bad buffer 0x%x\n", daddr);
+ return ERR_PTR(-EIO);
+ }
+ b->state = IPU3_CSS_BUFFER_DONE;
+ list_del(&b->list);
+ spin_unlock(&css->qlock);
+ break;
+ case IMGU_ABI_EVTTYPE_PIPELINE_DONE:
+ dev_dbg(css->dev, "event: pipeline done 0x%x for frame %ld\n",
+ event, css->frame);
+
+ if (css->frame == LONG_MAX)
+ css->frame = 0;
+ else
+ css->frame++;
+ break;
+ case IMGU_ABI_EVTTYPE_TIMER:
+ r = ipu3_css_dequeue_data(css, IMGU_ABI_QUEUE_EVENT_ID, &event);
+ if (r < 0)
+ return ERR_PTR(r);
+
+ if ((event & IMGU_ABI_EVTTYPE_EVENT_MASK) >>
+ IMGU_ABI_EVTTYPE_EVENT_SHIFT == IMGU_ABI_EVTTYPE_TIMER)
+ dev_dbg(css->dev, "event: timer\n");
+ else
+ dev_warn(css->dev, "half of timer event missing\n");
+ break;
+ case IMGU_ABI_EVTTYPE_FW_WARNING:
+ dev_warn(css->dev, "event: firmware warning 0x%x\n", event);
+ break;
+ case IMGU_ABI_EVTTYPE_FW_ASSERT:
+ dev_err(css->dev,
+ "event: firmware assert 0x%x module_id %i line_no %i\n",
+ event,
+ (event & IMGU_ABI_EVTTYPE_MODULEID_MASK) >>
+ IMGU_ABI_EVTTYPE_MODULEID_SHIFT,
+ swab16((event & IMGU_ABI_EVTTYPE_LINENO_MASK) >>
+ IMGU_ABI_EVTTYPE_LINENO_SHIFT));
+ break;
+ default:
+ dev_warn(css->dev, "received unknown event 0x%x\n", event);
+ }
+
+ return b;
+}
+
+/*
+ * Get a new set of parameters from pool and initialize them based on
+ * the parameters params, gdc, and obgrid. Any of these may be NULL,
+ * in which case the previously set parameters are used.
+ * If parameters haven't been set previously, initialize from scratch.
+ *
+ * Return index to css->parameter_set_info which has the newly created
+ * parameters or negative value on error.
+ */
+int ipu3_css_set_parameters(struct ipu3_css *css,
+ struct ipu3_uapi_params *set_params)
+{
+ struct ipu3_uapi_flags *use = set_params ? &set_params->use : NULL;
+ static const unsigned int queue_id = IMGU_ABI_QUEUE_A_ID;
+ const int stage = 0, thread = 0;
+ const struct imgu_fw_info *bi;
+ unsigned int stripes, i;
+ int obgrid_size;
+
+ /* Destination buffers which are filled here */
+ struct imgu_abi_parameter_set_info *param_set;
+ struct imgu_abi_acc_param *acc = NULL;
+ struct imgu_abi_gdc_warp_param *gdc = NULL;
+ struct ipu3_uapi_obgrid_param *obgrid = NULL;
+ const struct ipu3_css_map *map;
+ void *vmem0 = NULL;
+ void *dmem0 = NULL;
+
+ enum imgu_abi_memories m;
+ int r = -EBUSY;
+
+ if (!css->streaming)
+ return -EPROTO;
+
+ bi = &css->fwp->binary_header[css->current_binary];
+ obgrid_size = ipu3_css_fw_obgrid_size(bi);
+ stripes = bi->info.isp.sp.iterator.num_stripes ? : 1;
+
+ /*
+ * Check that we can get a new parameter_set_info from the pool.
+ * If this succeeds, then all of the other pool_get() calls below
+ * should also succeed.
+ */
+ if (ipu3_css_pool_get(&css->pool.parameter_set_info, css->frame) < 0)
+ goto fail_no_put;
+ param_set = ipu3_css_pool_last(&css->pool.parameter_set_info, 0)->vaddr;
+
+ map = ipu3_css_pool_last(&css->pool.acc, 0);
+ /* Get a new acc only if new parameters given, or none yet */
+ if (set_params || !map->vaddr) {
+ if (ipu3_css_pool_get(&css->pool.acc, css->frame) < 0)
+ goto fail;
+ map = ipu3_css_pool_last(&css->pool.acc, 0);
+ acc = map->vaddr;
+ }
+
+ /* Get new VMEM0 only if needed, or none yet */
+ m = IMGU_ABI_MEM_ISP_VMEM0;
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ if (!map->vaddr || (set_params && (set_params->use.lin_vmem_params ||
+ set_params->use.tnr3_vmem_params ||
+ set_params->use.xnr3_vmem_params))) {
+ if (ipu3_css_pool_get(&css->pool.binary_params_p[m],
+ css->frame) < 0)
+ goto fail;
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ vmem0 = map->vaddr;
+ }
+
+ /* Get new DMEM0 only if needed, or none yet */
+ m = IMGU_ABI_MEM_ISP_DMEM0;
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ if (!map->vaddr || (set_params && (set_params->use.tnr3_dmem_params ||
+ set_params->use.xnr3_dmem_params))) {
+ if (ipu3_css_pool_get(&css->pool.binary_params_p[m],
+ css->frame) < 0)
+ goto fail;
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ dmem0 = map->vaddr;
+ }
+
+ /* Configure acc parameter cluster */
+ if (acc) {
+ map = ipu3_css_pool_last(&css->pool.acc, 1);
+ r = ipu3_css_cfg_acc(css, use, acc, map->vaddr, set_params ?
+ &set_params->acc_param : NULL);
+ if (r < 0)
+ goto fail;
+ }
+
+ /* Configure late binding parameters */
+ if (vmem0) {
+ m = IMGU_ABI_MEM_ISP_VMEM0;
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 1);
+ r = ipu3_css_cfg_vmem0(css, use, vmem0, map->vaddr, set_params);
+ if (r < 0)
+ goto fail;
+ }
+
+ if (dmem0) {
+ m = IMGU_ABI_MEM_ISP_DMEM0;
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 1);
+ r = ipu3_css_cfg_dmem0(css, use, dmem0, map->vaddr, set_params);
+ if (r < 0)
+ goto fail;
+ }
+
+ /* Get a new gdc only if a new gdc is given, or none yet */
+ if (bi->info.isp.sp.enable.dvs_6axis) {
+ unsigned int a = IPU3_CSS_AUX_FRAME_REF;
+ unsigned int g = IPU3_CSS_RECT_GDC;
+ unsigned int e = IPU3_CSS_RECT_ENVELOPE;
+
+ map = ipu3_css_pool_last(&css->pool.gdc, 0);
+ if (!map->vaddr) {
+ if (ipu3_css_pool_get(&css->pool.gdc, css->frame) < 0)
+ goto fail;
+ map = ipu3_css_pool_last(&css->pool.gdc, 0);
+ gdc = map->vaddr;
+ ipu3_css_cfg_gdc_table(gdc,
+ css->aux_frames[a].bytesperline /
+ css->aux_frames[a].bytesperpixel,
+ css->aux_frames[a].height,
+ css->rect[g].width,
+ css->rect[g].height,
+ css->rect[e].width + FILTER_SIZE,
+ css->rect[e].height +
+ FILTER_SIZE);
+ }
+ }
+
+ /* Get a new obgrid only if a new obgrid is given, or none yet */
+ map = ipu3_css_pool_last(&css->pool.obgrid, 0);
+ if (!map->vaddr || (set_params && set_params->use.obgrid_param)) {
+ if (ipu3_css_pool_get(&css->pool.obgrid, css->frame) < 0)
+ goto fail;
+ map = ipu3_css_pool_last(&css->pool.obgrid, 0);
+ obgrid = map->vaddr;
+
+ /* Configure optical black level grid (obgrid) */
+ if (set_params && set_params->use.obgrid_param)
+ for (i = 0; i < obgrid_size / sizeof(*obgrid); i++)
+ obgrid[i] = set_params->obgrid_param;
+ else
+ memset(obgrid, 0, obgrid_size);
+ }
+
+ /* Configure parameter set info, queued to `queue_id' */
+
+ memset(param_set, 0, sizeof(*param_set));
+ map = ipu3_css_pool_last(&css->pool.acc, 0);
+ param_set->mem_map.acc_cluster_params_for_sp = map->daddr;
+
+ map = ipu3_css_pool_last(&css->pool.gdc, 0);
+ param_set->mem_map.dvs_6axis_params_y = map->daddr;
+
+ map = ipu3_css_pool_last(&css->pool.obgrid, 0);
+ for (i = 0; i < stripes; i++)
+ param_set->mem_map.obgrid_tbl[i] =
+ map->daddr + (obgrid_size / stripes) * i;
+
+ for (m = 0; m < IMGU_ABI_NUM_MEMORIES; m++) {
+ map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ param_set->mem_map.isp_mem_param[stage][m] = map->daddr;
+ }
+ /* Then queue the new parameter buffer */
+ map = ipu3_css_pool_last(&css->pool.parameter_set_info, 0);
+ r = ipu3_css_queue_data(css, queue_id, thread, map->daddr);
+ if (r < 0)
+ goto fail;
+
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0,
+ IMGU_ABI_EVENT_BUFFER_ENQUEUED(thread,
+ queue_id));
+ if (r < 0)
+ goto fail_no_put;
+
+ /* Finally dequeue all old parameter buffers */
+
+ do {
+ u32 daddr;
+
+ r = ipu3_css_dequeue_data(css, queue_id, &daddr);
+ if (r == -EBUSY)
+ break;
+ if (r)
+ goto fail_no_put;
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, thread,
+ IMGU_ABI_EVENT_BUFFER_DEQUEUED
+ (queue_id));
+ if (r < 0) {
+ dev_err(css->dev, "failed to queue parameter event\n");
+ goto fail_no_put;
+ }
+ } while (1);
+
+ return 0;
+
+fail:
+ /*
+ * A failure, most likely the parameter queue was full.
+ * Return error but continue streaming. User can try submitting new
+ * parameters again later.
+ */
+
+ ipu3_css_pool_put(&css->pool.parameter_set_info);
+ if (acc)
+ ipu3_css_pool_put(&css->pool.acc);
+ if (gdc)
+ ipu3_css_pool_put(&css->pool.gdc);
+ if (obgrid)
+ ipu3_css_pool_put(&css->pool.obgrid);
+ if (vmem0)
+ ipu3_css_pool_put(
+ &css->pool.binary_params_p[IMGU_ABI_MEM_ISP_VMEM0]);
+ if (dmem0)
+ ipu3_css_pool_put(
+ &css->pool.binary_params_p[IMGU_ABI_MEM_ISP_DMEM0]);
+
+fail_no_put:
+ return r;
+}
+
int ipu3_css_irq_ack(struct ipu3_css *css)
{
static const int NUM_SWIRQS = 3;
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (11 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 13/16] intel-ipu3: Add css pipeline programming Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-09 12:36 ` Sakari Ailus
2018-11-15 12:51 ` Hans Verkuil
2018-10-29 22:23 ` [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver Yong Zhi
` (4 subsequent siblings)
17 siblings, 2 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
Implement video driver that utilizes v4l2, vb2 queue support
and media controller APIs. The driver exposes single
subdevice and six nodes.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1091 ++++++++++++++++++++++++++++++
1 file changed, 1091 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
diff --git a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
new file mode 100644
index 0000000..31a3514
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
@@ -0,0 +1,1091 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Intel Corporation
+
+#include <linux/module.h>
+#include <linux/pm_runtime.h>
+
+#include <media/v4l2-ioctl.h>
+
+#include "ipu3.h"
+#include "ipu3-dmamap.h"
+
+/******************** v4l2_subdev_ops ********************/
+
+static int ipu3_subdev_open(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh)
+{
+ struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct v4l2_rect try_crop = {
+ .top = 0,
+ .left = 0,
+ .height = imgu->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.height,
+ .width = imgu->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.width,
+ };
+ unsigned int i;
+
+ /* Initialize try_fmt */
+ for (i = 0; i < IMGU_NODE_NUM; i++)
+ *v4l2_subdev_get_try_format(sd, fh->pad, i) =
+ imgu->nodes[i].pad_fmt;
+
+ *v4l2_subdev_get_try_crop(sd, fh->pad, IMGU_NODE_IN) = try_crop;
+
+ return 0;
+}
+
+static int ipu3_subdev_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ int r = 0;
+
+ r = imgu_s_stream(imgu, enable);
+ if (!r)
+ imgu->streaming = enable;
+
+ return r;
+}
+
+static int ipu3_subdev_get_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *fmt)
+{
+ struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct v4l2_mbus_framefmt *mf;
+ u32 pad = fmt->pad;
+
+ if (fmt->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
+ fmt->format = imgu->nodes[pad].pad_fmt;
+ } else {
+ mf = v4l2_subdev_get_try_format(sd, cfg, pad);
+ fmt->format = *mf;
+ }
+
+ return 0;
+}
+
+static int ipu3_subdev_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *fmt)
+{
+ struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct v4l2_mbus_framefmt *mf;
+ u32 pad = fmt->pad;
+
+ if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
+ mf = v4l2_subdev_get_try_format(sd, cfg, pad);
+ else
+ mf = &imgu->nodes[pad].pad_fmt;
+
+ fmt->format.code = mf->code;
+ /* Clamp the w and h based on the hardware capabilities */
+ if (imgu->subdev_pads[pad].flags & MEDIA_PAD_FL_SOURCE) {
+ fmt->format.width = clamp(fmt->format.width,
+ IPU3_OUTPUT_MIN_WIDTH,
+ IPU3_OUTPUT_MAX_WIDTH);
+ fmt->format.height = clamp(fmt->format.height,
+ IPU3_OUTPUT_MIN_HEIGHT,
+ IPU3_OUTPUT_MAX_HEIGHT);
+ } else {
+ fmt->format.width = clamp(fmt->format.width,
+ IPU3_INPUT_MIN_WIDTH,
+ IPU3_INPUT_MAX_WIDTH);
+ fmt->format.height = clamp(fmt->format.height,
+ IPU3_INPUT_MIN_HEIGHT,
+ IPU3_INPUT_MAX_HEIGHT);
+ }
+
+ *mf = fmt->format;
+
+ return 0;
+}
+
+static int ipu3_subdev_get_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_selection *sel)
+{
+ struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct v4l2_rect *try_sel, *r;
+
+ if (sel->pad != IMGU_NODE_IN)
+ return -EINVAL;
+
+ switch (sel->target) {
+ case V4L2_SEL_TGT_CROP:
+ try_sel = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
+ r = &imgu->rect.eff;
+ break;
+ case V4L2_SEL_TGT_COMPOSE:
+ try_sel = v4l2_subdev_get_try_compose(sd, cfg, sel->pad);
+ r = &imgu->rect.bds;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+ sel->r = *try_sel;
+ else
+ sel->r = *r;
+
+ return 0;
+}
+
+static int ipu3_subdev_set_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_selection *sel)
+{
+ struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct v4l2_rect *rect, *try_sel;
+
+ if (sel->pad != IMGU_NODE_IN)
+ return -EINVAL;
+
+ switch (sel->target) {
+ case V4L2_SEL_TGT_CROP:
+ try_sel = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
+ rect = &imgu->rect.eff;
+ break;
+ case V4L2_SEL_TGT_COMPOSE:
+ try_sel = v4l2_subdev_get_try_compose(sd, cfg, sel->pad);
+ rect = &imgu->rect.bds;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+ *try_sel = sel->r;
+ else
+ *rect = sel->r;
+
+ return 0;
+}
+
+/******************** media_entity_operations ********************/
+
+static int ipu3_link_setup(struct media_entity *entity,
+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct imgu_device *imgu = container_of(entity, struct imgu_device,
+ subdev.entity);
+ u32 pad = local->index;
+
+ WARN_ON(pad >= IMGU_NODE_NUM);
+
+ imgu->nodes[pad].enabled = flags & MEDIA_LNK_FL_ENABLED;
+
+ return 0;
+}
+
+/******************** vb2_ops ********************/
+
+static int ipu3_vb2_buf_init(struct vb2_buffer *vb)
+{
+ struct sg_table *sg = vb2_dma_sg_plane_desc(vb, 0);
+ struct imgu_device *imgu = vb2_get_drv_priv(vb->vb2_queue);
+ struct imgu_buffer *buf = container_of(vb,
+ struct imgu_buffer, vid_buf.vbb.vb2_buf);
+ struct imgu_video_device *node =
+ container_of(vb->vb2_queue, struct imgu_video_device, vbq);
+ unsigned int queue = imgu_node_to_queue(node - imgu->nodes);
+
+ if (queue == IPU3_CSS_QUEUE_PARAMS)
+ return 0;
+
+ return ipu3_dmamap_map_sg(imgu, sg->sgl, sg->nents, &buf->map);
+}
+
+/* Called when each buffer is freed */
+static void ipu3_vb2_buf_cleanup(struct vb2_buffer *vb)
+{
+ struct imgu_device *imgu = vb2_get_drv_priv(vb->vb2_queue);
+ struct imgu_buffer *buf = container_of(vb,
+ struct imgu_buffer, vid_buf.vbb.vb2_buf);
+ struct imgu_video_device *node =
+ container_of(vb->vb2_queue, struct imgu_video_device, vbq);
+ unsigned int queue = imgu_node_to_queue(node - imgu->nodes);
+
+ if (queue == IPU3_CSS_QUEUE_PARAMS)
+ return;
+
+ ipu3_dmamap_unmap(imgu, &buf->map);
+}
+
+/* Transfer buffer ownership to me */
+static void ipu3_vb2_buf_queue(struct vb2_buffer *vb)
+{
+ struct imgu_device *imgu = vb2_get_drv_priv(vb->vb2_queue);
+ struct imgu_video_device *node =
+ container_of(vb->vb2_queue, struct imgu_video_device, vbq);
+ unsigned int queue = imgu_node_to_queue(node - imgu->nodes);
+ unsigned long need_bytes;
+
+ if (vb->vb2_queue->type == V4L2_BUF_TYPE_META_CAPTURE ||
+ vb->vb2_queue->type == V4L2_BUF_TYPE_META_OUTPUT)
+ need_bytes = node->vdev_fmt.fmt.meta.buffersize;
+ else
+ need_bytes = node->vdev_fmt.fmt.pix_mp.plane_fmt[0].sizeimage;
+
+ if (queue == IPU3_CSS_QUEUE_PARAMS) {
+ unsigned long payload = vb2_get_plane_payload(vb, 0);
+ struct vb2_v4l2_buffer *buf =
+ container_of(vb, struct vb2_v4l2_buffer, vb2_buf);
+ int r = -EINVAL;
+
+ if (payload == 0) {
+ payload = need_bytes;
+ vb2_set_plane_payload(vb, 0, payload);
+ }
+ if (payload >= need_bytes)
+ r = ipu3_css_set_parameters(&imgu->css,
+ vb2_plane_vaddr(vb, 0));
+ buf->flags = V4L2_BUF_FLAG_DONE;
+ vb2_buffer_done(vb, r == 0 ? VB2_BUF_STATE_DONE
+ : VB2_BUF_STATE_ERROR);
+
+ } else {
+ struct imgu_buffer *buf = container_of(vb, struct imgu_buffer,
+ vid_buf.vbb.vb2_buf);
+
+ mutex_lock(&imgu->lock);
+ ipu3_css_buf_init(&buf->css_buf, queue, buf->map.daddr);
+ list_add_tail(&buf->vid_buf.list,
+ &imgu->nodes[node - imgu->nodes].buffers);
+ mutex_unlock(&imgu->lock);
+
+ vb2_set_plane_payload(&buf->vid_buf.vbb.vb2_buf, 0, need_bytes);
+
+ if (imgu->streaming)
+ imgu_queue_buffers(imgu, false);
+ }
+}
+
+static int ipu3_vb2_queue_setup(struct vb2_queue *vq,
+ unsigned int *num_buffers,
+ unsigned int *num_planes,
+ unsigned int sizes[],
+ struct device *alloc_devs[])
+{
+ struct imgu_device *imgu = vb2_get_drv_priv(vq);
+ struct imgu_video_device *node =
+ container_of(vq, struct imgu_video_device, vbq);
+ const struct v4l2_format *fmt = &node->vdev_fmt;
+ unsigned int size;
+
+ *num_buffers = clamp_val(*num_buffers, 1, VB2_MAX_FRAME);
+ alloc_devs[0] = &imgu->pci_dev->dev;
+
+ if (vq->type == V4L2_BUF_TYPE_META_CAPTURE ||
+ vq->type == V4L2_BUF_TYPE_META_OUTPUT)
+ size = fmt->fmt.meta.buffersize;
+ else
+ size = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
+
+ if (*num_planes) {
+ if (sizes[0] < size)
+ return -EINVAL;
+ size = sizes[0];
+ }
+
+ *num_planes = 1;
+ sizes[0] = size;
+ /* Initialize buffer queue */
+ INIT_LIST_HEAD(&node->buffers);
+
+ return 0;
+}
+
+/* Check if all enabled video nodes are streaming, exception ignored */
+static bool ipu3_all_nodes_streaming(struct imgu_device *imgu,
+ struct imgu_video_device *except)
+{
+ unsigned int i;
+
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ struct imgu_video_device *node = &imgu->nodes[i];
+
+ if (node == except)
+ continue;
+ if (node->enabled && !vb2_start_streaming_called(&node->vbq))
+ return false;
+ }
+
+ return true;
+}
+
+static void ipu3_return_all_buffers(struct imgu_device *imgu,
+ struct imgu_video_device *node,
+ enum vb2_buffer_state state)
+{
+ struct ipu3_vb2_buffer *b, *b0;
+
+ /* Return all buffers */
+ mutex_lock(&imgu->lock);
+ list_for_each_entry_safe(b, b0, &node->buffers, list) {
+ list_del(&b->list);
+ vb2_buffer_done(&b->vbb.vb2_buf, state);
+ }
+ mutex_unlock(&imgu->lock);
+}
+
+static int ipu3_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+ struct imgu_device *imgu = vb2_get_drv_priv(vq);
+ struct imgu_video_device *node =
+ container_of(vq, struct imgu_video_device, vbq);
+ int r;
+
+ if (imgu->streaming) {
+ r = -EBUSY;
+ goto fail_return_bufs;
+ }
+
+ if (!node->enabled) {
+ r = -EINVAL;
+ goto fail_return_bufs;
+ }
+ r = media_pipeline_start(&node->vdev.entity, &imgu->pipeline);
+ if (r < 0)
+ goto fail_return_bufs;
+
+ if (!ipu3_all_nodes_streaming(imgu, node))
+ return 0;
+
+ /* Start streaming of the whole pipeline now */
+
+ r = v4l2_subdev_call(&imgu->subdev, video, s_stream, 1);
+ if (r < 0)
+ goto fail_stop_pipeline;
+
+ return 0;
+
+fail_stop_pipeline:
+ media_pipeline_stop(&node->vdev.entity);
+fail_return_bufs:
+ ipu3_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
+
+ return r;
+}
+
+static void ipu3_vb2_stop_streaming(struct vb2_queue *vq)
+{
+ struct imgu_device *imgu = vb2_get_drv_priv(vq);
+ struct imgu_video_device *node =
+ container_of(vq, struct imgu_video_device, vbq);
+ int r;
+
+ WARN_ON(!node->enabled);
+
+ /* Was this the first node with streaming disabled? */
+ if (ipu3_all_nodes_streaming(imgu, node)) {
+ /* Yes, really stop streaming now */
+ r = v4l2_subdev_call(&imgu->subdev, video, s_stream, 0);
+ if (r)
+ dev_err(&imgu->pci_dev->dev,
+ "failed to stop streaming\n");
+ }
+
+ ipu3_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
+ media_pipeline_stop(&node->vdev.entity);
+}
+
+/******************** v4l2_ioctl_ops ********************/
+
+#define VID_CAPTURE 0
+#define VID_OUTPUT 1
+#define DEF_VID_CAPTURE 0
+#define DEF_VID_OUTPUT 1
+
+struct ipu3_fmt {
+ u32 fourcc;
+ u16 type; /* VID_CAPTURE or VID_OUTPUT not both */
+};
+
+/* format descriptions for capture and preview */
+static const struct ipu3_fmt formats[] = {
+ { V4L2_PIX_FMT_NV12, VID_CAPTURE },
+ { V4L2_PIX_FMT_IPU3_SGRBG10, VID_OUTPUT },
+ { V4L2_PIX_FMT_IPU3_SBGGR10, VID_OUTPUT },
+ { V4L2_PIX_FMT_IPU3_SGBRG10, VID_OUTPUT },
+ { V4L2_PIX_FMT_IPU3_SRGGB10, VID_OUTPUT },
+};
+
+/* Find the first matched format, return default if not found */
+static const struct ipu3_fmt *find_format(struct v4l2_format *f, u32 type)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(formats); i++) {
+ if (formats[i].fourcc == f->fmt.pix_mp.pixelformat &&
+ formats[i].type == type)
+ return &formats[i];
+ }
+
+ return type == VID_CAPTURE ? &formats[DEF_VID_CAPTURE] :
+ &formats[DEF_VID_OUTPUT];
+}
+
+static int ipu3_vidioc_querycap(struct file *file, void *fh,
+ struct v4l2_capability *cap)
+{
+ struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+
+ strlcpy(cap->driver, IMGU_NAME, sizeof(cap->driver));
+ strlcpy(cap->card, IMGU_NAME, sizeof(cap->card));
+ snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s", node->name);
+
+ return 0;
+}
+
+static int enum_fmts(struct v4l2_fmtdesc *f, u32 type)
+{
+ unsigned int i, j;
+
+ for (i = j = 0; i < ARRAY_SIZE(formats); ++i) {
+ if (formats[i].type == type) {
+ if (j == f->index)
+ break;
+ ++j;
+ }
+ }
+
+ if (i < ARRAY_SIZE(formats)) {
+ f->pixelformat = formats[i].fourcc;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static int vidioc_enum_fmt_vid_cap(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+ return -EINVAL;
+
+ return enum_fmts(f, VID_CAPTURE);
+}
+
+static int vidioc_enum_fmt_vid_out(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ if (f->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+ return -EINVAL;
+
+ return enum_fmts(f, VID_OUTPUT);
+}
+
+/* Propagate forward always the format from the CIO2 subdev */
+static int ipu3_vidioc_g_fmt(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+
+ f->fmt = node->vdev_fmt.fmt;
+
+ return 0;
+}
+
+/*
+ * Set input/output format. Unless it is just a try, this also resets
+ * selections (ie. effective and BDS resolutions) to defaults.
+ */
+static int imgu_fmt(struct imgu_device *imgu, int node,
+ struct v4l2_format *f, bool try)
+{
+ struct v4l2_pix_format_mplane try_fmts[IPU3_CSS_QUEUES];
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
+ struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
+ struct v4l2_mbus_framefmt pad_fmt;
+ unsigned int i, css_q;
+ int r;
+
+ if (imgu->nodes[IMGU_NODE_PV].enabled &&
+ imgu->nodes[IMGU_NODE_VF].enabled) {
+ dev_err(&imgu->pci_dev->dev,
+ "Postview and vf are not supported simultaneously\n");
+ return -EINVAL;
+ }
+ /*
+ * Tell css that the vf q is used for PV
+ */
+ if (imgu->nodes[IMGU_NODE_PV].enabled)
+ imgu->css.vf_output_en = IPU3_NODE_PV_ENABLED;
+ else if (imgu->nodes[IMGU_NODE_VF].enabled)
+ imgu->css.vf_output_en = IPU3_NODE_VF_ENABLED;
+
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ unsigned int inode = imgu_map_node(imgu, i);
+
+ /* Skip the meta node */
+ if (inode == IMGU_NODE_STAT_3A || inode == IMGU_NODE_PARAMS)
+ continue;
+ /* imgu_map_node defauls to PV if VF not enabled */
+ if (inode == IMGU_NODE_PV && node == IMGU_NODE_VF &&
+ imgu->css.vf_output_en == IPU3_NODE_VF_DISABLED)
+ inode = node;
+
+ if (try) {
+ try_fmts[i] = imgu->nodes[inode].vdev_fmt.fmt.pix_mp;
+ fmts[i] = &try_fmts[i];
+ } else {
+ fmts[i] = &imgu->nodes[inode].vdev_fmt.fmt.pix_mp;
+ }
+
+ /* CSS expects some format on OUT queue */
+ if (i != IPU3_CSS_QUEUE_OUT &&
+ !imgu->nodes[inode].enabled && inode != node)
+ fmts[i] = NULL;
+ }
+
+ if (!try) {
+ /* eff and bds res got by imgu_s_sel */
+ rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu->rect.eff;
+ rects[IPU3_CSS_RECT_BDS] = &imgu->rect.bds;
+ rects[IPU3_CSS_RECT_GDC] = &imgu->rect.gdc;
+
+ /* suppose that pad fmt was set by subdev s_fmt before */
+ pad_fmt = imgu->nodes[IMGU_NODE_IN].pad_fmt;
+ rects[IPU3_CSS_RECT_GDC]->width = pad_fmt.width;
+ rects[IPU3_CSS_RECT_GDC]->height = pad_fmt.height;
+ }
+
+ /*
+ * imgu doesn't set the node to the value given by user
+ * before we return success from this function, so set it here.
+ */
+ css_q = imgu_node_to_queue(node);
+ if (fmts[css_q])
+ *fmts[css_q] = f->fmt.pix_mp;
+ else
+ return -EINVAL;
+
+ if (try)
+ r = ipu3_css_fmt_try(&imgu->css, fmts, rects);
+ else
+ r = ipu3_css_fmt_set(&imgu->css, fmts, rects);
+
+ /* r is the binary number in the firmware blob */
+ if (r < 0)
+ return r;
+
+ if (try)
+ f->fmt.pix_mp = *fmts[css_q];
+ else
+ f->fmt = imgu->nodes[node].vdev_fmt.fmt;
+
+ return 0;
+}
+
+static int ipu3_try_fmt(struct file *file, void *fh, struct v4l2_format *f)
+{
+ struct v4l2_pix_format_mplane *pixm = &f->fmt.pix_mp;
+ const struct ipu3_fmt *fmt;
+
+ if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+ fmt = find_format(f, VID_CAPTURE);
+ else if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+ fmt = find_format(f, VID_OUTPUT);
+ else
+ return -EINVAL;
+
+ pixm->pixelformat = fmt->fourcc;
+
+ memset(pixm->plane_fmt[0].reserved, 0,
+ sizeof(pixm->plane_fmt[0].reserved));
+
+ return 0;
+}
+
+static int ipu3_vidioc_try_fmt(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct imgu_device *imgu = video_drvdata(file);
+ struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+ int r;
+
+ r = ipu3_try_fmt(file, fh, f);
+ if (r)
+ return r;
+
+ return imgu_fmt(imgu, node - imgu->nodes, f, true);
+}
+
+static int ipu3_vidioc_s_fmt(struct file *file, void *fh, struct v4l2_format *f)
+{
+ struct imgu_device *imgu = video_drvdata(file);
+ struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+ int r;
+
+ r = ipu3_try_fmt(file, fh, f);
+ if (r)
+ return r;
+
+ return imgu_fmt(imgu, node - imgu->nodes, f, false);
+}
+
+static int ipu3_meta_enum_format(struct file *file, void *fh,
+ struct v4l2_fmtdesc *f)
+{
+ struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+
+ /* Each node is dedicated to only one meta format */
+ if (f->index > 0 || f->type != node->vbq.type)
+ return -EINVAL;
+
+ f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
+
+ return 0;
+}
+
+static int ipu3_vidioc_g_meta_fmt(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+
+ if (f->type != node->vbq.type)
+ return -EINVAL;
+
+ f->fmt = node->vdev_fmt.fmt;
+
+ return 0;
+}
+
+static int ipu3_vidioc_enum_input(struct file *file, void *fh,
+ struct v4l2_input *input)
+{
+ if (input->index > 0)
+ return -EINVAL;
+ strlcpy(input->name, "camera", sizeof(input->name));
+ input->type = V4L2_INPUT_TYPE_CAMERA;
+
+ return 0;
+}
+
+static int ipu3_vidioc_g_input(struct file *file, void *fh, unsigned int *input)
+{
+ *input = 0;
+
+ return 0;
+}
+
+static int ipu3_vidioc_s_input(struct file *file, void *fh, unsigned int input)
+{
+ return input == 0 ? 0 : -EINVAL;
+}
+
+static int ipu3_vidioc_enum_output(struct file *file, void *fh,
+ struct v4l2_output *output)
+{
+ if (output->index > 0)
+ return -EINVAL;
+ strlcpy(output->name, "camera", sizeof(output->name));
+ output->type = V4L2_INPUT_TYPE_CAMERA;
+
+ return 0;
+}
+
+static int ipu3_vidioc_g_output(struct file *file, void *fh,
+ unsigned int *output)
+{
+ *output = 0;
+
+ return 0;
+}
+
+static int ipu3_vidioc_s_output(struct file *file, void *fh,
+ unsigned int output)
+{
+ return output == 0 ? 0 : -EINVAL;
+}
+
+/******************** function pointers ********************/
+
+static struct v4l2_subdev_internal_ops ipu3_subdev_internal_ops = {
+ .open = ipu3_subdev_open,
+};
+
+static const struct v4l2_subdev_video_ops ipu3_subdev_video_ops = {
+ .s_stream = ipu3_subdev_s_stream,
+};
+
+static const struct v4l2_subdev_pad_ops ipu3_subdev_pad_ops = {
+ .link_validate = v4l2_subdev_link_validate_default,
+ .get_fmt = ipu3_subdev_get_fmt,
+ .set_fmt = ipu3_subdev_set_fmt,
+ .get_selection = ipu3_subdev_get_selection,
+ .set_selection = ipu3_subdev_set_selection,
+};
+
+static const struct v4l2_subdev_ops ipu3_subdev_ops = {
+ .video = &ipu3_subdev_video_ops,
+ .pad = &ipu3_subdev_pad_ops,
+};
+
+static const struct media_entity_operations ipu3_media_ops = {
+ .link_setup = ipu3_link_setup,
+ .link_validate = v4l2_subdev_link_validate,
+};
+
+/****************** vb2_ops of the Q ********************/
+
+static const struct vb2_ops ipu3_vb2_ops = {
+ .buf_init = ipu3_vb2_buf_init,
+ .buf_cleanup = ipu3_vb2_buf_cleanup,
+ .buf_queue = ipu3_vb2_buf_queue,
+ .queue_setup = ipu3_vb2_queue_setup,
+ .start_streaming = ipu3_vb2_start_streaming,
+ .stop_streaming = ipu3_vb2_stop_streaming,
+ .wait_prepare = vb2_ops_wait_prepare,
+ .wait_finish = vb2_ops_wait_finish,
+};
+
+/****************** v4l2_file_operations *****************/
+
+static const struct v4l2_file_operations ipu3_v4l2_fops = {
+ .unlocked_ioctl = video_ioctl2,
+ .open = v4l2_fh_open,
+ .release = vb2_fop_release,
+ .poll = vb2_fop_poll,
+ .mmap = vb2_fop_mmap,
+};
+
+/******************** v4l2_ioctl_ops ********************/
+
+static const struct v4l2_ioctl_ops ipu3_v4l2_ioctl_ops = {
+ .vidioc_querycap = ipu3_vidioc_querycap,
+
+ .vidioc_enum_fmt_vid_cap_mplane = vidioc_enum_fmt_vid_cap,
+ .vidioc_g_fmt_vid_cap_mplane = ipu3_vidioc_g_fmt,
+ .vidioc_s_fmt_vid_cap_mplane = ipu3_vidioc_s_fmt,
+ .vidioc_try_fmt_vid_cap_mplane = ipu3_vidioc_try_fmt,
+
+ .vidioc_enum_fmt_vid_out_mplane = vidioc_enum_fmt_vid_out,
+ .vidioc_g_fmt_vid_out_mplane = ipu3_vidioc_g_fmt,
+ .vidioc_s_fmt_vid_out_mplane = ipu3_vidioc_s_fmt,
+ .vidioc_try_fmt_vid_out_mplane = ipu3_vidioc_try_fmt,
+
+ .vidioc_enum_output = ipu3_vidioc_enum_output,
+ .vidioc_g_output = ipu3_vidioc_g_output,
+ .vidioc_s_output = ipu3_vidioc_s_output,
+
+ .vidioc_enum_input = ipu3_vidioc_enum_input,
+ .vidioc_g_input = ipu3_vidioc_g_input,
+ .vidioc_s_input = ipu3_vidioc_s_input,
+
+ /* buffer queue management */
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+};
+
+static const struct v4l2_ioctl_ops ipu3_v4l2_meta_ioctl_ops = {
+ .vidioc_querycap = ipu3_vidioc_querycap,
+
+ /* meta capture */
+ .vidioc_enum_fmt_meta_cap = ipu3_meta_enum_format,
+ .vidioc_g_fmt_meta_cap = ipu3_vidioc_g_meta_fmt,
+ .vidioc_s_fmt_meta_cap = ipu3_vidioc_g_meta_fmt,
+ .vidioc_try_fmt_meta_cap = ipu3_vidioc_g_meta_fmt,
+
+ /* meta output */
+ .vidioc_enum_fmt_meta_out = ipu3_meta_enum_format,
+ .vidioc_g_fmt_meta_out = ipu3_vidioc_g_meta_fmt,
+ .vidioc_s_fmt_meta_out = ipu3_vidioc_g_meta_fmt,
+ .vidioc_try_fmt_meta_out = ipu3_vidioc_g_meta_fmt,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+};
+
+/******************** Framework registration ********************/
+
+/* helper function to config node's video properties */
+static void ipu3_node_to_v4l2(u32 node, struct video_device *vdev,
+ struct v4l2_format *f)
+{
+ u32 cap;
+
+ /* Should not happen */
+ WARN_ON(node >= IMGU_NODE_NUM);
+
+ switch (node) {
+ case IMGU_NODE_IN:
+ cap = V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+ f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+ vdev->ioctl_ops = &ipu3_v4l2_ioctl_ops;
+ break;
+ case IMGU_NODE_PARAMS:
+ cap = V4L2_CAP_META_OUTPUT;
+ f->type = V4L2_BUF_TYPE_META_OUTPUT;
+ f->fmt.meta.dataformat = V4L2_META_FMT_IPU3_PARAMS;
+ vdev->ioctl_ops = &ipu3_v4l2_meta_ioctl_ops;
+ ipu3_css_meta_fmt_set(&f->fmt.meta);
+ break;
+ case IMGU_NODE_STAT_3A:
+ cap = V4L2_CAP_META_CAPTURE;
+ f->type = V4L2_BUF_TYPE_META_CAPTURE;
+ f->fmt.meta.dataformat = V4L2_META_FMT_IPU3_STAT_3A;
+ vdev->ioctl_ops = &ipu3_v4l2_meta_ioctl_ops;
+ ipu3_css_meta_fmt_set(&f->fmt.meta);
+ break;
+ default:
+ cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE;
+ f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+ vdev->ioctl_ops = &ipu3_v4l2_ioctl_ops;
+ }
+
+ vdev->device_caps = V4L2_CAP_STREAMING | cap;
+}
+
+int ipu3_v4l2_register(struct imgu_device *imgu)
+{
+ struct v4l2_mbus_framefmt def_bus_fmt = { 0 };
+ struct v4l2_pix_format_mplane def_pix_fmt = { 0 };
+
+ int i, r;
+
+ /* Initialize miscellaneous variables */
+ imgu->streaming = false;
+
+ /* Init media device */
+ media_device_pci_init(&imgu->media_dev, imgu->pci_dev, IMGU_NAME);
+
+ /* Set up v4l2 device */
+ imgu->v4l2_dev.mdev = &imgu->media_dev;
+ imgu->v4l2_dev.ctrl_handler = imgu->ctrl_handler;
+ r = v4l2_device_register(&imgu->pci_dev->dev, &imgu->v4l2_dev);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register V4L2 device (%d)\n", r);
+ goto fail_v4l2_dev;
+ }
+
+ /* Initialize subdev media entity */
+ imgu->subdev_pads = kzalloc(sizeof(*imgu->subdev_pads) *
+ IMGU_NODE_NUM, GFP_KERNEL);
+ if (!imgu->subdev_pads) {
+ r = -ENOMEM;
+ goto fail_subdev_pads;
+ }
+ r = media_entity_pads_init(&imgu->subdev.entity, IMGU_NODE_NUM,
+ imgu->subdev_pads);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed initialize subdev media entity (%d)\n", r);
+ goto fail_media_entity;
+ }
+ imgu->subdev.entity.ops = &ipu3_media_ops;
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ imgu->subdev_pads[i].flags = imgu->nodes[i].output ?
+ MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+ }
+
+ /* Initialize subdev */
+ v4l2_subdev_init(&imgu->subdev, &ipu3_subdev_ops);
+ imgu->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
+ imgu->subdev.internal_ops = &ipu3_subdev_internal_ops;
+ imgu->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+ strlcpy(imgu->subdev.name, IMGU_NAME, sizeof(imgu->subdev.name));
+ v4l2_set_subdevdata(&imgu->subdev, imgu);
+ imgu->subdev.ctrl_handler = imgu->ctrl_handler;
+ r = v4l2_device_register_subdev(&imgu->v4l2_dev, &imgu->subdev);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed initialize subdev (%d)\n", r);
+ goto fail_subdev;
+ }
+ r = v4l2_device_register_subdev_nodes(&imgu->v4l2_dev);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register subdevs (%d)\n", r);
+ goto fail_subdevs;
+ }
+
+ /* Initialize formats to default values */
+ def_bus_fmt.width = 1920;
+ def_bus_fmt.height = 1080;
+ def_bus_fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
+ def_bus_fmt.field = V4L2_FIELD_NONE;
+ def_bus_fmt.colorspace = V4L2_COLORSPACE_RAW;
+ def_bus_fmt.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+ def_bus_fmt.quantization = V4L2_QUANTIZATION_DEFAULT;
+ def_bus_fmt.xfer_func = V4L2_XFER_FUNC_DEFAULT;
+
+ def_pix_fmt.width = def_bus_fmt.width;
+ def_pix_fmt.height = def_bus_fmt.height;
+ def_pix_fmt.field = def_bus_fmt.field;
+ def_pix_fmt.num_planes = 1;
+ def_pix_fmt.plane_fmt[0].bytesperline = def_pix_fmt.width * 2;
+ def_pix_fmt.plane_fmt[0].sizeimage =
+ def_pix_fmt.height * def_pix_fmt.plane_fmt[0].bytesperline;
+ def_pix_fmt.flags = 0;
+ def_pix_fmt.colorspace = def_bus_fmt.colorspace;
+ def_pix_fmt.ycbcr_enc = def_bus_fmt.ycbcr_enc;
+ def_pix_fmt.quantization = def_bus_fmt.quantization;
+ def_pix_fmt.xfer_func = def_bus_fmt.xfer_func;
+
+ /* Create video nodes and links */
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ struct imgu_video_device *node = &imgu->nodes[i];
+ struct video_device *vdev = &node->vdev;
+ struct vb2_queue *vbq = &node->vbq;
+ u32 flags;
+
+ /* Initialize miscellaneous variables */
+ mutex_init(&node->lock);
+ INIT_LIST_HEAD(&node->buffers);
+
+ /* Initialize formats to default values */
+ node->pad_fmt = def_bus_fmt;
+ ipu3_node_to_v4l2(i, vdev, &node->vdev_fmt);
+ if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
+ node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+ def_pix_fmt.pixelformat = node->output ?
+ V4L2_PIX_FMT_IPU3_SGRBG10 :
+ V4L2_PIX_FMT_NV12;
+ node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
+ }
+ /* Initialize media entities */
+ r = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed initialize media entity (%d)\n", r);
+ goto fail_vdev_media_entity;
+ }
+ node->vdev_pad.flags = node->output ?
+ MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
+ vdev->entity.ops = NULL;
+
+ /* Initialize vbq */
+ vbq->type = node->vdev_fmt.type;
+ vbq->io_modes = VB2_USERPTR | VB2_MMAP | VB2_DMABUF;
+ vbq->ops = &ipu3_vb2_ops;
+ vbq->mem_ops = &vb2_dma_sg_memops;
+ if (imgu->buf_struct_size <= 0)
+ imgu->buf_struct_size = sizeof(struct ipu3_vb2_buffer);
+ vbq->buf_struct_size = imgu->buf_struct_size;
+ vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+ vbq->min_buffers_needed = 0; /* Can streamon w/o buffers */
+ vbq->drv_priv = imgu;
+ vbq->lock = &node->lock;
+ r = vb2_queue_init(vbq);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to initialize video queue (%d)\n", r);
+ goto fail_vdev;
+ }
+
+ /* Initialize vdev */
+ snprintf(vdev->name, sizeof(vdev->name), "%s %s",
+ IMGU_NAME, node->name);
+ vdev->release = video_device_release_empty;
+ vdev->fops = &ipu3_v4l2_fops;
+ vdev->lock = &node->lock;
+ vdev->v4l2_dev = &imgu->v4l2_dev;
+ vdev->queue = &node->vbq;
+ vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
+ video_set_drvdata(vdev, imgu);
+ r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register video device (%d)\n", r);
+ goto fail_vdev;
+ }
+
+ /* Create link between video node and the subdev pad */
+ flags = 0;
+ if (node->enabled)
+ flags |= MEDIA_LNK_FL_ENABLED;
+ if (node->immutable)
+ flags |= MEDIA_LNK_FL_IMMUTABLE;
+ if (node->output) {
+ r = media_create_pad_link(&vdev->entity, 0,
+ &imgu->subdev.entity,
+ i, flags);
+ } else {
+ r = media_create_pad_link(&imgu->subdev.entity,
+ i, &vdev->entity, 0, flags);
+ }
+ if (r)
+ goto fail_link;
+ }
+
+ r = media_device_register(&imgu->media_dev);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register media device (%d)\n", r);
+ i--;
+ goto fail_link;
+ }
+
+ return 0;
+
+ for (; i >= 0; i--) {
+fail_link:
+ video_unregister_device(&imgu->nodes[i].vdev);
+fail_vdev:
+ media_entity_cleanup(&imgu->nodes[i].vdev.entity);
+fail_vdev_media_entity:
+ mutex_destroy(&imgu->nodes[i].lock);
+ }
+fail_subdevs:
+ v4l2_device_unregister_subdev(&imgu->subdev);
+fail_subdev:
+ media_entity_cleanup(&imgu->subdev.entity);
+fail_media_entity:
+ kfree(imgu->subdev_pads);
+fail_subdev_pads:
+ v4l2_device_unregister(&imgu->v4l2_dev);
+fail_v4l2_dev:
+ media_device_cleanup(&imgu->media_dev);
+
+ return r;
+}
+EXPORT_SYMBOL_GPL(ipu3_v4l2_register);
+
+int ipu3_v4l2_unregister(struct imgu_device *imgu)
+{
+ unsigned int i;
+
+ media_device_unregister(&imgu->media_dev);
+ media_device_cleanup(&imgu->media_dev);
+
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ video_unregister_device(&imgu->nodes[i].vdev);
+ media_entity_cleanup(&imgu->nodes[i].vdev.entity);
+ mutex_destroy(&imgu->nodes[i].lock);
+ }
+
+ v4l2_device_unregister_subdev(&imgu->subdev);
+ media_entity_cleanup(&imgu->subdev.entity);
+ kfree(imgu->subdev_pads);
+ v4l2_device_unregister(&imgu->v4l2_dev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ipu3_v4l2_unregister);
+
+void ipu3_v4l2_buffer_done(struct vb2_buffer *vb,
+ enum vb2_buffer_state state)
+{
+ struct ipu3_vb2_buffer *b =
+ container_of(vb, struct ipu3_vb2_buffer, vbb.vb2_buf);
+
+ list_del(&b->list);
+ vb2_buffer_done(&b->vbb.vb2_buf, state);
+}
+EXPORT_SYMBOL_GPL(ipu3_v4l2_buffer_done);
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework
2018-10-29 22:23 ` [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework Yong Zhi
@ 2018-11-09 12:36 ` Sakari Ailus
2018-11-09 23:26 ` Zhi, Yong
2018-11-15 12:51 ` Hans Verkuil
1 sibling, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-09 12:36 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:23:08PM -0700, Yong Zhi wrote:
> Implement video driver that utilizes v4l2, vb2 queue support
> and media controller APIs. The driver exposes single
> subdevice and six nodes.
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> ---
> drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1091 ++++++++++++++++++++++++++++++
> 1 file changed, 1091 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
>
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> new file mode 100644
> index 0000000..31a3514
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> @@ -0,0 +1,1091 @@
> +// SPDX-License-Identifier: GPL-2.0
> +// Copyright (C) 2018 Intel Corporation
> +
> +#include <linux/module.h>
> +#include <linux/pm_runtime.h>
> +
> +#include <media/v4l2-ioctl.h>
> +
> +#include "ipu3.h"
> +#include "ipu3-dmamap.h"
> +
> +/******************** v4l2_subdev_ops ********************/
> +
> +static int ipu3_subdev_open(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh)
> +{
> + struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
> + struct v4l2_rect try_crop = {
> + .top = 0,
> + .left = 0,
> + .height = imgu->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.height,
> + .width = imgu->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.width,
> + };
> + unsigned int i;
> +
> + /* Initialize try_fmt */
> + for (i = 0; i < IMGU_NODE_NUM; i++)
> + *v4l2_subdev_get_try_format(sd, fh->pad, i) =
> + imgu->nodes[i].pad_fmt;
The try formats should reflect the defaults, not the current device state.
> +
> + *v4l2_subdev_get_try_crop(sd, fh->pad, IMGU_NODE_IN) = try_crop;
Same for the crop. How about the compose rectangle?
> +
> + return 0;
> +}
...
> +int ipu3_v4l2_register(struct imgu_device *imgu)
> +{
> + struct v4l2_mbus_framefmt def_bus_fmt = { 0 };
> + struct v4l2_pix_format_mplane def_pix_fmt = { 0 };
> +
Extra newline.
> + int i, r;
> +
> + /* Initialize miscellaneous variables */
> + imgu->streaming = false;
> +
> + /* Init media device */
> + media_device_pci_init(&imgu->media_dev, imgu->pci_dev, IMGU_NAME);
> +
> + /* Set up v4l2 device */
> + imgu->v4l2_dev.mdev = &imgu->media_dev;
> + imgu->v4l2_dev.ctrl_handler = imgu->ctrl_handler;
> + r = v4l2_device_register(&imgu->pci_dev->dev, &imgu->v4l2_dev);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to register V4L2 device (%d)\n", r);
> + goto fail_v4l2_dev;
> + }
> +
> + /* Initialize subdev media entity */
> + imgu->subdev_pads = kzalloc(sizeof(*imgu->subdev_pads) *
> + IMGU_NODE_NUM, GFP_KERNEL);
As the number of pads is static, could you instead put the array directly
to the struct, instead of using a pointer? Remember to remove to
corresponding kfree, too.
> + if (!imgu->subdev_pads) {
> + r = -ENOMEM;
> + goto fail_subdev_pads;
> + }
> + r = media_entity_pads_init(&imgu->subdev.entity, IMGU_NODE_NUM,
> + imgu->subdev_pads);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed initialize subdev media entity (%d)\n", r);
> + goto fail_media_entity;
> + }
> + imgu->subdev.entity.ops = &ipu3_media_ops;
> + for (i = 0; i < IMGU_NODE_NUM; i++) {
> + imgu->subdev_pads[i].flags = imgu->nodes[i].output ?
> + MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
> + }
> +
> + /* Initialize subdev */
> + v4l2_subdev_init(&imgu->subdev, &ipu3_subdev_ops);
> + imgu->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
> + imgu->subdev.internal_ops = &ipu3_subdev_internal_ops;
> + imgu->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> + strlcpy(imgu->subdev.name, IMGU_NAME, sizeof(imgu->subdev.name));
strscpy(), please. Same elsewhere in this and also on the 13th.
> + v4l2_set_subdevdata(&imgu->subdev, imgu);
> + imgu->subdev.ctrl_handler = imgu->ctrl_handler;
> + r = v4l2_device_register_subdev(&imgu->v4l2_dev, &imgu->subdev);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed initialize subdev (%d)\n", r);
> + goto fail_subdev;
> + }
> + r = v4l2_device_register_subdev_nodes(&imgu->v4l2_dev);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to register subdevs (%d)\n", r);
> + goto fail_subdevs;
> + }
> +
> + /* Initialize formats to default values */
> + def_bus_fmt.width = 1920;
> + def_bus_fmt.height = 1080;
> + def_bus_fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> + def_bus_fmt.field = V4L2_FIELD_NONE;
> + def_bus_fmt.colorspace = V4L2_COLORSPACE_RAW;
> + def_bus_fmt.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> + def_bus_fmt.quantization = V4L2_QUANTIZATION_DEFAULT;
> + def_bus_fmt.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> +
> + def_pix_fmt.width = def_bus_fmt.width;
> + def_pix_fmt.height = def_bus_fmt.height;
> + def_pix_fmt.field = def_bus_fmt.field;
> + def_pix_fmt.num_planes = 1;
> + def_pix_fmt.plane_fmt[0].bytesperline = def_pix_fmt.width * 2;
> + def_pix_fmt.plane_fmt[0].sizeimage =
> + def_pix_fmt.height * def_pix_fmt.plane_fmt[0].bytesperline;
> + def_pix_fmt.flags = 0;
> + def_pix_fmt.colorspace = def_bus_fmt.colorspace;
> + def_pix_fmt.ycbcr_enc = def_bus_fmt.ycbcr_enc;
> + def_pix_fmt.quantization = def_bus_fmt.quantization;
> + def_pix_fmt.xfer_func = def_bus_fmt.xfer_func;
> +
> + /* Create video nodes and links */
> + for (i = 0; i < IMGU_NODE_NUM; i++) {
> + struct imgu_video_device *node = &imgu->nodes[i];
> + struct video_device *vdev = &node->vdev;
> + struct vb2_queue *vbq = &node->vbq;
> + u32 flags;
> +
> + /* Initialize miscellaneous variables */
> + mutex_init(&node->lock);
> + INIT_LIST_HEAD(&node->buffers);
> +
> + /* Initialize formats to default values */
> + node->pad_fmt = def_bus_fmt;
> + ipu3_node_to_v4l2(i, vdev, &node->vdev_fmt);
> + if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
> + node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
> + def_pix_fmt.pixelformat = node->output ?
> + V4L2_PIX_FMT_IPU3_SGRBG10 :
> + V4L2_PIX_FMT_NV12;
> + node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
> + }
> + /* Initialize media entities */
> + r = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed initialize media entity (%d)\n", r);
> + goto fail_vdev_media_entity;
> + }
> + node->vdev_pad.flags = node->output ?
> + MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
> + vdev->entity.ops = NULL;
> +
> + /* Initialize vbq */
> + vbq->type = node->vdev_fmt.type;
> + vbq->io_modes = VB2_USERPTR | VB2_MMAP | VB2_DMABUF;
> + vbq->ops = &ipu3_vb2_ops;
> + vbq->mem_ops = &vb2_dma_sg_memops;
> + if (imgu->buf_struct_size <= 0)
> + imgu->buf_struct_size = sizeof(struct ipu3_vb2_buffer);
> + vbq->buf_struct_size = imgu->buf_struct_size;
> + vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> + vbq->min_buffers_needed = 0; /* Can streamon w/o buffers */
> + vbq->drv_priv = imgu;
> + vbq->lock = &node->lock;
> + r = vb2_queue_init(vbq);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to initialize video queue (%d)\n", r);
> + goto fail_vdev;
> + }
> +
> + /* Initialize vdev */
> + snprintf(vdev->name, sizeof(vdev->name), "%s %s",
> + IMGU_NAME, node->name);
> + vdev->release = video_device_release_empty;
> + vdev->fops = &ipu3_v4l2_fops;
> + vdev->lock = &node->lock;
> + vdev->v4l2_dev = &imgu->v4l2_dev;
> + vdev->queue = &node->vbq;
> + vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
> + video_set_drvdata(vdev, imgu);
> + r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to register video device (%d)\n", r);
> + goto fail_vdev;
> + }
> +
> + /* Create link between video node and the subdev pad */
> + flags = 0;
> + if (node->enabled)
> + flags |= MEDIA_LNK_FL_ENABLED;
> + if (node->immutable)
> + flags |= MEDIA_LNK_FL_IMMUTABLE;
> + if (node->output) {
> + r = media_create_pad_link(&vdev->entity, 0,
> + &imgu->subdev.entity,
> + i, flags);
> + } else {
> + r = media_create_pad_link(&imgu->subdev.entity,
> + i, &vdev->entity, 0, flags);
> + }
> + if (r)
> + goto fail_link;
> + }
> +
> + r = media_device_register(&imgu->media_dev);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to register media device (%d)\n", r);
> + i--;
> + goto fail_link;
> + }
> +
> + return 0;
> +
> + for (; i >= 0; i--) {
> +fail_link:
> + video_unregister_device(&imgu->nodes[i].vdev);
> +fail_vdev:
> + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> +fail_vdev_media_entity:
> + mutex_destroy(&imgu->nodes[i].lock);
> + }
> +fail_subdevs:
> + v4l2_device_unregister_subdev(&imgu->subdev);
> +fail_subdev:
> + media_entity_cleanup(&imgu->subdev.entity);
> +fail_media_entity:
> + kfree(imgu->subdev_pads);
> +fail_subdev_pads:
> + v4l2_device_unregister(&imgu->v4l2_dev);
> +fail_v4l2_dev:
> + media_device_cleanup(&imgu->media_dev);
> +
> + return r;
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_register);
> +
> +int ipu3_v4l2_unregister(struct imgu_device *imgu)
> +{
> + unsigned int i;
> +
> + media_device_unregister(&imgu->media_dev);
> + media_device_cleanup(&imgu->media_dev);
media_device_cleanup() should take place after unregistering the
sub-devices.
> +
> + for (i = 0; i < IMGU_NODE_NUM; i++) {
> + video_unregister_device(&imgu->nodes[i].vdev);
> + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> + mutex_destroy(&imgu->nodes[i].lock);
> + }
> +
> + v4l2_device_unregister_subdev(&imgu->subdev);
> + media_entity_cleanup(&imgu->subdev.entity);
> + kfree(imgu->subdev_pads);
> + v4l2_device_unregister(&imgu->v4l2_dev);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_unregister);
> +
> +void ipu3_v4l2_buffer_done(struct vb2_buffer *vb,
> + enum vb2_buffer_state state)
> +{
> + struct ipu3_vb2_buffer *b =
> + container_of(vb, struct ipu3_vb2_buffer, vbb.vb2_buf);
> +
> + list_del(&b->list);
> + vb2_buffer_done(&b->vbb.vb2_buf, state);
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_buffer_done);
--
Kind regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework
2018-11-09 12:36 ` Sakari Ailus
@ 2018-11-09 23:26 ` Zhi, Yong
0 siblings, 0 replies; 123+ messages in thread
From: Zhi, Yong @ 2018-11-09 23:26 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi, Sakari,
Thanks for the feedback.
> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> Sent: Friday, November 9, 2018 6:37 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media
> framework
>
> Hi Yong,
>
> On Mon, Oct 29, 2018 at 03:23:08PM -0700, Yong Zhi wrote:
> > Implement video driver that utilizes v4l2, vb2 queue support and media
> > controller APIs. The driver exposes single subdevice and six nodes.
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > ---
> > drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1091
> > ++++++++++++++++++++++++++++++
> > 1 file changed, 1091 insertions(+)
> > create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> >
> > diff --git a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> > b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> > new file mode 100644
> > index 0000000..31a3514
> > --- /dev/null
> > +++ b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> > @@ -0,0 +1,1091 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +// Copyright (C) 2018 Intel Corporation
> > +
> > +#include <linux/module.h>
> > +#include <linux/pm_runtime.h>
> > +
> > +#include <media/v4l2-ioctl.h>
> > +
> > +#include "ipu3.h"
> > +#include "ipu3-dmamap.h"
> > +
> > +/******************** v4l2_subdev_ops ********************/
> > +
> > +static int ipu3_subdev_open(struct v4l2_subdev *sd, struct
> > +v4l2_subdev_fh *fh) {
> > + struct imgu_device *imgu = container_of(sd, struct imgu_device,
> subdev);
> > + struct v4l2_rect try_crop = {
> > + .top = 0,
> > + .left = 0,
> > + .height = imgu-
> >nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.height,
> > + .width = imgu-
> >nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.width,
> > + };
> > + unsigned int i;
> > +
> > + /* Initialize try_fmt */
> > + for (i = 0; i < IMGU_NODE_NUM; i++)
> > + *v4l2_subdev_get_try_format(sd, fh->pad, i) =
> > + imgu->nodes[i].pad_fmt;
>
> The try formats should reflect the defaults, not the current device state.
>
Ack, will fix in next update.
> > +
> > + *v4l2_subdev_get_try_crop(sd, fh->pad, IMGU_NODE_IN) = try_crop;
>
> Same for the crop. How about the compose rectangle?
>
Ok, will check both crop and compose rectangle.
> > +
> > + return 0;
> > +}
>
> ...
>
> > +int ipu3_v4l2_register(struct imgu_device *imgu) {
> > + struct v4l2_mbus_framefmt def_bus_fmt = { 0 };
> > + struct v4l2_pix_format_mplane def_pix_fmt = { 0 };
> > +
>
> Extra newline.
Ack.
>
> > + int i, r;
> > +
> > + /* Initialize miscellaneous variables */
> > + imgu->streaming = false;
> > +
> > + /* Init media device */
> > + media_device_pci_init(&imgu->media_dev, imgu->pci_dev,
> IMGU_NAME);
> > +
> > + /* Set up v4l2 device */
> > + imgu->v4l2_dev.mdev = &imgu->media_dev;
> > + imgu->v4l2_dev.ctrl_handler = imgu->ctrl_handler;
> > + r = v4l2_device_register(&imgu->pci_dev->dev, &imgu->v4l2_dev);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to register V4L2 device (%d)\n", r);
> > + goto fail_v4l2_dev;
> > + }
> > +
> > + /* Initialize subdev media entity */
> > + imgu->subdev_pads = kzalloc(sizeof(*imgu->subdev_pads) *
> > + IMGU_NODE_NUM, GFP_KERNEL);
>
> As the number of pads is static, could you instead put the array directly to
> the struct, instead of using a pointer? Remember to remove to
> corresponding kfree, too.
Good point, kzalloc does not serve its purpose here.
>
> > + if (!imgu->subdev_pads) {
> > + r = -ENOMEM;
> > + goto fail_subdev_pads;
> > + }
> > + r = media_entity_pads_init(&imgu->subdev.entity,
> IMGU_NODE_NUM,
> > + imgu->subdev_pads);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed initialize subdev media entity (%d)\n", r);
> > + goto fail_media_entity;
> > + }
> > + imgu->subdev.entity.ops = &ipu3_media_ops;
> > + for (i = 0; i < IMGU_NODE_NUM; i++) {
> > + imgu->subdev_pads[i].flags = imgu->nodes[i].output ?
> > + MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
> > + }
> > +
> > + /* Initialize subdev */
> > + v4l2_subdev_init(&imgu->subdev, &ipu3_subdev_ops);
> > + imgu->subdev.entity.function =
> MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
> > + imgu->subdev.internal_ops = &ipu3_subdev_internal_ops;
> > + imgu->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> > + strlcpy(imgu->subdev.name, IMGU_NAME, sizeof(imgu-
> >subdev.name));
>
> strscpy(), please. Same elsewhere in this and also on the 13th.
>
> > + v4l2_set_subdevdata(&imgu->subdev, imgu);
> > + imgu->subdev.ctrl_handler = imgu->ctrl_handler;
> > + r = v4l2_device_register_subdev(&imgu->v4l2_dev, &imgu->subdev);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed initialize subdev (%d)\n", r);
> > + goto fail_subdev;
> > + }
> > + r = v4l2_device_register_subdev_nodes(&imgu->v4l2_dev);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to register subdevs (%d)\n", r);
> > + goto fail_subdevs;
> > + }
> > +
> > + /* Initialize formats to default values */
> > + def_bus_fmt.width = 1920;
> > + def_bus_fmt.height = 1080;
> > + def_bus_fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
> > + def_bus_fmt.field = V4L2_FIELD_NONE;
> > + def_bus_fmt.colorspace = V4L2_COLORSPACE_RAW;
> > + def_bus_fmt.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > + def_bus_fmt.quantization = V4L2_QUANTIZATION_DEFAULT;
> > + def_bus_fmt.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> > +
> > + def_pix_fmt.width = def_bus_fmt.width;
> > + def_pix_fmt.height = def_bus_fmt.height;
> > + def_pix_fmt.field = def_bus_fmt.field;
> > + def_pix_fmt.num_planes = 1;
> > + def_pix_fmt.plane_fmt[0].bytesperline = def_pix_fmt.width * 2;
> > + def_pix_fmt.plane_fmt[0].sizeimage =
> > + def_pix_fmt.height * def_pix_fmt.plane_fmt[0].bytesperline;
> > + def_pix_fmt.flags = 0;
> > + def_pix_fmt.colorspace = def_bus_fmt.colorspace;
> > + def_pix_fmt.ycbcr_enc = def_bus_fmt.ycbcr_enc;
> > + def_pix_fmt.quantization = def_bus_fmt.quantization;
> > + def_pix_fmt.xfer_func = def_bus_fmt.xfer_func;
> > +
> > + /* Create video nodes and links */
> > + for (i = 0; i < IMGU_NODE_NUM; i++) {
> > + struct imgu_video_device *node = &imgu->nodes[i];
> > + struct video_device *vdev = &node->vdev;
> > + struct vb2_queue *vbq = &node->vbq;
> > + u32 flags;
> > +
> > + /* Initialize miscellaneous variables */
> > + mutex_init(&node->lock);
> > + INIT_LIST_HEAD(&node->buffers);
> > +
> > + /* Initialize formats to default values */
> > + node->pad_fmt = def_bus_fmt;
> > + ipu3_node_to_v4l2(i, vdev, &node->vdev_fmt);
> > + if (node->vdev_fmt.type ==
> V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
> > + node->vdev_fmt.type ==
> V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
> > + def_pix_fmt.pixelformat = node->output ?
> > +
> V4L2_PIX_FMT_IPU3_SGRBG10 :
> > + V4L2_PIX_FMT_NV12;
> > + node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
> > + }
> > + /* Initialize media entities */
> > + r = media_entity_pads_init(&vdev->entity, 1, &node-
> >vdev_pad);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed initialize media entity (%d)\n", r);
> > + goto fail_vdev_media_entity;
> > + }
> > + node->vdev_pad.flags = node->output ?
> > + MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
> > + vdev->entity.ops = NULL;
> > +
> > + /* Initialize vbq */
> > + vbq->type = node->vdev_fmt.type;
> > + vbq->io_modes = VB2_USERPTR | VB2_MMAP |
> VB2_DMABUF;
> > + vbq->ops = &ipu3_vb2_ops;
> > + vbq->mem_ops = &vb2_dma_sg_memops;
> > + if (imgu->buf_struct_size <= 0)
> > + imgu->buf_struct_size = sizeof(struct
> ipu3_vb2_buffer);
> > + vbq->buf_struct_size = imgu->buf_struct_size;
> > + vbq->timestamp_flags =
> V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> > + vbq->min_buffers_needed = 0; /* Can streamon w/o buffers
> */
> > + vbq->drv_priv = imgu;
> > + vbq->lock = &node->lock;
> > + r = vb2_queue_init(vbq);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to initialize video queue (%d)\n", r);
> > + goto fail_vdev;
> > + }
> > +
> > + /* Initialize vdev */
> > + snprintf(vdev->name, sizeof(vdev->name), "%s %s",
> > + IMGU_NAME, node->name);
> > + vdev->release = video_device_release_empty;
> > + vdev->fops = &ipu3_v4l2_fops;
> > + vdev->lock = &node->lock;
> > + vdev->v4l2_dev = &imgu->v4l2_dev;
> > + vdev->queue = &node->vbq;
> > + vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
> > + video_set_drvdata(vdev, imgu);
> > + r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to register video device (%d)\n", r);
> > + goto fail_vdev;
> > + }
> > +
> > + /* Create link between video node and the subdev pad */
> > + flags = 0;
> > + if (node->enabled)
> > + flags |= MEDIA_LNK_FL_ENABLED;
> > + if (node->immutable)
> > + flags |= MEDIA_LNK_FL_IMMUTABLE;
> > + if (node->output) {
> > + r = media_create_pad_link(&vdev->entity, 0,
> > + &imgu->subdev.entity,
> > + i, flags);
> > + } else {
> > + r = media_create_pad_link(&imgu->subdev.entity,
> > + i, &vdev->entity, 0, flags);
> > + }
> > + if (r)
> > + goto fail_link;
> > + }
> > +
> > + r = media_device_register(&imgu->media_dev);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to register media device (%d)\n", r);
> > + i--;
> > + goto fail_link;
> > + }
> > +
> > + return 0;
> > +
> > + for (; i >= 0; i--) {
> > +fail_link:
> > + video_unregister_device(&imgu->nodes[i].vdev);
> > +fail_vdev:
> > + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> > +fail_vdev_media_entity:
> > + mutex_destroy(&imgu->nodes[i].lock);
> > + }
> > +fail_subdevs:
> > + v4l2_device_unregister_subdev(&imgu->subdev);
> > +fail_subdev:
> > + media_entity_cleanup(&imgu->subdev.entity);
> > +fail_media_entity:
> > + kfree(imgu->subdev_pads);
> > +fail_subdev_pads:
> > + v4l2_device_unregister(&imgu->v4l2_dev);
> > +fail_v4l2_dev:
> > + media_device_cleanup(&imgu->media_dev);
> > +
> > + return r;
> > +}
> > +EXPORT_SYMBOL_GPL(ipu3_v4l2_register);
> > +
> > +int ipu3_v4l2_unregister(struct imgu_device *imgu) {
> > + unsigned int i;
> > +
> > + media_device_unregister(&imgu->media_dev);
> > + media_device_cleanup(&imgu->media_dev);
>
> media_device_cleanup() should take place after unregistering the sub-
> devices.
>
Thanks for catching this.
Yong
> > +
> > + for (i = 0; i < IMGU_NODE_NUM; i++) {
> > + video_unregister_device(&imgu->nodes[i].vdev);
> > + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> > + mutex_destroy(&imgu->nodes[i].lock);
> > + }
> > +
> > + v4l2_device_unregister_subdev(&imgu->subdev);
> > + media_entity_cleanup(&imgu->subdev.entity);
> > + kfree(imgu->subdev_pads);
> > + v4l2_device_unregister(&imgu->v4l2_dev);
> > +
> > + return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(ipu3_v4l2_unregister);
> > +
> > +void ipu3_v4l2_buffer_done(struct vb2_buffer *vb,
> > + enum vb2_buffer_state state)
> > +{
> > + struct ipu3_vb2_buffer *b =
> > + container_of(vb, struct ipu3_vb2_buffer, vbb.vb2_buf);
> > +
> > + list_del(&b->list);
> > + vb2_buffer_done(&b->vbb.vb2_buf, state); }
> > +EXPORT_SYMBOL_GPL(ipu3_v4l2_buffer_done);
>
> --
> Kind regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework
2018-10-29 22:23 ` [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework Yong Zhi
2018-11-09 12:36 ` Sakari Ailus
@ 2018-11-15 12:51 ` Hans Verkuil
2018-11-15 16:09 ` Zhi, Yong
1 sibling, 1 reply; 123+ messages in thread
From: Hans Verkuil @ 2018-11-15 12:51 UTC (permalink / raw)
To: Yong Zhi, linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao
On 10/29/18 23:23, Yong Zhi wrote:
> Implement video driver that utilizes v4l2, vb2 queue support
> and media controller APIs. The driver exposes single
> subdevice and six nodes.
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> ---
> drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1091 ++++++++++++++++++++++++++++++
> 1 file changed, 1091 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
>
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
<snip>
> +int ipu3_v4l2_register(struct imgu_device *imgu)
> +{
<snip>
> + /* Initialize vbq */
> + vbq->type = node->vdev_fmt.type;
> + vbq->io_modes = VB2_USERPTR | VB2_MMAP | VB2_DMABUF;
Are you sure USERPTR works? If you have alignment requirements that
the buffer starts at a multiple of more than (I think) 8 bytes, then
USERPTR won't work.
> + vbq->ops = &ipu3_vb2_ops;
> + vbq->mem_ops = &vb2_dma_sg_memops;
> + if (imgu->buf_struct_size <= 0)
> + imgu->buf_struct_size = sizeof(struct ipu3_vb2_buffer);
> + vbq->buf_struct_size = imgu->buf_struct_size;
> + vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> + vbq->min_buffers_needed = 0; /* Can streamon w/o buffers */
> + vbq->drv_priv = imgu;
> + vbq->lock = &node->lock;
> + r = vb2_queue_init(vbq);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to initialize video queue (%d)\n", r);
> + goto fail_vdev;
> + }
> +
> + /* Initialize vdev */
> + snprintf(vdev->name, sizeof(vdev->name), "%s %s",
> + IMGU_NAME, node->name);
> + vdev->release = video_device_release_empty;
> + vdev->fops = &ipu3_v4l2_fops;
> + vdev->lock = &node->lock;
> + vdev->v4l2_dev = &imgu->v4l2_dev;
> + vdev->queue = &node->vbq;
> + vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
> + video_set_drvdata(vdev, imgu);
> + r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to register video device (%d)\n", r);
> + goto fail_vdev;
> + }
> +
> + /* Create link between video node and the subdev pad */
> + flags = 0;
> + if (node->enabled)
> + flags |= MEDIA_LNK_FL_ENABLED;
> + if (node->immutable)
> + flags |= MEDIA_LNK_FL_IMMUTABLE;
> + if (node->output) {
> + r = media_create_pad_link(&vdev->entity, 0,
> + &imgu->subdev.entity,
> + i, flags);
> + } else {
> + r = media_create_pad_link(&imgu->subdev.entity,
> + i, &vdev->entity, 0, flags);
> + }
> + if (r)
> + goto fail_link;
> + }
> +
> + r = media_device_register(&imgu->media_dev);
> + if (r) {
> + dev_err(&imgu->pci_dev->dev,
> + "failed to register media device (%d)\n", r);
> + i--;
> + goto fail_link;
> + }
> +
> + return 0;
> +
> + for (; i >= 0; i--) {
> +fail_link:
> + video_unregister_device(&imgu->nodes[i].vdev);
> +fail_vdev:
> + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> +fail_vdev_media_entity:
> + mutex_destroy(&imgu->nodes[i].lock);
> + }
> +fail_subdevs:
> + v4l2_device_unregister_subdev(&imgu->subdev);
> +fail_subdev:
> + media_entity_cleanup(&imgu->subdev.entity);
> +fail_media_entity:
> + kfree(imgu->subdev_pads);
> +fail_subdev_pads:
> + v4l2_device_unregister(&imgu->v4l2_dev);
> +fail_v4l2_dev:
> + media_device_cleanup(&imgu->media_dev);
> +
> + return r;
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_register);
> +
> +int ipu3_v4l2_unregister(struct imgu_device *imgu)
> +{
> + unsigned int i;
> +
> + media_device_unregister(&imgu->media_dev);
> + media_device_cleanup(&imgu->media_dev);
> +
> + for (i = 0; i < IMGU_NODE_NUM; i++) {
> + video_unregister_device(&imgu->nodes[i].vdev);
> + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> + mutex_destroy(&imgu->nodes[i].lock);
> + }
> +
> + v4l2_device_unregister_subdev(&imgu->subdev);
> + media_entity_cleanup(&imgu->subdev.entity);
> + kfree(imgu->subdev_pads);
> + v4l2_device_unregister(&imgu->v4l2_dev);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_unregister);
> +
> +void ipu3_v4l2_buffer_done(struct vb2_buffer *vb,
> + enum vb2_buffer_state state)
> +{
> + struct ipu3_vb2_buffer *b =
> + container_of(vb, struct ipu3_vb2_buffer, vbb.vb2_buf);
> +
> + list_del(&b->list);
> + vb2_buffer_done(&b->vbb.vb2_buf, state);
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_buffer_done);
>
Regards,
Hans
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework
2018-11-15 12:51 ` Hans Verkuil
@ 2018-11-15 16:09 ` Zhi, Yong
0 siblings, 0 replies; 123+ messages in thread
From: Zhi, Yong @ 2018-11-15 16:09 UTC (permalink / raw)
To: Hans Verkuil, linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, Mani, Rajmohan,
Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu,
Cao, Bingbu
Hi, Hans,
Thanks for the review.
> -----Original Message-----
> From: Hans Verkuil [mailto:hverkuil@xs4all.nl]
> Sent: Thursday, November 15, 2018 6:51 AM
> To: Zhi, Yong <yong.zhi@intel.com>; linux-media@vger.kernel.org;
> sakari.ailus@linux.intel.com
> Cc: tfiga@chromium.org; mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media
> framework
>
> On 10/29/18 23:23, Yong Zhi wrote:
> > Implement video driver that utilizes v4l2, vb2 queue support and media
> > controller APIs. The driver exposes single subdevice and six nodes.
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > ---
> > drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1091
> > ++++++++++++++++++++++++++++++
> > 1 file changed, 1091 insertions(+)
> > create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> >
> > diff --git a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> > b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
>
> <snip>
>
> > +int ipu3_v4l2_register(struct imgu_device *imgu) {
>
> <snip>
>
> > + /* Initialize vbq */
> > + vbq->type = node->vdev_fmt.type;
> > + vbq->io_modes = VB2_USERPTR | VB2_MMAP |
> VB2_DMABUF;
>
> Are you sure USERPTR works? If you have alignment requirements that the
> buffer starts at a multiple of more than (I think) 8 bytes, then USERPTR won't
> work.
USRPTR was used at the beginning of project, we then switched to dma buffer mainly for performance reason:
https://chromium-review.googlesource.com/c/chromiumos/platform/arc-camera/+/620252
>
> > + vbq->ops = &ipu3_vb2_ops;
> > + vbq->mem_ops = &vb2_dma_sg_memops;
> > + if (imgu->buf_struct_size <= 0)
> > + imgu->buf_struct_size = sizeof(struct
> ipu3_vb2_buffer);
> > + vbq->buf_struct_size = imgu->buf_struct_size;
> > + vbq->timestamp_flags =
> V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> > + vbq->min_buffers_needed = 0; /* Can streamon w/o buffers
> */
> > + vbq->drv_priv = imgu;
> > + vbq->lock = &node->lock;
> > + r = vb2_queue_init(vbq);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to initialize video queue (%d)\n", r);
> > + goto fail_vdev;
> > + }
> > +
> > + /* Initialize vdev */
> > + snprintf(vdev->name, sizeof(vdev->name), "%s %s",
> > + IMGU_NAME, node->name);
> > + vdev->release = video_device_release_empty;
> > + vdev->fops = &ipu3_v4l2_fops;
> > + vdev->lock = &node->lock;
> > + vdev->v4l2_dev = &imgu->v4l2_dev;
> > + vdev->queue = &node->vbq;
> > + vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
> > + video_set_drvdata(vdev, imgu);
> > + r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to register video device (%d)\n", r);
> > + goto fail_vdev;
> > + }
> > +
> > + /* Create link between video node and the subdev pad */
> > + flags = 0;
> > + if (node->enabled)
> > + flags |= MEDIA_LNK_FL_ENABLED;
> > + if (node->immutable)
> > + flags |= MEDIA_LNK_FL_IMMUTABLE;
> > + if (node->output) {
> > + r = media_create_pad_link(&vdev->entity, 0,
> > + &imgu->subdev.entity,
> > + i, flags);
> > + } else {
> > + r = media_create_pad_link(&imgu->subdev.entity,
> > + i, &vdev->entity, 0, flags);
> > + }
> > + if (r)
> > + goto fail_link;
> > + }
> > +
> > + r = media_device_register(&imgu->media_dev);
> > + if (r) {
> > + dev_err(&imgu->pci_dev->dev,
> > + "failed to register media device (%d)\n", r);
> > + i--;
> > + goto fail_link;
> > + }
> > +
> > + return 0;
> > +
> > + for (; i >= 0; i--) {
> > +fail_link:
> > + video_unregister_device(&imgu->nodes[i].vdev);
> > +fail_vdev:
> > + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> > +fail_vdev_media_entity:
> > + mutex_destroy(&imgu->nodes[i].lock);
> > + }
> > +fail_subdevs:
> > + v4l2_device_unregister_subdev(&imgu->subdev);
> > +fail_subdev:
> > + media_entity_cleanup(&imgu->subdev.entity);
> > +fail_media_entity:
> > + kfree(imgu->subdev_pads);
> > +fail_subdev_pads:
> > + v4l2_device_unregister(&imgu->v4l2_dev);
> > +fail_v4l2_dev:
> > + media_device_cleanup(&imgu->media_dev);
> > +
> > + return r;
> > +}
> > +EXPORT_SYMBOL_GPL(ipu3_v4l2_register);
> > +
> > +int ipu3_v4l2_unregister(struct imgu_device *imgu) {
> > + unsigned int i;
> > +
> > + media_device_unregister(&imgu->media_dev);
> > + media_device_cleanup(&imgu->media_dev);
> > +
> > + for (i = 0; i < IMGU_NODE_NUM; i++) {
> > + video_unregister_device(&imgu->nodes[i].vdev);
> > + media_entity_cleanup(&imgu->nodes[i].vdev.entity);
> > + mutex_destroy(&imgu->nodes[i].lock);
> > + }
> > +
> > + v4l2_device_unregister_subdev(&imgu->subdev);
> > + media_entity_cleanup(&imgu->subdev.entity);
> > + kfree(imgu->subdev_pads);
> > + v4l2_device_unregister(&imgu->v4l2_dev);
> > +
> > + return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(ipu3_v4l2_unregister);
> > +
> > +void ipu3_v4l2_buffer_done(struct vb2_buffer *vb,
> > + enum vb2_buffer_state state)
> > +{
> > + struct ipu3_vb2_buffer *b =
> > + container_of(vb, struct ipu3_vb2_buffer, vbb.vb2_buf);
> > +
> > + list_del(&b->list);
> > + vb2_buffer_done(&b->vbb.vb2_buf, state); }
> > +EXPORT_SYMBOL_GPL(ipu3_v4l2_buffer_done);
> >
>
> Regards,
>
> Hans
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (12 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 14/16] intel-ipu3: Add v4l2 driver based on media framework Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-09 12:54 ` Sakari Ailus
2018-10-29 22:23 ` [PATCH v7 16/16] intel-ipu3: Add dual pipe support Yong Zhi
` (3 subsequent siblings)
17 siblings, 1 reply; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
This patch adds support for the Intel IPU v3 as found
on Skylake and Kaby Lake SoCs.
The driver glues v4l2, css(camera sub system) and other
pieces together to perform its functions, it also loads
the IPU3 firmware binary as part of its initialization.
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
---
drivers/media/pci/intel/ipu3/Kconfig | 16 +
drivers/media/pci/intel/ipu3/Makefile | 12 +
drivers/media/pci/intel/ipu3/ipu3.c | 844 ++++++++++++++++++++++++++++++++++
drivers/media/pci/intel/ipu3/ipu3.h | 153 ++++++
4 files changed, 1025 insertions(+)
create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
diff --git a/drivers/media/pci/intel/ipu3/Kconfig b/drivers/media/pci/intel/ipu3/Kconfig
index 715f776..44ebcbb 100644
--- a/drivers/media/pci/intel/ipu3/Kconfig
+++ b/drivers/media/pci/intel/ipu3/Kconfig
@@ -15,3 +15,19 @@ config VIDEO_IPU3_CIO2
Say Y or M here if you have a Skylake/Kaby Lake SoC with MIPI CSI-2
connected camera.
The module will be called ipu3-cio2.
+
+config VIDEO_IPU3_IMGU
+ tristate "Intel ipu3-imgu driver"
+ depends on PCI && VIDEO_V4L2
+ depends on MEDIA_CONTROLLER && VIDEO_V4L2_SUBDEV_API
+ depends on X86
+ select IOMMU_IOVA
+ select VIDEOBUF2_DMA_SG
+
+ ---help---
+ This is the video4linux2 driver for Intel IPU3 image processing unit,
+ found in Intel Skylake and Kaby Lake SoCs and used for processing
+ images and video.
+
+ Say Y or M here if you have a Skylake/Kaby Lake SoC with a MIPI
+ camera. The module will be called ipu3-imgu.
diff --git a/drivers/media/pci/intel/ipu3/Makefile b/drivers/media/pci/intel/ipu3/Makefile
index 20186e3..60bd5db 100644
--- a/drivers/media/pci/intel/ipu3/Makefile
+++ b/drivers/media/pci/intel/ipu3/Makefile
@@ -1 +1,13 @@
+#
+# Makefile for the IPU3 cio2 and ImgU drivers
+#
+
obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
+
+ipu3-imgu-objs += \
+ ipu3-mmu.o ipu3-dmamap.o \
+ ipu3-tables.o ipu3-css-pool.o \
+ ipu3-css-fw.o ipu3-css-params.o \
+ ipu3-css.o ipu3-v4l2.o ipu3.o
+
+obj-$(CONFIG_VIDEO_IPU3_IMGU) += ipu3-imgu.o
diff --git a/drivers/media/pci/intel/ipu3/ipu3.c b/drivers/media/pci/intel/ipu3/ipu3.c
new file mode 100644
index 0000000..eda7299
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3.c
@@ -0,0 +1,844 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2017 Intel Corporation
+ * Copyright 2017 Google LLC
+ *
+ * Based on Intel IPU4 driver.
+ *
+ */
+
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pm_runtime.h>
+
+#include "ipu3.h"
+#include "ipu3-dmamap.h"
+#include "ipu3-mmu.h"
+
+#define IMGU_PCI_ID 0x1919
+#define IMGU_PCI_BAR 0
+#define IMGU_DMA_MASK DMA_BIT_MASK(39)
+#define IMGU_MAX_QUEUE_DEPTH (2 + 2)
+
+/*
+ * pre-allocated buffer size for IMGU dummy buffers. Those
+ * values should be tuned to big enough to avoid buffer
+ * re-allocation when streaming to lower streaming latency.
+ */
+#define CSS_QUEUE_IN_BUF_SIZE 0
+#define CSS_QUEUE_PARAMS_BUF_SIZE 0
+#define CSS_QUEUE_OUT_BUF_SIZE (4160 * 3120 * 12 / 8)
+#define CSS_QUEUE_VF_BUF_SIZE (1920 * 1080 * 12 / 8)
+#define CSS_QUEUE_STAT_3A_BUF_SIZE 125664
+
+static const size_t css_queue_buf_size_map[IPU3_CSS_QUEUES] = {
+ [IPU3_CSS_QUEUE_IN] = CSS_QUEUE_IN_BUF_SIZE,
+ [IPU3_CSS_QUEUE_PARAMS] = CSS_QUEUE_PARAMS_BUF_SIZE,
+ [IPU3_CSS_QUEUE_OUT] = CSS_QUEUE_OUT_BUF_SIZE,
+ [IPU3_CSS_QUEUE_VF] = CSS_QUEUE_VF_BUF_SIZE,
+ [IPU3_CSS_QUEUE_STAT_3A] = CSS_QUEUE_STAT_3A_BUF_SIZE,
+};
+
+static const struct imgu_node_mapping imgu_node_map[IMGU_NODE_NUM] = {
+ [IMGU_NODE_IN] = {IPU3_CSS_QUEUE_IN, "input"},
+ [IMGU_NODE_PARAMS] = {IPU3_CSS_QUEUE_PARAMS, "parameters"},
+ [IMGU_NODE_OUT] = {IPU3_CSS_QUEUE_OUT, "output"},
+ [IMGU_NODE_VF] = {IPU3_CSS_QUEUE_VF, "viewfinder"},
+ [IMGU_NODE_PV] = {IPU3_CSS_QUEUE_VF, "postview"},
+ [IMGU_NODE_STAT_3A] = {IPU3_CSS_QUEUE_STAT_3A, "3a stat"},
+};
+
+unsigned int imgu_node_to_queue(unsigned int node)
+{
+ return imgu_node_map[node].css_queue;
+}
+
+unsigned int imgu_map_node(struct imgu_device *imgu, unsigned int css_queue)
+{
+ unsigned int i;
+
+ if (css_queue == IPU3_CSS_QUEUE_VF)
+ return imgu->nodes[IMGU_NODE_VF].enabled ?
+ IMGU_NODE_VF : IMGU_NODE_PV;
+
+ for (i = 0; i < IMGU_NODE_NUM; i++)
+ if (imgu_node_map[i].css_queue == css_queue)
+ break;
+
+ return i;
+}
+
+/**************** Dummy buffers ****************/
+
+static void imgu_dummybufs_cleanup(struct imgu_device *imgu)
+{
+ unsigned int i;
+
+ for (i = 0; i < IPU3_CSS_QUEUES; i++)
+ ipu3_dmamap_free(imgu, &imgu->queues[i].dmap);
+}
+
+static int imgu_dummybufs_preallocate(struct imgu_device *imgu)
+{
+ unsigned int i;
+ size_t size;
+
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ size = css_queue_buf_size_map[i];
+ /*
+ * Do not enable dummy buffers for master queue,
+ * always require that real buffers from user are
+ * available.
+ */
+ if (i == IMGU_QUEUE_MASTER || size == 0)
+ continue;
+
+ if (!ipu3_dmamap_alloc(imgu, &imgu->queues[i].dmap, size)) {
+ imgu_dummybufs_cleanup(imgu);
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+static int imgu_dummybufs_init(struct imgu_device *imgu)
+{
+ const struct v4l2_pix_format_mplane *mpix;
+ const struct v4l2_meta_format *meta;
+ unsigned int i, j, node;
+ size_t size;
+
+ /* Allocate a dummy buffer for each queue where buffer is optional */
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ node = imgu_map_node(imgu, i);
+ if (!imgu->queue_enabled[node] || i == IMGU_QUEUE_MASTER)
+ continue;
+
+ if (!imgu->nodes[IMGU_NODE_VF].enabled &&
+ !imgu->nodes[IMGU_NODE_PV].enabled &&
+ i == IPU3_CSS_QUEUE_VF)
+ /*
+ * Do not enable dummy buffers for VF/PV if it is not
+ * requested by the user.
+ */
+ continue;
+
+ meta = &imgu->nodes[node].vdev_fmt.fmt.meta;
+ mpix = &imgu->nodes[node].vdev_fmt.fmt.pix_mp;
+
+ if (node == IMGU_NODE_STAT_3A || node == IMGU_NODE_PARAMS)
+ size = meta->buffersize;
+ else
+ size = mpix->plane_fmt[0].sizeimage;
+
+ if (ipu3_css_dma_buffer_resize(imgu, &imgu->queues[i].dmap,
+ size)) {
+ imgu_dummybufs_cleanup(imgu);
+ return -ENOMEM;
+ }
+
+ for (j = 0; j < IMGU_MAX_QUEUE_DEPTH; j++)
+ ipu3_css_buf_init(&imgu->queues[i].dummybufs[j], i,
+ imgu->queues[i].dmap.daddr);
+ }
+
+ return 0;
+}
+
+/* May be called from atomic context */
+static struct ipu3_css_buffer *imgu_dummybufs_get(struct imgu_device *imgu,
+ int queue)
+{
+ unsigned int i;
+
+ /* dummybufs are not allocated for master q */
+ if (queue == IPU3_CSS_QUEUE_IN)
+ return NULL;
+
+ if (WARN_ON(!imgu->queues[queue].dmap.vaddr))
+ /* Buffer should not be allocated here */
+ return NULL;
+
+ for (i = 0; i < IMGU_MAX_QUEUE_DEPTH; i++)
+ if (ipu3_css_buf_state(&imgu->queues[queue].dummybufs[i]) !=
+ IPU3_CSS_BUFFER_QUEUED)
+ break;
+
+ if (i == IMGU_MAX_QUEUE_DEPTH)
+ return NULL;
+
+ ipu3_css_buf_init(&imgu->queues[queue].dummybufs[i], queue,
+ imgu->queues[queue].dmap.daddr);
+
+ return &imgu->queues[queue].dummybufs[i];
+}
+
+/* Check if given buffer is a dummy buffer */
+static bool imgu_dummybufs_check(struct imgu_device *imgu,
+ struct ipu3_css_buffer *buf)
+{
+ unsigned int i;
+
+ for (i = 0; i < IMGU_MAX_QUEUE_DEPTH; i++)
+ if (buf == &imgu->queues[buf->queue].dummybufs[i])
+ break;
+
+ return i < IMGU_MAX_QUEUE_DEPTH;
+}
+
+static void imgu_buffer_done(struct imgu_device *imgu, struct vb2_buffer *vb,
+ enum vb2_buffer_state state)
+{
+ mutex_lock(&imgu->lock);
+ ipu3_v4l2_buffer_done(vb, state);
+ mutex_unlock(&imgu->lock);
+}
+
+static struct ipu3_css_buffer *imgu_queue_getbuf(struct imgu_device *imgu,
+ unsigned int node)
+{
+ struct imgu_buffer *buf;
+
+ if (WARN_ON(node >= IMGU_NODE_NUM))
+ return NULL;
+
+ /* Find first free buffer from the node */
+ list_for_each_entry(buf, &imgu->nodes[node].buffers, vid_buf.list) {
+ if (ipu3_css_buf_state(&buf->css_buf) == IPU3_CSS_BUFFER_NEW)
+ return &buf->css_buf;
+ }
+
+ /* There were no free buffers, try to return a dummy buffer */
+ return imgu_dummybufs_get(imgu, imgu_node_map[node].css_queue);
+}
+
+/*
+ * Queue as many buffers to CSS as possible. If all buffers don't fit into
+ * CSS buffer queues, they remain unqueued and will be queued later.
+ */
+int imgu_queue_buffers(struct imgu_device *imgu, bool initial)
+{
+ unsigned int node;
+ int r = 0;
+ struct imgu_buffer *ibuf;
+
+ if (!ipu3_css_is_streaming(&imgu->css))
+ return 0;
+
+ mutex_lock(&imgu->lock);
+
+ /* Buffer set is queued to FW only when input buffer is ready */
+ for (node = IMGU_NODE_NUM - 1;
+ imgu_queue_getbuf(imgu, IMGU_NODE_IN);
+ node = node ? node - 1 : IMGU_NODE_NUM - 1) {
+
+ if (node == IMGU_NODE_VF &&
+ (imgu->css.pipe_id == IPU3_CSS_PIPE_ID_CAPTURE ||
+ !imgu->nodes[IMGU_NODE_VF].enabled)) {
+ continue;
+ } else if (node == IMGU_NODE_PV &&
+ (imgu->css.pipe_id == IPU3_CSS_PIPE_ID_VIDEO ||
+ !imgu->nodes[IMGU_NODE_PV].enabled)) {
+ continue;
+ } else if (imgu->queue_enabled[node]) {
+ struct ipu3_css_buffer *buf =
+ imgu_queue_getbuf(imgu, node);
+ int dummy;
+
+ if (!buf)
+ break;
+
+ r = ipu3_css_buf_queue(&imgu->css, buf);
+ if (r)
+ break;
+ dummy = imgu_dummybufs_check(imgu, buf);
+ if (!dummy)
+ ibuf = container_of(buf, struct imgu_buffer,
+ css_buf);
+ dev_dbg(&imgu->pci_dev->dev,
+ "queue %s %s buffer %d to css da: 0x%08x\n",
+ dummy ? "dummy" : "user",
+ imgu_node_map[node].name,
+ dummy ? 0 : ibuf->vid_buf.vbb.vb2_buf.index,
+ (u32)buf->daddr);
+ }
+ }
+ mutex_unlock(&imgu->lock);
+
+ if (r && r != -EBUSY)
+ goto failed;
+
+ return 0;
+
+failed:
+ /*
+ * On error, mark all buffers as failed which are not
+ * yet queued to CSS
+ */
+ dev_err(&imgu->pci_dev->dev,
+ "failed to queue buffer to CSS on queue %i (%d)\n",
+ node, r);
+
+ if (initial)
+ /* If we were called from streamon(), no need to finish bufs */
+ return r;
+
+ for (node = 0; node < IMGU_NODE_NUM; node++) {
+ struct imgu_buffer *buf, *buf0;
+
+ if (!imgu->queue_enabled[node])
+ continue; /* Skip disabled queues */
+
+ mutex_lock(&imgu->lock);
+ list_for_each_entry_safe(buf, buf0, &imgu->nodes[node].buffers,
+ vid_buf.list) {
+ if (ipu3_css_buf_state(&buf->css_buf) ==
+ IPU3_CSS_BUFFER_QUEUED)
+ continue; /* Was already queued, skip */
+
+ ipu3_v4l2_buffer_done(&buf->vid_buf.vbb.vb2_buf,
+ VB2_BUF_STATE_ERROR);
+ }
+ mutex_unlock(&imgu->lock);
+ }
+
+ return r;
+}
+
+static int imgu_powerup(struct imgu_device *imgu)
+{
+ int r;
+
+ r = ipu3_css_set_powerup(&imgu->pci_dev->dev, imgu->base);
+ if (r)
+ return r;
+
+ ipu3_mmu_resume(imgu->mmu);
+ return 0;
+}
+
+static void imgu_powerdown(struct imgu_device *imgu)
+{
+ ipu3_mmu_suspend(imgu->mmu);
+ ipu3_css_set_powerdown(&imgu->pci_dev->dev, imgu->base);
+}
+
+int imgu_s_stream(struct imgu_device *imgu, int enable)
+{
+ struct device *dev = &imgu->pci_dev->dev;
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
+ struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
+ unsigned int i, node;
+ int r;
+
+ if (!enable) {
+ /* Stop streaming */
+ dev_dbg(dev, "stream off\n");
+ /* Block new buffers to be queued to CSS. */
+ atomic_set(&imgu->qbuf_barrier, 1);
+ ipu3_css_stop_streaming(&imgu->css);
+ synchronize_irq(imgu->pci_dev->irq);
+ atomic_set(&imgu->qbuf_barrier, 0);
+ imgu_powerdown(imgu);
+ pm_runtime_put(&imgu->pci_dev->dev);
+
+ return 0;
+ }
+
+ /* Start streaming */
+
+ dev_dbg(dev, "stream on\n");
+ for (i = 0; i < IMGU_NODE_NUM; i++)
+ imgu->queue_enabled[i] = imgu->nodes[i].enabled;
+
+ /*
+ * CSS library expects that the following queues are
+ * always enabled; if buffers are not provided to some of the
+ * queues, it stalls due to lack of buffers.
+ * Force the queues to be enabled and if the user really hasn't
+ * enabled them, use dummy buffers.
+ */
+ imgu->queue_enabled[IMGU_NODE_OUT] = true;
+ imgu->queue_enabled[IMGU_NODE_VF] = true;
+ imgu->queue_enabled[IMGU_NODE_PV] = true;
+ imgu->queue_enabled[IMGU_NODE_STAT_3A] = true;
+
+ /* This is handled specially */
+ imgu->queue_enabled[IPU3_CSS_QUEUE_PARAMS] = false;
+
+ /* Initialize CSS formats */
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ node = imgu_map_node(imgu, i);
+ /* No need to reconfig meta nodes */
+ if (node == IMGU_NODE_STAT_3A || node == IMGU_NODE_PARAMS)
+ continue;
+ fmts[i] = imgu->queue_enabled[node] ?
+ &imgu->nodes[node].vdev_fmt.fmt.pix_mp : NULL;
+ }
+
+ /* Enable VF output only when VF or PV queue requested by user */
+ imgu->css.vf_output_en = IPU3_NODE_VF_DISABLED;
+ if (imgu->nodes[IMGU_NODE_VF].enabled)
+ imgu->css.vf_output_en = IPU3_NODE_VF_ENABLED;
+ else if (imgu->nodes[IMGU_NODE_PV].enabled)
+ imgu->css.vf_output_en = IPU3_NODE_PV_ENABLED;
+
+ rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu->rect.eff;
+ rects[IPU3_CSS_RECT_BDS] = &imgu->rect.bds;
+ rects[IPU3_CSS_RECT_GDC] = &imgu->rect.gdc;
+
+ r = ipu3_css_fmt_set(&imgu->css, fmts, rects);
+ if (r) {
+ dev_err(dev, "failed to set initial formats (%d)", r);
+ return r;
+ }
+
+ /* Set Power */
+ r = pm_runtime_get_sync(dev);
+ if (r < 0) {
+ dev_err(dev, "failed to set imgu power\n");
+ pm_runtime_put(dev);
+ return r;
+ }
+
+ r = imgu_powerup(imgu);
+ if (r) {
+ dev_err(dev, "failed to power up imgu\n");
+ pm_runtime_put(dev);
+ return r;
+ }
+
+ /* Start CSS streaming */
+ r = ipu3_css_start_streaming(&imgu->css);
+ if (r) {
+ dev_err(dev, "failed to start css streaming (%d)", r);
+ goto fail_start_streaming;
+ }
+
+ /* Initialize dummy buffers */
+ r = imgu_dummybufs_init(imgu);
+ if (r) {
+ dev_err(dev, "failed to initialize dummy buffers (%d)", r);
+ goto fail_dummybufs;
+ }
+
+ /* Queue as many buffers from queue as possible */
+ r = imgu_queue_buffers(imgu, true);
+ if (r) {
+ dev_err(dev, "failed to queue initial buffers (%d)", r);
+ goto fail_queueing;
+ }
+
+ return 0;
+
+fail_queueing:
+ imgu_dummybufs_cleanup(imgu);
+fail_dummybufs:
+ ipu3_css_stop_streaming(&imgu->css);
+fail_start_streaming:
+ pm_runtime_put(dev);
+
+ return r;
+}
+
+static int imgu_video_nodes_init(struct imgu_device *imgu)
+{
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
+ struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
+ unsigned int i;
+ int r;
+
+ imgu->buf_struct_size = sizeof(struct imgu_buffer);
+
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ imgu->nodes[i].name = imgu_node_map[i].name;
+ imgu->nodes[i].output = i < IMGU_QUEUE_FIRST_INPUT;
+ imgu->nodes[i].immutable = false;
+ imgu->nodes[i].enabled = false;
+
+ if (i != IMGU_NODE_PARAMS && i != IMGU_NODE_STAT_3A)
+ fmts[imgu_node_map[i].css_queue] =
+ &imgu->nodes[i].vdev_fmt.fmt.pix_mp;
+ atomic_set(&imgu->nodes[i].sequence, 0);
+ }
+
+ /* Master queue is always enabled */
+ imgu->nodes[IMGU_QUEUE_MASTER].immutable = true;
+ imgu->nodes[IMGU_QUEUE_MASTER].enabled = true;
+
+ r = ipu3_v4l2_register(imgu);
+ if (r)
+ return r;
+
+ /* Set initial formats and initialize formats of video nodes */
+ rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu->rect.eff;
+ rects[IPU3_CSS_RECT_BDS] = &imgu->rect.bds;
+ ipu3_css_fmt_set(&imgu->css, fmts, rects);
+
+ /* Pre-allocate dummy buffers */
+ r = imgu_dummybufs_preallocate(imgu);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to pre-allocate dummy buffers (%d)", r);
+ imgu_dummybufs_cleanup(imgu);
+ ipu3_v4l2_unregister(imgu);
+ }
+
+ return 0;
+}
+
+static void imgu_video_nodes_exit(struct imgu_device *imgu)
+{
+ imgu_dummybufs_cleanup(imgu);
+ ipu3_v4l2_unregister(imgu);
+}
+
+/**************** PCI interface ****************/
+
+static irqreturn_t imgu_isr_threaded(int irq, void *imgu_ptr)
+{
+ struct imgu_device *imgu = imgu_ptr;
+
+ /* Dequeue / queue buffers */
+ do {
+ u64 ns = ktime_get_ns();
+ struct ipu3_css_buffer *b;
+ struct imgu_buffer *buf;
+ unsigned int node;
+ bool dummy;
+
+ do {
+ mutex_lock(&imgu->lock);
+ b = ipu3_css_buf_dequeue(&imgu->css);
+ mutex_unlock(&imgu->lock);
+ } while (PTR_ERR(b) == -EAGAIN);
+
+ if (IS_ERR_OR_NULL(b)) {
+ if (!b || PTR_ERR(b) == -EBUSY) /* All done */
+ break;
+ dev_err(&imgu->pci_dev->dev,
+ "failed to dequeue buffers (%ld)\n",
+ PTR_ERR(b));
+ break;
+ }
+
+ node = imgu_map_node(imgu, b->queue);
+ dummy = imgu_dummybufs_check(imgu, b);
+ if (!dummy)
+ buf = container_of(b, struct imgu_buffer, css_buf);
+ dev_dbg(&imgu->pci_dev->dev,
+ "dequeue %s %s buffer %d from css\n",
+ dummy ? "dummy" : "user",
+ imgu_node_map[node].name,
+ dummy ? 0 : buf->vid_buf.vbb.vb2_buf.index);
+
+ if (dummy)
+ /* It was a dummy buffer, skip it */
+ continue;
+
+ /* Fill vb2 buffer entries and tell it's ready */
+ if (!imgu->nodes[node].output) {
+ buf->vid_buf.vbb.vb2_buf.timestamp = ns;
+ buf->vid_buf.vbb.field = V4L2_FIELD_NONE;
+ buf->vid_buf.vbb.sequence =
+ atomic_inc_return(&imgu->nodes[node].sequence);
+ }
+ imgu_buffer_done(imgu, &buf->vid_buf.vbb.vb2_buf,
+ ipu3_css_buf_state(&buf->css_buf) ==
+ IPU3_CSS_BUFFER_DONE ?
+ VB2_BUF_STATE_DONE :
+ VB2_BUF_STATE_ERROR);
+ mutex_lock(&imgu->lock);
+ if (ipu3_css_queue_empty(&imgu->css))
+ wake_up_all(&imgu->buf_drain_wq);
+ mutex_unlock(&imgu->lock);
+ } while (1);
+
+ /*
+ * Try to queue more buffers for CSS.
+ * qbuf_barrier is used to disable new buffers
+ * to be queued to CSS.
+ */
+ if (!atomic_read(&imgu->qbuf_barrier))
+ imgu_queue_buffers(imgu, false);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t imgu_isr(int irq, void *imgu_ptr)
+{
+ struct imgu_device *imgu = imgu_ptr;
+
+ /* acknowledge interruption */
+ if (ipu3_css_irq_ack(&imgu->css) < 0)
+ return IRQ_NONE;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static int imgu_pci_config_setup(struct pci_dev *dev)
+{
+ u16 pci_command;
+ int r = pci_enable_msi(dev);
+
+ if (r) {
+ dev_err(&dev->dev, "failed to enable MSI (%d)\n", r);
+ return r;
+ }
+
+ pci_read_config_word(dev, PCI_COMMAND, &pci_command);
+ pci_command |= PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER |
+ PCI_COMMAND_INTX_DISABLE;
+ pci_write_config_word(dev, PCI_COMMAND, pci_command);
+
+ return 0;
+}
+
+static int imgu_pci_probe(struct pci_dev *pci_dev,
+ const struct pci_device_id *id)
+{
+ struct imgu_device *imgu;
+ phys_addr_t phys;
+ unsigned long phys_len;
+ void __iomem *const *iomap;
+ int r;
+
+ imgu = devm_kzalloc(&pci_dev->dev, sizeof(*imgu), GFP_KERNEL);
+ if (!imgu)
+ return -ENOMEM;
+
+ imgu->pci_dev = pci_dev;
+
+ r = pcim_enable_device(pci_dev);
+ if (r) {
+ dev_err(&pci_dev->dev, "failed to enable device (%d)\n", r);
+ return r;
+ }
+
+ dev_info(&pci_dev->dev, "device 0x%x (rev: 0x%x)\n",
+ pci_dev->device, pci_dev->revision);
+
+ phys = pci_resource_start(pci_dev, IMGU_PCI_BAR);
+ phys_len = pci_resource_len(pci_dev, IMGU_PCI_BAR);
+
+ r = pcim_iomap_regions(pci_dev, 1 << IMGU_PCI_BAR, pci_name(pci_dev));
+ if (r) {
+ dev_err(&pci_dev->dev, "failed to remap I/O memory (%d)\n", r);
+ return r;
+ }
+ dev_info(&pci_dev->dev, "physical base address %pap, %lu bytes\n",
+ &phys, phys_len);
+
+ iomap = pcim_iomap_table(pci_dev);
+ if (!iomap) {
+ dev_err(&pci_dev->dev, "failed to iomap table\n");
+ return -ENODEV;
+ }
+
+ imgu->base = iomap[IMGU_PCI_BAR];
+
+ pci_set_drvdata(pci_dev, imgu);
+
+ pci_set_master(pci_dev);
+
+ r = dma_coerce_mask_and_coherent(&pci_dev->dev, IMGU_DMA_MASK);
+ if (r) {
+ dev_err(&pci_dev->dev, "failed to set DMA mask (%d)\n", r);
+ return -ENODEV;
+ }
+
+ r = imgu_pci_config_setup(pci_dev);
+ if (r)
+ return r;
+
+ mutex_init(&imgu->lock);
+ atomic_set(&imgu->qbuf_barrier, 0);
+ init_waitqueue_head(&imgu->buf_drain_wq);
+
+ r = ipu3_css_set_powerup(&pci_dev->dev, imgu->base);
+ if (r) {
+ dev_err(&pci_dev->dev,
+ "failed to power up CSS (%d)\n", r);
+ goto out_mutex_destroy;
+ }
+
+ imgu->mmu = ipu3_mmu_init(&pci_dev->dev, imgu->base);
+ if (IS_ERR(imgu->mmu)) {
+ r = PTR_ERR(imgu->mmu);
+ dev_err(&pci_dev->dev, "failed to initialize MMU (%d)\n", r);
+ goto out_css_powerdown;
+ }
+
+ r = ipu3_dmamap_init(imgu);
+ if (r) {
+ dev_err(&pci_dev->dev,
+ "failed to initialize DMA mapping (%d)\n", r);
+ goto out_mmu_exit;
+ }
+
+ /* ISP programming */
+ r = ipu3_css_init(&pci_dev->dev, &imgu->css, imgu->base, phys_len);
+ if (r) {
+ dev_err(&pci_dev->dev, "failed to initialize CSS (%d)\n", r);
+ goto out_dmamap_exit;
+ }
+
+ /* v4l2 sub-device registration */
+ r = imgu_video_nodes_init(imgu);
+ if (r) {
+ dev_err(&pci_dev->dev, "failed to create V4L2 devices (%d)\n",
+ r);
+ goto out_css_cleanup;
+ }
+
+ r = devm_request_threaded_irq(&pci_dev->dev, pci_dev->irq,
+ imgu_isr, imgu_isr_threaded,
+ IRQF_SHARED, IMGU_NAME, imgu);
+ if (r) {
+ dev_err(&pci_dev->dev, "failed to request IRQ (%d)\n", r);
+ goto out_video_exit;
+ }
+
+ pm_runtime_put_noidle(&pci_dev->dev);
+ pm_runtime_allow(&pci_dev->dev);
+
+ return 0;
+
+out_video_exit:
+ imgu_video_nodes_exit(imgu);
+out_css_cleanup:
+ ipu3_css_cleanup(&imgu->css);
+out_dmamap_exit:
+ ipu3_dmamap_exit(imgu);
+out_mmu_exit:
+ ipu3_mmu_exit(imgu->mmu);
+out_css_powerdown:
+ ipu3_css_set_powerdown(&pci_dev->dev, imgu->base);
+out_mutex_destroy:
+ mutex_destroy(&imgu->lock);
+
+ return r;
+}
+
+static void imgu_pci_remove(struct pci_dev *pci_dev)
+{
+ struct imgu_device *imgu = pci_get_drvdata(pci_dev);
+
+ pm_runtime_forbid(&pci_dev->dev);
+ pm_runtime_get_noresume(&pci_dev->dev);
+
+ imgu_video_nodes_exit(imgu);
+ ipu3_css_cleanup(&imgu->css);
+ ipu3_css_set_powerdown(&pci_dev->dev, imgu->base);
+ ipu3_dmamap_exit(imgu);
+ ipu3_mmu_exit(imgu->mmu);
+ mutex_destroy(&imgu->lock);
+}
+
+static int __maybe_unused imgu_suspend(struct device *dev)
+{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+ struct imgu_device *imgu = pci_get_drvdata(pci_dev);
+
+ dev_dbg(dev, "enter %s\n", __func__);
+ imgu->suspend_in_stream = ipu3_css_is_streaming(&imgu->css);
+ if (!imgu->suspend_in_stream)
+ goto out;
+ /* Block new buffers to be queued to CSS. */
+ atomic_set(&imgu->qbuf_barrier, 1);
+ /*
+ * Wait for currently running irq handler to be done so that
+ * no new buffers will be queued to fw later.
+ */
+ synchronize_irq(pci_dev->irq);
+ /* Wait until all buffers in CSS are done. */
+ if (!wait_event_timeout(imgu->buf_drain_wq,
+ ipu3_css_queue_empty(&imgu->css), msecs_to_jiffies(1000)))
+ dev_err(dev, "wait buffer drain timeout.\n");
+
+ ipu3_css_stop_streaming(&imgu->css);
+ atomic_set(&imgu->qbuf_barrier, 0);
+ imgu_powerdown(imgu);
+ pm_runtime_force_suspend(dev);
+out:
+ dev_dbg(dev, "leave %s\n", __func__);
+ return 0;
+}
+
+static int __maybe_unused imgu_resume(struct device *dev)
+{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+ struct imgu_device *imgu = pci_get_drvdata(pci_dev);
+ int r = 0;
+
+ dev_dbg(dev, "enter %s\n", __func__);
+
+ if (!imgu->suspend_in_stream)
+ goto out;
+
+ pm_runtime_force_resume(dev);
+
+ r = imgu_powerup(imgu);
+ if (r) {
+ dev_err(dev, "failed to power up imgu\n");
+ goto out;
+ }
+
+ /* Start CSS streaming */
+ r = ipu3_css_start_streaming(&imgu->css);
+ if (r) {
+ dev_err(dev, "failed to resume css streaming (%d)", r);
+ goto out;
+ }
+
+ r = imgu_queue_buffers(imgu, true);
+ if (r)
+ dev_err(dev, "failed to queue buffers (%d)", r);
+out:
+ dev_dbg(dev, "leave %s\n", __func__);
+
+ return r;
+}
+
+/*
+ * PCI rpm framework checks the existence of driver rpm callbacks.
+ * Place a dummy callback here to avoid rpm going into error state.
+ */
+static int imgu_rpm_dummy_cb(struct device *dev)
+{
+ return 0;
+}
+
+static const struct dev_pm_ops imgu_pm_ops = {
+ SET_RUNTIME_PM_OPS(&imgu_rpm_dummy_cb, &imgu_rpm_dummy_cb, NULL)
+ SET_SYSTEM_SLEEP_PM_OPS(&imgu_suspend, &imgu_resume)
+};
+
+static const struct pci_device_id imgu_pci_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_INTEL, IMGU_PCI_ID) },
+ { 0, }
+};
+
+MODULE_DEVICE_TABLE(pci, imgu_pci_tbl);
+
+static struct pci_driver imgu_pci_driver = {
+ .name = IMGU_NAME,
+ .id_table = imgu_pci_tbl,
+ .probe = imgu_pci_probe,
+ .remove = imgu_pci_remove,
+ .driver = {
+ .pm = &imgu_pm_ops,
+ },
+};
+
+module_pci_driver(imgu_pci_driver);
+
+MODULE_AUTHOR("Tuukka Toivonen <tuukka.toivonen@intel.com>");
+MODULE_AUTHOR("Tianshu Qiu <tian.shu.qiu@intel.com>");
+MODULE_AUTHOR("Jian Xu Zheng <jian.xu.zheng@intel.com>");
+MODULE_AUTHOR("Yuning Pu <yuning.pu@intel.com>");
+MODULE_AUTHOR("Yong Zhi <yong.zhi@intel.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Intel ipu3_imgu PCI driver");
diff --git a/drivers/media/pci/intel/ipu3/ipu3.h b/drivers/media/pci/intel/ipu3/ipu3.h
new file mode 100644
index 0000000..5c2b420
--- /dev/null
+++ b/drivers/media/pci/intel/ipu3/ipu3.h
@@ -0,0 +1,153 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2018 Intel Corporation */
+
+#ifndef __IPU3_H
+#define __IPU3_H
+
+#include <linux/iova.h>
+#include <linux/pci.h>
+
+#include <media/v4l2-device.h>
+#include <media/videobuf2-dma-sg.h>
+
+#include "ipu3-css.h"
+
+#define IMGU_NAME "ipu3-imgu"
+
+/*
+ * The semantics of the driver is that whenever there is a buffer available in
+ * master queue, the driver queues a buffer also to all other active nodes.
+ * If user space hasn't provided a buffer to all other video nodes first,
+ * the driver gets an internal dummy buffer and queues it.
+ */
+#define IMGU_QUEUE_MASTER IPU3_CSS_QUEUE_IN
+#define IMGU_QUEUE_FIRST_INPUT IPU3_CSS_QUEUE_OUT
+#define IMGU_MAX_QUEUE_DEPTH (2 + 2)
+
+#define IMGU_NODE_IN 0 /* Input RAW image */
+#define IMGU_NODE_PARAMS 1 /* Input parameters */
+#define IMGU_NODE_OUT 2 /* Main output for still or video */
+#define IMGU_NODE_VF 3 /* Preview */
+#define IMGU_NODE_PV 4 /* Postview for still capture */
+#define IMGU_NODE_STAT_3A 5 /* 3A statistics */
+#define IMGU_NODE_NUM 6
+
+#define file_to_intel_ipu3_node(__file) \
+ container_of(video_devdata(__file), struct imgu_video_device, vdev)
+
+#define IPU3_INPUT_MIN_WIDTH 0U
+#define IPU3_INPUT_MIN_HEIGHT 0U
+#define IPU3_INPUT_MAX_WIDTH 5120U
+#define IPU3_INPUT_MAX_HEIGHT 38404U
+#define IPU3_OUTPUT_MIN_WIDTH 2U
+#define IPU3_OUTPUT_MIN_HEIGHT 2U
+#define IPU3_OUTPUT_MAX_WIDTH 4480U
+#define IPU3_OUTPUT_MAX_HEIGHT 34004U
+
+struct ipu3_vb2_buffer {
+ /* Public fields */
+ struct vb2_v4l2_buffer vbb; /* Must be the first field */
+
+ /* Private fields */
+ struct list_head list;
+};
+
+struct imgu_buffer {
+ struct ipu3_vb2_buffer vid_buf; /* Must be the first field */
+ struct ipu3_css_buffer css_buf;
+ struct ipu3_css_map map;
+};
+
+struct imgu_node_mapping {
+ unsigned int css_queue;
+ const char *name;
+};
+
+/**
+ * struct imgu_video_device
+ * each node registers as video device and maintains its
+ * own vb2_queue.
+ */
+struct imgu_video_device {
+ const char *name;
+ bool output; /* Frames to the driver? */
+ bool immutable; /* Can not be enabled/disabled */
+ bool enabled;
+ int queued; /* Buffers already queued */
+ struct v4l2_format vdev_fmt; /* Currently set format */
+
+ /* Private fields */
+ struct video_device vdev;
+ struct media_pad vdev_pad;
+ struct v4l2_mbus_framefmt pad_fmt;
+ struct vb2_queue vbq;
+ struct list_head buffers;
+ /* Protect vb2_queue and vdev structs*/
+ struct mutex lock;
+ atomic_t sequence;
+};
+
+/*
+ * imgu_device -- ImgU (Imaging Unit) driver
+ */
+struct imgu_device {
+ struct pci_dev *pci_dev;
+ void __iomem *base;
+
+ /* Internally enabled queues */
+ struct {
+ struct ipu3_css_map dmap;
+ struct ipu3_css_buffer dummybufs[IMGU_MAX_QUEUE_DEPTH];
+ } queues[IPU3_CSS_QUEUES];
+ struct imgu_video_device nodes[IMGU_NODE_NUM];
+ bool queue_enabled[IMGU_NODE_NUM];
+
+ /* Public fields, fill before registering */
+ unsigned int buf_struct_size;
+ bool streaming; /* Public read only */
+ struct v4l2_ctrl_handler *ctrl_handler;
+
+ /* Private fields */
+ struct v4l2_device v4l2_dev;
+ struct media_device media_dev;
+ struct media_pipeline pipeline;
+ struct v4l2_subdev subdev;
+ struct media_pad *subdev_pads;
+ struct v4l2_file_operations v4l2_file_ops;
+
+ /* MMU driver for css */
+ struct ipu3_mmu_info *mmu;
+ struct iova_domain iova_domain;
+
+ /* css - Camera Sub-System */
+ struct ipu3_css css;
+
+ /*
+ * Coarse-grained lock to protect
+ * vid_buf.list and css->queue
+ */
+ struct mutex lock;
+ /* Forbit streaming and buffer queuing during system suspend. */
+ atomic_t qbuf_barrier;
+ struct {
+ struct v4l2_rect eff; /* effective resolution */
+ struct v4l2_rect bds; /* bayer-domain scaled resolution*/
+ struct v4l2_rect gdc; /* gdc output resolution */
+ } rect;
+ /* Indicate if system suspend take place while imgu is streaming. */
+ bool suspend_in_stream;
+ /* Used to wait for FW buffer queue drain. */
+ wait_queue_head_t buf_drain_wq;
+};
+
+unsigned int imgu_node_to_queue(unsigned int node);
+unsigned int imgu_map_node(struct imgu_device *imgu, unsigned int css_queue);
+int imgu_queue_buffers(struct imgu_device *imgu, bool initial);
+
+int ipu3_v4l2_register(struct imgu_device *dev);
+int ipu3_v4l2_unregister(struct imgu_device *dev);
+void ipu3_v4l2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state);
+
+int imgu_s_stream(struct imgu_device *imgu, int enable);
+
+#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver
2018-10-29 22:23 ` [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver Yong Zhi
@ 2018-11-09 12:54 ` Sakari Ailus
2018-11-12 22:16 ` Zhi, Yong
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-09 12:54 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
On Mon, Oct 29, 2018 at 03:23:09PM -0700, Yong Zhi wrote:
> This patch adds support for the Intel IPU v3 as found
> on Skylake and Kaby Lake SoCs.
>
> The driver glues v4l2, css(camera sub system) and other
> pieces together to perform its functions, it also loads
> the IPU3 firmware binary as part of its initialization.
>
> Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> ---
> drivers/media/pci/intel/ipu3/Kconfig | 16 +
> drivers/media/pci/intel/ipu3/Makefile | 12 +
> drivers/media/pci/intel/ipu3/ipu3.c | 844 ++++++++++++++++++++++++++++++++++
> drivers/media/pci/intel/ipu3/ipu3.h | 153 ++++++
> 4 files changed, 1025 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
>
> diff --git a/drivers/media/pci/intel/ipu3/Kconfig b/drivers/media/pci/intel/ipu3/Kconfig
> index 715f776..44ebcbb 100644
> --- a/drivers/media/pci/intel/ipu3/Kconfig
> +++ b/drivers/media/pci/intel/ipu3/Kconfig
> @@ -15,3 +15,19 @@ config VIDEO_IPU3_CIO2
> Say Y or M here if you have a Skylake/Kaby Lake SoC with MIPI CSI-2
> connected camera.
> The module will be called ipu3-cio2.
> +
> +config VIDEO_IPU3_IMGU
> + tristate "Intel ipu3-imgu driver"
> + depends on PCI && VIDEO_V4L2
> + depends on MEDIA_CONTROLLER && VIDEO_V4L2_SUBDEV_API
> + depends on X86
> + select IOMMU_IOVA
> + select VIDEOBUF2_DMA_SG
> +
Extra newline.
> + ---help---
> + This is the video4linux2 driver for Intel IPU3 image processing unit,
"Video4Linux2"
> + found in Intel Skylake and Kaby Lake SoCs and used for processing
> + images and video.
> +
> + Say Y or M here if you have a Skylake/Kaby Lake SoC with a MIPI
> + camera. The module will be called ipu3-imgu.
The latter tab should be a space only.
> diff --git a/drivers/media/pci/intel/ipu3/Makefile b/drivers/media/pci/intel/ipu3/Makefile
> index 20186e3..60bd5db 100644
> --- a/drivers/media/pci/intel/ipu3/Makefile
> +++ b/drivers/media/pci/intel/ipu3/Makefile
> @@ -1 +1,13 @@
> +#
> +# Makefile for the IPU3 cio2 and ImgU drivers
> +#
> +
> obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
> +
> +ipu3-imgu-objs += \
> + ipu3-mmu.o ipu3-dmamap.o \
> + ipu3-tables.o ipu3-css-pool.o \
> + ipu3-css-fw.o ipu3-css-params.o \
> + ipu3-css.o ipu3-v4l2.o ipu3.o
> +
> +obj-$(CONFIG_VIDEO_IPU3_IMGU) += ipu3-imgu.o
> diff --git a/drivers/media/pci/intel/ipu3/ipu3.c b/drivers/media/pci/intel/ipu3/ipu3.c
> new file mode 100644
> index 0000000..eda7299
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3.c
> @@ -0,0 +1,844 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2017 Intel Corporation
> + * Copyright 2017 Google LLC
> + *
> + * Based on Intel IPU4 driver.
> + *
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pm_runtime.h>
> +
> +#include "ipu3.h"
> +#include "ipu3-dmamap.h"
> +#include "ipu3-mmu.h"
> +
> +#define IMGU_PCI_ID 0x1919
> +#define IMGU_PCI_BAR 0
> +#define IMGU_DMA_MASK DMA_BIT_MASK(39)
> +#define IMGU_MAX_QUEUE_DEPTH (2 + 2)
> +
> +/*
> + * pre-allocated buffer size for IMGU dummy buffers. Those
> + * values should be tuned to big enough to avoid buffer
> + * re-allocation when streaming to lower streaming latency.
> + */
> +#define CSS_QUEUE_IN_BUF_SIZE 0
> +#define CSS_QUEUE_PARAMS_BUF_SIZE 0
> +#define CSS_QUEUE_OUT_BUF_SIZE (4160 * 3120 * 12 / 8)
> +#define CSS_QUEUE_VF_BUF_SIZE (1920 * 1080 * 12 / 8)
> +#define CSS_QUEUE_STAT_3A_BUF_SIZE 125664
Could you use sizeof(struct ipu3_uapi_stats_3a) instead?
That said, it might not be a bad idea to add a sanity check on the size:
BUILD_BUG_ON(sizeof(struct ipu3_uapi_stats_3a) !=
CSS_QUEUE_STAT_3A_BUF_SIZE);
...
> diff --git a/drivers/media/pci/intel/ipu3/ipu3.h b/drivers/media/pci/intel/ipu3/ipu3.h
> new file mode 100644
> index 0000000..5c2b420
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3.h
> @@ -0,0 +1,153 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2018 Intel Corporation */
> +
> +#ifndef __IPU3_H
> +#define __IPU3_H
> +
> +#include <linux/iova.h>
> +#include <linux/pci.h>
> +
> +#include <media/v4l2-device.h>
> +#include <media/videobuf2-dma-sg.h>
> +
> +#include "ipu3-css.h"
> +
> +#define IMGU_NAME "ipu3-imgu"
> +
> +/*
> + * The semantics of the driver is that whenever there is a buffer available in
> + * master queue, the driver queues a buffer also to all other active nodes.
> + * If user space hasn't provided a buffer to all other video nodes first,
> + * the driver gets an internal dummy buffer and queues it.
> + */
> +#define IMGU_QUEUE_MASTER IPU3_CSS_QUEUE_IN
> +#define IMGU_QUEUE_FIRST_INPUT IPU3_CSS_QUEUE_OUT
> +#define IMGU_MAX_QUEUE_DEPTH (2 + 2)
> +
> +#define IMGU_NODE_IN 0 /* Input RAW image */
> +#define IMGU_NODE_PARAMS 1 /* Input parameters */
> +#define IMGU_NODE_OUT 2 /* Main output for still or video */
> +#define IMGU_NODE_VF 3 /* Preview */
> +#define IMGU_NODE_PV 4 /* Postview for still capture */
> +#define IMGU_NODE_STAT_3A 5 /* 3A statistics */
> +#define IMGU_NODE_NUM 6
> +
> +#define file_to_intel_ipu3_node(__file) \
> + container_of(video_devdata(__file), struct imgu_video_device, vdev)
> +
> +#define IPU3_INPUT_MIN_WIDTH 0U
> +#define IPU3_INPUT_MIN_HEIGHT 0U
> +#define IPU3_INPUT_MAX_WIDTH 5120U
> +#define IPU3_INPUT_MAX_HEIGHT 38404U
> +#define IPU3_OUTPUT_MIN_WIDTH 2U
> +#define IPU3_OUTPUT_MIN_HEIGHT 2U
> +#define IPU3_OUTPUT_MAX_WIDTH 4480U
> +#define IPU3_OUTPUT_MAX_HEIGHT 34004U
> +
> +struct ipu3_vb2_buffer {
> + /* Public fields */
> + struct vb2_v4l2_buffer vbb; /* Must be the first field */
> +
> + /* Private fields */
> + struct list_head list;
> +};
> +
> +struct imgu_buffer {
> + struct ipu3_vb2_buffer vid_buf; /* Must be the first field */
> + struct ipu3_css_buffer css_buf;
> + struct ipu3_css_map map;
> +};
> +
> +struct imgu_node_mapping {
> + unsigned int css_queue;
> + const char *name;
> +};
> +
> +/**
> + * struct imgu_video_device
> + * each node registers as video device and maintains its
> + * own vb2_queue.
> + */
> +struct imgu_video_device {
> + const char *name;
> + bool output; /* Frames to the driver? */
> + bool immutable; /* Can not be enabled/disabled */
> + bool enabled;
> + int queued; /* Buffers already queued */
The queued field is unused.
> + struct v4l2_format vdev_fmt; /* Currently set format */
> +
> + /* Private fields */
> + struct video_device vdev;
> + struct media_pad vdev_pad;
> + struct v4l2_mbus_framefmt pad_fmt;
> + struct vb2_queue vbq;
> + struct list_head buffers;
> + /* Protect vb2_queue and vdev structs*/
> + struct mutex lock;
> + atomic_t sequence;
> +};
> +
--
Kind regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver
2018-11-09 12:54 ` Sakari Ailus
@ 2018-11-12 22:16 ` Zhi, Yong
0 siblings, 0 replies; 123+ messages in thread
From: Zhi, Yong @ 2018-11-12 22:16 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi, Sakari,
Thanks again for the code review.
> -----Original Message-----
> From: Sakari Ailus [mailto:sakari.ailus@linux.intel.com]
> Sent: Friday, November 9, 2018 6:54 AM
> To: Zhi, Yong <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver
>
> Hi Yong,
>
> On Mon, Oct 29, 2018 at 03:23:09PM -0700, Yong Zhi wrote:
> > This patch adds support for the Intel IPU v3 as found on Skylake and
> > Kaby Lake SoCs.
> >
> > The driver glues v4l2, css(camera sub system) and other pieces
> > together to perform its functions, it also loads the IPU3 firmware
> > binary as part of its initialization.
> >
> > Signed-off-by: Yong Zhi <yong.zhi@intel.com>
> > Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > ---
> > drivers/media/pci/intel/ipu3/Kconfig | 16 +
> > drivers/media/pci/intel/ipu3/Makefile | 12 +
> > drivers/media/pci/intel/ipu3/ipu3.c | 844
> ++++++++++++++++++++++++++++++++++
> > drivers/media/pci/intel/ipu3/ipu3.h | 153 ++++++
> > 4 files changed, 1025 insertions(+)
> > create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
> > create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
> >
> > diff --git a/drivers/media/pci/intel/ipu3/Kconfig
> > b/drivers/media/pci/intel/ipu3/Kconfig
> > index 715f776..44ebcbb 100644
> > --- a/drivers/media/pci/intel/ipu3/Kconfig
> > +++ b/drivers/media/pci/intel/ipu3/Kconfig
> > @@ -15,3 +15,19 @@ config VIDEO_IPU3_CIO2
> > Say Y or M here if you have a Skylake/Kaby Lake SoC with MIPI CSI-2
> > connected camera.
> > The module will be called ipu3-cio2.
> > +
> > +config VIDEO_IPU3_IMGU
> > + tristate "Intel ipu3-imgu driver"
> > + depends on PCI && VIDEO_V4L2
> > + depends on MEDIA_CONTROLLER && VIDEO_V4L2_SUBDEV_API
> > + depends on X86
> > + select IOMMU_IOVA
> > + select VIDEOBUF2_DMA_SG
> > +
>
> Extra newline.
>
Ack.
> > + ---help---
> > + This is the video4linux2 driver for Intel IPU3 image processing
> > +unit,
>
> "Video4Linux2"
>
Ack.
> > + found in Intel Skylake and Kaby Lake SoCs and used for processing
> > + images and video.
> > +
> > + Say Y or M here if you have a Skylake/Kaby Lake SoC with a MIPI
> > + camera. The module will be called ipu3-imgu.
>
> The latter tab should be a space only.
>
Ack.
> > diff --git a/drivers/media/pci/intel/ipu3/Makefile
> > b/drivers/media/pci/intel/ipu3/Makefile
> > index 20186e3..60bd5db 100644
> > --- a/drivers/media/pci/intel/ipu3/Makefile
> > +++ b/drivers/media/pci/intel/ipu3/Makefile
> > @@ -1 +1,13 @@
> > +#
> > +# Makefile for the IPU3 cio2 and ImgU drivers #
> > +
> > obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
> > +
> > +ipu3-imgu-objs += \
> > + ipu3-mmu.o ipu3-dmamap.o \
> > + ipu3-tables.o ipu3-css-pool.o \
> > + ipu3-css-fw.o ipu3-css-params.o \
> > + ipu3-css.o ipu3-v4l2.o ipu3.o
> > +
> > +obj-$(CONFIG_VIDEO_IPU3_IMGU) += ipu3-imgu.o
> > diff --git a/drivers/media/pci/intel/ipu3/ipu3.c
> > b/drivers/media/pci/intel/ipu3/ipu3.c
> > new file mode 100644
> > index 0000000..eda7299
> > --- /dev/null
> > +++ b/drivers/media/pci/intel/ipu3/ipu3.c
> > @@ -0,0 +1,844 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Copyright (C) 2017 Intel Corporation
> > + * Copyright 2017 Google LLC
> > + *
> > + * Based on Intel IPU4 driver.
> > + *
> > + */
> > +
> > +#include <linux/delay.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/module.h>
> > +#include <linux/pm_runtime.h>
> > +
> > +#include "ipu3.h"
> > +#include "ipu3-dmamap.h"
> > +#include "ipu3-mmu.h"
> > +
> > +#define IMGU_PCI_ID 0x1919
> > +#define IMGU_PCI_BAR 0
> > +#define IMGU_DMA_MASK DMA_BIT_MASK(39)
> > +#define IMGU_MAX_QUEUE_DEPTH (2 + 2)
> > +
> > +/*
> > + * pre-allocated buffer size for IMGU dummy buffers. Those
> > + * values should be tuned to big enough to avoid buffer
> > + * re-allocation when streaming to lower streaming latency.
> > + */
> > +#define CSS_QUEUE_IN_BUF_SIZE 0
> > +#define CSS_QUEUE_PARAMS_BUF_SIZE 0
> > +#define CSS_QUEUE_OUT_BUF_SIZE (4160 * 3120 * 12 / 8)
> > +#define CSS_QUEUE_VF_BUF_SIZE (1920 * 1080 * 12 / 8)
> > +#define CSS_QUEUE_STAT_3A_BUF_SIZE 125664
>
> Could you use sizeof(struct ipu3_uapi_stats_3a) instead?
>
> That said, it might not be a bad idea to add a sanity check on the size:
>
> BUILD_BUG_ON(sizeof(struct ipu3_uapi_stats_3a) !=
> CSS_QUEUE_STAT_3A_BUF_SIZE);
>
Ack.
> ...
>
> > diff --git a/drivers/media/pci/intel/ipu3/ipu3.h
> > b/drivers/media/pci/intel/ipu3/ipu3.h
> > new file mode 100644
> > index 0000000..5c2b420
> > --- /dev/null
> > +++ b/drivers/media/pci/intel/ipu3/ipu3.h
> > @@ -0,0 +1,153 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2018 Intel Corporation */
> > +
> > +#ifndef __IPU3_H
> > +#define __IPU3_H
> > +
> > +#include <linux/iova.h>
> > +#include <linux/pci.h>
> > +
> > +#include <media/v4l2-device.h>
> > +#include <media/videobuf2-dma-sg.h>
> > +
> > +#include "ipu3-css.h"
> > +
> > +#define IMGU_NAME "ipu3-imgu"
> > +
> > +/*
> > + * The semantics of the driver is that whenever there is a buffer
> > +available in
> > + * master queue, the driver queues a buffer also to all other active nodes.
> > + * If user space hasn't provided a buffer to all other video nodes
> > +first,
> > + * the driver gets an internal dummy buffer and queues it.
> > + */
> > +#define IMGU_QUEUE_MASTER IPU3_CSS_QUEUE_IN
> > +#define IMGU_QUEUE_FIRST_INPUT IPU3_CSS_QUEUE_OUT
> > +#define IMGU_MAX_QUEUE_DEPTH (2 + 2)
> > +
> > +#define IMGU_NODE_IN 0 /* Input RAW image */
> > +#define IMGU_NODE_PARAMS 1 /* Input parameters */
> > +#define IMGU_NODE_OUT 2 /* Main output for still or
> video */
> > +#define IMGU_NODE_VF 3 /* Preview */
> > +#define IMGU_NODE_PV 4 /* Postview for still capture
> */
> > +#define IMGU_NODE_STAT_3A 5 /* 3A statistics */
> > +#define IMGU_NODE_NUM 6
> > +
> > +#define file_to_intel_ipu3_node(__file) \
> > + container_of(video_devdata(__file), struct imgu_video_device, vdev)
> > +
> > +#define IPU3_INPUT_MIN_WIDTH 0U
> > +#define IPU3_INPUT_MIN_HEIGHT 0U
> > +#define IPU3_INPUT_MAX_WIDTH 5120U
> > +#define IPU3_INPUT_MAX_HEIGHT 38404U
> > +#define IPU3_OUTPUT_MIN_WIDTH 2U
> > +#define IPU3_OUTPUT_MIN_HEIGHT 2U
> > +#define IPU3_OUTPUT_MAX_WIDTH 4480U
> > +#define IPU3_OUTPUT_MAX_HEIGHT 34004U
> > +
> > +struct ipu3_vb2_buffer {
> > + /* Public fields */
> > + struct vb2_v4l2_buffer vbb; /* Must be the first field */
> > +
> > + /* Private fields */
> > + struct list_head list;
> > +};
> > +
> > +struct imgu_buffer {
> > + struct ipu3_vb2_buffer vid_buf; /* Must be the first field */
> > + struct ipu3_css_buffer css_buf;
> > + struct ipu3_css_map map;
> > +};
> > +
> > +struct imgu_node_mapping {
> > + unsigned int css_queue;
> > + const char *name;
> > +};
> > +
> > +/**
> > + * struct imgu_video_device
> > + * each node registers as video device and maintains its
> > + * own vb2_queue.
> > + */
> > +struct imgu_video_device {
> > + const char *name;
> > + bool output; /* Frames to the driver? */
> > + bool immutable; /* Can not be enabled/disabled */
> > + bool enabled;
> > + int queued; /* Buffers already queued */
>
> The queued field is unused.
>
Ack, thanks for catching this.
Best regards,
Yong
> > + struct v4l2_format vdev_fmt; /* Currently set format */
> > +
> > + /* Private fields */
> > + struct video_device vdev;
> > + struct media_pad vdev_pad;
> > + struct v4l2_mbus_framefmt pad_fmt;
> > + struct vb2_queue vbq;
> > + struct list_head buffers;
> > + /* Protect vb2_queue and vdev structs*/
> > + struct mutex lock;
> > + atomic_t sequence;
> > +};
> > +
>
> --
> Kind regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* [PATCH v7 16/16] intel-ipu3: Add dual pipe support
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (13 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 15/16] intel-ipu3: Add imgu top level pci device driver Yong Zhi
@ 2018-10-29 22:23 ` Yong Zhi
2018-11-01 12:03 ` [PATCH v7 00/16] Intel IPU3 ImgU patchset Sakari Ailus
` (2 subsequent siblings)
17 siblings, 0 replies; 123+ messages in thread
From: Yong Zhi @ 2018-10-29 22:23 UTC (permalink / raw)
To: linux-media, sakari.ailus
Cc: tfiga, mchehab, hans.verkuil, laurent.pinchart, rajmohan.mani,
jian.xu.zheng, jerry.w.hu, tuukka.toivonen, tian.shu.qiu,
bingbu.cao, Yong Zhi
From: "Cao,Bing Bu" <bingbu.cao@intel.com>
This patch adds support to run dual pipes simultaneously.
A private ioctl to configure the pipe mode (video or still)
is also implemented.
IPU3 hardware supports a maximum of 2 streams per pipe.
With the support of dual pipes, more than 2 stream outputs
can be achieved.
This helps to support advanced camera features like
Continuous View Finder (CVF) and Snapshot During Video(SDV).
Extend ipu3 IMGU driver to support dual pipes
1. Extend current IMGU device to contain 2 groups
of video nodes and 2 subdevs
2. Extend current css to support 2 pipeline and make
CSS APIs to support 2 pipe
3. Add a v4l2 ctrl to allow user to specify the mode
of the pipe
4. Check media pipeline link status to get enabled
pipes
Signed-off-by: Tian Shu Qiu <tian.shu.qiu@intel.com>
Signed-off-by: Yong Zhi <yong.zhi@intel.com>
---
drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 11 +-
drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 6 +-
drivers/media/pci/intel/ipu3/ipu3-css-params.c | 309 +++++----
drivers/media/pci/intel/ipu3/ipu3-css-params.h | 9 +-
drivers/media/pci/intel/ipu3/ipu3-css.c | 864 ++++++++++++++-----------
drivers/media/pci/intel/ipu3/ipu3-css.h | 84 +--
drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 797 ++++++++++++++++-------
drivers/media/pci/intel/ipu3/ipu3.c | 284 ++++----
drivers/media/pci/intel/ipu3/ipu3.h | 56 +-
include/uapi/linux/intel-ipu3.h | 8 +
10 files changed, 1469 insertions(+), 959 deletions(-)
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-fw.c b/drivers/media/pci/intel/ipu3/ipu3-css-fw.c
index ba459e9..55861aa 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css-fw.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-fw.c
@@ -69,16 +69,17 @@ unsigned int ipu3_css_fw_obgrid_size(const struct imgu_fw_info *bi)
return obgrid_size;
}
-void *ipu3_css_fw_pipeline_params(struct ipu3_css *css,
- enum imgu_abi_param_class c,
- enum imgu_abi_memories m,
+void *ipu3_css_fw_pipeline_params(struct ipu3_css *css, unsigned int pipe,
+ enum imgu_abi_param_class cls,
+ enum imgu_abi_memories mem,
struct imgu_fw_isp_parameter *par,
size_t par_size, void *binary_params)
{
- struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+ struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css->pipes[pipe].bindex];
if (par->offset + par->size >
- bi->info.isp.sp.mem_initializers.params[c][m].size)
+ bi->info.isp.sp.mem_initializers.params[cls][mem].size)
return NULL;
if (par->size != par_size)
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-fw.h b/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
index 954bb31..d1ffe51 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-fw.h
@@ -179,9 +179,9 @@ int ipu3_css_fw_init(struct ipu3_css *css);
void ipu3_css_fw_cleanup(struct ipu3_css *css);
unsigned int ipu3_css_fw_obgrid_size(const struct imgu_fw_info *bi);
-void *ipu3_css_fw_pipeline_params(struct ipu3_css *css,
- enum imgu_abi_param_class c,
- enum imgu_abi_memories m,
+void *ipu3_css_fw_pipeline_params(struct ipu3_css *css, unsigned int pipe,
+ enum imgu_abi_param_class cls,
+ enum imgu_abi_memories mem,
struct imgu_fw_isp_parameter *par,
size_t par_size, void *binary_params);
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-params.c b/drivers/media/pci/intel/ipu3/ipu3-css-params.c
index add2be4..7843631 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css-params.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-params.c
@@ -364,55 +364,59 @@ static int ipu3_css_osys_calc_frame_and_stripe_params(
struct ipu3_css_scaler_info *scaler_luma,
struct ipu3_css_scaler_info *scaler_chroma,
struct ipu3_css_frame_params frame_params[],
- struct ipu3_css_stripe_params stripe_params[])
+ struct ipu3_css_stripe_params stripe_params[],
+ unsigned int pipe)
{
- u32 input_width = css->rect[IPU3_CSS_RECT_GDC].width;
- u32 input_height = css->rect[IPU3_CSS_RECT_GDC].height;
- u32 target_width = css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
- u32 target_height = css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
- unsigned int procmode = 0;
struct ipu3_css_reso reso;
unsigned int output_width, pin, s;
+ u32 input_width, input_height, target_width, target_height;
+ unsigned int procmode = 0;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+
+ input_width = css_pipe->rect[IPU3_CSS_RECT_GDC].width;
+ input_height = css_pipe->rect[IPU3_CSS_RECT_GDC].height;
+ target_width = css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ target_height = css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
/* Frame parameters */
/* Input width for Output System is output width of DVS (with GDC) */
- reso.input_width = css->rect[IPU3_CSS_RECT_GDC].width;
+ reso.input_width = css_pipe->rect[IPU3_CSS_RECT_GDC].width;
/* Input height for Output System is output height of DVS (with GDC) */
- reso.input_height = css->rect[IPU3_CSS_RECT_GDC].height;
+ reso.input_height = css_pipe->rect[IPU3_CSS_RECT_GDC].height;
reso.input_format =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
reso.pin_width[IMGU_ABI_OSYS_PIN_OUT] =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
reso.pin_height[IMGU_ABI_OSYS_PIN_OUT] =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
reso.pin_stride[IMGU_ABI_OSYS_PIN_OUT] =
- css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].width_pad;
reso.pin_format[IMGU_ABI_OSYS_PIN_OUT] =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
reso.pin_width[IMGU_ABI_OSYS_PIN_VF] =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
reso.pin_height[IMGU_ABI_OSYS_PIN_VF] =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
reso.pin_stride[IMGU_ABI_OSYS_PIN_VF] =
- css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].width_pad;
reso.pin_format[IMGU_ABI_OSYS_PIN_VF] =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
/* Configure the frame parameters for all output pins */
frame_params[IMGU_ABI_OSYS_PIN_OUT].width =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
frame_params[IMGU_ABI_OSYS_PIN_OUT].height =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
frame_params[IMGU_ABI_OSYS_PIN_VF].width =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
frame_params[IMGU_ABI_OSYS_PIN_VF].height =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
frame_params[IMGU_ABI_OSYS_PIN_VF].crop_top = 0;
frame_params[IMGU_ABI_OSYS_PIN_VF].crop_left = 0;
@@ -842,7 +846,8 @@ static int ipu3_css_osys_calc_frame_and_stripe_params(
* This function configures the Output Formatter System, given the number of
* stripes, scaler luma and chrome parameters
*/
-static void ipu3_css_osys_calc(struct ipu3_css *css, unsigned int stripes,
+static void ipu3_css_osys_calc(struct ipu3_css *css, unsigned int pipe,
+ unsigned int stripes,
struct imgu_abi_osys_config *osys,
struct ipu3_css_scaler_info *scaler_luma,
struct ipu3_css_scaler_info *scaler_chroma,
@@ -852,13 +857,15 @@ static void ipu3_css_osys_calc(struct ipu3_css *css, unsigned int stripes,
struct ipu3_css_stripe_params stripe_params[IPU3_UAPI_MAX_STRIPES];
struct imgu_abi_osys_formatter_params *param;
unsigned int pin, s;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
memset(osys, 0, sizeof(*osys));
/* Compute the frame and stripe params */
ipu3_css_osys_calc_frame_and_stripe_params(css, stripes, osys,
scaler_luma, scaler_chroma,
- frame_params, stripe_params);
+ frame_params, stripe_params,
+ pipe);
/* Output formatter system parameters */
@@ -1180,19 +1187,20 @@ static void ipu3_css_osys_calc(struct ipu3_css *css, unsigned int stripes,
block_stripes[0].height = stripe_params[0].input_height;
} else {
struct imgu_fw_info *bi =
- &css->fwp->binary_header[css->current_binary];
- unsigned int sp_block_width = IPU3_UAPI_ISP_VEC_ELEMS *
- bi->info.isp.sp.block.block_width;
+ &css->fwp->binary_header[css_pipe->bindex];
+ unsigned int sp_block_width =
+ bi->info.isp.sp.block.block_width *
+ IPU3_UAPI_ISP_VEC_ELEMS;
block_stripes[0].width = roundup(stripe_params[0].input_width,
sp_block_width);
block_stripes[1].offset =
- rounddown(css->rect[IPU3_CSS_RECT_GDC].width -
+ rounddown(css_pipe->rect[IPU3_CSS_RECT_GDC].width -
stripe_params[1].input_width, sp_block_width);
block_stripes[1].width =
- roundup(css->rect[IPU3_CSS_RECT_GDC].width -
+ roundup(css_pipe->rect[IPU3_CSS_RECT_GDC].width -
block_stripes[1].offset, sp_block_width);
- block_stripes[0].height = css->rect[IPU3_CSS_RECT_GDC].height;
+ block_stripes[0].height = css_pipe->rect[IPU3_CSS_RECT_GDC].height;
block_stripes[1].height = block_stripes[0].height;
}
}
@@ -1620,15 +1628,17 @@ ipu3_css_acc_process_lines(const struct process_lines *pl,
return 0;
}
-static int ipu3_css_af_ops_calc(struct ipu3_css *css,
+static int ipu3_css_af_ops_calc(struct ipu3_css *css, unsigned int pipe,
struct imgu_abi_af_config *af_config)
{
struct imgu_abi_af_intra_frame_operations_data *to =
&af_config->operations_data;
- struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+ struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css_pipe->bindex];
struct process_lines pl = {
- .image_height = css->rect[IPU3_CSS_RECT_BDS].height,
+ .image_height = css_pipe->rect[IPU3_CSS_RECT_BDS].height,
.grid_height = af_config->config.grid_cfg.height,
.block_height =
1 << af_config->config.grid_cfg.block_height_log2,
@@ -1646,14 +1656,16 @@ static int ipu3_css_af_ops_calc(struct ipu3_css *css,
}
static int
-ipu3_css_awb_fr_ops_calc(struct ipu3_css *css,
+ipu3_css_awb_fr_ops_calc(struct ipu3_css *css, unsigned int pipe,
struct imgu_abi_awb_fr_config *awb_fr_config)
{
struct imgu_abi_awb_fr_intra_frame_operations_data *to =
&awb_fr_config->operations_data;
- struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+ struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css_pipe->bindex];
struct process_lines pl = {
- .image_height = css->rect[IPU3_CSS_RECT_BDS].height,
+ .image_height = css_pipe->rect[IPU3_CSS_RECT_BDS].height,
.grid_height = awb_fr_config->config.grid_cfg.height,
.block_height =
1 << awb_fr_config->config.grid_cfg.block_height_log2,
@@ -1670,15 +1682,17 @@ ipu3_css_awb_fr_ops_calc(struct ipu3_css *css,
NULL);
}
-static int ipu3_css_awb_ops_calc(struct ipu3_css *css,
+static int ipu3_css_awb_ops_calc(struct ipu3_css *css, unsigned int pipe,
struct imgu_abi_awb_config *awb_config)
{
struct imgu_abi_awb_intra_frame_operations_data *to =
&awb_config->operations_data;
- struct imgu_fw_info *bi = &css->fwp->binary_header[css->current_binary];
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+ struct imgu_fw_info *bi =
+ &css->fwp->binary_header[css_pipe->bindex];
struct process_lines pl = {
- .image_height = css->rect[IPU3_CSS_RECT_BDS].height,
+ .image_height = css_pipe->rect[IPU3_CSS_RECT_BDS].height,
.grid_height = awb_config->config.grid.height,
.block_height =
1 << awb_config->config.grid.block_height_log2,
@@ -1710,21 +1724,22 @@ static void ipu3_css_grid_end_calc(struct ipu3_uapi_grid_config *grid_cfg)
/****************** config computation *****************************/
-static void ipu3_css_cfg_acc_stripe(struct ipu3_css *css,
+static void ipu3_css_cfg_acc_stripe(struct ipu3_css *css, unsigned int pipe,
struct imgu_abi_acc_param *acc)
{
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
const struct imgu_fw_info *bi =
- &css->fwp->binary_header[css->current_binary];
- const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
- const unsigned int F = IPU3_UAPI_ISP_VEC_ELEMS * 2;
+ &css->fwp->binary_header[css_pipe->bindex];
struct ipu3_css_scaler_info scaler_luma, scaler_chroma;
+ const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
+ const unsigned int f = IPU3_UAPI_ISP_VEC_ELEMS * 2;
unsigned int bds_ds, i;
memset(acc, 0, sizeof(*acc));
/* acc_param: osys_config */
- ipu3_css_osys_calc(css, stripes, &acc->osys, &scaler_luma,
+ ipu3_css_osys_calc(css, pipe, stripes, &acc->osys, &scaler_luma,
&scaler_chroma, acc->stripe.block_stripes);
/* acc_param: stripe data */
@@ -1740,77 +1755,78 @@ static void ipu3_css_cfg_acc_stripe(struct ipu3_css *css,
acc->stripe.num_of_stripes = stripes;
acc->stripe.input_frame.width =
- css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
acc->stripe.input_frame.height =
- css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
acc->stripe.input_frame.bayer_order =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
for (i = 0; i < stripes; i++)
acc->stripe.bds_out_stripes[i].height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->stripe.bds_out_stripes[0].offset = 0;
if (stripes <= 1) {
acc->stripe.bds_out_stripes[0].width =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, F);
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width, f);
} else {
/* Image processing is divided into two stripes */
acc->stripe.bds_out_stripes[0].width =
acc->stripe.bds_out_stripes[1].width =
- (css->rect[IPU3_CSS_RECT_BDS].width / 2 & ~(F - 1)) + F;
+ (css_pipe->rect[IPU3_CSS_RECT_BDS].width / 2 & ~(f - 1)) + f;
/*
* Sum of width of the two stripes should not be smaller
* than output width and must be even times of overlapping
* unit f.
*/
- if ((css->rect[IPU3_CSS_RECT_BDS].width / F & 1) !=
- !!(css->rect[IPU3_CSS_RECT_BDS].width & (F - 1)))
- acc->stripe.bds_out_stripes[0].width += F;
- if ((css->rect[IPU3_CSS_RECT_BDS].width / F & 1) &&
- (css->rect[IPU3_CSS_RECT_BDS].width & (F - 1))) {
- acc->stripe.bds_out_stripes[0].width += F;
- acc->stripe.bds_out_stripes[1].width += F;
+ if ((css_pipe->rect[IPU3_CSS_RECT_BDS].width / f & 1) !=
+ !!(css_pipe->rect[IPU3_CSS_RECT_BDS].width & (f - 1)))
+ acc->stripe.bds_out_stripes[0].width += f;
+ if ((css_pipe->rect[IPU3_CSS_RECT_BDS].width / f & 1) &&
+ (css_pipe->rect[IPU3_CSS_RECT_BDS].width & (f - 1))) {
+ acc->stripe.bds_out_stripes[0].width += f;
+ acc->stripe.bds_out_stripes[1].width += f;
}
/* Overlap between stripes is IPU3_UAPI_ISP_VEC_ELEMS * 4 */
acc->stripe.bds_out_stripes[1].offset =
- acc->stripe.bds_out_stripes[0].width - 2 * F;
+ acc->stripe.bds_out_stripes[0].width - 2 * f;
}
acc->stripe.effective_stripes[0].height =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].height;
acc->stripe.effective_stripes[0].offset = 0;
acc->stripe.bds_out_stripes_no_overlap[0].height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->stripe.bds_out_stripes_no_overlap[0].offset = 0;
acc->stripe.output_stripes[0].height =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
acc->stripe.output_stripes[0].offset = 0;
if (stripes <= 1) {
acc->stripe.down_scaled_stripes[0].width =
- css->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].width;
acc->stripe.down_scaled_stripes[0].height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->stripe.down_scaled_stripes[0].offset = 0;
acc->stripe.effective_stripes[0].width =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].width;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].width;
acc->stripe.bds_out_stripes_no_overlap[0].width =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, F);
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width, f);
acc->stripe.output_stripes[0].width =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
} else { /* Two stripes */
- bds_ds = css->rect[IPU3_CSS_RECT_EFFECTIVE].width *
+ bds_ds = css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].width *
IMGU_BDS_GRANULARITY /
- css->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].width;
acc->stripe.down_scaled_stripes[0] =
acc->stripe.bds_out_stripes[0];
acc->stripe.down_scaled_stripes[1] =
acc->stripe.bds_out_stripes[1];
- if (!IS_ALIGNED(css->rect[IPU3_CSS_RECT_BDS].width, F))
- acc->stripe.down_scaled_stripes[1].width += -F +
- (css->rect[IPU3_CSS_RECT_BDS].width & (F - 1));
+ if (!IS_ALIGNED(css_pipe->rect[IPU3_CSS_RECT_BDS].width, f))
+ acc->stripe.down_scaled_stripes[1].width +=
+ (css_pipe->rect[IPU3_CSS_RECT_BDS].width
+ & (f - 1)) - f;
acc->stripe.effective_stripes[0].width = bds_ds *
acc->stripe.down_scaled_stripes[0].width /
@@ -1819,55 +1835,55 @@ static void ipu3_css_cfg_acc_stripe(struct ipu3_css *css,
acc->stripe.down_scaled_stripes[1].width /
IMGU_BDS_GRANULARITY;
acc->stripe.effective_stripes[1].height =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].height;
acc->stripe.effective_stripes[1].offset = bds_ds *
acc->stripe.down_scaled_stripes[1].offset /
IMGU_BDS_GRANULARITY;
acc->stripe.bds_out_stripes_no_overlap[0].width =
acc->stripe.bds_out_stripes_no_overlap[1].offset =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, 2 * F) / 2;
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width, 2 * f) / 2;
acc->stripe.bds_out_stripes_no_overlap[1].width =
- DIV_ROUND_UP(css->rect[IPU3_CSS_RECT_BDS].width, F) /
- 2 * F;
+ DIV_ROUND_UP(css_pipe->rect[IPU3_CSS_RECT_BDS].width, f)
+ / 2 * f;
acc->stripe.bds_out_stripes_no_overlap[1].height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->stripe.output_stripes[0].width =
- acc->stripe.down_scaled_stripes[0].width - F;
+ acc->stripe.down_scaled_stripes[0].width - f;
acc->stripe.output_stripes[1].width =
- acc->stripe.down_scaled_stripes[1].width - F;
+ acc->stripe.down_scaled_stripes[1].width - f;
acc->stripe.output_stripes[1].height =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
acc->stripe.output_stripes[1].offset =
acc->stripe.output_stripes[0].width;
}
acc->stripe.output_system_in_frame_width =
- css->rect[IPU3_CSS_RECT_GDC].width;
+ css_pipe->rect[IPU3_CSS_RECT_GDC].width;
acc->stripe.output_system_in_frame_height =
- css->rect[IPU3_CSS_RECT_GDC].height;
+ css_pipe->rect[IPU3_CSS_RECT_GDC].height;
acc->stripe.effective_frame_width =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].width;
- acc->stripe.bds_frame_width = css->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].width;
+ acc->stripe.bds_frame_width = css_pipe->rect[IPU3_CSS_RECT_BDS].width;
acc->stripe.out_frame_width =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
acc->stripe.out_frame_height =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
acc->stripe.gdc_in_buffer_width =
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline /
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline /
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel;
acc->stripe.gdc_in_buffer_height =
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
acc->stripe.gdc_in_buffer_offset_x = IMGU_GDC_BUF_X;
acc->stripe.gdc_in_buffer_offset_y = IMGU_GDC_BUF_Y;
acc->stripe.display_frame_width =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
acc->stripe.display_frame_height =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
acc->stripe.bds_aligned_frame_width =
- roundup(css->rect[IPU3_CSS_RECT_BDS].width,
+ roundup(css_pipe->rect[IPU3_CSS_RECT_BDS].width,
2 * IPU3_UAPI_ISP_VEC_ELEMS);
if (stripes > 1)
@@ -1878,13 +1894,15 @@ static void ipu3_css_cfg_acc_stripe(struct ipu3_css *css,
}
static void ipu3_css_cfg_acc_dvs(struct ipu3_css *css,
- struct imgu_abi_acc_param *acc)
+ struct imgu_abi_acc_param *acc,
+ unsigned int pipe)
{
unsigned int i;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
/* Disable DVS statistics */
acc->dvs_stat.operations_data.process_lines_data[0].lines =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->dvs_stat.operations_data.process_lines_data[0].cfg_set = 0;
acc->dvs_stat.operations_data.ops[0].op_type =
IMGU_ABI_ACC_OPTYPE_PROCESS_LINES;
@@ -1896,8 +1914,10 @@ static void ipu3_css_cfg_acc_dvs(struct ipu3_css *css,
static void acc_bds_per_stripe_data(struct ipu3_css *css,
struct imgu_abi_acc_param *acc,
- const int i)
+ const int i, unsigned int pipe)
{
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+
acc->bds.per_stripe.aligned_data[i].data.crop.hor_crop_en = 0;
acc->bds.per_stripe.aligned_data[i].data.crop.hor_crop_start = 0;
acc->bds.per_stripe.aligned_data[i].data.crop.hor_crop_end = 0;
@@ -1908,7 +1928,7 @@ static void acc_bds_per_stripe_data(struct ipu3_css *css,
acc->bds.per_stripe.aligned_data[i].data.ver_ctrl1.out_frame_width =
acc->stripe.down_scaled_stripes[i].width;
acc->bds.per_stripe.aligned_data[i].data.ver_ctrl1.out_frame_height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
}
/*
@@ -1917,19 +1937,21 @@ static void acc_bds_per_stripe_data(struct ipu3_css *css,
* telling which fields to take from the old values (or generate if it is NULL)
* and which to take from the new user values.
*/
-int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+int ipu3_css_cfg_acc(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_uapi_flags *use,
struct imgu_abi_acc_param *acc,
struct imgu_abi_acc_param *acc_old,
struct ipu3_uapi_acc_param *acc_user)
{
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
const struct imgu_fw_info *bi =
- &css->fwp->binary_header[css->current_binary];
+ &css->fwp->binary_header[css_pipe->bindex];
const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
const unsigned int tnr_frame_width =
acc->stripe.bds_aligned_frame_width;
const unsigned int min_overlap = 10;
const struct v4l2_pix_format_mplane *pixm =
- &css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix;
+ &css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix;
const struct ipu3_css_bds_config *cfg_bds;
struct imgu_abi_input_feeder_data *feeder_data;
@@ -1938,21 +1960,21 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
/* Update stripe using chroma and luma */
- ipu3_css_cfg_acc_stripe(css, acc);
+ ipu3_css_cfg_acc_stripe(css, pipe, acc);
/* acc_param: input_feeder_config */
ofs_x = ((pixm->width -
- css->rect[IPU3_CSS_RECT_EFFECTIVE].width) >> 1) & ~1;
- ofs_x += css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].width) >> 1) & ~1;
+ ofs_x += css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
IMGU_ABI_BAYER_ORDER_RGGB ||
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
IMGU_ABI_BAYER_ORDER_GBRG ? 1 : 0;
ofs_y = ((pixm->height -
- css->rect[IPU3_CSS_RECT_EFFECTIVE].height) >> 1) & ~1;
- ofs_y += css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].height) >> 1) & ~1;
+ ofs_y += css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
IMGU_ABI_BAYER_ORDER_BGGR ||
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order ==
IMGU_ABI_BAYER_ORDER_GBRG ? 1 : 0;
acc->input_feeder.data.row_stride = pixm->plane_fmt[0].bytesperline;
acc->input_feeder.data.start_row_address =
@@ -2108,11 +2130,11 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->shd.shd.grid.grid_height_per_slice;
if (ipu3_css_shd_ops_calc(&acc->shd.shd_ops, &acc->shd.shd.grid,
- css->rect[IPU3_CSS_RECT_BDS].height))
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height))
return -EINVAL;
/* acc_param: dvs_stat_config */
- ipu3_css_cfg_acc_dvs(css, acc);
+ ipu3_css_cfg_acc_dvs(css, acc, pipe);
/* acc_param: yuvp1_iefd_config */
@@ -2254,8 +2276,8 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
/* acc_param: bds_config */
- bds_ds = (css->rect[IPU3_CSS_RECT_EFFECTIVE].height *
- IMGU_BDS_GRANULARITY) / css->rect[IPU3_CSS_RECT_BDS].height;
+ bds_ds = (css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].height *
+ IMGU_BDS_GRANULARITY) / css_pipe->rect[IPU3_CSS_RECT_BDS].height;
if (bds_ds < IMGU_BDS_MIN_SF_INV ||
bds_ds - IMGU_BDS_MIN_SF_INV >= ARRAY_SIZE(ipu3_css_bds_configs))
return -EINVAL;
@@ -2270,11 +2292,11 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->bds.hor.hor_ctrl0.min_clip_val = IMGU_BDS_MIN_CLIP_VAL;
acc->bds.hor.hor_ctrl0.max_clip_val = IMGU_BDS_MAX_CLIP_VAL;
acc->bds.hor.hor_ctrl0.out_frame_width =
- css->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].width;
acc->bds.hor.hor_ptrn_arr = cfg_bds->ptrn_arr;
acc->bds.hor.hor_phase_arr = cfg_bds->hor_phase_arr;
acc->bds.hor.hor_ctrl2.input_frame_height =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].height;
acc->bds.ver.ver_ctrl0.min_clip_val = IMGU_BDS_MIN_CLIP_VAL;
acc->bds.ver.ver_ctrl0.max_clip_val = IMGU_BDS_MAX_CLIP_VAL;
acc->bds.ver.ver_ctrl0.sample_patrn_length =
@@ -2283,11 +2305,11 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->bds.ver.ver_ptrn_arr = cfg_bds->ptrn_arr;
acc->bds.ver.ver_phase_arr = cfg_bds->ver_phase_arr;
acc->bds.ver.ver_ctrl1.out_frame_width =
- css->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].width;
acc->bds.ver.ver_ctrl1.out_frame_height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
for (i = 0; i < stripes; i++)
- acc_bds_per_stripe_data(css, acc, i);
+ acc_bds_per_stripe_data(css, acc, i, pipe);
acc->bds.enabled = cfg_bds->hor_ds_en || cfg_bds->ver_ds_en;
@@ -2317,15 +2339,15 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->anr.transform.enable = 1;
acc->anr.tile2strm.enable = 1;
acc->anr.tile2strm.frame_width =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
acc->anr.search.frame_width = acc->anr.tile2strm.frame_width;
acc->anr.stitch.frame_width = acc->anr.tile2strm.frame_width;
- acc->anr.tile2strm.frame_height = css->rect[IPU3_CSS_RECT_BDS].height;
+ acc->anr.tile2strm.frame_height = css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->anr.search.frame_height = acc->anr.tile2strm.frame_height;
acc->anr.stitch.frame_height = acc->anr.tile2strm.frame_height;
- width = ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
- height = css->rect[IPU3_CSS_RECT_BDS].height;
+ width = ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
+ height = css_pipe->rect[IPU3_CSS_RECT_BDS].height;
if (acc->anr.transform.xreset + width > IPU3_UAPI_ANR_MAX_RESET)
acc->anr.transform.xreset = IPU3_UAPI_ANR_MAX_RESET - width;
@@ -2409,7 +2431,7 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->awb_fr.stripes[i].grid_cfg.height_per_slice = 1;
}
- if (ipu3_css_awb_fr_ops_calc(css, &acc->awb_fr))
+ if (ipu3_css_awb_fr_ops_calc(css, pipe, &acc->awb_fr))
return -EINVAL;
/* acc_param: ae_config */
@@ -2511,9 +2533,9 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->af.config.grid_cfg.height_per_slice =
IMGU_ABI_AF_MAX_CELLS_PER_SET / acc->af.config.grid_cfg.width;
acc->af.config.frame_size.width =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width, IMGU_ISP_VMEM_ALIGN);
acc->af.config.frame_size.height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
if (acc->stripe.bds_out_stripes[0].width <= min_overlap)
return -EINVAL;
@@ -2521,7 +2543,7 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
for (i = 0; i < stripes; i++) {
acc->af.stripes[i].grid_cfg = acc->af.config.grid_cfg;
acc->af.stripes[i].frame_size.height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
acc->af.stripes[i].frame_size.width =
acc->stripe.bds_out_stripes[i].width;
}
@@ -2572,7 +2594,7 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->af.stripes[i].grid_cfg.height_per_slice = 1;
}
- if (ipu3_css_af_ops_calc(css, &acc->af))
+ if (ipu3_css_af_ops_calc(css, pipe, &acc->af))
return -EINVAL;
/* acc_param: awb_config */
@@ -2641,7 +2663,7 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
acc->awb.stripes[i].grid.height_per_slice = 1;
}
- if (ipu3_css_awb_ops_calc(css, &acc->awb))
+ if (ipu3_css_awb_ops_calc(css, pipe, &acc->awb))
return -EINVAL;
return 0;
@@ -2656,7 +2678,8 @@ int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
* to the structure inside `new_binary_params'. In that case the caller
* should calculate and fill the structure from scratch.
*/
-static void *ipu3_css_cfg_copy(struct ipu3_css *css, bool use_user,
+static void *ipu3_css_cfg_copy(struct ipu3_css *css,
+ unsigned int pipe, bool use_user,
void *user_setting, void *old_binary_params,
void *new_binary_params,
enum imgu_abi_memories m,
@@ -2666,8 +2689,8 @@ static void *ipu3_css_cfg_copy(struct ipu3_css *css, bool use_user,
const enum imgu_abi_param_class c = IMGU_ABI_PARAM_CLASS_PARAM;
void *new_setting, *old_setting;
- new_setting = ipu3_css_fw_pipeline_params(css, c, m, par, par_size,
- new_binary_params);
+ new_setting = ipu3_css_fw_pipeline_params(css, pipe, c, m, par,
+ par_size, new_binary_params);
if (!new_setting)
return ERR_PTR(-EPROTO); /* Corrupted firmware */
@@ -2676,7 +2699,7 @@ static void *ipu3_css_cfg_copy(struct ipu3_css *css, bool use_user,
memcpy(new_setting, user_setting, par_size);
} else if (old_binary_params) {
/* Take previous value */
- old_setting = ipu3_css_fw_pipeline_params(css, c, m, par,
+ old_setting = ipu3_css_fw_pipeline_params(css, pipe, c, m, par,
par_size,
old_binary_params);
if (!old_setting)
@@ -2692,12 +2715,13 @@ static void *ipu3_css_cfg_copy(struct ipu3_css *css, bool use_user,
/*
* Configure VMEM0 parameters (late binding parameters).
*/
-int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+int ipu3_css_cfg_vmem0(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_uapi_flags *use,
void *vmem0, void *vmem0_old,
struct ipu3_uapi_params *user)
{
const struct imgu_fw_info *bi =
- &css->fwp->binary_header[css->current_binary];
+ &css->fwp->binary_header[css->pipes[pipe].bindex];
struct imgu_fw_param_memory_offsets *pofs = (void *)css->fwp +
bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_PARAM];
struct ipu3_uapi_isp_lin_vmem_params *lin_vmem = NULL;
@@ -2713,7 +2737,7 @@ int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
/* Configure Linearization VMEM0 parameters */
- lin_vmem = ipu3_css_cfg_copy(css, use && use->lin_vmem_params,
+ lin_vmem = ipu3_css_cfg_copy(css, pipe, use && use->lin_vmem_params,
&user->lin_vmem_params, vmem0_old, vmem0,
m, &pofs->vmem.lin, sizeof(*lin_vmem));
if (!IS_ERR_OR_NULL(lin_vmem)) {
@@ -2732,8 +2756,9 @@ int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
}
/* Configure TNR3 VMEM parameters */
- if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
- tnr_vmem = ipu3_css_cfg_copy(css, use && use->tnr3_vmem_params,
+ if (css->pipes[pipe].pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ tnr_vmem = ipu3_css_cfg_copy(css, pipe,
+ use && use->tnr3_vmem_params,
&user->tnr3_vmem_params,
vmem0_old, vmem0, m,
&pofs->vmem.tnr3,
@@ -2748,7 +2773,7 @@ int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
/* Configure XNR3 VMEM parameters */
- xnr_vmem = ipu3_css_cfg_copy(css, use && use->xnr3_vmem_params,
+ xnr_vmem = ipu3_css_cfg_copy(css, pipe, use && use->xnr3_vmem_params,
&user->xnr3_vmem_params, vmem0_old, vmem0,
m, &pofs->vmem.xnr3, sizeof(*xnr_vmem));
if (!IS_ERR_OR_NULL(xnr_vmem)) {
@@ -2769,12 +2794,14 @@ int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
/*
* Configure DMEM0 parameters (late binding parameters).
*/
-int ipu3_css_cfg_dmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+int ipu3_css_cfg_dmem0(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_uapi_flags *use,
void *dmem0, void *dmem0_old,
struct ipu3_uapi_params *user)
{
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
const struct imgu_fw_info *bi =
- &css->fwp->binary_header[css->current_binary];
+ &css->fwp->binary_header[css_pipe->bindex];
struct imgu_fw_param_memory_offsets *pofs = (void *)css->fwp +
bi->blob.memory_offsets.offsets[IMGU_ABI_PARAM_CLASS_PARAM];
@@ -2789,10 +2816,12 @@ int ipu3_css_cfg_dmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
memset(dmem0, 0, bi->info.isp.sp.mem_initializers.params[c][m].size);
/* Configure TNR3 DMEM0 parameters */
- if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
- tnr_dmem = ipu3_css_cfg_copy(css, use && use->tnr3_dmem_params,
- &user->tnr3_dmem_params, dmem0_old,
- dmem0, m, &pofs->dmem.tnr3,
+ if (css_pipe->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ tnr_dmem = ipu3_css_cfg_copy(css, pipe,
+ use && use->tnr3_dmem_params,
+ &user->tnr3_dmem_params,
+ dmem0_old, dmem0, m,
+ &pofs->dmem.tnr3,
sizeof(*tnr_dmem));
if (!IS_ERR_OR_NULL(tnr_dmem)) {
/* Generate parameter from scratch */
@@ -2803,7 +2832,7 @@ int ipu3_css_cfg_dmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
/* Configure XNR3 DMEM0 parameters */
- xnr_dmem = ipu3_css_cfg_copy(css, use && use->xnr3_dmem_params,
+ xnr_dmem = ipu3_css_cfg_copy(css, pipe, use && use->xnr3_dmem_params,
&user->xnr3_dmem_params, dmem0_old, dmem0,
m, &pofs->dmem.xnr3, sizeof(*xnr_dmem));
if (!IS_ERR_OR_NULL(xnr_dmem)) {
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-params.h b/drivers/media/pci/intel/ipu3/ipu3-css-params.h
index f93ed027..f3a0a47 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css-params.h
+++ b/drivers/media/pci/intel/ipu3/ipu3-css-params.h
@@ -4,16 +4,19 @@
#ifndef __IPU3_PARAMS_H
#define __IPU3_PARAMS_H
-int ipu3_css_cfg_acc(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+int ipu3_css_cfg_acc(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_uapi_flags *use,
struct imgu_abi_acc_param *acc,
struct imgu_abi_acc_param *acc_old,
struct ipu3_uapi_acc_param *acc_user);
-int ipu3_css_cfg_vmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+int ipu3_css_cfg_vmem0(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_uapi_flags *use,
void *vmem0, void *vmem0_old,
struct ipu3_uapi_params *user);
-int ipu3_css_cfg_dmem0(struct ipu3_css *css, struct ipu3_uapi_flags *use,
+int ipu3_css_cfg_dmem0(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_uapi_flags *use,
void *dmem0, void *dmem0_old,
struct ipu3_uapi_params *user);
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css.c b/drivers/media/pci/intel/ipu3/ipu3-css.c
index c63b387..753913c 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-css.c
@@ -659,27 +659,28 @@ static void ipu3_css_hw_cleanup(struct ipu3_css *css)
usleep_range(200, 300);
}
-static void ipu3_css_pipeline_cleanup(struct ipu3_css *css)
+static void ipu3_css_pipeline_cleanup(struct ipu3_css *css, unsigned int pipe)
{
struct imgu_device *imgu = dev_get_drvdata(css->dev);
unsigned int i;
- ipu3_css_pool_cleanup(imgu, &css->pool.parameter_set_info);
- ipu3_css_pool_cleanup(imgu, &css->pool.acc);
- ipu3_css_pool_cleanup(imgu, &css->pool.gdc);
- ipu3_css_pool_cleanup(imgu, &css->pool.obgrid);
+ ipu3_css_pool_cleanup(imgu,
+ &css->pipes[pipe].pool.parameter_set_info);
+ ipu3_css_pool_cleanup(imgu, &css->pipes[pipe].pool.acc);
+ ipu3_css_pool_cleanup(imgu, &css->pipes[pipe].pool.gdc);
+ ipu3_css_pool_cleanup(imgu, &css->pipes[pipe].pool.obgrid);
for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
- ipu3_css_pool_cleanup(imgu, &css->pool.binary_params_p[i]);
+ ipu3_css_pool_cleanup(imgu,
+ &css->pipes[pipe].pool.binary_params_p[i]);
}
/*
* This function initializes various stages of the
* IPU3 CSS ISP pipeline
*/
-static int ipu3_css_pipeline_init(struct ipu3_css *css)
+static int ipu3_css_pipeline_init(struct ipu3_css *css, unsigned int pipe)
{
- static const unsigned int PIPE_ID = IPU3_CSS_PIPE_ID_VIDEO;
static const int BYPC = 2; /* Bytes per component */
static const struct imgu_abi_buffer_sp buffer_sp_init = {
.buf_src = {.queue_id = IMGU_ABI_QUEUE_EVENT_ID},
@@ -693,11 +694,12 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
struct imgu_abi_isp_ref_dmem_state *cfg_ref_state;
struct imgu_abi_isp_tnr3_dmem_state *cfg_tnr_state;
- const int pipe = 0, stage = 0, thread = 0;
+ const int stage = 0;
unsigned int i, j;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
const struct imgu_fw_info *bi =
- &css->fwp->binary_header[css->current_binary];
+ &css->fwp->binary_header[css_pipe->bindex];
const unsigned int stripes = bi->info.isp.sp.iterator.num_stripes;
struct imgu_fw_config_memory_offsets *cofs = (void *)css->fwp +
@@ -710,103 +712,107 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
struct imgu_abi_sp_group *sp_group;
const unsigned int bds_width_pad =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].width,
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width,
2 * IPU3_UAPI_ISP_VEC_ELEMS);
const enum imgu_abi_memories m0 = IMGU_ABI_MEM_ISP_DMEM0;
enum imgu_abi_param_class cfg = IMGU_ABI_PARAM_CLASS_CONFIG;
- void *vaddr = css->binary_params_cs[cfg - 1][m0].vaddr;
+ void *vaddr = css_pipe->binary_params_cs[cfg - 1][m0].vaddr;
struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ dev_dbg(css->dev, "%s for pipe %d", __func__, pipe);
+
/* Configure iterator */
- cfg_iter = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ cfg_iter = ipu3_css_fw_pipeline_params(css, pipe, cfg, m0,
&cofs->dmem.iterator,
sizeof(*cfg_iter), vaddr);
if (!cfg_iter)
goto bad_firmware;
cfg_iter->input_info.res.width =
- css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
cfg_iter->input_info.res.height =
- css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
cfg_iter->input_info.padded_width =
- css->queue[IPU3_CSS_QUEUE_IN].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].width_pad;
cfg_iter->input_info.format =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->frame_format;
cfg_iter->input_info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bit_depth;
cfg_iter->input_info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
cfg_iter->input_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
- cfg_iter->internal_info.res.width = css->rect[IPU3_CSS_RECT_BDS].width;
+ cfg_iter->internal_info.res.width = css_pipe->rect[IPU3_CSS_RECT_BDS].width;
cfg_iter->internal_info.res.height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
cfg_iter->internal_info.padded_width = bds_width_pad;
cfg_iter->internal_info.format =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
cfg_iter->internal_info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
cfg_iter->internal_info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
cfg_iter->internal_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
cfg_iter->output_info.res.width =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
cfg_iter->output_info.res.height =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
cfg_iter->output_info.padded_width =
- css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].width_pad;
cfg_iter->output_info.format =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
cfg_iter->output_info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
cfg_iter->output_info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
cfg_iter->output_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
cfg_iter->vf_info.res.width =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
cfg_iter->vf_info.res.height =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
cfg_iter->vf_info.padded_width =
- css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].width_pad;
cfg_iter->vf_info.format =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
cfg_iter->vf_info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->bit_depth;
cfg_iter->vf_info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->bayer_order;
cfg_iter->vf_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
- cfg_iter->dvs_envelope.width = css->rect[IPU3_CSS_RECT_ENVELOPE].width;
+ cfg_iter->dvs_envelope.width = css_pipe->rect[IPU3_CSS_RECT_ENVELOPE].width;
cfg_iter->dvs_envelope.height =
- css->rect[IPU3_CSS_RECT_ENVELOPE].height;
+ css_pipe->rect[IPU3_CSS_RECT_ENVELOPE].height;
/* Configure reference (delay) frames */
- cfg_ref = ipu3_css_fw_pipeline_params(css, cfg, m0, &cofs->dmem.ref,
+ cfg_ref = ipu3_css_fw_pipeline_params(css, pipe, cfg, m0,
+ &cofs->dmem.ref,
sizeof(*cfg_ref), vaddr);
if (!cfg_ref)
goto bad_firmware;
cfg_ref->port_b.crop = 0;
cfg_ref->port_b.elems = IMGU_ABI_ISP_DDR_WORD_BYTES / BYPC;
- cfg_ref->port_b.width = css->aux_frames[IPU3_CSS_AUX_FRAME_REF].width;
+ cfg_ref->port_b.width =
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].width;
cfg_ref->port_b.stride =
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline;
cfg_ref->width_a_over_b =
IPU3_UAPI_ISP_VEC_ELEMS / cfg_ref->port_b.elems;
cfg_ref->dvs_frame_delay = IPU3_CSS_AUX_FRAMES - 1;
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++) {
cfg_ref->ref_frame_addr_y[i] =
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i].daddr;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i].daddr;
cfg_ref->ref_frame_addr_c[i] =
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i].daddr +
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline *
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i].daddr +
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline *
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
}
for (; i < IMGU_ABI_FRAMES_REF; i++) {
cfg_ref->ref_frame_addr_y[i] = 0;
@@ -815,23 +821,23 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
/* Configure DVS (digital video stabilization) */
- cfg_dvs = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ cfg_dvs = ipu3_css_fw_pipeline_params(css, pipe, cfg, m0,
&cofs->dmem.dvs, sizeof(*cfg_dvs),
vaddr);
if (!cfg_dvs)
goto bad_firmware;
cfg_dvs->num_horizontal_blocks =
- ALIGN(DIV_ROUND_UP(css->rect[IPU3_CSS_RECT_GDC].width,
+ ALIGN(DIV_ROUND_UP(css_pipe->rect[IPU3_CSS_RECT_GDC].width,
IMGU_DVS_BLOCK_W), 2);
cfg_dvs->num_vertical_blocks =
- DIV_ROUND_UP(css->rect[IPU3_CSS_RECT_GDC].height,
+ DIV_ROUND_UP(css_pipe->rect[IPU3_CSS_RECT_GDC].height,
IMGU_DVS_BLOCK_H);
/* Configure TNR (temporal noise reduction) */
- if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
- cfg_tnr = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ if (css_pipe->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ cfg_tnr = ipu3_css_fw_pipeline_params(css, pipe, cfg, m0,
&cofs->dmem.tnr3,
sizeof(*cfg_tnr),
vaddr);
@@ -841,17 +847,17 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
cfg_tnr->port_b.crop = 0;
cfg_tnr->port_b.elems = IMGU_ABI_ISP_DDR_WORD_BYTES;
cfg_tnr->port_b.width =
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width;
cfg_tnr->port_b.stride =
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperline;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperline;
cfg_tnr->width_a_over_b =
- IPU3_UAPI_ISP_VEC_ELEMS / cfg_tnr->port_b.elems;
+ IPU3_UAPI_ISP_VEC_ELEMS / cfg_tnr->port_b.elems;
cfg_tnr->frame_height =
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height;
cfg_tnr->delay_frame = IPU3_CSS_AUX_FRAMES - 1;
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
cfg_tnr->frame_addr[i] =
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR]
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR]
.mem[i].daddr;
for (; i < IMGU_ABI_FRAMES_TNR; i++)
cfg_tnr->frame_addr[i] = 0;
@@ -860,9 +866,9 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
/* Configure ref dmem state parameters */
cfg = IMGU_ABI_PARAM_CLASS_STATE;
- vaddr = css->binary_params_cs[cfg - 1][m0].vaddr;
+ vaddr = css_pipe->binary_params_cs[cfg - 1][m0].vaddr;
- cfg_ref_state = ipu3_css_fw_pipeline_params(css, cfg, m0,
+ cfg_ref_state = ipu3_css_fw_pipeline_params(css, pipe, cfg, m0,
&sofs->dmem.ref,
sizeof(*cfg_ref_state),
vaddr);
@@ -873,9 +879,9 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
cfg_ref_state->ref_out_buf_idx = 1;
/* Configure tnr dmem state parameters */
- if (css->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
+ if (css_pipe->pipe_id == IPU3_CSS_PIPE_ID_VIDEO) {
cfg_tnr_state =
- ipu3_css_fw_pipeline_params(css, cfg, m0,
+ ipu3_css_fw_pipeline_params(css, pipe, cfg, m0,
&sofs->dmem.tnr3,
sizeof(*cfg_tnr_state),
vaddr);
@@ -892,22 +898,22 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
/* Configure ISP stage */
- isp_stage = css->xmem_isp_stage_ptrs[pipe][stage].vaddr;
+ isp_stage = css_pipe->xmem_isp_stage_ptrs[pipe][stage].vaddr;
memset(isp_stage, 0, sizeof(*isp_stage));
isp_stage->blob_info = bi->blob;
isp_stage->binary_info = bi->info.isp.sp;
strlcpy(isp_stage->binary_name,
- (char *)css->fwp + bi->blob.prog_name_offset,
- sizeof(isp_stage->binary_name));
+ (char *)css->fwp + bi->blob.prog_name_offset,
+ sizeof(isp_stage->binary_name));
isp_stage->mem_initializers = bi->info.isp.sp.mem_initializers;
for (i = IMGU_ABI_PARAM_CLASS_CONFIG; i < IMGU_ABI_PARAM_CLASS_NUM; i++)
for (j = 0; j < IMGU_ABI_NUM_MEMORIES; j++)
isp_stage->mem_initializers.params[i][j].address =
- css->binary_params_cs[i - 1][j].daddr;
+ css_pipe->binary_params_cs[i - 1][j].daddr;
/* Configure SP stage */
- sp_stage = css->xmem_sp_stage_ptrs[pipe][stage].vaddr;
+ sp_stage = css_pipe->xmem_sp_stage_ptrs[pipe][stage].vaddr;
memset(sp_stage, 0, sizeof(*sp_stage));
sp_stage->frames.in.buf_attr = buffer_sp_init;
@@ -923,48 +929,45 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
sp_stage->isp_copy_vf = 0;
sp_stage->isp_copy_output = 0;
- /* Enable VF output only when VF or PV queue requested by user */
-
- sp_stage->enable.vf_output =
- (css->vf_output_en != IPU3_NODE_VF_DISABLED);
+ sp_stage->enable.vf_output = css_pipe->vf_output_en;
sp_stage->frames.effective_in_res.width =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].width;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].width;
sp_stage->frames.effective_in_res.height =
- css->rect[IPU3_CSS_RECT_EFFECTIVE].height;
+ css_pipe->rect[IPU3_CSS_RECT_EFFECTIVE].height;
sp_stage->frames.in.info.res.width =
- css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.width;
sp_stage->frames.in.info.res.height =
- css->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].fmt.mpix.height;
sp_stage->frames.in.info.padded_width =
- css->queue[IPU3_CSS_QUEUE_IN].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].width_pad;
sp_stage->frames.in.info.format =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->frame_format;
sp_stage->frames.in.info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bit_depth;
sp_stage->frames.in.info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_IN].css_fmt->bayer_order;
sp_stage->frames.in.info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
sp_stage->frames.in.buf_attr.buf_src.queue_id = IMGU_ABI_QUEUE_C_ID;
sp_stage->frames.in.buf_attr.buf_type =
IMGU_ABI_BUFFER_TYPE_INPUT_FRAME;
sp_stage->frames.out[0].info.res.width =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.width;
sp_stage->frames.out[0].info.res.height =
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
sp_stage->frames.out[0].info.padded_width =
- css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].width_pad;
sp_stage->frames.out[0].info.format =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
sp_stage->frames.out[0].info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
sp_stage->frames.out[0].info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
sp_stage->frames.out[0].info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
sp_stage->frames.out[0].planes.nv.uv.offset =
- css->queue[IPU3_CSS_QUEUE_OUT].width_pad *
- css->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].width_pad *
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].fmt.mpix.height;
sp_stage->frames.out[0].buf_attr.buf_src.queue_id = IMGU_ABI_QUEUE_D_ID;
sp_stage->frames.out[0].buf_attr.buf_type =
IMGU_ABI_BUFFER_TYPE_OUTPUT_FRAME;
@@ -973,38 +976,38 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
IMGU_ABI_QUEUE_EVENT_ID;
sp_stage->frames.internal_frame_info.res.width =
- css->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].width;
sp_stage->frames.internal_frame_info.res.height =
- css->rect[IPU3_CSS_RECT_BDS].height;
+ css_pipe->rect[IPU3_CSS_RECT_BDS].height;
sp_stage->frames.internal_frame_info.padded_width = bds_width_pad;
sp_stage->frames.internal_frame_info.format =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->frame_format;
sp_stage->frames.internal_frame_info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bit_depth;
sp_stage->frames.internal_frame_info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].css_fmt->bayer_order;
sp_stage->frames.internal_frame_info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
sp_stage->frames.out_vf.info.res.width =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.width;
sp_stage->frames.out_vf.info.res.height =
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
sp_stage->frames.out_vf.info.padded_width =
- css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].width_pad;
sp_stage->frames.out_vf.info.format =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->frame_format;
sp_stage->frames.out_vf.info.raw_bit_depth =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bit_depth;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->bit_depth;
sp_stage->frames.out_vf.info.raw_bayer_order =
- css->queue[IPU3_CSS_QUEUE_VF].css_fmt->bayer_order;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].css_fmt->bayer_order;
sp_stage->frames.out_vf.info.raw_type = IMGU_ABI_RAW_TYPE_BAYER;
sp_stage->frames.out_vf.planes.yuv.u.offset =
- css->queue[IPU3_CSS_QUEUE_VF].width_pad *
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].width_pad *
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height;
sp_stage->frames.out_vf.planes.yuv.v.offset =
- css->queue[IPU3_CSS_QUEUE_VF].width_pad *
- css->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height * 5 / 4;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].width_pad *
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].fmt.mpix.height * 5 / 4;
sp_stage->frames.out_vf.buf_attr.buf_src.queue_id = IMGU_ABI_QUEUE_E_ID;
sp_stage->frames.out_vf.buf_attr.buf_type =
IMGU_ABI_BUFFER_TYPE_VF_OUTPUT_FRAME;
@@ -1015,16 +1018,16 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
sp_stage->frames.dvs_buf.buf_src.queue_id = IMGU_ABI_QUEUE_G_ID;
sp_stage->frames.dvs_buf.buf_type = IMGU_ABI_BUFFER_TYPE_DIS_STATISTICS;
- sp_stage->dvs_envelope.width = css->rect[IPU3_CSS_RECT_ENVELOPE].width;
+ sp_stage->dvs_envelope.width = css_pipe->rect[IPU3_CSS_RECT_ENVELOPE].width;
sp_stage->dvs_envelope.height =
- css->rect[IPU3_CSS_RECT_ENVELOPE].height;
+ css_pipe->rect[IPU3_CSS_RECT_ENVELOPE].height;
sp_stage->isp_pipe_version =
bi->info.isp.sp.pipeline.isp_pipe_version;
sp_stage->isp_deci_log_factor =
- clamp(max(fls(css->rect[IPU3_CSS_RECT_BDS].width /
+ clamp(max(fls(css_pipe->rect[IPU3_CSS_RECT_BDS].width /
IMGU_MAX_BQ_GRID_WIDTH),
- fls(css->rect[IPU3_CSS_RECT_BDS].height /
+ fls(css_pipe->rect[IPU3_CSS_RECT_BDS].height /
IMGU_MAX_BQ_GRID_HEIGHT)) - 1, 3, 5);
sp_stage->isp_vf_downscale_bits = 0;
sp_stage->if_config_index = 255;
@@ -1033,52 +1036,54 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
sp_stage->enable.s3a = 1;
sp_stage->enable.dvs_stats = 0;
- sp_stage->xmem_bin_addr = css->binary[css->current_binary].daddr;
- sp_stage->xmem_map_addr = css->sp_ddr_ptrs.daddr;
- sp_stage->isp_stage_addr = css->xmem_isp_stage_ptrs[pipe][stage].daddr;
+ sp_stage->xmem_bin_addr = css->binary[css_pipe->bindex].daddr;
+ sp_stage->xmem_map_addr = css_pipe->sp_ddr_ptrs.daddr;
+ sp_stage->isp_stage_addr =
+ css_pipe->xmem_isp_stage_ptrs[pipe][stage].daddr;
/* Configure SP group */
sp_group = css->xmem_sp_group_ptrs.vaddr;
- memset(sp_group, 0, sizeof(*sp_group));
-
- sp_group->pipe[thread].num_stages = 1;
- sp_group->pipe[thread].pipe_id = PIPE_ID;
- sp_group->pipe[thread].thread_id = thread;
- sp_group->pipe[thread].pipe_num = pipe;
- sp_group->pipe[thread].num_execs = -1;
- sp_group->pipe[thread].pipe_qos_config = -1;
- sp_group->pipe[thread].required_bds_factor = 0;
- sp_group->pipe[thread].dvs_frame_delay = IPU3_CSS_AUX_FRAMES - 1;
- sp_group->pipe[thread].inout_port_config =
+ memset(&sp_group->pipe[pipe], 0, sizeof(struct imgu_abi_sp_pipeline));
+
+ sp_group->pipe[pipe].num_stages = 1;
+ sp_group->pipe[pipe].pipe_id = css_pipe->pipe_id;
+ sp_group->pipe[pipe].thread_id = pipe;
+ sp_group->pipe[pipe].pipe_num = pipe;
+ sp_group->pipe[pipe].num_execs = -1;
+ sp_group->pipe[pipe].pipe_qos_config = -1;
+ sp_group->pipe[pipe].required_bds_factor = 0;
+ sp_group->pipe[pipe].dvs_frame_delay = IPU3_CSS_AUX_FRAMES - 1;
+ sp_group->pipe[pipe].inout_port_config =
IMGU_ABI_PORT_CONFIG_TYPE_INPUT_HOST |
IMGU_ABI_PORT_CONFIG_TYPE_OUTPUT_HOST;
- sp_group->pipe[thread].scaler_pp_lut = 0;
- sp_group->pipe[thread].shading.internal_frame_origin_x_bqs_on_sctbl = 0;
- sp_group->pipe[thread].shading.internal_frame_origin_y_bqs_on_sctbl = 0;
- sp_group->pipe[thread].sp_stage_addr[stage] =
- css->xmem_sp_stage_ptrs[pipe][stage].daddr;
- sp_group->pipe[thread].pipe_config =
- bi->info.isp.sp.enable.params ? (1 << thread) : 0;
- sp_group->pipe[thread].pipe_config |= IMGU_ABI_PIPE_CONFIG_ACQUIRE_ISP;
+ sp_group->pipe[pipe].scaler_pp_lut = 0;
+ sp_group->pipe[pipe].shading.internal_frame_origin_x_bqs_on_sctbl = 0;
+ sp_group->pipe[pipe].shading.internal_frame_origin_y_bqs_on_sctbl = 0;
+ sp_group->pipe[pipe].sp_stage_addr[stage] =
+ css_pipe->xmem_sp_stage_ptrs[pipe][stage].daddr;
+ sp_group->pipe[pipe].pipe_config =
+ bi->info.isp.sp.enable.params ? (1 << pipe) : 0;
+ sp_group->pipe[pipe].pipe_config |= IMGU_ABI_PIPE_CONFIG_ACQUIRE_ISP;
/* Initialize parameter pools */
- if (ipu3_css_pool_init(imgu, &css->pool.parameter_set_info,
+ if (ipu3_css_pool_init(imgu, &css_pipe->pool.parameter_set_info,
sizeof(struct imgu_abi_parameter_set_info)) ||
- ipu3_css_pool_init(imgu, &css->pool.acc,
+ ipu3_css_pool_init(imgu, &css_pipe->pool.acc,
sizeof(struct imgu_abi_acc_param)) ||
- ipu3_css_pool_init(imgu, &css->pool.gdc,
+ ipu3_css_pool_init(imgu, &css_pipe->pool.gdc,
sizeof(struct imgu_abi_gdc_warp_param) *
3 * cfg_dvs->num_horizontal_blocks / 2 *
cfg_dvs->num_vertical_blocks) ||
- ipu3_css_pool_init(imgu, &css->pool.obgrid,
+ ipu3_css_pool_init(imgu, &css_pipe->pool.obgrid,
ipu3_css_fw_obgrid_size(
- &css->fwp->binary_header[css->current_binary])))
+ &css->fwp->binary_header[css_pipe->bindex])))
goto out_of_memory;
for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
- if (ipu3_css_pool_init(imgu, &css->pool.binary_params_p[i],
+ if (ipu3_css_pool_init(imgu,
+ &css_pipe->pool.binary_params_p[i],
bi->info.isp.sp.mem_initializers.params
[IMGU_ABI_PARAM_CLASS_PARAM][i].size))
goto out_of_memory;
@@ -1086,11 +1091,11 @@ static int ipu3_css_pipeline_init(struct ipu3_css *css)
return 0;
bad_firmware:
- ipu3_css_pipeline_cleanup(css);
+ ipu3_css_pipeline_cleanup(css, pipe);
return -EPROTO;
out_of_memory:
- ipu3_css_pipeline_cleanup(css);
+ ipu3_css_pipeline_cleanup(css, pipe);
return -ENOMEM;
}
@@ -1193,134 +1198,147 @@ static int ipu3_css_dequeue_data(struct ipu3_css *css, int queue, u32 *data)
}
/* Free binary-specific resources */
-static void ipu3_css_binary_cleanup(struct ipu3_css *css)
+static void ipu3_css_binary_cleanup(struct ipu3_css *css, unsigned int pipe)
{
struct imgu_device *imgu = dev_get_drvdata(css->dev);
unsigned int i, j;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+
for (j = 0; j < IMGU_ABI_PARAM_CLASS_NUM - 1; j++)
for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
- ipu3_dmamap_free(imgu, &css->binary_params_cs[j][i]);
+ ipu3_dmamap_free(imgu,
+ &css_pipe->binary_params_cs[j][i]);
j = IPU3_CSS_AUX_FRAME_REF;
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
- ipu3_dmamap_free(imgu, &css->aux_frames[j].mem[i]);
+ ipu3_dmamap_free(imgu,
+ &css_pipe->aux_frames[j].mem[i]);
j = IPU3_CSS_AUX_FRAME_TNR;
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
- ipu3_dmamap_free(imgu, &css->aux_frames[j].mem[i]);
+ ipu3_dmamap_free(imgu,
+ &css_pipe->aux_frames[j].mem[i]);
}
-static int ipu3_css_binary_preallocate(struct ipu3_css *css)
+static int ipu3_css_binary_preallocate(struct ipu3_css *css, unsigned int pipe)
{
struct imgu_device *imgu = dev_get_drvdata(css->dev);
unsigned int i, j;
- for (j = IMGU_ABI_PARAM_CLASS_CONFIG; j < IMGU_ABI_PARAM_CLASS_NUM; j++)
- for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++) {
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+
+ for (j = IMGU_ABI_PARAM_CLASS_CONFIG;
+ j < IMGU_ABI_PARAM_CLASS_NUM; j++)
+ for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++)
if (!ipu3_dmamap_alloc(imgu,
- &css->binary_params_cs[j - 1][i],
- CSS_ABI_SIZE))
+ &css_pipe->binary_params_cs[j - 1][i],
+ CSS_ABI_SIZE))
goto out_of_memory;
- }
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
if (!ipu3_dmamap_alloc(imgu,
- &css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i],
- CSS_BDS_SIZE))
+ &css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].
+ mem[i], CSS_BDS_SIZE))
goto out_of_memory;
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
if (!ipu3_dmamap_alloc(imgu,
- &css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].mem[i],
- CSS_GDC_SIZE))
+ &css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].
+ mem[i], CSS_GDC_SIZE))
goto out_of_memory;
return 0;
out_of_memory:
- ipu3_css_binary_cleanup(css);
+ ipu3_css_binary_cleanup(css, pipe);
return -ENOMEM;
}
/* allocate binary-specific resources */
-static int ipu3_css_binary_setup(struct ipu3_css *css)
+static int ipu3_css_binary_setup(struct ipu3_css *css, unsigned int pipe)
{
- const struct imgu_abi_binary_info *sp =
- &css->fwp->binary_header[css->current_binary].info.isp.sp;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+ struct imgu_fw_info *bi = &css->fwp->binary_header[css_pipe->bindex];
struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ int i, j, size;
static const int BYPC = 2; /* Bytes per component */
- unsigned int w, h, size, i, j;
+ unsigned int w, h;
/* Allocate parameter memory blocks for this binary */
for (j = IMGU_ABI_PARAM_CLASS_CONFIG; j < IMGU_ABI_PARAM_CLASS_NUM; j++)
for (i = 0; i < IMGU_ABI_NUM_MEMORIES; i++) {
if (ipu3_css_dma_buffer_resize(
- imgu, &css->binary_params_cs[j - 1][i],
- sp->mem_initializers.params[j][i].size))
+ imgu,
+ &css_pipe->binary_params_cs[j - 1][i],
+ bi->info.isp.sp.mem_initializers.params[j][i].size))
goto out_of_memory;
}
/* Allocate internal frame buffers */
/* Reference frames for DVS, FRAME_FORMAT_YUV420_16 */
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel = BYPC;
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].width =
- css->rect[IPU3_CSS_RECT_BDS].width;
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height =
- ALIGN(css->rect[IPU3_CSS_RECT_BDS].height,
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel = BYPC;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].width =
+ css_pipe->rect[IPU3_CSS_RECT_BDS].width;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].height =
+ ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].height,
IMGU_DVS_BLOCK_H) + 2 * IMGU_GDC_BUF_Y;
- h = css->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
- w = ALIGN(css->rect[IPU3_CSS_RECT_BDS].width,
+ h = css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].height;
+ w = ALIGN(css_pipe->rect[IPU3_CSS_RECT_BDS].width,
2 * IPU3_UAPI_ISP_VEC_ELEMS) + 2 * IMGU_GDC_BUF_X;
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline =
- css->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel * w;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperline =
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].bytesperpixel * w;
size = w * h * BYPC + (w / 2) * (h / 2) * BYPC * 2;
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
if (ipu3_css_dma_buffer_resize(
imgu,
- &css->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i], size))
+ &css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_REF].mem[i],
+ size))
goto out_of_memory;
/* TNR frames for temporal noise reduction, FRAME_FORMAT_YUV_LINE */
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperpixel = 1;
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width =
- roundup(css->rect[IPU3_CSS_RECT_GDC].width,
- sp->block.block_width *
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperpixel = 1;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width =
+ roundup(css_pipe->rect[IPU3_CSS_RECT_GDC].width,
+ bi->info.isp.sp.block.block_width *
IPU3_UAPI_ISP_VEC_ELEMS);
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height =
- roundup(css->rect[IPU3_CSS_RECT_GDC].height,
- sp->block.output_block_height);
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height =
+ roundup(css_pipe->rect[IPU3_CSS_RECT_GDC].height,
+ bi->info.isp.sp.block.output_block_height);
- w = css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width;
- css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperline = w;
- h = css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height;
+ w = css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].width;
+ css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].bytesperline = w;
+ h = css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].height;
size = w * ALIGN(h * 3 / 2 + 3, 2); /* +3 for vf_pp prefetch */
for (i = 0; i < IPU3_CSS_AUX_FRAMES; i++)
if (ipu3_css_dma_buffer_resize(
imgu,
- &css->aux_frames[IPU3_CSS_AUX_FRAME_TNR].mem[i], size))
+ &css_pipe->aux_frames[IPU3_CSS_AUX_FRAME_TNR].mem[i],
+ size))
goto out_of_memory;
return 0;
out_of_memory:
- ipu3_css_binary_cleanup(css);
+ ipu3_css_binary_cleanup(css, pipe);
return -ENOMEM;
}
int ipu3_css_start_streaming(struct ipu3_css *css)
{
u32 data;
- int r;
+ int r, pipe;
if (css->streaming)
return -EPROTO;
- r = ipu3_css_binary_setup(css);
- if (r < 0)
- return r;
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ r = ipu3_css_binary_setup(css, pipe);
+ if (r < 0)
+ return r;
+ }
r = ipu3_css_hw_init(css);
if (r < 0)
@@ -1330,19 +1348,23 @@ int ipu3_css_start_streaming(struct ipu3_css *css)
if (r < 0)
goto fail;
- r = ipu3_css_pipeline_init(css);
- if (r < 0)
- goto fail;
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ css->pipes[pipe].frame = 0;
+ r = ipu3_css_pipeline_init(css, pipe);
+ if (r < 0)
+ goto fail;
+ }
css->streaming = true;
- css->frame = 0;
ipu3_css_hw_enable_irq(css);
/* Initialize parameters to default */
- r = ipu3_css_set_parameters(css, NULL);
- if (r < 0)
- goto fail;
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ r = ipu3_css_set_parameters(css, pipe, NULL);
+ if (r < 0)
+ goto fail;
+ }
while (!(r = ipu3_css_dequeue_data(css, IMGU_ABI_QUEUE_A_ID, &data)))
;
@@ -1354,18 +1376,23 @@ int ipu3_css_start_streaming(struct ipu3_css *css)
if (r != -EBUSY)
goto fail;
- r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0,
- IMGU_ABI_EVENT_START_STREAM);
- if (r < 0)
- goto fail;
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, pipe,
+ IMGU_ABI_EVENT_START_STREAM |
+ pipe << 16);
+ if (r < 0)
+ goto fail;
+ }
return 0;
fail:
css->streaming = false;
ipu3_css_hw_cleanup(css);
- ipu3_css_pipeline_cleanup(css);
- ipu3_css_binary_cleanup(css);
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ ipu3_css_pipeline_cleanup(css, pipe);
+ ipu3_css_binary_cleanup(css, pipe);
+ }
return r;
}
@@ -1373,13 +1400,14 @@ int ipu3_css_start_streaming(struct ipu3_css *css)
void ipu3_css_stop_streaming(struct ipu3_css *css)
{
struct ipu3_css_buffer *b, *b0;
- int q, r;
-
- r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0,
- IMGU_ABI_EVENT_STOP_STREAM);
+ int q, r, pipe;
- if (r < 0)
- dev_warn(css->dev, "failed on stop stream event\n");
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, pipe,
+ IMGU_ABI_EVENT_STOP_STREAM);
+ if (r < 0)
+ dev_warn(css->dev, "failed on stop stream event\n");
+ }
if (!css->streaming)
return;
@@ -1388,58 +1416,132 @@ void ipu3_css_stop_streaming(struct ipu3_css *css)
ipu3_css_hw_cleanup(css);
- ipu3_css_pipeline_cleanup(css);
+ for_each_set_bit(pipe, css->enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
- spin_lock(&css->qlock);
- for (q = 0; q < IPU3_CSS_QUEUES; q++)
- list_for_each_entry_safe(b, b0, &css->queue[q].bufs, list) {
- b->state = IPU3_CSS_BUFFER_FAILED;
- list_del(&b->list);
- }
- spin_unlock(&css->qlock);
+ ipu3_css_pipeline_cleanup(css, pipe);
+
+ spin_lock(&css_pipe->qlock);
+ for (q = 0; q < IPU3_CSS_QUEUES; q++)
+ list_for_each_entry_safe(b, b0,
+ &css_pipe->queue[q].bufs,
+ list) {
+ b->state = IPU3_CSS_BUFFER_FAILED;
+ list_del(&b->list);
+ }
+ spin_unlock(&css_pipe->qlock);
+ }
css->streaming = false;
}
-bool ipu3_css_queue_empty(struct ipu3_css *css)
+bool ipu3_css_pipe_queue_empty(struct ipu3_css *css, unsigned int pipe)
{
int q;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
- spin_lock(&css->qlock);
+ spin_lock(&css_pipe->qlock);
for (q = 0; q < IPU3_CSS_QUEUES; q++)
- if (!list_empty(&css->queue[q].bufs))
+ if (!list_empty(&css_pipe->queue[q].bufs))
break;
- spin_unlock(&css->qlock);
-
+ spin_unlock(&css_pipe->qlock);
return (q == IPU3_CSS_QUEUES);
}
+bool ipu3_css_queue_empty(struct ipu3_css *css)
+{
+ unsigned int pipe;
+ bool ret = 0;
+
+ for (pipe = 0; pipe < IMGU_MAX_PIPE_NUM; pipe++)
+ ret &= ipu3_css_pipe_queue_empty(css, pipe);
+
+ return ret;
+}
+
bool ipu3_css_is_streaming(struct ipu3_css *css)
{
return css->streaming;
}
-void ipu3_css_cleanup(struct ipu3_css *css)
+static int ipu3_css_map_init(struct ipu3_css *css, unsigned int pipe)
{
struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
unsigned int p, q, i;
- ipu3_css_stop_streaming(css);
- ipu3_css_binary_cleanup(css);
+ /* Allocate and map common structures with imgu hardware */
+ for (p = 0; p < IPU3_CSS_PIPE_ID_NUM; p++)
+ for (i = 0; i < IMGU_ABI_MAX_STAGES; i++) {
+ if (!ipu3_dmamap_alloc(imgu,
+ &css_pipe->
+ xmem_sp_stage_ptrs[p][i],
+ sizeof(struct imgu_abi_sp_stage)))
+ return -ENOMEM;
+ if (!ipu3_dmamap_alloc(imgu,
+ &css_pipe->
+ xmem_isp_stage_ptrs[p][i],
+ sizeof(struct imgu_abi_isp_stage)))
+ return -ENOMEM;
+ }
- for (q = 0; q < IPU3_CSS_QUEUES; q++)
- for (i = 0; i < ARRAY_SIZE(css->abi_buffers[q]); i++)
- ipu3_dmamap_free(imgu, &css->abi_buffers[q][i]);
+ if (!ipu3_dmamap_alloc(imgu, &css_pipe->sp_ddr_ptrs,
+ ALIGN(sizeof(struct imgu_abi_ddr_address_map),
+ IMGU_ABI_ISP_DDR_WORD_BYTES)))
+ return -ENOMEM;
+
+ for (q = 0; q < IPU3_CSS_QUEUES; q++) {
+ unsigned int abi_buf_num = ARRAY_SIZE(css_pipe->abi_buffers[q]);
+
+ for (i = 0; i < abi_buf_num; i++)
+ if (!ipu3_dmamap_alloc(imgu,
+ &css_pipe->abi_buffers[q][i],
+ sizeof(struct imgu_abi_buffer)))
+ return -ENOMEM;
+ }
+
+ if (ipu3_css_binary_preallocate(css, pipe)) {
+ ipu3_css_binary_cleanup(css, pipe);
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void ipu3_css_pipe_cleanup(struct ipu3_css *css, unsigned int pipe)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+ unsigned int p, q, i, abi_buf_num;
+
+ ipu3_css_binary_cleanup(css, pipe);
+
+ for (q = 0; q < IPU3_CSS_QUEUES; q++) {
+ abi_buf_num = ARRAY_SIZE(css_pipe->abi_buffers[q]);
+ for (i = 0; i < abi_buf_num; i++)
+ ipu3_dmamap_free(imgu, &css_pipe->abi_buffers[q][i]);
+ }
for (p = 0; p < IPU3_CSS_PIPE_ID_NUM; p++)
for (i = 0; i < IMGU_ABI_MAX_STAGES; i++) {
- ipu3_dmamap_free(imgu, &css->xmem_sp_stage_ptrs[p][i]);
- ipu3_dmamap_free(imgu, &css->xmem_isp_stage_ptrs[p][i]);
+ ipu3_dmamap_free(imgu,
+ &css_pipe->xmem_sp_stage_ptrs[p][i]);
+ ipu3_dmamap_free(imgu,
+ &css_pipe->xmem_isp_stage_ptrs[p][i]);
}
- ipu3_dmamap_free(imgu, &css->sp_ddr_ptrs);
- ipu3_dmamap_free(imgu, &css->xmem_sp_group_ptrs);
+ ipu3_dmamap_free(imgu, &css_pipe->sp_ddr_ptrs);
+}
+
+void ipu3_css_cleanup(struct ipu3_css *css)
+{
+ struct imgu_device *imgu = dev_get_drvdata(css->dev);
+ unsigned int pipe;
+ ipu3_css_stop_streaming(css);
+ for (pipe = 0; pipe < IMGU_MAX_PIPE_NUM; pipe++)
+ ipu3_css_pipe_cleanup(css, pipe);
+ ipu3_dmamap_free(imgu, &css->xmem_sp_group_ptrs);
ipu3_css_fw_cleanup(css);
}
@@ -1447,67 +1549,40 @@ int ipu3_css_init(struct device *dev, struct ipu3_css *css,
void __iomem *base, int length)
{
struct imgu_device *imgu = dev_get_drvdata(dev);
- int r, p, q, i;
+ int r, q, pipe;
/* Initialize main data structure */
css->dev = dev;
css->base = base;
css->iomem_length = length;
- css->current_binary = IPU3_CSS_DEFAULT_BINARY;
- css->pipe_id = IPU3_CSS_PIPE_ID_NUM;
- css->vf_output_en = IPU3_NODE_VF_DISABLED;
- spin_lock_init(&css->qlock);
- for (q = 0; q < IPU3_CSS_QUEUES; q++) {
- r = ipu3_css_queue_init(&css->queue[q], NULL, 0);
- if (r)
+ for (pipe = 0; pipe < IMGU_MAX_PIPE_NUM; pipe++) {
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+
+ css_pipe->vf_output_en = false;
+ spin_lock_init(&css_pipe->qlock);
+ css_pipe->bindex = IPU3_CSS_DEFAULT_BINARY;
+ css_pipe->pipe_id = IPU3_CSS_PIPE_ID_VIDEO;
+ for (q = 0; q < IPU3_CSS_QUEUES; q++) {
+ r = ipu3_css_queue_init(&css_pipe->queue[q], NULL, 0);
+ if (r)
+ return r;
+ }
+ r = ipu3_css_map_init(css, pipe);
+ if (r) {
+ ipu3_css_cleanup(css);
return r;
+ }
}
+ if (!ipu3_dmamap_alloc(imgu, &css->xmem_sp_group_ptrs,
+ sizeof(struct imgu_abi_sp_group)))
+ return -ENOMEM;
r = ipu3_css_fw_init(css);
if (r)
return r;
- /* Allocate and map common structures with imgu hardware */
-
- for (p = 0; p < IPU3_CSS_PIPE_ID_NUM; p++)
- for (i = 0; i < IMGU_ABI_MAX_STAGES; i++) {
- if (!ipu3_dmamap_alloc(imgu,
- &css->xmem_sp_stage_ptrs[p][i],
- sizeof(struct imgu_abi_sp_stage)))
- goto error_no_memory;
- if (!ipu3_dmamap_alloc(imgu,
- &css->xmem_isp_stage_ptrs[p][i],
- sizeof(struct imgu_abi_isp_stage)))
- goto error_no_memory;
- }
-
- if (!ipu3_dmamap_alloc(imgu, &css->sp_ddr_ptrs,
- ALIGN(sizeof(struct imgu_abi_ddr_address_map),
- IMGU_ABI_ISP_DDR_WORD_BYTES)))
- goto error_no_memory;
-
- if (!ipu3_dmamap_alloc(imgu, &css->xmem_sp_group_ptrs,
- sizeof(struct imgu_abi_sp_group)))
- goto error_no_memory;
-
- for (q = 0; q < IPU3_CSS_QUEUES; q++)
- for (i = 0; i < ARRAY_SIZE(css->abi_buffers[q]); i++)
- if (!ipu3_dmamap_alloc(imgu, &css->abi_buffers[q][i],
- sizeof(struct imgu_abi_buffer)))
- goto error_no_memory;
-
- if (ipu3_css_binary_preallocate(css))
- goto error_binary_setup;
-
return 0;
-
-error_binary_setup:
- ipu3_css_binary_cleanup(css);
-error_no_memory:
- ipu3_css_cleanup(css);
-
- return -ENOMEM;
}
static u32 ipu3_css_adjust(u32 res, u32 align)
@@ -1519,11 +1594,13 @@ static u32 ipu3_css_adjust(u32 res, u32 align)
/* Select a binary matching the required resolutions and formats */
static int ipu3_css_find_binary(struct ipu3_css *css,
+ unsigned int pipe,
struct ipu3_css_queue queue[IPU3_CSS_QUEUES],
struct v4l2_rect rects[IPU3_CSS_RECTS])
{
const int binary_nr = css->fwp->file_header.binary_nr;
- unsigned int binary_mode = (css->pipe_id == IPU3_CSS_PIPE_ID_CAPTURE) ?
+ unsigned int binary_mode =
+ (css->pipes[pipe].pipe_id == IPU3_CSS_PIPE_ID_CAPTURE) ?
IA_CSS_BINARY_MODE_PRIMARY : IA_CSS_BINARY_MODE_VIDEO;
const struct v4l2_pix_format_mplane *in =
&queue[IPU3_CSS_QUEUE_IN].fmt.mpix;
@@ -1624,7 +1701,8 @@ static int ipu3_css_find_binary(struct ipu3_css *css,
}
/* All checks passed, select the binary */
- dev_dbg(css->dev, "using binary %s\n", name);
+ dev_dbg(css->dev, "using binary %s id = %u\n", name,
+ bi->info.isp.sp.id);
return i;
}
@@ -1641,7 +1719,8 @@ static int ipu3_css_find_binary(struct ipu3_css *css,
*/
int ipu3_css_fmt_try(struct ipu3_css *css,
struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
- struct v4l2_rect *rects[IPU3_CSS_RECTS])
+ struct v4l2_rect *rects[IPU3_CSS_RECTS],
+ unsigned int pipe)
{
static const u32 EFF_ALIGN_W = 2;
static const u32 BDS_ALIGN_W = 4;
@@ -1673,13 +1752,7 @@ int ipu3_css_fmt_try(struct ipu3_css *css,
&q[IPU3_CSS_QUEUE_OUT].fmt.mpix;
struct v4l2_pix_format_mplane *const vf =
&q[IPU3_CSS_QUEUE_VF].fmt.mpix;
- int binary, i, s;
-
- /* Decide which pipe to use */
- if (css->vf_output_en == IPU3_NODE_PV_ENABLED)
- css->pipe_id = IPU3_CSS_PIPE_ID_CAPTURE;
- else if (css->vf_output_en == IPU3_NODE_VF_ENABLED)
- css->pipe_id = IPU3_CSS_PIPE_ID_VIDEO;
+ int i, s;
/* Adjust all formats, get statistics buffer sizes and formats */
for (i = 0; i < IPU3_CSS_QUEUES; i++) {
@@ -1714,9 +1787,8 @@ int ipu3_css_fmt_try(struct ipu3_css *css,
/* Always require one input and vf only if out is also enabled */
if (!ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_IN]) ||
- (ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_VF]) &&
- !ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_OUT]))) {
- dev_dbg(css->dev, "required queues are disabled\n");
+ !ipu3_css_queue_enabled(&q[IPU3_CSS_QUEUE_OUT])) {
+ dev_warn(css->dev, "required queues are disabled\n");
return -EINVAL;
}
@@ -1755,12 +1827,16 @@ int ipu3_css_fmt_try(struct ipu3_css *css,
s = (bds->height - gdc->height) / 2 - FILTER_SIZE;
env->height = s < MIN_ENVELOPE ? MIN_ENVELOPE : s;
- binary = ipu3_css_find_binary(css, q, r);
- if (binary < 0) {
+ css->pipes[pipe].bindex =
+ ipu3_css_find_binary(css, pipe, q, r);
+ if (css->pipes[pipe].bindex < 0) {
dev_err(css->dev, "failed to find suitable binary\n");
return -EINVAL;
}
+ dev_dbg(css->dev, "Binary index %d for pipe %d found.",
+ css->pipes[pipe].bindex, pipe);
+
/* Final adjustment and set back the queried formats */
for (i = 0; i < IPU3_CSS_QUEUES; i++) {
if (fmts[i]) {
@@ -1784,16 +1860,18 @@ int ipu3_css_fmt_try(struct ipu3_css *css,
bds->width, bds->height, gdc->width, gdc->height,
out->width, out->height, vf->width, vf->height);
- return binary;
+ return 0;
}
int ipu3_css_fmt_set(struct ipu3_css *css,
struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
- struct v4l2_rect *rects[IPU3_CSS_RECTS])
+ struct v4l2_rect *rects[IPU3_CSS_RECTS],
+ unsigned int pipe)
{
struct v4l2_rect rect_data[IPU3_CSS_RECTS];
struct v4l2_rect *all_rects[IPU3_CSS_RECTS];
int i, r;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
for (i = 0; i < IPU3_CSS_RECTS; i++) {
if (rects[i])
@@ -1802,17 +1880,16 @@ int ipu3_css_fmt_set(struct ipu3_css *css,
memset(&rect_data[i], 0, sizeof(rect_data[i]));
all_rects[i] = &rect_data[i];
}
- r = ipu3_css_fmt_try(css, fmts, all_rects);
+ r = ipu3_css_fmt_try(css, fmts, all_rects, pipe);
if (r < 0)
return r;
- css->current_binary = (unsigned int)r;
for (i = 0; i < IPU3_CSS_QUEUES; i++)
- if (ipu3_css_queue_init(&css->queue[i], fmts[i],
+ if (ipu3_css_queue_init(&css_pipe->queue[i], fmts[i],
IPU3_CSS_QUEUE_TO_FLAGS(i)))
return -EINVAL;
for (i = 0; i < IPU3_CSS_RECTS; i++) {
- css->rect[i] = rect_data[i];
+ css_pipe->rect[i] = rect_data[i];
if (rects[i])
*rects[i] = rect_data[i];
}
@@ -1842,13 +1919,14 @@ int ipu3_css_meta_fmt_set(struct v4l2_meta_format *fmt)
* Returns 0 on success, -EBUSY if the buffer queue is full, or some other
* code on error conditions.
*/
-int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b)
+int ipu3_css_buf_queue(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_css_buffer *b)
{
- static const int thread;
struct imgu_abi_buffer *abi_buf;
struct imgu_addr_t *buf_addr;
u32 data;
int r;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
if (!css->streaming)
return -EPROTO; /* CSS or buffer in wrong state */
@@ -1857,11 +1935,11 @@ int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b)
return -EINVAL;
b->queue_pos = ipu3_css_queue_pos(css, ipu3_css_queues[b->queue].qid,
- thread);
+ pipe);
- if (b->queue_pos >= ARRAY_SIZE(css->abi_buffers[b->queue]))
+ if (b->queue_pos >= ARRAY_SIZE(css->pipes[pipe].abi_buffers[b->queue]))
return -EIO;
- abi_buf = css->abi_buffers[b->queue][b->queue_pos].vaddr;
+ abi_buf = css->pipes[pipe].abi_buffers[b->queue][b->queue_pos].vaddr;
/* Fill struct abi_buffer for firmware */
memset(abi_buf, 0, sizeof(*abi_buf));
@@ -1874,30 +1952,31 @@ int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b)
if (b->queue == IPU3_CSS_QUEUE_OUT)
abi_buf->payload.frame.padded_width =
- css->queue[IPU3_CSS_QUEUE_OUT].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_OUT].width_pad;
if (b->queue == IPU3_CSS_QUEUE_VF)
abi_buf->payload.frame.padded_width =
- css->queue[IPU3_CSS_QUEUE_VF].width_pad;
+ css_pipe->queue[IPU3_CSS_QUEUE_VF].width_pad;
- spin_lock(&css->qlock);
- list_add_tail(&b->list, &css->queue[b->queue].bufs);
- spin_unlock(&css->qlock);
+ spin_lock(&css_pipe->qlock);
+ list_add_tail(&b->list, &css_pipe->queue[b->queue].bufs);
+ spin_unlock(&css_pipe->qlock);
b->state = IPU3_CSS_BUFFER_QUEUED;
- data = css->abi_buffers[b->queue][b->queue_pos].daddr;
+ data = css->pipes[pipe].abi_buffers[b->queue][b->queue_pos].daddr;
r = ipu3_css_queue_data(css, ipu3_css_queues[b->queue].qid,
- thread, data);
+ pipe, data);
if (r < 0)
goto queueing_failed;
- data = IMGU_ABI_EVENT_BUFFER_ENQUEUED(thread,
+ data = IMGU_ABI_EVENT_BUFFER_ENQUEUED(pipe,
ipu3_css_queues[b->queue].qid);
- r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0, data);
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, pipe, data);
if (r < 0)
goto queueing_failed;
- dev_dbg(css->dev, "queued buffer %p to css queue %i\n", b, b->queue);
+ dev_dbg(css->dev, "queued buffer %p to css queue %i in pipe %d\n",
+ b, b->queue, pipe);
return 0;
@@ -1916,7 +1995,6 @@ int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b)
*/
struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css)
{
- static const int thread;
static const unsigned char evtype_to_queue[] = {
[IMGU_ABI_EVTTYPE_INPUT_FRAME_DONE] = IPU3_CSS_QUEUE_IN,
[IMGU_ABI_EVTTYPE_OUT_FRAME_DONE] = IPU3_CSS_QUEUE_OUT,
@@ -1926,6 +2004,7 @@ struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css)
struct ipu3_css_buffer *b = ERR_PTR(-EAGAIN);
u32 event, daddr;
int evtype, pipe, pipeid, queue, qid, r;
+ struct ipu3_css_pipe *css_pipe;
if (!css->streaming)
return ERR_PTR(-EPROTO);
@@ -1949,11 +2028,16 @@ struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css)
queue = evtype_to_queue[evtype];
qid = ipu3_css_queues[queue].qid;
+ if (pipe >= IMGU_MAX_PIPE_NUM) {
+ dev_err(css->dev, "Invalid pipe: %i\n", pipe);
+ return ERR_PTR(-EIO);
+ }
+
if (qid >= IMGU_ABI_QUEUE_NUM) {
dev_err(css->dev, "Invalid qid: %i\n", qid);
return ERR_PTR(-EIO);
}
-
+ css_pipe = &css->pipes[pipe];
dev_dbg(css->dev,
"event: buffer done 0x%x queue %i pipe %i pipeid %i\n",
event, queue, pipe, pipeid);
@@ -1965,39 +2049,52 @@ struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css)
return ERR_PTR(-EIO);
}
- r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, thread,
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, pipe,
IMGU_ABI_EVENT_BUFFER_DEQUEUED(qid));
if (r < 0) {
dev_err(css->dev, "failed to queue event\n");
return ERR_PTR(-EIO);
}
- spin_lock(&css->qlock);
- if (list_empty(&css->queue[queue].bufs)) {
- spin_unlock(&css->qlock);
+ spin_lock(&css_pipe->qlock);
+ if (list_empty(&css_pipe->queue[queue].bufs)) {
+ spin_unlock(&css_pipe->qlock);
dev_err(css->dev, "event on empty queue\n");
return ERR_PTR(-EIO);
}
- b = list_first_entry(&css->queue[queue].bufs,
+ b = list_first_entry(&css_pipe->queue[queue].bufs,
struct ipu3_css_buffer, list);
if (queue != b->queue ||
- daddr != css->abi_buffers[b->queue][b->queue_pos].daddr) {
- spin_unlock(&css->qlock);
+ daddr != css_pipe->abi_buffers
+ [b->queue][b->queue_pos].daddr) {
+ spin_unlock(&css_pipe->qlock);
dev_err(css->dev, "dequeued bad buffer 0x%x\n", daddr);
return ERR_PTR(-EIO);
}
+
+ dev_dbg(css->dev, "buffer 0x%8x done from pipe %d\n", daddr, pipe);
+ b->pipe = pipe;
b->state = IPU3_CSS_BUFFER_DONE;
list_del(&b->list);
- spin_unlock(&css->qlock);
+ spin_unlock(&css_pipe->qlock);
break;
case IMGU_ABI_EVTTYPE_PIPELINE_DONE:
- dev_dbg(css->dev, "event: pipeline done 0x%x for frame %ld\n",
- event, css->frame);
+ pipe = (event & IMGU_ABI_EVTTYPE_PIPE_MASK) >>
+ IMGU_ABI_EVTTYPE_PIPE_SHIFT;
+ if (pipe >= IMGU_MAX_PIPE_NUM) {
+ dev_err(css->dev, "Invalid pipe: %i\n", pipe);
+ return ERR_PTR(-EIO);
+ }
+
+ css_pipe = &css->pipes[pipe];
+ dev_dbg(css->dev,
+ "event: pipeline done 0x%8x for frame %ld pipe %d\n",
+ event, css_pipe->frame, pipe);
- if (css->frame == LONG_MAX)
- css->frame = 0;
+ if (css_pipe->frame == LONG_MAX)
+ css_pipe->frame = 0;
else
- css->frame++;
+ css_pipe->frame++;
break;
case IMGU_ABI_EVTTYPE_TIMER:
r = ipu3_css_dequeue_data(css, IMGU_ABI_QUEUE_EVENT_ID, &event);
@@ -2038,15 +2135,16 @@ struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css)
* Return index to css->parameter_set_info which has the newly created
* parameters or negative value on error.
*/
-int ipu3_css_set_parameters(struct ipu3_css *css,
+int ipu3_css_set_parameters(struct ipu3_css *css, unsigned int pipe,
struct ipu3_uapi_params *set_params)
{
- struct ipu3_uapi_flags *use = set_params ? &set_params->use : NULL;
static const unsigned int queue_id = IMGU_ABI_QUEUE_A_ID;
- const int stage = 0, thread = 0;
+ struct ipu3_css_pipe *css_pipe = &css->pipes[pipe];
+ const int stage = 0;
const struct imgu_fw_info *bi;
- unsigned int stripes, i;
int obgrid_size;
+ unsigned int stripes, i;
+ struct ipu3_uapi_flags *use = set_params ? &set_params->use : NULL;
/* Destination buffers which are filled here */
struct imgu_abi_parameter_set_info *param_set;
@@ -2063,7 +2161,9 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
if (!css->streaming)
return -EPROTO;
- bi = &css->fwp->binary_header[css->current_binary];
+ dev_dbg(css->dev, "%s for pipe %d", __func__, pipe);
+
+ bi = &css->fwp->binary_header[css_pipe->bindex];
obgrid_size = ipu3_css_fw_obgrid_size(bi);
stripes = bi->info.isp.sp.iterator.num_stripes ? : 1;
@@ -2072,49 +2172,53 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
* If this succeeds, then all of the other pool_get() calls below
* should also succeed.
*/
- if (ipu3_css_pool_get(&css->pool.parameter_set_info, css->frame) < 0)
+ if (ipu3_css_pool_get(&css_pipe->pool.parameter_set_info,
+ css_pipe->frame) < 0)
goto fail_no_put;
- param_set = ipu3_css_pool_last(&css->pool.parameter_set_info, 0)->vaddr;
+ param_set = ipu3_css_pool_last(&css_pipe->pool.parameter_set_info,
+ 0)->vaddr;
- map = ipu3_css_pool_last(&css->pool.acc, 0);
/* Get a new acc only if new parameters given, or none yet */
+ map = ipu3_css_pool_last(&css_pipe->pool.acc, 0);
if (set_params || !map->vaddr) {
- if (ipu3_css_pool_get(&css->pool.acc, css->frame) < 0)
+ if (ipu3_css_pool_get(&css_pipe->pool.acc, css_pipe->frame) < 0)
goto fail;
- map = ipu3_css_pool_last(&css->pool.acc, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.acc, 0);
acc = map->vaddr;
}
/* Get new VMEM0 only if needed, or none yet */
m = IMGU_ABI_MEM_ISP_VMEM0;
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 0);
if (!map->vaddr || (set_params && (set_params->use.lin_vmem_params ||
set_params->use.tnr3_vmem_params ||
set_params->use.xnr3_vmem_params))) {
- if (ipu3_css_pool_get(&css->pool.binary_params_p[m],
- css->frame) < 0)
+ if (ipu3_css_pool_get(&css_pipe->pool.binary_params_p[m],
+ css_pipe->frame) < 0)
goto fail;
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 0);
vmem0 = map->vaddr;
}
/* Get new DMEM0 only if needed, or none yet */
m = IMGU_ABI_MEM_ISP_DMEM0;
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 0);
if (!map->vaddr || (set_params && (set_params->use.tnr3_dmem_params ||
set_params->use.xnr3_dmem_params))) {
- if (ipu3_css_pool_get(&css->pool.binary_params_p[m],
- css->frame) < 0)
+ if (ipu3_css_pool_get(&css_pipe->pool.binary_params_p[m],
+ css_pipe->frame) < 0)
goto fail;
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 0);
dmem0 = map->vaddr;
}
/* Configure acc parameter cluster */
if (acc) {
- map = ipu3_css_pool_last(&css->pool.acc, 1);
- r = ipu3_css_cfg_acc(css, use, acc, map->vaddr, set_params ?
- &set_params->acc_param : NULL);
+ /* get acc_old */
+ map = ipu3_css_pool_last(&css_pipe->pool.acc, 1);
+ /* user acc */
+ r = ipu3_css_cfg_acc(css, pipe, use, acc, map->vaddr,
+ set_params ? &set_params->acc_param : NULL);
if (r < 0)
goto fail;
}
@@ -2122,16 +2226,18 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
/* Configure late binding parameters */
if (vmem0) {
m = IMGU_ABI_MEM_ISP_VMEM0;
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 1);
- r = ipu3_css_cfg_vmem0(css, use, vmem0, map->vaddr, set_params);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 1);
+ r = ipu3_css_cfg_vmem0(css, pipe, use, vmem0,
+ map->vaddr, set_params);
if (r < 0)
goto fail;
}
if (dmem0) {
m = IMGU_ABI_MEM_ISP_DMEM0;
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 1);
- r = ipu3_css_cfg_dmem0(css, use, dmem0, map->vaddr, set_params);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 1);
+ r = ipu3_css_cfg_dmem0(css, pipe, use, dmem0,
+ map->vaddr, set_params);
if (r < 0)
goto fail;
}
@@ -2142,30 +2248,30 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
unsigned int g = IPU3_CSS_RECT_GDC;
unsigned int e = IPU3_CSS_RECT_ENVELOPE;
- map = ipu3_css_pool_last(&css->pool.gdc, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.gdc, 0);
if (!map->vaddr) {
- if (ipu3_css_pool_get(&css->pool.gdc, css->frame) < 0)
+ if (ipu3_css_pool_get(&css_pipe->pool.gdc, css_pipe->frame) < 0)
goto fail;
- map = ipu3_css_pool_last(&css->pool.gdc, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.gdc, 0);
gdc = map->vaddr;
- ipu3_css_cfg_gdc_table(gdc,
- css->aux_frames[a].bytesperline /
- css->aux_frames[a].bytesperpixel,
- css->aux_frames[a].height,
- css->rect[g].width,
- css->rect[g].height,
- css->rect[e].width + FILTER_SIZE,
- css->rect[e].height +
- FILTER_SIZE);
+ ipu3_css_cfg_gdc_table(map->vaddr,
+ css_pipe->aux_frames[a].bytesperline /
+ css_pipe->aux_frames[a].bytesperpixel,
+ css_pipe->aux_frames[a].height,
+ css_pipe->rect[g].width,
+ css_pipe->rect[g].height,
+ css_pipe->rect[e].width + FILTER_SIZE,
+ css_pipe->rect[e].height +
+ FILTER_SIZE);
}
}
/* Get a new obgrid only if a new obgrid is given, or none yet */
- map = ipu3_css_pool_last(&css->pool.obgrid, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.obgrid, 0);
if (!map->vaddr || (set_params && set_params->use.obgrid_param)) {
- if (ipu3_css_pool_get(&css->pool.obgrid, css->frame) < 0)
+ if (ipu3_css_pool_get(&css_pipe->pool.obgrid, css_pipe->frame) < 0)
goto fail;
- map = ipu3_css_pool_last(&css->pool.obgrid, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.obgrid, 0);
obgrid = map->vaddr;
/* Configure optical black level grid (obgrid) */
@@ -2179,29 +2285,31 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
/* Configure parameter set info, queued to `queue_id' */
memset(param_set, 0, sizeof(*param_set));
- map = ipu3_css_pool_last(&css->pool.acc, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.acc, 0);
param_set->mem_map.acc_cluster_params_for_sp = map->daddr;
- map = ipu3_css_pool_last(&css->pool.gdc, 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.gdc, 0);
param_set->mem_map.dvs_6axis_params_y = map->daddr;
- map = ipu3_css_pool_last(&css->pool.obgrid, 0);
- for (i = 0; i < stripes; i++)
+ for (i = 0; i < stripes; i++) {
+ map = ipu3_css_pool_last(&css_pipe->pool.obgrid, 0);
param_set->mem_map.obgrid_tbl[i] =
- map->daddr + (obgrid_size / stripes) * i;
+ map->daddr + (obgrid_size / stripes) * i;
+ }
for (m = 0; m < IMGU_ABI_NUM_MEMORIES; m++) {
- map = ipu3_css_pool_last(&css->pool.binary_params_p[m], 0);
+ map = ipu3_css_pool_last(&css_pipe->pool.binary_params_p[m], 0);
param_set->mem_map.isp_mem_param[stage][m] = map->daddr;
}
+
/* Then queue the new parameter buffer */
- map = ipu3_css_pool_last(&css->pool.parameter_set_info, 0);
- r = ipu3_css_queue_data(css, queue_id, thread, map->daddr);
+ map = ipu3_css_pool_last(&css_pipe->pool.parameter_set_info, 0);
+ r = ipu3_css_queue_data(css, queue_id, pipe, map->daddr);
if (r < 0)
goto fail;
- r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, 0,
- IMGU_ABI_EVENT_BUFFER_ENQUEUED(thread,
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, pipe,
+ IMGU_ABI_EVENT_BUFFER_ENQUEUED(pipe,
queue_id));
if (r < 0)
goto fail_no_put;
@@ -2216,7 +2324,7 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
break;
if (r)
goto fail_no_put;
- r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, thread,
+ r = ipu3_css_queue_data(css, IMGU_ABI_QUEUE_EVENT_ID, pipe,
IMGU_ABI_EVENT_BUFFER_DEQUEUED
(queue_id));
if (r < 0) {
@@ -2234,19 +2342,21 @@ int ipu3_css_set_parameters(struct ipu3_css *css,
* parameters again later.
*/
- ipu3_css_pool_put(&css->pool.parameter_set_info);
+ ipu3_css_pool_put(&css_pipe->pool.parameter_set_info);
if (acc)
- ipu3_css_pool_put(&css->pool.acc);
+ ipu3_css_pool_put(&css_pipe->pool.acc);
if (gdc)
- ipu3_css_pool_put(&css->pool.gdc);
+ ipu3_css_pool_put(&css_pipe->pool.gdc);
if (obgrid)
- ipu3_css_pool_put(&css->pool.obgrid);
+ ipu3_css_pool_put(&css_pipe->pool.obgrid);
if (vmem0)
ipu3_css_pool_put(
- &css->pool.binary_params_p[IMGU_ABI_MEM_ISP_VMEM0]);
+ &css_pipe->pool.binary_params_p
+ [IMGU_ABI_MEM_ISP_VMEM0]);
if (dmem0)
ipu3_css_pool_put(
- &css->pool.binary_params_p[IMGU_ABI_MEM_ISP_DMEM0]);
+ &css_pipe->pool.binary_params_p
+ [IMGU_ABI_MEM_ISP_DMEM0]);
fail_no_put:
return r;
diff --git a/drivers/media/pci/intel/ipu3/ipu3-css.h b/drivers/media/pci/intel/ipu3/ipu3-css.h
index d16d0c4..11b1437 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-css.h
+++ b/drivers/media/pci/intel/ipu3/ipu3-css.h
@@ -13,6 +13,7 @@
/* 2 stages for split isp pipeline, 1 for scaling */
#define IMGU_NUM_SP 2
#define IMGU_MAX_PIPELINE_NUM 20
+#define IMGU_MAX_PIPE_NUM 2
/* For DVS etc., format FRAME_FMT_YUV420_16 */
#define IPU3_CSS_AUX_FRAME_REF 0
@@ -57,12 +58,6 @@ struct ipu3_css_resolution {
u32 h;
};
-enum ipu3_css_vf_status {
- IPU3_NODE_VF_ENABLED,
- IPU3_NODE_PV_ENABLED,
- IPU3_NODE_VF_DISABLED
-};
-
enum ipu3_css_buffer_state {
IPU3_CSS_BUFFER_NEW, /* Not yet queued */
IPU3_CSS_BUFFER_QUEUED, /* Queued, waiting to be filled */
@@ -77,6 +72,7 @@ struct ipu3_css_buffer {
enum ipu3_css_buffer_state state;
struct list_head list;
u8 queue_pos;
+ unsigned int pipe;
};
struct ipu3_css_format {
@@ -100,34 +96,32 @@ struct ipu3_css_queue {
} fmt;
const struct ipu3_css_format *css_fmt;
- unsigned int width_pad; /* bytesperline / byp */
+ unsigned int width_pad;
struct list_head bufs;
};
-/* IPU3 Camera Sub System structure */
-struct ipu3_css {
- struct device *dev;
- void __iomem *base;
- const struct firmware *fw;
- struct imgu_fw_header *fwp;
- int iomem_length;
- int fw_bl, fw_sp[IMGU_NUM_SP]; /* Indices of bl and SP binaries */
- struct ipu3_css_map *binary; /* fw binaries mapped to device */
- unsigned int current_binary; /* Currently selected binary */
- bool streaming; /* true when streaming is enabled */
- long frame; /* Latest frame not yet processed */
- enum ipu3_css_pipe_id pipe_id; /* CSS pipe ID. */
+struct ipu3_css_pipe {
+ enum ipu3_css_pipe_id pipe_id;
+ unsigned int bindex;
+
+ struct ipu3_css_queue queue[IPU3_CSS_QUEUES];
+ struct v4l2_rect rect[IPU3_CSS_RECTS];
+
+ bool vf_output_en;
+ /* Protect access to queue[IPU3_CSS_QUEUES] */
+ spinlock_t qlock;
/* Data structures shared with IMGU and driver, always allocated */
+ struct ipu3_css_map sp_ddr_ptrs;
struct ipu3_css_map xmem_sp_stage_ptrs[IPU3_CSS_PIPE_ID_NUM]
[IMGU_ABI_MAX_STAGES];
struct ipu3_css_map xmem_isp_stage_ptrs[IPU3_CSS_PIPE_ID_NUM]
[IMGU_ABI_MAX_STAGES];
- struct ipu3_css_map sp_ddr_ptrs;
- struct ipu3_css_map xmem_sp_group_ptrs;
- /* Data structures shared with IMGU and driver, binary specific */
- /* PARAM_CLASS_CONFIG and PARAM_CLASS_STATE parameters */
+ /*
+ * Data structures shared with IMGU and driver, binary specific.
+ * PARAM_CLASS_CONFIG and PARAM_CLASS_STATE parameters.
+ */
struct ipu3_css_map binary_params_cs[IMGU_ABI_PARAM_CLASS_NUM - 1]
[IMGU_ABI_NUM_MEMORIES];
@@ -139,10 +133,6 @@ struct ipu3_css {
unsigned int bytesperpixel;
} aux_frames[IPU3_CSS_AUX_FRAME_TYPES];
- struct ipu3_css_queue queue[IPU3_CSS_QUEUES];
- struct v4l2_rect rect[IPU3_CSS_RECTS];
- struct ipu3_css_map abi_buffers[IPU3_CSS_QUEUES]
- [IMGU_ABI_HOST2SP_BUFQ_SIZE];
struct {
struct ipu3_css_pool parameter_set_info;
@@ -153,9 +143,27 @@ struct ipu3_css {
struct ipu3_css_pool binary_params_p[IMGU_ABI_NUM_MEMORIES];
} pool;
- enum ipu3_css_vf_status vf_output_en;
- /* Protect access to css->queue[] */
- spinlock_t qlock;
+ struct ipu3_css_map abi_buffers[IPU3_CSS_QUEUES]
+ [IMGU_ABI_HOST2SP_BUFQ_SIZE];
+ long frame; /* Latest frame not yet processed */
+};
+
+/* IPU3 Camera Sub System structure */
+struct ipu3_css {
+ struct device *dev;
+ void __iomem *base;
+ const struct firmware *fw;
+ struct imgu_fw_header *fwp;
+ int iomem_length;
+ int fw_bl, fw_sp[IMGU_NUM_SP]; /* Indices of bl and SP binaries */
+ struct ipu3_css_map *binary; /* fw binaries mapped to device */
+ bool streaming; /* true when streaming is enabled */
+
+ struct ipu3_css_pipe pipes[IMGU_MAX_PIPE_NUM];
+ struct ipu3_css_map xmem_sp_group_ptrs;
+
+ /* enabled pipe(s) */
+ DECLARE_BITMAP(enabled_pipes, IMGU_MAX_PIPE_NUM);
};
/******************* css v4l *******************/
@@ -164,17 +172,21 @@ int ipu3_css_init(struct device *dev, struct ipu3_css *css,
void ipu3_css_cleanup(struct ipu3_css *css);
int ipu3_css_fmt_try(struct ipu3_css *css,
struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
- struct v4l2_rect *rects[IPU3_CSS_RECTS]);
+ struct v4l2_rect *rects[IPU3_CSS_RECTS],
+ unsigned int pipe);
int ipu3_css_fmt_set(struct ipu3_css *css,
struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES],
- struct v4l2_rect *rects[IPU3_CSS_RECTS]);
+ struct v4l2_rect *rects[IPU3_CSS_RECTS],
+ unsigned int pipe);
int ipu3_css_meta_fmt_set(struct v4l2_meta_format *fmt);
-int ipu3_css_buf_queue(struct ipu3_css *css, struct ipu3_css_buffer *b);
+int ipu3_css_buf_queue(struct ipu3_css *css, unsigned int pipe,
+ struct ipu3_css_buffer *b);
struct ipu3_css_buffer *ipu3_css_buf_dequeue(struct ipu3_css *css);
int ipu3_css_start_streaming(struct ipu3_css *css);
void ipu3_css_stop_streaming(struct ipu3_css *css);
bool ipu3_css_queue_empty(struct ipu3_css *css);
bool ipu3_css_is_streaming(struct ipu3_css *css);
+bool ipu3_css_pipe_queue_empty(struct ipu3_css *css, unsigned int pipe);
/******************* css hw *******************/
int ipu3_css_set_powerup(struct device *dev, void __iomem *base);
@@ -182,10 +194,10 @@ void ipu3_css_set_powerdown(struct device *dev, void __iomem *base);
int ipu3_css_irq_ack(struct ipu3_css *css);
/******************* set parameters ************/
-int ipu3_css_set_parameters(struct ipu3_css *css,
+int ipu3_css_set_parameters(struct ipu3_css *css, unsigned int pipe,
struct ipu3_uapi_params *set_params);
-/******************* css misc *******************/
+/******************* auxiliary helpers *******************/
static inline enum ipu3_css_buffer_state
ipu3_css_buf_state(struct ipu3_css_buffer *b)
{
diff --git a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
index 31a3514..4066383 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-v4l2.c
@@ -4,6 +4,7 @@
#include <linux/module.h>
#include <linux/pm_runtime.h>
+#include <media/v4l2-event.h>
#include <media/v4l2-ioctl.h>
#include "ipu3.h"
@@ -13,19 +14,28 @@
static int ipu3_subdev_open(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh)
{
- struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct imgu_device *imgu = v4l2_get_subdevdata(sd);
+ struct imgu_media_pipe *imgu_pipe;
struct v4l2_rect try_crop = {
.top = 0,
.left = 0,
- .height = imgu->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.height,
- .width = imgu->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.width,
};
unsigned int i;
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
+ unsigned int pipe = imgu_sd->pipe;
+
+ imgu_pipe = &imgu->imgu_pipe[pipe];
+ try_crop.width =
+ imgu_pipe->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.width;
+ try_crop.height =
+ imgu_pipe->nodes[IMGU_NODE_IN].vdev_fmt.fmt.pix_mp.height;
/* Initialize try_fmt */
for (i = 0; i < IMGU_NODE_NUM; i++)
*v4l2_subdev_get_try_format(sd, fh->pad, i) =
- imgu->nodes[i].pad_fmt;
+ imgu_pipe->nodes[i].pad_fmt;
*v4l2_subdev_get_try_crop(sd, fh->pad, IMGU_NODE_IN) = try_crop;
@@ -34,26 +44,89 @@ static int ipu3_subdev_open(struct v4l2_subdev *sd, struct v4l2_subdev_fh *fh)
static int ipu3_subdev_s_stream(struct v4l2_subdev *sd, int enable)
{
- struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ int i;
+ unsigned int node;
int r = 0;
+ struct imgu_device *imgu = v4l2_get_subdevdata(sd);
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
+ unsigned int pipe = imgu_sd->pipe;
+ struct device *dev = &imgu->pci_dev->dev;
+ struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
+ struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
+ struct ipu3_css_pipe *css_pipe = &imgu->css.pipes[pipe];
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
- r = imgu_s_stream(imgu, enable);
- if (!r)
- imgu->streaming = enable;
+ dev_dbg(dev, "%s %d for pipe %d", __func__, enable, pipe);
+ /* grab ctrl after streamon and return after off */
+ v4l2_ctrl_grab(imgu_sd->ctrl, enable);
- return r;
+ if (!enable) {
+ imgu_sd->active = false;
+ return 0;
+ }
+
+ for (i = 0; i < IMGU_NODE_NUM; i++)
+ imgu_pipe->queue_enabled[i] = imgu_pipe->nodes[i].enabled;
+
+ /* This is handled specially */
+ imgu_pipe->queue_enabled[IPU3_CSS_QUEUE_PARAMS] = false;
+
+ /* Initialize CSS formats */
+ for (i = 0; i < IPU3_CSS_QUEUES; i++) {
+ node = imgu_map_node(imgu, i);
+ /* No need to reconfig meta nodes */
+ if (node == IMGU_NODE_STAT_3A || node == IMGU_NODE_PARAMS)
+ continue;
+ fmts[i] = imgu_pipe->queue_enabled[node] ?
+ &imgu_pipe->nodes[node].vdev_fmt.fmt.pix_mp : NULL;
+ }
+
+ /* Enable VF output only when VF queue requested by user */
+ css_pipe->vf_output_en = false;
+ if (imgu_pipe->nodes[IMGU_NODE_VF].enabled)
+ css_pipe->vf_output_en = true;
+
+ if (atomic_read(&imgu_sd->running_mode) == IPU3_RUNNING_MODE_VIDEO)
+ css_pipe->pipe_id = IPU3_CSS_PIPE_ID_VIDEO;
+ else
+ css_pipe->pipe_id = IPU3_CSS_PIPE_ID_CAPTURE;
+
+ dev_dbg(dev, "IPU3 pipe %d pipe_id %d", pipe, css_pipe->pipe_id);
+
+ rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu_sd->rect.eff;
+ rects[IPU3_CSS_RECT_BDS] = &imgu_sd->rect.bds;
+ rects[IPU3_CSS_RECT_GDC] = &imgu_sd->rect.gdc;
+
+ r = ipu3_css_fmt_set(&imgu->css, fmts, rects, pipe);
+ if (r) {
+ dev_err(dev, "failed to set initial formats pipe %d with (%d)",
+ pipe, r);
+ return r;
+ }
+
+ imgu_sd->active = true;
+
+ return 0;
}
static int ipu3_subdev_get_fmt(struct v4l2_subdev *sd,
struct v4l2_subdev_pad_config *cfg,
struct v4l2_subdev_format *fmt)
{
- struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct imgu_device *imgu = v4l2_get_subdevdata(sd);
struct v4l2_mbus_framefmt *mf;
+ struct imgu_media_pipe *imgu_pipe;
u32 pad = fmt->pad;
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
+ unsigned int pipe = imgu_sd->pipe;
+ imgu_pipe = &imgu->imgu_pipe[pipe];
if (fmt->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
- fmt->format = imgu->nodes[pad].pad_fmt;
+ fmt->format = imgu_pipe->nodes[pad].pad_fmt;
} else {
mf = v4l2_subdev_get_try_format(sd, cfg, pad);
fmt->format = *mf;
@@ -66,18 +139,28 @@ static int ipu3_subdev_set_fmt(struct v4l2_subdev *sd,
struct v4l2_subdev_pad_config *cfg,
struct v4l2_subdev_format *fmt)
{
- struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct imgu_media_pipe *imgu_pipe;
+ struct imgu_device *imgu = v4l2_get_subdevdata(sd);
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
+
struct v4l2_mbus_framefmt *mf;
u32 pad = fmt->pad;
+ unsigned int pipe = imgu_sd->pipe;
+
+ dev_dbg(&imgu->pci_dev->dev, "set subdev %d pad %d fmt to [%dx%d]",
+ pipe, pad, fmt->format.width, fmt->format.height);
+ imgu_pipe = &imgu->imgu_pipe[pipe];
if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
mf = v4l2_subdev_get_try_format(sd, cfg, pad);
else
- mf = &imgu->nodes[pad].pad_fmt;
+ mf = &imgu_pipe->nodes[pad].pad_fmt;
fmt->format.code = mf->code;
/* Clamp the w and h based on the hardware capabilities */
- if (imgu->subdev_pads[pad].flags & MEDIA_PAD_FL_SOURCE) {
+ if (imgu_sd->subdev_pads[pad].flags & MEDIA_PAD_FL_SOURCE) {
fmt->format.width = clamp(fmt->format.width,
IPU3_OUTPUT_MIN_WIDTH,
IPU3_OUTPUT_MAX_WIDTH);
@@ -102,8 +185,10 @@ static int ipu3_subdev_get_selection(struct v4l2_subdev *sd,
struct v4l2_subdev_pad_config *cfg,
struct v4l2_subdev_selection *sel)
{
- struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
struct v4l2_rect *try_sel, *r;
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
if (sel->pad != IMGU_NODE_IN)
return -EINVAL;
@@ -111,11 +196,11 @@ static int ipu3_subdev_get_selection(struct v4l2_subdev *sd,
switch (sel->target) {
case V4L2_SEL_TGT_CROP:
try_sel = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
- r = &imgu->rect.eff;
+ r = &imgu_sd->rect.eff;
break;
case V4L2_SEL_TGT_COMPOSE:
try_sel = v4l2_subdev_get_try_compose(sd, cfg, sel->pad);
- r = &imgu->rect.bds;
+ r = &imgu_sd->rect.bds;
break;
default:
return -EINVAL;
@@ -133,20 +218,28 @@ static int ipu3_subdev_set_selection(struct v4l2_subdev *sd,
struct v4l2_subdev_pad_config *cfg,
struct v4l2_subdev_selection *sel)
{
- struct imgu_device *imgu = container_of(sd, struct imgu_device, subdev);
+ struct imgu_device *imgu = v4l2_get_subdevdata(sd);
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
struct v4l2_rect *rect, *try_sel;
+ dev_dbg(&imgu->pci_dev->dev,
+ "set subdev %d sel which %d target 0x%4x rect [%dx%d]",
+ imgu_sd->pipe, sel->which, sel->target,
+ sel->r.width, sel->r.height);
+
if (sel->pad != IMGU_NODE_IN)
return -EINVAL;
switch (sel->target) {
case V4L2_SEL_TGT_CROP:
try_sel = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
- rect = &imgu->rect.eff;
+ rect = &imgu_sd->rect.eff;
break;
case V4L2_SEL_TGT_COMPOSE:
try_sel = v4l2_subdev_get_try_compose(sd, cfg, sel->pad);
- rect = &imgu->rect.bds;
+ rect = &imgu_sd->rect.bds;
break;
default:
return -EINVAL;
@@ -166,13 +259,35 @@ static int ipu3_link_setup(struct media_entity *entity,
const struct media_pad *local,
const struct media_pad *remote, u32 flags)
{
- struct imgu_device *imgu = container_of(entity, struct imgu_device,
- subdev.entity);
+ struct imgu_media_pipe *imgu_pipe;
+ struct v4l2_subdev *sd = container_of(entity, struct v4l2_subdev,
+ entity);
+ struct imgu_device *imgu = v4l2_get_subdevdata(sd);
+ struct imgu_v4l2_subdev *imgu_sd = container_of(sd,
+ struct imgu_v4l2_subdev,
+ subdev);
+ unsigned int pipe = imgu_sd->pipe;
u32 pad = local->index;
WARN_ON(pad >= IMGU_NODE_NUM);
- imgu->nodes[pad].enabled = flags & MEDIA_LNK_FL_ENABLED;
+ dev_dbg(&imgu->pci_dev->dev, "pipe %d pad %d is %s", pipe, pad,
+ flags & MEDIA_LNK_FL_ENABLED ? "enabled" : "disabled");
+
+ imgu_pipe = &imgu->imgu_pipe[pipe];
+ imgu_pipe->nodes[pad].enabled = flags & MEDIA_LNK_FL_ENABLED;
+
+ /* enable input node to enable the pipe */
+ if (pad != IMGU_NODE_IN)
+ return 0;
+
+ if (flags & MEDIA_LNK_FL_ENABLED)
+ __set_bit(pipe, imgu->css.enabled_pipes);
+ else
+ __clear_bit(pipe, imgu->css.enabled_pipes);
+
+ dev_dbg(&imgu->pci_dev->dev, "pipe %d is %s", pipe,
+ flags & MEDIA_LNK_FL_ENABLED ? "enabled" : "disabled");
return 0;
}
@@ -187,7 +302,7 @@ static int ipu3_vb2_buf_init(struct vb2_buffer *vb)
struct imgu_buffer, vid_buf.vbb.vb2_buf);
struct imgu_video_device *node =
container_of(vb->vb2_queue, struct imgu_video_device, vbq);
- unsigned int queue = imgu_node_to_queue(node - imgu->nodes);
+ unsigned int queue = imgu_node_to_queue(node->id);
if (queue == IPU3_CSS_QUEUE_PARAMS)
return 0;
@@ -203,7 +318,7 @@ static void ipu3_vb2_buf_cleanup(struct vb2_buffer *vb)
struct imgu_buffer, vid_buf.vbb.vb2_buf);
struct imgu_video_device *node =
container_of(vb->vb2_queue, struct imgu_video_device, vbq);
- unsigned int queue = imgu_node_to_queue(node - imgu->nodes);
+ unsigned int queue = imgu_node_to_queue(node->id);
if (queue == IPU3_CSS_QUEUE_PARAMS)
return;
@@ -217,8 +332,9 @@ static void ipu3_vb2_buf_queue(struct vb2_buffer *vb)
struct imgu_device *imgu = vb2_get_drv_priv(vb->vb2_queue);
struct imgu_video_device *node =
container_of(vb->vb2_queue, struct imgu_video_device, vbq);
- unsigned int queue = imgu_node_to_queue(node - imgu->nodes);
+ unsigned int queue = imgu_node_to_queue(node->id);
unsigned long need_bytes;
+ unsigned int pipe = node->pipe;
if (vb->vb2_queue->type == V4L2_BUF_TYPE_META_CAPTURE ||
vb->vb2_queue->type == V4L2_BUF_TYPE_META_OUTPUT)
@@ -237,7 +353,7 @@ static void ipu3_vb2_buf_queue(struct vb2_buffer *vb)
vb2_set_plane_payload(vb, 0, payload);
}
if (payload >= need_bytes)
- r = ipu3_css_set_parameters(&imgu->css,
+ r = ipu3_css_set_parameters(&imgu->css, pipe,
vb2_plane_vaddr(vb, 0));
buf->flags = V4L2_BUF_FLAG_DONE;
vb2_buffer_done(vb, r == 0 ? VB2_BUF_STATE_DONE
@@ -250,14 +366,18 @@ static void ipu3_vb2_buf_queue(struct vb2_buffer *vb)
mutex_lock(&imgu->lock);
ipu3_css_buf_init(&buf->css_buf, queue, buf->map.daddr);
list_add_tail(&buf->vid_buf.list,
- &imgu->nodes[node - imgu->nodes].buffers);
+ &node->buffers);
mutex_unlock(&imgu->lock);
vb2_set_plane_payload(&buf->vid_buf.vbb.vb2_buf, 0, need_bytes);
if (imgu->streaming)
- imgu_queue_buffers(imgu, false);
+ imgu_queue_buffers(imgu, false, pipe);
}
+
+ dev_dbg(&imgu->pci_dev->dev, "%s for pipe %d node %d", __func__,
+ node->pipe, node->id);
+
}
static int ipu3_vb2_queue_setup(struct vb2_queue *vq,
@@ -289,6 +409,7 @@ static int ipu3_vb2_queue_setup(struct vb2_queue *vq,
*num_planes = 1;
sizes[0] = size;
+
/* Initialize buffer queue */
INIT_LIST_HEAD(&node->buffers);
@@ -299,15 +420,27 @@ static int ipu3_vb2_queue_setup(struct vb2_queue *vq,
static bool ipu3_all_nodes_streaming(struct imgu_device *imgu,
struct imgu_video_device *except)
{
- unsigned int i;
-
- for (i = 0; i < IMGU_NODE_NUM; i++) {
- struct imgu_video_device *node = &imgu->nodes[i];
+ unsigned int i, pipe, p;
+ struct imgu_video_device *node;
+ struct device *dev = &imgu->pci_dev->dev;
+
+ pipe = except->pipe;
+ if (!test_bit(pipe, imgu->css.enabled_pipes)) {
+ dev_warn(&imgu->pci_dev->dev,
+ "pipe %d link is not ready yet", pipe);
+ return false;
+ }
- if (node == except)
- continue;
- if (node->enabled && !vb2_start_streaming_called(&node->vbq))
- return false;
+ for_each_set_bit(p, imgu->css.enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ node = &imgu->imgu_pipe[p].nodes[i];
+ dev_dbg(dev, "%s pipe %u queue %u name %s enabled = %u",
+ __func__, p, i, node->name, node->enabled);
+ if (node == except)
+ continue;
+ if (node->enabled && !vb2_start_streaming_called(&node->vbq))
+ return false;
+ }
}
return true;
@@ -330,10 +463,16 @@ static void ipu3_return_all_buffers(struct imgu_device *imgu,
static int ipu3_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
{
+ struct imgu_media_pipe *imgu_pipe;
struct imgu_device *imgu = vb2_get_drv_priv(vq);
+ struct device *dev = &imgu->pci_dev->dev;
struct imgu_video_device *node =
container_of(vq, struct imgu_video_device, vbq);
int r;
+ unsigned int pipe;
+
+ dev_dbg(dev, "%s node name %s pipe %d id %u", __func__,
+ node->name, node->pipe, node->id);
if (imgu->streaming) {
r = -EBUSY;
@@ -341,21 +480,33 @@ static int ipu3_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
}
if (!node->enabled) {
+ dev_err(dev, "IMGU node is not enabled");
r = -EINVAL;
goto fail_return_bufs;
}
- r = media_pipeline_start(&node->vdev.entity, &imgu->pipeline);
+
+ pipe = node->pipe;
+ imgu_pipe = &imgu->imgu_pipe[pipe];
+ r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
if (r < 0)
goto fail_return_bufs;
+
if (!ipu3_all_nodes_streaming(imgu, node))
return 0;
- /* Start streaming of the whole pipeline now */
+ for_each_set_bit(pipe, imgu->css.enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ r = v4l2_subdev_call(&imgu->imgu_pipe[pipe].imgu_sd.subdev,
+ video, s_stream, 1);
+ if (r < 0)
+ goto fail_stop_pipeline;
+ }
- r = v4l2_subdev_call(&imgu->subdev, video, s_stream, 1);
- if (r < 0)
- goto fail_stop_pipeline;
+ /* Start streaming of the whole pipeline now */
+ dev_dbg(dev, "IMGU streaming is ready to start");
+ r = imgu_s_stream(imgu, true);
+ if (!r)
+ imgu->streaming = true;
return 0;
@@ -369,20 +520,31 @@ static int ipu3_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
static void ipu3_vb2_stop_streaming(struct vb2_queue *vq)
{
+ struct imgu_media_pipe *imgu_pipe;
struct imgu_device *imgu = vb2_get_drv_priv(vq);
+ struct device *dev = &imgu->pci_dev->dev;
struct imgu_video_device *node =
container_of(vq, struct imgu_video_device, vbq);
int r;
+ unsigned int pipe;
WARN_ON(!node->enabled);
+ pipe = node->pipe;
+ dev_dbg(dev, "Try to stream off node [%d][%d]", pipe, node->id);
+ imgu_pipe = &imgu->imgu_pipe[pipe];
+ r = v4l2_subdev_call(&imgu_pipe->imgu_sd.subdev, video, s_stream, 0);
+ if (r)
+ dev_err(&imgu->pci_dev->dev,
+ "failed to stop subdev streaming\n");
+
/* Was this the first node with streaming disabled? */
- if (ipu3_all_nodes_streaming(imgu, node)) {
+ if (imgu->streaming && ipu3_all_nodes_streaming(imgu, node)) {
/* Yes, really stop streaming now */
- r = v4l2_subdev_call(&imgu->subdev, video, s_stream, 0);
- if (r)
- dev_err(&imgu->pci_dev->dev,
- "failed to stop streaming\n");
+ dev_dbg(dev, "IMGU streaming is ready to stop");
+ r = imgu_s_stream(imgu, false);
+ if (!r)
+ imgu->streaming = false;
}
ipu3_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
@@ -490,29 +652,35 @@ static int ipu3_vidioc_g_fmt(struct file *file, void *fh,
* Set input/output format. Unless it is just a try, this also resets
* selections (ie. effective and BDS resolutions) to defaults.
*/
-static int imgu_fmt(struct imgu_device *imgu, int node,
+static int imgu_fmt(struct imgu_device *imgu, unsigned int pipe, int node,
struct v4l2_format *f, bool try)
{
+ struct device *dev = &imgu->pci_dev->dev;
struct v4l2_pix_format_mplane try_fmts[IPU3_CSS_QUEUES];
struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
struct v4l2_mbus_framefmt pad_fmt;
unsigned int i, css_q;
int r;
+ struct ipu3_css_pipe *css_pipe = &imgu->css.pipes[pipe];
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
+ struct imgu_v4l2_subdev *imgu_sd = &imgu_pipe->imgu_sd;
- if (imgu->nodes[IMGU_NODE_PV].enabled &&
- imgu->nodes[IMGU_NODE_VF].enabled) {
- dev_err(&imgu->pci_dev->dev,
- "Postview and vf are not supported simultaneously\n");
- return -EINVAL;
- }
- /*
- * Tell css that the vf q is used for PV
- */
- if (imgu->nodes[IMGU_NODE_PV].enabled)
- imgu->css.vf_output_en = IPU3_NODE_PV_ENABLED;
- else if (imgu->nodes[IMGU_NODE_VF].enabled)
- imgu->css.vf_output_en = IPU3_NODE_VF_ENABLED;
+ dev_dbg(dev, "set fmt node [%u][%u](try = %d)", pipe, node, try);
+
+ for (i = 0; i < IMGU_NODE_NUM; i++)
+ dev_dbg(dev, "IMGU pipe %d node %d enabled = %d",
+ pipe, i, imgu_pipe->nodes[i].enabled);
+
+ if (imgu_pipe->nodes[IMGU_NODE_VF].enabled)
+ css_pipe->vf_output_en = true;
+
+ if (atomic_read(&imgu_sd->running_mode) == IPU3_RUNNING_MODE_VIDEO)
+ css_pipe->pipe_id = IPU3_CSS_PIPE_ID_VIDEO;
+ else
+ css_pipe->pipe_id = IPU3_CSS_PIPE_ID_CAPTURE;
+
+ dev_dbg(dev, "IPU3 pipe %d pipe_id = %d", pipe, css_pipe->pipe_id);
for (i = 0; i < IPU3_CSS_QUEUES; i++) {
unsigned int inode = imgu_map_node(imgu, i);
@@ -520,32 +688,31 @@ static int imgu_fmt(struct imgu_device *imgu, int node,
/* Skip the meta node */
if (inode == IMGU_NODE_STAT_3A || inode == IMGU_NODE_PARAMS)
continue;
- /* imgu_map_node defauls to PV if VF not enabled */
- if (inode == IMGU_NODE_PV && node == IMGU_NODE_VF &&
- imgu->css.vf_output_en == IPU3_NODE_VF_DISABLED)
- inode = node;
if (try) {
- try_fmts[i] = imgu->nodes[inode].vdev_fmt.fmt.pix_mp;
+ try_fmts[i] =
+ imgu_pipe->nodes[inode].vdev_fmt.fmt.pix_mp;
fmts[i] = &try_fmts[i];
} else {
- fmts[i] = &imgu->nodes[inode].vdev_fmt.fmt.pix_mp;
+ fmts[i] = &imgu_pipe->nodes[inode].vdev_fmt.fmt.pix_mp;
}
/* CSS expects some format on OUT queue */
if (i != IPU3_CSS_QUEUE_OUT &&
- !imgu->nodes[inode].enabled && inode != node)
+ !imgu_pipe->nodes[inode].enabled)
fmts[i] = NULL;
}
if (!try) {
/* eff and bds res got by imgu_s_sel */
- rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu->rect.eff;
- rects[IPU3_CSS_RECT_BDS] = &imgu->rect.bds;
- rects[IPU3_CSS_RECT_GDC] = &imgu->rect.gdc;
+ struct imgu_v4l2_subdev *imgu_sd = &imgu_pipe->imgu_sd;
+
+ rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu_sd->rect.eff;
+ rects[IPU3_CSS_RECT_BDS] = &imgu_sd->rect.bds;
+ rects[IPU3_CSS_RECT_GDC] = &imgu_sd->rect.gdc;
/* suppose that pad fmt was set by subdev s_fmt before */
- pad_fmt = imgu->nodes[IMGU_NODE_IN].pad_fmt;
+ pad_fmt = imgu_pipe->nodes[IMGU_NODE_IN].pad_fmt;
rects[IPU3_CSS_RECT_GDC]->width = pad_fmt.width;
rects[IPU3_CSS_RECT_GDC]->height = pad_fmt.height;
}
@@ -561,9 +728,9 @@ static int imgu_fmt(struct imgu_device *imgu, int node,
return -EINVAL;
if (try)
- r = ipu3_css_fmt_try(&imgu->css, fmts, rects);
+ r = ipu3_css_fmt_try(&imgu->css, fmts, rects, pipe);
else
- r = ipu3_css_fmt_set(&imgu->css, fmts, rects);
+ r = ipu3_css_fmt_set(&imgu->css, fmts, rects, pipe);
/* r is the binary number in the firmware blob */
if (r < 0)
@@ -572,7 +739,7 @@ static int imgu_fmt(struct imgu_device *imgu, int node,
if (try)
f->fmt.pix_mp = *fmts[css_q];
else
- f->fmt = imgu->nodes[node].vdev_fmt.fmt;
+ f->fmt = imgu_pipe->nodes[node].vdev_fmt.fmt;
return 0;
}
@@ -601,27 +768,37 @@ static int ipu3_vidioc_try_fmt(struct file *file, void *fh,
struct v4l2_format *f)
{
struct imgu_device *imgu = video_drvdata(file);
+ struct device *dev = &imgu->pci_dev->dev;
struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+ struct v4l2_pix_format_mplane *pix_mp = &f->fmt.pix_mp;
int r;
+ dev_dbg(dev, "%s [%ux%u] for node %d\n", __func__,
+ pix_mp->width, pix_mp->height, node->id);
+
r = ipu3_try_fmt(file, fh, f);
if (r)
return r;
- return imgu_fmt(imgu, node - imgu->nodes, f, true);
+ return imgu_fmt(imgu, node->pipe, node->id, f, true);
}
static int ipu3_vidioc_s_fmt(struct file *file, void *fh, struct v4l2_format *f)
{
struct imgu_device *imgu = video_drvdata(file);
+ struct device *dev = &imgu->pci_dev->dev;
struct imgu_video_device *node = file_to_intel_ipu3_node(file);
+ struct v4l2_pix_format_mplane *pix_mp = &f->fmt.pix_mp;
int r;
+ dev_dbg(dev, "%s [%ux%u] for node %d\n", __func__,
+ pix_mp->width, pix_mp->height, node->id);
+
r = ipu3_try_fmt(file, fh, f);
if (r)
return r;
- return imgu_fmt(imgu, node - imgu->nodes, f, false);
+ return imgu_fmt(imgu, node->pipe, node->id, f, false);
}
static int ipu3_meta_enum_format(struct file *file, void *fh,
@@ -705,6 +882,11 @@ static struct v4l2_subdev_internal_ops ipu3_subdev_internal_ops = {
.open = ipu3_subdev_open,
};
+static const struct v4l2_subdev_core_ops ipu3_subdev_core_ops = {
+ .subscribe_event = v4l2_ctrl_subdev_subscribe_event,
+ .unsubscribe_event = v4l2_event_subdev_unsubscribe,
+};
+
static const struct v4l2_subdev_video_ops ipu3_subdev_video_ops = {
.s_stream = ipu3_subdev_s_stream,
};
@@ -718,6 +900,7 @@ static const struct v4l2_subdev_pad_ops ipu3_subdev_pad_ops = {
};
static const struct v4l2_subdev_ops ipu3_subdev_ops = {
+ .core = &ipu3_subdev_core_ops,
.video = &ipu3_subdev_video_ops,
.pad = &ipu3_subdev_pad_ops,
};
@@ -811,6 +994,40 @@ static const struct v4l2_ioctl_ops ipu3_v4l2_meta_ioctl_ops = {
.vidioc_expbuf = vb2_ioctl_expbuf,
};
+static int ipu3_sd_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct imgu_v4l2_subdev *imgu_sd =
+ container_of(ctrl->handler, struct imgu_v4l2_subdev, ctrl_handler);
+ struct imgu_device *imgu = v4l2_get_subdevdata(&imgu_sd->subdev);
+ struct device *dev = &imgu->pci_dev->dev;
+
+ dev_dbg(dev, "set val %d to ctrl 0x%8x for subdev %d",
+ ctrl->val, ctrl->id, imgu_sd->pipe);
+
+ switch (ctrl->id) {
+ case V4L2_CID_INTEL_IPU3_MODE:
+ atomic_set(&imgu_sd->running_mode, ctrl->val);
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+static const struct v4l2_ctrl_ops ipu3_subdev_ctrl_ops = {
+ .s_ctrl = ipu3_sd_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config ipu3_subdev_ctrl_mode = {
+ .ops = &ipu3_subdev_ctrl_ops,
+ .id = V4L2_CID_INTEL_IPU3_MODE,
+ .name = "IPU3 Pipe Mode",
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .min = IPU3_RUNNING_MODE_VIDEO,
+ .max = IPU3_RUNNING_MODE_STILL,
+ .step = 1,
+ .def = IPU3_RUNNING_MODE_VIDEO,
+};
+
/******************** Framework registration ********************/
/* helper function to config node's video properties */
@@ -851,70 +1068,85 @@ static void ipu3_node_to_v4l2(u32 node, struct video_device *vdev,
vdev->device_caps = V4L2_CAP_STREAMING | cap;
}
-int ipu3_v4l2_register(struct imgu_device *imgu)
+static int ipu3_v4l2_subdev_register(struct imgu_device *imgu,
+ struct imgu_v4l2_subdev *imgu_sd,
+ unsigned int pipe)
{
- struct v4l2_mbus_framefmt def_bus_fmt = { 0 };
- struct v4l2_pix_format_mplane def_pix_fmt = { 0 };
-
int i, r;
-
- /* Initialize miscellaneous variables */
- imgu->streaming = false;
-
- /* Init media device */
- media_device_pci_init(&imgu->media_dev, imgu->pci_dev, IMGU_NAME);
-
- /* Set up v4l2 device */
- imgu->v4l2_dev.mdev = &imgu->media_dev;
- imgu->v4l2_dev.ctrl_handler = imgu->ctrl_handler;
- r = v4l2_device_register(&imgu->pci_dev->dev, &imgu->v4l2_dev);
- if (r) {
- dev_err(&imgu->pci_dev->dev,
- "failed to register V4L2 device (%d)\n", r);
- goto fail_v4l2_dev;
- }
+ struct v4l2_ctrl_handler *hdl = &imgu_sd->ctrl_handler;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
/* Initialize subdev media entity */
- imgu->subdev_pads = kzalloc(sizeof(*imgu->subdev_pads) *
+ imgu_sd->subdev_pads = kzalloc(sizeof(*imgu_sd->subdev_pads) *
IMGU_NODE_NUM, GFP_KERNEL);
- if (!imgu->subdev_pads) {
- r = -ENOMEM;
- goto fail_subdev_pads;
- }
- r = media_entity_pads_init(&imgu->subdev.entity, IMGU_NODE_NUM,
- imgu->subdev_pads);
+ if (!imgu_sd->subdev_pads)
+ return -ENOMEM;
+
+ r = media_entity_pads_init(&imgu_sd->subdev.entity, IMGU_NODE_NUM,
+ imgu_sd->subdev_pads);
if (r) {
dev_err(&imgu->pci_dev->dev,
"failed initialize subdev media entity (%d)\n", r);
goto fail_media_entity;
}
- imgu->subdev.entity.ops = &ipu3_media_ops;
+ imgu_sd->subdev.entity.ops = &ipu3_media_ops;
for (i = 0; i < IMGU_NODE_NUM; i++) {
- imgu->subdev_pads[i].flags = imgu->nodes[i].output ?
+ imgu_sd->subdev_pads[i].flags = imgu_pipe->nodes[i].output ?
MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
}
/* Initialize subdev */
- v4l2_subdev_init(&imgu->subdev, &ipu3_subdev_ops);
- imgu->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
- imgu->subdev.internal_ops = &ipu3_subdev_internal_ops;
- imgu->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
- strlcpy(imgu->subdev.name, IMGU_NAME, sizeof(imgu->subdev.name));
- v4l2_set_subdevdata(&imgu->subdev, imgu);
- imgu->subdev.ctrl_handler = imgu->ctrl_handler;
- r = v4l2_device_register_subdev(&imgu->v4l2_dev, &imgu->subdev);
- if (r) {
+ v4l2_subdev_init(&imgu_sd->subdev, &ipu3_subdev_ops);
+ imgu_sd->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_STATISTICS;
+ imgu_sd->subdev.internal_ops = &ipu3_subdev_internal_ops;
+ imgu_sd->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
+ V4L2_SUBDEV_FL_HAS_EVENTS;
+ snprintf(imgu_sd->subdev.name, sizeof(imgu_sd->subdev.name),
+ "%s %d", IMGU_NAME, pipe);
+ v4l2_set_subdevdata(&imgu_sd->subdev, imgu);
+ atomic_set(&imgu_sd->running_mode, IPU3_RUNNING_MODE_VIDEO);
+ v4l2_ctrl_handler_init(hdl, 1);
+ imgu_sd->subdev.ctrl_handler = hdl;
+ imgu_sd->ctrl = v4l2_ctrl_new_custom(hdl, &ipu3_subdev_ctrl_mode, NULL);
+ if (hdl->error) {
+ r = hdl->error;
dev_err(&imgu->pci_dev->dev,
- "failed initialize subdev (%d)\n", r);
+ "failed to create subdev v4l2 ctrl with err %d", r);
goto fail_subdev;
}
- r = v4l2_device_register_subdev_nodes(&imgu->v4l2_dev);
+ r = v4l2_device_register_subdev(&imgu->v4l2_dev, &imgu_sd->subdev);
if (r) {
dev_err(&imgu->pci_dev->dev,
- "failed to register subdevs (%d)\n", r);
- goto fail_subdevs;
+ "failed initialize subdev (%d)\n", r);
+ goto fail_subdev;
}
+ imgu_sd->pipe = pipe;
+ return 0;
+
+fail_subdev:
+ v4l2_ctrl_handler_free(imgu_sd->subdev.ctrl_handler);
+ media_entity_cleanup(&imgu_sd->subdev.entity);
+fail_media_entity:
+ kfree(imgu_sd->subdev_pads);
+
+ return r;
+}
+
+static int ipu3_v4l2_node_setup(struct imgu_device *imgu, unsigned int pipe,
+ int node_num)
+{
+ int r;
+ u32 flags;
+ struct v4l2_mbus_framefmt def_bus_fmt = { 0 };
+ struct v4l2_pix_format_mplane def_pix_fmt = { 0 };
+ struct device *dev = &imgu->pci_dev->dev;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
+ struct v4l2_subdev *sd = &imgu_pipe->imgu_sd.subdev;
+ struct imgu_video_device *node = &imgu_pipe->nodes[node_num];
+ struct video_device *vdev = &node->vdev;
+ struct vb2_queue *vbq = &node->vbq;
+
/* Initialize formats to default values */
def_bus_fmt.width = 1920;
def_bus_fmt.height = 1080;
@@ -938,117 +1170,240 @@ int ipu3_v4l2_register(struct imgu_device *imgu)
def_pix_fmt.quantization = def_bus_fmt.quantization;
def_pix_fmt.xfer_func = def_bus_fmt.xfer_func;
- /* Create video nodes and links */
+ /* Initialize miscellaneous variables */
+ mutex_init(&node->lock);
+ INIT_LIST_HEAD(&node->buffers);
+
+ /* Initialize formats to default values */
+ node->pad_fmt = def_bus_fmt;
+ node->id = node_num;
+ node->pipe = pipe;
+ ipu3_node_to_v4l2(node_num, vdev, &node->vdev_fmt);
+ if (node->vdev_fmt.type ==
+ V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
+ node->vdev_fmt.type ==
+ V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+ def_pix_fmt.pixelformat = node->output ?
+ V4L2_PIX_FMT_IPU3_SGRBG10 :
+ V4L2_PIX_FMT_NV12;
+ node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
+ }
+
+ /* Initialize media entities */
+ r = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
+ if (r) {
+ dev_err(dev, "failed initialize media entity (%d)\n", r);
+ mutex_destroy(&node->lock);
+ return r;
+ }
+ node->vdev_pad.flags = node->output ?
+ MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
+ vdev->entity.ops = NULL;
+
+ /* Initialize vbq */
+ vbq->type = node->vdev_fmt.type;
+ vbq->io_modes = VB2_USERPTR | VB2_MMAP | VB2_DMABUF;
+ vbq->ops = &ipu3_vb2_ops;
+ vbq->mem_ops = &vb2_dma_sg_memops;
+ if (imgu->buf_struct_size <= 0)
+ imgu->buf_struct_size =
+ sizeof(struct ipu3_vb2_buffer);
+ vbq->buf_struct_size = imgu->buf_struct_size;
+ vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+ /* can streamon w/o buffers */
+ vbq->min_buffers_needed = 0;
+ vbq->drv_priv = imgu;
+ vbq->lock = &node->lock;
+ r = vb2_queue_init(vbq);
+ if (r) {
+ dev_err(dev, "failed to initialize video queue (%d)", r);
+ media_entity_cleanup(&vdev->entity);
+ return r;
+ }
+
+ /* Initialize vdev */
+ snprintf(vdev->name, sizeof(vdev->name), "%s %d %s",
+ IMGU_NAME, pipe, node->name);
+ vdev->release = video_device_release_empty;
+ vdev->fops = &ipu3_v4l2_fops;
+ vdev->lock = &node->lock;
+ vdev->v4l2_dev = &imgu->v4l2_dev;
+ vdev->queue = &node->vbq;
+ vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
+ video_set_drvdata(vdev, imgu);
+ r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
+ if (r) {
+ dev_err(dev, "failed to register video device (%d)", r);
+ media_entity_cleanup(&vdev->entity);
+ return r;
+ }
+
+ /* Create link between video node and the subdev pad */
+ flags = 0;
+ if (node->enabled)
+ flags |= MEDIA_LNK_FL_ENABLED;
+ if (node->output) {
+ r = media_create_pad_link(&vdev->entity, 0, &sd->entity,
+ node_num, flags);
+ } else {
+ r = media_create_pad_link(&sd->entity, node_num, &vdev->entity,
+ 0, flags);
+ }
+ if (r) {
+ dev_err(dev, "failed to create pad link (%d)", r);
+ video_unregister_device(vdev);
+ return r;
+ }
+
+ return 0;
+}
+
+static void ipu3_v4l2_nodes_cleanup_pipe(struct imgu_device *imgu,
+ unsigned int pipe, int node)
+{
+ int i;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
+
+ for (i = 0; i < node; i++) {
+ video_unregister_device(&imgu_pipe->nodes[i].vdev);
+ media_entity_cleanup(&imgu_pipe->nodes[i].vdev.entity);
+ mutex_destroy(&imgu_pipe->nodes[i].lock);
+ }
+}
+
+static void ipu3_v4l2_nodes_cleanup(struct imgu_device *imgu, unsigned int pipe)
+{
+ int i;
+
+ for (i = 0; i < pipe; i++) {
+ ipu3_v4l2_nodes_cleanup_pipe(imgu, i, IMGU_NODE_NUM);
+ }
+}
+
+static int ipu3_v4l2_nodes_setup_pipe(struct imgu_device *imgu, int pipe)
+{
+ int i, r;
+
for (i = 0; i < IMGU_NODE_NUM; i++) {
- struct imgu_video_device *node = &imgu->nodes[i];
- struct video_device *vdev = &node->vdev;
- struct vb2_queue *vbq = &node->vbq;
- u32 flags;
-
- /* Initialize miscellaneous variables */
- mutex_init(&node->lock);
- INIT_LIST_HEAD(&node->buffers);
-
- /* Initialize formats to default values */
- node->pad_fmt = def_bus_fmt;
- ipu3_node_to_v4l2(i, vdev, &node->vdev_fmt);
- if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
- node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
- def_pix_fmt.pixelformat = node->output ?
- V4L2_PIX_FMT_IPU3_SGRBG10 :
- V4L2_PIX_FMT_NV12;
- node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
- }
- /* Initialize media entities */
- r = media_entity_pads_init(&vdev->entity, 1, &node->vdev_pad);
- if (r) {
- dev_err(&imgu->pci_dev->dev,
- "failed initialize media entity (%d)\n", r);
- goto fail_vdev_media_entity;
- }
- node->vdev_pad.flags = node->output ?
- MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
- vdev->entity.ops = NULL;
-
- /* Initialize vbq */
- vbq->type = node->vdev_fmt.type;
- vbq->io_modes = VB2_USERPTR | VB2_MMAP | VB2_DMABUF;
- vbq->ops = &ipu3_vb2_ops;
- vbq->mem_ops = &vb2_dma_sg_memops;
- if (imgu->buf_struct_size <= 0)
- imgu->buf_struct_size = sizeof(struct ipu3_vb2_buffer);
- vbq->buf_struct_size = imgu->buf_struct_size;
- vbq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
- vbq->min_buffers_needed = 0; /* Can streamon w/o buffers */
- vbq->drv_priv = imgu;
- vbq->lock = &node->lock;
- r = vb2_queue_init(vbq);
- if (r) {
- dev_err(&imgu->pci_dev->dev,
- "failed to initialize video queue (%d)\n", r);
- goto fail_vdev;
- }
+ r = ipu3_v4l2_node_setup(imgu, pipe, i);
+ if (r)
+ goto cleanup;
+ }
+
+ return 0;
+
+cleanup:
+ ipu3_v4l2_nodes_cleanup_pipe(imgu, pipe, i);
+ return r;
+}
+
+static int ipu3_v4l2_nodes_setup(struct imgu_device *imgu)
+{
+ int i, r;
+
+ /* Create video nodes and links */
+ for (i = 0; i < IMGU_MAX_PIPE_NUM; i++) {
+ r = ipu3_v4l2_nodes_setup_pipe(imgu, i);
+ if (r)
+ goto cleanup;
+ }
+
+ return 0;
- /* Initialize vdev */
- snprintf(vdev->name, sizeof(vdev->name), "%s %s",
- IMGU_NAME, node->name);
- vdev->release = video_device_release_empty;
- vdev->fops = &ipu3_v4l2_fops;
- vdev->lock = &node->lock;
- vdev->v4l2_dev = &imgu->v4l2_dev;
- vdev->queue = &node->vbq;
- vdev->vfl_dir = node->output ? VFL_DIR_TX : VFL_DIR_RX;
- video_set_drvdata(vdev, imgu);
- r = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
+cleanup:
+ ipu3_v4l2_nodes_cleanup(imgu, i);
+ return r;
+}
+
+static void ipu3_v4l2_subdevs_cleanup(struct imgu_device *imgu,
+ unsigned int pipe)
+{
+ int i;
+ struct imgu_media_pipe *imgu_pipe;
+
+ for (i = 0; i < pipe; i++) {
+ imgu_pipe = &imgu->imgu_pipe[i];
+ v4l2_device_unregister_subdev(&imgu_pipe->imgu_sd.subdev);
+ v4l2_ctrl_handler_free(imgu_pipe->imgu_sd.subdev.ctrl_handler);
+ media_entity_cleanup(&imgu_pipe->imgu_sd.subdev.entity);
+ kfree(imgu_pipe->imgu_sd.subdev_pads);
+ }
+}
+
+static int ipu3_v4l2_register_pipes(struct imgu_device *imgu)
+{
+ struct imgu_media_pipe *imgu_pipe;
+ int i, r;
+
+ for (i = 0; i < IMGU_MAX_PIPE_NUM; i++) {
+ imgu_pipe = &imgu->imgu_pipe[i];
+ r = ipu3_v4l2_subdev_register(imgu, &imgu_pipe->imgu_sd, i);
if (r) {
dev_err(&imgu->pci_dev->dev,
- "failed to register video device (%d)\n", r);
- goto fail_vdev;
+ "failed to register subdev%d ret (%d)\n", i, r);
+ break;
}
+ }
- /* Create link between video node and the subdev pad */
- flags = 0;
- if (node->enabled)
- flags |= MEDIA_LNK_FL_ENABLED;
- if (node->immutable)
- flags |= MEDIA_LNK_FL_IMMUTABLE;
- if (node->output) {
- r = media_create_pad_link(&vdev->entity, 0,
- &imgu->subdev.entity,
- i, flags);
- } else {
- r = media_create_pad_link(&imgu->subdev.entity,
- i, &vdev->entity, 0, flags);
- }
- if (r)
- goto fail_link;
+ if (i == IMGU_MAX_PIPE_NUM)
+ return 0;
+
+ ipu3_v4l2_subdevs_cleanup(imgu, i);
+ return r;
+}
+
+int ipu3_v4l2_register(struct imgu_device *imgu)
+{
+ int r;
+
+ /* Initialize miscellaneous variables */
+ imgu->streaming = false;
+
+ /* Set up media device */
+ media_device_pci_init(&imgu->media_dev, imgu->pci_dev, IMGU_NAME);
+
+ /* Set up v4l2 device */
+ imgu->v4l2_dev.mdev = &imgu->media_dev;
+ imgu->v4l2_dev.ctrl_handler = NULL;
+ r = v4l2_device_register(&imgu->pci_dev->dev, &imgu->v4l2_dev);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register V4L2 device (%d)\n", r);
+ goto fail_v4l2_dev;
+ }
+
+ r = ipu3_v4l2_register_pipes(imgu);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register pipes (%d)\n", r);
+ goto fail_v4l2_pipes;
+ }
+
+ r = v4l2_device_register_subdev_nodes(&imgu->v4l2_dev);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to register subdevs (%d)\n", r);
+ goto fail_subdevs;
+ }
+
+ r = ipu3_v4l2_nodes_setup(imgu);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev, "failed to setup nodes (%d)", r);
+ goto fail_subdevs;
}
r = media_device_register(&imgu->media_dev);
if (r) {
dev_err(&imgu->pci_dev->dev,
"failed to register media device (%d)\n", r);
- i--;
- goto fail_link;
+ goto fail_subdevs;
}
return 0;
- for (; i >= 0; i--) {
-fail_link:
- video_unregister_device(&imgu->nodes[i].vdev);
-fail_vdev:
- media_entity_cleanup(&imgu->nodes[i].vdev.entity);
-fail_vdev_media_entity:
- mutex_destroy(&imgu->nodes[i].lock);
- }
fail_subdevs:
- v4l2_device_unregister_subdev(&imgu->subdev);
-fail_subdev:
- media_entity_cleanup(&imgu->subdev.entity);
-fail_media_entity:
- kfree(imgu->subdev_pads);
-fail_subdev_pads:
+ ipu3_v4l2_subdevs_cleanup(imgu, IMGU_MAX_PIPE_NUM);
+fail_v4l2_pipes:
v4l2_device_unregister(&imgu->v4l2_dev);
fail_v4l2_dev:
media_device_cleanup(&imgu->media_dev);
@@ -1059,21 +1414,11 @@ EXPORT_SYMBOL_GPL(ipu3_v4l2_register);
int ipu3_v4l2_unregister(struct imgu_device *imgu)
{
- unsigned int i;
-
media_device_unregister(&imgu->media_dev);
- media_device_cleanup(&imgu->media_dev);
-
- for (i = 0; i < IMGU_NODE_NUM; i++) {
- video_unregister_device(&imgu->nodes[i].vdev);
- media_entity_cleanup(&imgu->nodes[i].vdev.entity);
- mutex_destroy(&imgu->nodes[i].lock);
- }
-
- v4l2_device_unregister_subdev(&imgu->subdev);
- media_entity_cleanup(&imgu->subdev.entity);
- kfree(imgu->subdev_pads);
+ ipu3_v4l2_nodes_cleanup(imgu, IMGU_MAX_PIPE_NUM);
+ ipu3_v4l2_subdevs_cleanup(imgu, IMGU_MAX_PIPE_NUM);
v4l2_device_unregister(&imgu->v4l2_dev);
+ media_device_cleanup(&imgu->media_dev);
return 0;
}
diff --git a/drivers/media/pci/intel/ipu3/ipu3.c b/drivers/media/pci/intel/ipu3/ipu3.c
index eda7299..ff2c35a 100644
--- a/drivers/media/pci/intel/ipu3/ipu3.c
+++ b/drivers/media/pci/intel/ipu3/ipu3.c
@@ -45,7 +45,6 @@ static const struct imgu_node_mapping imgu_node_map[IMGU_NODE_NUM] = {
[IMGU_NODE_PARAMS] = {IPU3_CSS_QUEUE_PARAMS, "parameters"},
[IMGU_NODE_OUT] = {IPU3_CSS_QUEUE_OUT, "output"},
[IMGU_NODE_VF] = {IPU3_CSS_QUEUE_VF, "viewfinder"},
- [IMGU_NODE_PV] = {IPU3_CSS_QUEUE_VF, "postview"},
[IMGU_NODE_STAT_3A] = {IPU3_CSS_QUEUE_STAT_3A, "3a stat"},
};
@@ -58,10 +57,6 @@ unsigned int imgu_map_node(struct imgu_device *imgu, unsigned int css_queue)
{
unsigned int i;
- if (css_queue == IPU3_CSS_QUEUE_VF)
- return imgu->nodes[IMGU_NODE_VF].enabled ?
- IMGU_NODE_VF : IMGU_NODE_PV;
-
for (i = 0; i < IMGU_NODE_NUM; i++)
if (imgu_node_map[i].css_queue == css_queue)
break;
@@ -71,18 +66,22 @@ unsigned int imgu_map_node(struct imgu_device *imgu, unsigned int css_queue)
/**************** Dummy buffers ****************/
-static void imgu_dummybufs_cleanup(struct imgu_device *imgu)
+static void imgu_dummybufs_cleanup(struct imgu_device *imgu, unsigned int pipe)
{
unsigned int i;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
for (i = 0; i < IPU3_CSS_QUEUES; i++)
- ipu3_dmamap_free(imgu, &imgu->queues[i].dmap);
+ ipu3_dmamap_free(imgu,
+ &imgu_pipe->queues[i].dmap);
}
-static int imgu_dummybufs_preallocate(struct imgu_device *imgu)
+static int imgu_dummybufs_preallocate(struct imgu_device *imgu,
+ unsigned int pipe)
{
unsigned int i;
size_t size;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
for (i = 0; i < IPU3_CSS_QUEUES; i++) {
size = css_queue_buf_size_map[i];
@@ -94,8 +93,9 @@ static int imgu_dummybufs_preallocate(struct imgu_device *imgu)
if (i == IMGU_QUEUE_MASTER || size == 0)
continue;
- if (!ipu3_dmamap_alloc(imgu, &imgu->queues[i].dmap, size)) {
- imgu_dummybufs_cleanup(imgu);
+ if (!ipu3_dmamap_alloc(imgu,
+ &imgu_pipe->queues[i].dmap, size)) {
+ imgu_dummybufs_cleanup(imgu, pipe);
return -ENOMEM;
}
}
@@ -103,45 +103,46 @@ static int imgu_dummybufs_preallocate(struct imgu_device *imgu)
return 0;
}
-static int imgu_dummybufs_init(struct imgu_device *imgu)
+static int imgu_dummybufs_init(struct imgu_device *imgu, unsigned int pipe)
{
const struct v4l2_pix_format_mplane *mpix;
const struct v4l2_meta_format *meta;
- unsigned int i, j, node;
+ unsigned int i, k, node;
size_t size;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
/* Allocate a dummy buffer for each queue where buffer is optional */
for (i = 0; i < IPU3_CSS_QUEUES; i++) {
node = imgu_map_node(imgu, i);
- if (!imgu->queue_enabled[node] || i == IMGU_QUEUE_MASTER)
+ if (!imgu_pipe->queue_enabled[node] || i == IMGU_QUEUE_MASTER)
continue;
- if (!imgu->nodes[IMGU_NODE_VF].enabled &&
- !imgu->nodes[IMGU_NODE_PV].enabled &&
+ if (!imgu_pipe->nodes[IMGU_NODE_VF].enabled &&
i == IPU3_CSS_QUEUE_VF)
/*
- * Do not enable dummy buffers for VF/PV if it is not
+ * Do not enable dummy buffers for VF if it is not
* requested by the user.
*/
continue;
- meta = &imgu->nodes[node].vdev_fmt.fmt.meta;
- mpix = &imgu->nodes[node].vdev_fmt.fmt.pix_mp;
+ meta = &imgu_pipe->nodes[node].vdev_fmt.fmt.meta;
+ mpix = &imgu_pipe->nodes[node].vdev_fmt.fmt.pix_mp;
if (node == IMGU_NODE_STAT_3A || node == IMGU_NODE_PARAMS)
size = meta->buffersize;
else
size = mpix->plane_fmt[0].sizeimage;
- if (ipu3_css_dma_buffer_resize(imgu, &imgu->queues[i].dmap,
+ if (ipu3_css_dma_buffer_resize(imgu,
+ &imgu_pipe->queues[i].dmap,
size)) {
- imgu_dummybufs_cleanup(imgu);
+ imgu_dummybufs_cleanup(imgu, pipe);
return -ENOMEM;
}
- for (j = 0; j < IMGU_MAX_QUEUE_DEPTH; j++)
- ipu3_css_buf_init(&imgu->queues[i].dummybufs[j], i,
- imgu->queues[i].dmap.daddr);
+ for (k = 0; k < IMGU_MAX_QUEUE_DEPTH; k++)
+ ipu3_css_buf_init(&imgu_pipe->queues[i].dummybufs[k], i,
+ imgu_pipe->queues[i].dmap.daddr);
}
return 0;
@@ -149,40 +150,43 @@ static int imgu_dummybufs_init(struct imgu_device *imgu)
/* May be called from atomic context */
static struct ipu3_css_buffer *imgu_dummybufs_get(struct imgu_device *imgu,
- int queue)
+ int queue, unsigned int pipe)
{
unsigned int i;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
/* dummybufs are not allocated for master q */
if (queue == IPU3_CSS_QUEUE_IN)
return NULL;
- if (WARN_ON(!imgu->queues[queue].dmap.vaddr))
+ if (WARN_ON(!imgu_pipe->queues[queue].dmap.vaddr))
/* Buffer should not be allocated here */
return NULL;
for (i = 0; i < IMGU_MAX_QUEUE_DEPTH; i++)
- if (ipu3_css_buf_state(&imgu->queues[queue].dummybufs[i]) !=
+ if (ipu3_css_buf_state(&imgu_pipe->queues[queue].dummybufs[i]) !=
IPU3_CSS_BUFFER_QUEUED)
break;
if (i == IMGU_MAX_QUEUE_DEPTH)
return NULL;
- ipu3_css_buf_init(&imgu->queues[queue].dummybufs[i], queue,
- imgu->queues[queue].dmap.daddr);
+ ipu3_css_buf_init(&imgu_pipe->queues[queue].dummybufs[i], queue,
+ imgu_pipe->queues[queue].dmap.daddr);
- return &imgu->queues[queue].dummybufs[i];
+ return &imgu_pipe->queues[queue].dummybufs[i];
}
/* Check if given buffer is a dummy buffer */
static bool imgu_dummybufs_check(struct imgu_device *imgu,
- struct ipu3_css_buffer *buf)
+ struct ipu3_css_buffer *buf,
+ unsigned int pipe)
{
unsigned int i;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
for (i = 0; i < IMGU_MAX_QUEUE_DEPTH; i++)
- if (buf == &imgu->queues[buf->queue].dummybufs[i])
+ if (buf == &imgu_pipe->queues[buf->queue].dummybufs[i])
break;
return i < IMGU_MAX_QUEUE_DEPTH;
@@ -197,63 +201,64 @@ static void imgu_buffer_done(struct imgu_device *imgu, struct vb2_buffer *vb,
}
static struct ipu3_css_buffer *imgu_queue_getbuf(struct imgu_device *imgu,
- unsigned int node)
+ unsigned int node,
+ unsigned int pipe)
{
struct imgu_buffer *buf;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
if (WARN_ON(node >= IMGU_NODE_NUM))
return NULL;
/* Find first free buffer from the node */
- list_for_each_entry(buf, &imgu->nodes[node].buffers, vid_buf.list) {
+ list_for_each_entry(buf, &imgu_pipe->nodes[node].buffers, vid_buf.list) {
if (ipu3_css_buf_state(&buf->css_buf) == IPU3_CSS_BUFFER_NEW)
return &buf->css_buf;
}
/* There were no free buffers, try to return a dummy buffer */
- return imgu_dummybufs_get(imgu, imgu_node_map[node].css_queue);
+ return imgu_dummybufs_get(imgu, imgu_node_map[node].css_queue, pipe);
}
/*
* Queue as many buffers to CSS as possible. If all buffers don't fit into
* CSS buffer queues, they remain unqueued and will be queued later.
*/
-int imgu_queue_buffers(struct imgu_device *imgu, bool initial)
+int imgu_queue_buffers(struct imgu_device *imgu, bool initial, unsigned int pipe)
{
unsigned int node;
int r = 0;
struct imgu_buffer *ibuf;
+ struct imgu_media_pipe *imgu_pipe = &imgu->imgu_pipe[pipe];
if (!ipu3_css_is_streaming(&imgu->css))
return 0;
+ dev_dbg(&imgu->pci_dev->dev, "Queue buffers to pipe %d", pipe);
mutex_lock(&imgu->lock);
/* Buffer set is queued to FW only when input buffer is ready */
for (node = IMGU_NODE_NUM - 1;
- imgu_queue_getbuf(imgu, IMGU_NODE_IN);
+ imgu_queue_getbuf(imgu, IMGU_NODE_IN, pipe);
node = node ? node - 1 : IMGU_NODE_NUM - 1) {
if (node == IMGU_NODE_VF &&
- (imgu->css.pipe_id == IPU3_CSS_PIPE_ID_CAPTURE ||
- !imgu->nodes[IMGU_NODE_VF].enabled)) {
- continue;
- } else if (node == IMGU_NODE_PV &&
- (imgu->css.pipe_id == IPU3_CSS_PIPE_ID_VIDEO ||
- !imgu->nodes[IMGU_NODE_PV].enabled)) {
+ !imgu_pipe->nodes[IMGU_NODE_VF].enabled) {
+ dev_warn(&imgu->pci_dev->dev,
+ "Vf not enabled, ignore queue");
continue;
- } else if (imgu->queue_enabled[node]) {
+ } else if (imgu_pipe->queue_enabled[node]) {
struct ipu3_css_buffer *buf =
- imgu_queue_getbuf(imgu, node);
+ imgu_queue_getbuf(imgu, node, pipe);
int dummy;
if (!buf)
break;
- r = ipu3_css_buf_queue(&imgu->css, buf);
+ r = ipu3_css_buf_queue(&imgu->css, pipe, buf);
if (r)
break;
- dummy = imgu_dummybufs_check(imgu, buf);
+ dummy = imgu_dummybufs_check(imgu, buf, pipe);
if (!dummy)
ibuf = container_of(buf, struct imgu_buffer,
css_buf);
@@ -288,14 +293,15 @@ int imgu_queue_buffers(struct imgu_device *imgu, bool initial)
for (node = 0; node < IMGU_NODE_NUM; node++) {
struct imgu_buffer *buf, *buf0;
- if (!imgu->queue_enabled[node])
+ if (!imgu_pipe->queue_enabled[node])
continue; /* Skip disabled queues */
mutex_lock(&imgu->lock);
- list_for_each_entry_safe(buf, buf0, &imgu->nodes[node].buffers,
+ list_for_each_entry_safe(buf, buf0,
+ &imgu_pipe->nodes[node].buffers,
vid_buf.list) {
if (ipu3_css_buf_state(&buf->css_buf) ==
- IPU3_CSS_BUFFER_QUEUED)
+ IPU3_CSS_BUFFER_QUEUED)
continue; /* Was already queued, skip */
ipu3_v4l2_buffer_done(&buf->vid_buf.vbb.vb2_buf,
@@ -328,10 +334,7 @@ static void imgu_powerdown(struct imgu_device *imgu)
int imgu_s_stream(struct imgu_device *imgu, int enable)
{
struct device *dev = &imgu->pci_dev->dev;
- struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
- struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
- unsigned int i, node;
- int r;
+ int r, pipe;
if (!enable) {
/* Stop streaming */
@@ -347,54 +350,6 @@ int imgu_s_stream(struct imgu_device *imgu, int enable)
return 0;
}
- /* Start streaming */
-
- dev_dbg(dev, "stream on\n");
- for (i = 0; i < IMGU_NODE_NUM; i++)
- imgu->queue_enabled[i] = imgu->nodes[i].enabled;
-
- /*
- * CSS library expects that the following queues are
- * always enabled; if buffers are not provided to some of the
- * queues, it stalls due to lack of buffers.
- * Force the queues to be enabled and if the user really hasn't
- * enabled them, use dummy buffers.
- */
- imgu->queue_enabled[IMGU_NODE_OUT] = true;
- imgu->queue_enabled[IMGU_NODE_VF] = true;
- imgu->queue_enabled[IMGU_NODE_PV] = true;
- imgu->queue_enabled[IMGU_NODE_STAT_3A] = true;
-
- /* This is handled specially */
- imgu->queue_enabled[IPU3_CSS_QUEUE_PARAMS] = false;
-
- /* Initialize CSS formats */
- for (i = 0; i < IPU3_CSS_QUEUES; i++) {
- node = imgu_map_node(imgu, i);
- /* No need to reconfig meta nodes */
- if (node == IMGU_NODE_STAT_3A || node == IMGU_NODE_PARAMS)
- continue;
- fmts[i] = imgu->queue_enabled[node] ?
- &imgu->nodes[node].vdev_fmt.fmt.pix_mp : NULL;
- }
-
- /* Enable VF output only when VF or PV queue requested by user */
- imgu->css.vf_output_en = IPU3_NODE_VF_DISABLED;
- if (imgu->nodes[IMGU_NODE_VF].enabled)
- imgu->css.vf_output_en = IPU3_NODE_VF_ENABLED;
- else if (imgu->nodes[IMGU_NODE_PV].enabled)
- imgu->css.vf_output_en = IPU3_NODE_PV_ENABLED;
-
- rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu->rect.eff;
- rects[IPU3_CSS_RECT_BDS] = &imgu->rect.bds;
- rects[IPU3_CSS_RECT_GDC] = &imgu->rect.gdc;
-
- r = ipu3_css_fmt_set(&imgu->css, fmts, rects);
- if (r) {
- dev_err(dev, "failed to set initial formats (%d)", r);
- return r;
- }
-
/* Set Power */
r = pm_runtime_get_sync(dev);
if (r < 0) {
@@ -417,24 +372,26 @@ int imgu_s_stream(struct imgu_device *imgu, int enable)
goto fail_start_streaming;
}
- /* Initialize dummy buffers */
- r = imgu_dummybufs_init(imgu);
- if (r) {
- dev_err(dev, "failed to initialize dummy buffers (%d)", r);
- goto fail_dummybufs;
- }
+ for_each_set_bit(pipe, imgu->css.enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ /* Initialize dummy buffers */
+ r = imgu_dummybufs_init(imgu, pipe);
+ if (r) {
+ dev_err(dev, "failed to initialize dummy buffers (%d)", r);
+ goto fail_dummybufs;
+ }
- /* Queue as many buffers from queue as possible */
- r = imgu_queue_buffers(imgu, true);
- if (r) {
- dev_err(dev, "failed to queue initial buffers (%d)", r);
- goto fail_queueing;
+ /* Queue as many buffers from queue as possible */
+ r = imgu_queue_buffers(imgu, true, pipe);
+ if (r) {
+ dev_err(dev, "failed to queue initial buffers (%d)", r);
+ goto fail_queueing;
+ }
}
return 0;
-
fail_queueing:
- imgu_dummybufs_cleanup(imgu);
+ for_each_set_bit(pipe, imgu->css.enabled_pipes, IMGU_MAX_PIPE_NUM)
+ imgu_dummybufs_cleanup(imgu, pipe);
fail_dummybufs:
ipu3_css_stop_streaming(&imgu->css);
fail_start_streaming:
@@ -447,51 +404,66 @@ static int imgu_video_nodes_init(struct imgu_device *imgu)
{
struct v4l2_pix_format_mplane *fmts[IPU3_CSS_QUEUES] = { NULL };
struct v4l2_rect *rects[IPU3_CSS_RECTS] = { NULL };
- unsigned int i;
+ struct imgu_media_pipe *imgu_pipe;
+ unsigned int i, j;
int r;
imgu->buf_struct_size = sizeof(struct imgu_buffer);
- for (i = 0; i < IMGU_NODE_NUM; i++) {
- imgu->nodes[i].name = imgu_node_map[i].name;
- imgu->nodes[i].output = i < IMGU_QUEUE_FIRST_INPUT;
- imgu->nodes[i].immutable = false;
- imgu->nodes[i].enabled = false;
+ for (j = 0; j < IMGU_MAX_PIPE_NUM; j++) {
+ imgu_pipe = &imgu->imgu_pipe[j];
- if (i != IMGU_NODE_PARAMS && i != IMGU_NODE_STAT_3A)
- fmts[imgu_node_map[i].css_queue] =
- &imgu->nodes[i].vdev_fmt.fmt.pix_mp;
- atomic_set(&imgu->nodes[i].sequence, 0);
- }
+ for (i = 0; i < IMGU_NODE_NUM; i++) {
+ imgu_pipe->nodes[i].name = imgu_node_map[i].name;
+ imgu_pipe->nodes[i].output = i < IMGU_QUEUE_FIRST_INPUT;
+ imgu_pipe->nodes[i].enabled = false;
- /* Master queue is always enabled */
- imgu->nodes[IMGU_QUEUE_MASTER].immutable = true;
- imgu->nodes[IMGU_QUEUE_MASTER].enabled = true;
+ if (i != IMGU_NODE_PARAMS && i != IMGU_NODE_STAT_3A)
+ fmts[imgu_node_map[i].css_queue] =
+ &imgu_pipe->nodes[i].vdev_fmt.fmt.pix_mp;
+ atomic_set(&imgu_pipe->nodes[i].sequence, 0);
+ }
+ }
r = ipu3_v4l2_register(imgu);
if (r)
return r;
/* Set initial formats and initialize formats of video nodes */
- rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu->rect.eff;
- rects[IPU3_CSS_RECT_BDS] = &imgu->rect.bds;
- ipu3_css_fmt_set(&imgu->css, fmts, rects);
+ for (j = 0; j < IMGU_MAX_PIPE_NUM; j++) {
+ imgu_pipe = &imgu->imgu_pipe[j];
- /* Pre-allocate dummy buffers */
- r = imgu_dummybufs_preallocate(imgu);
- if (r) {
- dev_err(&imgu->pci_dev->dev,
- "failed to pre-allocate dummy buffers (%d)", r);
- imgu_dummybufs_cleanup(imgu);
- ipu3_v4l2_unregister(imgu);
+ rects[IPU3_CSS_RECT_EFFECTIVE] = &imgu_pipe->imgu_sd.rect.eff;
+ rects[IPU3_CSS_RECT_BDS] = &imgu_pipe->imgu_sd.rect.bds;
+ ipu3_css_fmt_set(&imgu->css, fmts, rects, j);
+
+ /* Pre-allocate dummy buffers */
+ r = imgu_dummybufs_preallocate(imgu, j);
+ if (r) {
+ dev_err(&imgu->pci_dev->dev,
+ "failed to pre-allocate dummy buffers (%d)", r);
+ goto out_cleanup;
+ }
}
return 0;
+
+out_cleanup:
+ for (j = 0; j < IMGU_MAX_PIPE_NUM; j++)
+ imgu_dummybufs_cleanup(imgu, j);
+
+ ipu3_v4l2_unregister(imgu);
+
+ return r;
}
static void imgu_video_nodes_exit(struct imgu_device *imgu)
{
- imgu_dummybufs_cleanup(imgu);
+ int i;
+
+ for (i = 0; i < IMGU_MAX_PIPE_NUM; i++)
+ imgu_dummybufs_cleanup(imgu, i);
+
ipu3_v4l2_unregister(imgu);
}
@@ -500,13 +472,15 @@ static void imgu_video_nodes_exit(struct imgu_device *imgu)
static irqreturn_t imgu_isr_threaded(int irq, void *imgu_ptr)
{
struct imgu_device *imgu = imgu_ptr;
+ struct imgu_media_pipe *imgu_pipe;
+ int p;
/* Dequeue / queue buffers */
do {
u64 ns = ktime_get_ns();
struct ipu3_css_buffer *b;
struct imgu_buffer *buf;
- unsigned int node;
+ unsigned int node, pipe;
bool dummy;
do {
@@ -525,25 +499,31 @@ static irqreturn_t imgu_isr_threaded(int irq, void *imgu_ptr)
}
node = imgu_map_node(imgu, b->queue);
- dummy = imgu_dummybufs_check(imgu, b);
+ pipe = b->pipe;
+ dummy = imgu_dummybufs_check(imgu, b, pipe);
if (!dummy)
buf = container_of(b, struct imgu_buffer, css_buf);
dev_dbg(&imgu->pci_dev->dev,
- "dequeue %s %s buffer %d from css\n",
+ "dequeue %s %s buffer %d daddr 0x%x from css\n",
dummy ? "dummy" : "user",
imgu_node_map[node].name,
- dummy ? 0 : buf->vid_buf.vbb.vb2_buf.index);
+ dummy ? 0 : buf->vid_buf.vbb.vb2_buf.index,
+ (u32)b->daddr);
if (dummy)
/* It was a dummy buffer, skip it */
continue;
/* Fill vb2 buffer entries and tell it's ready */
- if (!imgu->nodes[node].output) {
+ imgu_pipe = &imgu->imgu_pipe[pipe];
+ if (!imgu_pipe->nodes[node].output) {
buf->vid_buf.vbb.vb2_buf.timestamp = ns;
buf->vid_buf.vbb.field = V4L2_FIELD_NONE;
buf->vid_buf.vbb.sequence =
- atomic_inc_return(&imgu->nodes[node].sequence);
+ atomic_inc_return(
+ &imgu_pipe->nodes[node].sequence);
+ dev_dbg(&imgu->pci_dev->dev, "vb2 buffer sequence %d",
+ buf->vid_buf.vbb.sequence);
}
imgu_buffer_done(imgu, &buf->vid_buf.vbb.vb2_buf,
ipu3_css_buf_state(&buf->css_buf) ==
@@ -562,7 +542,8 @@ static irqreturn_t imgu_isr_threaded(int irq, void *imgu_ptr)
* to be queued to CSS.
*/
if (!atomic_read(&imgu->qbuf_barrier))
- imgu_queue_buffers(imgu, false);
+ for_each_set_bit(p, imgu->css.enabled_pipes, IMGU_MAX_PIPE_NUM)
+ imgu_queue_buffers(imgu, false, p);
return IRQ_HANDLED;
}
@@ -772,6 +753,7 @@ static int __maybe_unused imgu_resume(struct device *dev)
struct pci_dev *pci_dev = to_pci_dev(dev);
struct imgu_device *imgu = pci_get_drvdata(pci_dev);
int r = 0;
+ unsigned int pipe;
dev_dbg(dev, "enter %s\n", __func__);
@@ -793,9 +775,13 @@ static int __maybe_unused imgu_resume(struct device *dev)
goto out;
}
- r = imgu_queue_buffers(imgu, true);
- if (r)
- dev_err(dev, "failed to queue buffers (%d)", r);
+ for_each_set_bit(pipe, imgu->css.enabled_pipes, IMGU_MAX_PIPE_NUM) {
+ r = imgu_queue_buffers(imgu, true, pipe);
+ if (r)
+ dev_err(dev, "failed to queue buffers to pipe %d (%d)",
+ pipe, r);
+ }
+
out:
dev_dbg(dev, "leave %s\n", __func__);
diff --git a/drivers/media/pci/intel/ipu3/ipu3.h b/drivers/media/pci/intel/ipu3/ipu3.h
index 5c2b420..8abae6d 100644
--- a/drivers/media/pci/intel/ipu3/ipu3.h
+++ b/drivers/media/pci/intel/ipu3/ipu3.h
@@ -7,6 +7,7 @@
#include <linux/iova.h>
#include <linux/pci.h>
+#include <media/v4l2-ctrls.h>
#include <media/v4l2-device.h>
#include <media/videobuf2-dma-sg.h>
@@ -28,9 +29,8 @@
#define IMGU_NODE_PARAMS 1 /* Input parameters */
#define IMGU_NODE_OUT 2 /* Main output for still or video */
#define IMGU_NODE_VF 3 /* Preview */
-#define IMGU_NODE_PV 4 /* Postview for still capture */
-#define IMGU_NODE_STAT_3A 5 /* 3A statistics */
-#define IMGU_NODE_NUM 6
+#define IMGU_NODE_STAT_3A 4 /* 3A statistics */
+#define IMGU_NODE_NUM 5
#define file_to_intel_ipu3_node(__file) \
container_of(video_devdata(__file), struct imgu_video_device, vdev)
@@ -71,7 +71,6 @@ struct imgu_node_mapping {
struct imgu_video_device {
const char *name;
bool output; /* Frames to the driver? */
- bool immutable; /* Can not be enabled/disabled */
bool enabled;
int queued; /* Buffers already queued */
struct v4l2_format vdev_fmt; /* Currently set format */
@@ -85,14 +84,27 @@ struct imgu_video_device {
/* Protect vb2_queue and vdev structs*/
struct mutex lock;
atomic_t sequence;
+ unsigned int id;
+ unsigned int pipe;
};
-/*
- * imgu_device -- ImgU (Imaging Unit) driver
- */
-struct imgu_device {
- struct pci_dev *pci_dev;
- void __iomem *base;
+struct imgu_v4l2_subdev {
+ unsigned int pipe;
+ struct v4l2_subdev subdev;
+ struct media_pad *subdev_pads;
+ struct {
+ struct v4l2_rect eff; /* effective resolution */
+ struct v4l2_rect bds; /* bayer-domain scaled resolution*/
+ struct v4l2_rect gdc; /* gdc output resolution */
+ } rect;
+ struct v4l2_ctrl_handler ctrl_handler;
+ struct v4l2_ctrl *ctrl;
+ atomic_t running_mode;
+ bool active;
+};
+
+struct imgu_media_pipe {
+ unsigned int pipe;
/* Internally enabled queues */
struct {
@@ -101,18 +113,26 @@ struct imgu_device {
} queues[IPU3_CSS_QUEUES];
struct imgu_video_device nodes[IMGU_NODE_NUM];
bool queue_enabled[IMGU_NODE_NUM];
+ struct media_pipeline pipeline;
+ struct imgu_v4l2_subdev imgu_sd;
+};
+
+/*
+ * imgu_device -- ImgU (Imaging Unit) driver
+ */
+struct imgu_device {
+ struct pci_dev *pci_dev;
+ void __iomem *base;
/* Public fields, fill before registering */
unsigned int buf_struct_size;
bool streaming; /* Public read only */
- struct v4l2_ctrl_handler *ctrl_handler;
+
+ struct imgu_media_pipe imgu_pipe[IMGU_MAX_PIPE_NUM];
/* Private fields */
struct v4l2_device v4l2_dev;
struct media_device media_dev;
- struct media_pipeline pipeline;
- struct v4l2_subdev subdev;
- struct media_pad *subdev_pads;
struct v4l2_file_operations v4l2_file_ops;
/* MMU driver for css */
@@ -129,11 +149,6 @@ struct imgu_device {
struct mutex lock;
/* Forbit streaming and buffer queuing during system suspend. */
atomic_t qbuf_barrier;
- struct {
- struct v4l2_rect eff; /* effective resolution */
- struct v4l2_rect bds; /* bayer-domain scaled resolution*/
- struct v4l2_rect gdc; /* gdc output resolution */
- } rect;
/* Indicate if system suspend take place while imgu is streaming. */
bool suspend_in_stream;
/* Used to wait for FW buffer queue drain. */
@@ -142,7 +157,8 @@ struct imgu_device {
unsigned int imgu_node_to_queue(unsigned int node);
unsigned int imgu_map_node(struct imgu_device *imgu, unsigned int css_queue);
-int imgu_queue_buffers(struct imgu_device *imgu, bool initial);
+int imgu_queue_buffers(struct imgu_device *imgu, bool initial,
+ unsigned int pipe);
int ipu3_v4l2_register(struct imgu_device *dev);
int ipu3_v4l2_unregister(struct imgu_device *dev);
diff --git a/include/uapi/linux/intel-ipu3.h b/include/uapi/linux/intel-ipu3.h
index c2608b6..7ad42b6 100644
--- a/include/uapi/linux/intel-ipu3.h
+++ b/include/uapi/linux/intel-ipu3.h
@@ -2816,4 +2816,12 @@ struct ipu3_uapi_params {
/* Optical black level compensation */
struct ipu3_uapi_obgrid_param obgrid_param;
} __packed;
+
+/* custom ctrl to set pipe mode */
+#define V4L2_CID_INTEL_IPU3_BASE (V4L2_CID_USER_BASE + 0x10a0)
+#define V4L2_CID_INTEL_IPU3_MODE (V4L2_CID_INTEL_IPU3_BASE + 1)
+enum ipu3_running_mode {
+ IPU3_RUNNING_MODE_VIDEO = 0,
+ IPU3_RUNNING_MODE_STILL = 1,
+};
#endif
--
2.7.4
^ permalink raw reply related [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (14 preceding siblings ...)
2018-10-29 22:23 ` [PATCH v7 16/16] intel-ipu3: Add dual pipe support Yong Zhi
@ 2018-11-01 12:03 ` Sakari Ailus
2018-11-07 4:16 ` Bing Bu Cao
2018-11-14 0:25 ` jacopo mondi
2018-11-29 14:43 ` Laurent Pinchart
17 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-01 12:03 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Yong,
Thanks for the update!
On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> Hi,
>
> This series adds support for the Intel IPU3 (Image Processing Unit)
> ImgU which is essentially a modern memory-to-memory ISP. It implements
> raw Bayer to YUV image format conversion as well as a large number of
> other pixel processing algorithms for improving the image quality.
>
> Meta data formats are defined for image statistics (3A, i.e. automatic
> white balance, exposure and focus, histogram and local area contrast
> enhancement) as well as for the pixel processing algorithm parameters.
> The documentation for these formats is currently not included in the
> patchset but will be added in a future version of this set.
>
> The algorithm parameters need to be considered specific to a given frame
> and typically a large number of these parameters change on frame to frame
> basis. Additionally, the parameters are highly structured (and not a flat
> space of independent configuration primitives). They also reflect the
> data structures used by the firmware and the hardware. On top of that,
> the algorithms require highly specialized user space to make meaningful
> use of them. For these reasons it has been chosen video buffers to pass
> the parameters to the device.
>
> On individual patches:
>
> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> image processors and HW accelerators.
>
> The 3A statistics and other firmware parameter computation related
> functions are implemented in patch 11.
>
> All IPU3 pipeline default settings can be found in patch 10.
>
> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> its own MMU unit, the driver is implemented in patch 6.
>
> Patch 7 uses above driver for DMA mapping operation.
>
> The communication between IPU3 firmware and driver is implemented with circular
> queues in patch 8.
>
> Patch 9 provide some utility functions and manage IPU3 fw download and
> install.
>
> The firmware which is called ipu3-fw.bin can be downloaded from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>
> Firmware ABI is defined in patches 4 and 5.
>
> Patches 12 and 13 are of the same file, the former contains all h/w programming
> related code, the latter implements interface functions for access fw & hw
> capabilities.
>
> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>
> <URL:https://patchwork.kernel.org/patch/9976295/>
I've pushed the latest set here:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
You can just say the entire set depends on those going forward; the
documentation is needed, too.
>
> Patch 15 represents the top level that glues all of the other components together,
> passing arguments between the components.
>
> Patch 16 is a recent effort to extend v6 for advanced camera features like
> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>
> Link to user space implementation:
>
> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>
> ImgU media topology print:
>
> # media-ctl -d /dev/media0 -p
> Media controller API version 4.19.0
>
> Media device information
> ------------------------
> driver ipu3-imgu
> model ipu3-imgu
> serial
> bus info PCI:0000:00:05.0
> hw revision 0x80862015
> driver version 4.19.0
>
> Device topology
> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> type V4L2 subdev subtype Unknown flags 0
> device node name /dev/v4l-subdev0
> pad0: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
This doesn't seem right. Which formats can be enumerated from the pad?
> crop:(0,0)/1920x1080
> compose:(0,0)/1920x1080]
Does the compose rectangle affect the scaling on all outputs?
> <- "ipu3-imgu 0 input":0 []
Are there links that have no useful link configuration? If so, you should
set them enabled and immutable in the driver.
> pad1: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
I'd suggest to use MEDIA_BUS_FMT_FIXED here.
> <- "ipu3-imgu 0 parameters":0 []
> pad2: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 0 output":0 []
> pad3: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 0 viewfinder":0 []
Are there other differences between output and viewfinder?
> pad4: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 0 3a stat":0 []
FIXED here, too.
>
> - entity 7: ipu3-imgu 1 (5 pads, 5 links)
> type V4L2 subdev subtype Unknown flags 0
> device node name /dev/v4l-subdev1
> pad0: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> crop:(0,0)/1920x1080
> compose:(0,0)/1920x1080]
> <- "ipu3-imgu 1 input":0 []
> pad1: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> <- "ipu3-imgu 1 parameters":0 []
> pad2: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 1 output":0 []
> pad3: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 1 viewfinder":0 []
> pad4: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 1 3a stat":0 []
This is a minor matter but --- could you create the second sub-device after
the video device nodes related to the first one have been already created?
That'd make reading the output easier.
>
> - entity 17: ipu3-imgu 0 input (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video0
> pad0: Source
> -> "ipu3-imgu 0":0 []
>
> - entity 23: ipu3-imgu 0 parameters (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video1
> pad0: Source
> -> "ipu3-imgu 0":1 []
>
> - entity 29: ipu3-imgu 0 output (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video2
> pad0: Sink
> <- "ipu3-imgu 0":2 []
>
> - entity 35: ipu3-imgu 0 viewfinder (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video3
> pad0: Sink
> <- "ipu3-imgu 0":3 []
>
> - entity 41: ipu3-imgu 0 3a stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video4
> pad0: Sink
> <- "ipu3-imgu 0":4 []
>
> - entity 47: ipu3-imgu 1 input (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video5
> pad0: Source
> -> "ipu3-imgu 1":0 []
>
> - entity 53: ipu3-imgu 1 parameters (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video6
> pad0: Source
> -> "ipu3-imgu 1":1 []
>
> - entity 59: ipu3-imgu 1 output (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video7
> pad0: Sink
> <- "ipu3-imgu 1":2 []
>
> - entity 65: ipu3-imgu 1 viewfinder (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video8
> pad0: Sink
> <- "ipu3-imgu 1":3 []
>
> - entity 71: ipu3-imgu 1 3a stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video9
> pad0: Sink
> <- "ipu3-imgu 1":4 []
>
>
> v4l2-compliance utility is built with Sakari's patches for meta data
> output support(rebased):
>
> <URL:https://patchwork.linuxtv.org/patch/43370/>
> <URL:https://patchwork.linuxtv.org/patch/43369/>
>
> The test (v4l2-compliance -m 0) passes without error, outputs are appended at
> the end of revision history.
>
> Note:
>
> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be enabled
> prior to the test.
> 2. Stream tests are not performed since it requires pre-configuration for each case.
>
> ===========
> = history =
> ===========
>
> v7 update:
>
> 1. Add driver and uAPI documentation.
>
> Update based on v1 review from Tomasz, Hans, Sokari and Mauro:
> https://patchwork.kernel.org/patch/10465663/
> https://patchwork.kernel.org/patch/10465665/
>
> 2. Add dual pipe support which includes:
> - Extend current IMGU device to contain 2 subdevs and two groups of video nodes.
> - Add a v4l2 ctrl to allow user to specify the mode(video or still) of the pipe.
>
> 3. Kconfig
> - Restrict build for X86 arch to fix build error for ia64/sparc.
> (fatal error: asm/set_memory.h: No such file or directory)
>
> 4. ipu3-abi.h
> - Change __u32 to u32.
> - Use generic __attribute__((aligned(x))) format. (Mauro/Hans)
> - Split abi to 2 patches, one for register defines, enums, the other for structs. (Tomasz)
>
> 5. ipu3-mmu.c
> - Fix ipu3-mmu/dmamap exit functions. (Tomasz)
> (Port from https://chromium-review.googlesource.com/1084522)
> - Use free_page instead of kfree. (Tomasz)
> - document struct ipu3_mmu_info.
> - Fix copyright information.
>
> 6. ipu3-dmamap.c (Tomasz)
> - Update APIs based on v6 review.
> - Replace sizeof(struct page *) with sizeof(*pages).
> - Remove un-needed (WARN_ON(!dev)) inside void *ipu3_dmamap_alloc().
>
> 7. ipu3.c (Tomasz)
> - imgu_video_nodes_init()
> Fix the missing call to ipu3_v4l2_unregister() in the error path of
> imgu_dummybufs_preallocate().
> - imgu_queue_buffers()
> Evaluate loop condition explicitly for code clarity and simplicity.
> FW requires all output buffers to be queued at start, so adjust the order of
> buffer queuing accordingly. (bufix by Tianshu)
> - imgu_isr_threaded()
> Fix interrupt handler return value.
> (Port from https://chromium-review.googlesource.com/1088539)
> - Add back the buf_drain_wq from ("avoid sleep in wait_event condition")'
> (Port from https://chromium-review.googlesource.com/875420)
>
> 8. ipu3-v4l2.c
> - ipu3_v4l2_register(). (Tomasz)
> Split media initialization and registration, also change media device
> register/un-register order.
>
> - Fix v4l2-compliance fail on sub-devicef for VIDIOC_CREATE_BUFS and
> VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT.
>
> 9. ipu3-css.c, ipu3-css.h, ipu3-css-fw.h, ipu3-abi.h
> - Convert macros in structs to enums. (Tomasz)
>
> 10. ipu3-css-pool.c, ipu3-css-pool.h, ipu3.c
> - Document the structs. (Hans/Maruo)
>
> 11. ipu3-css-params.c
> - Fixup for noise reduction parameters processing. (bug fixing)
>
> version 6:
>
> - intel-ipu3.h uAPI
> Move out the definitions not used by user space. (suggested by Sakari)
> - ipu3-abi.h, ipu3-css-fw.h
> Clean up the header files.
> Remove enum type from ABI structs.
> - ipu3-css.h and ipu3-css.c
> Disable DVS support and remove related code.
> - ipu3-v4l2.c
> Fixes of v4l2_compliance test fails on ImgU sub-dev.
> - ipu3-css-params.c
> Refactor awb/awb_fr/af_ops_calc() functions. (Sakari)
> - Build mmu and dmamap driver as part of ImgU ko module; (Sakari)
> - Add "ipu3-imgu" prefix to media entity names; (Sakari)
> - Fix indentation and white space; (Sakari)
> - Rebase to kernel v4.16;
> - Use SPDX license identifiers in all drivers; (Sakari)
> - Internal fix and performance improvements such as:
> Stop fw gracefully during stream off.
> Enable irq only after start streaming to avoid unexpected interrupt.
> Use spinlock to protect IPU3_CSS_QUEUES access.
> Return NULL when dequeuing buffer before streaming.
>
> TODOs:
> - Documentation on ImgU driver programming interface to configure and enable
> ISP HW, which will include details on complete V4L2 Kernel driver interface
> and IO-Control parameters, except for the ISP internal algorithm and its
> parameters (which is Intel proprietary IP).
>
> version 5:
> - ipu3-css-pool.c/ipu3_css_pool_check().
> add handling of the framenum wrap around case in ipu3_css_pool_check().
> - ipu3.c, ipu3-v4l2.c, ipu3.h
> merge struct ipu3_mem2mem2_device into imgu_device and update the code
> accordingly. (Suggested by Sakari)
> - ipu3-mmu.c driver:
> use __get_free_page() for page-aligned allocations (Tomasz).
> optimize tlb invalidation by calling them at the end of map/unmap. (Tomasz).
> remove dependency on iommu. (Sakari)
> introduce few new functions from iommu.c.
> - ipu3-dmamap.c driver
> call mmu directly without IOMMU_SUPPORT (Sakari)
> update dmamap APIs. (Suggested by Tomasz)
> - ipu3_v4l2.c
> move g/s_selection callback to V4l2 sub-device (Sakari)
> remove colon from ImgU sub-device name. (Sakari)
> - ipu3-css-params.c
> fix indentation, 0-day scan warnings etc.
> - ipu3-css.c
> fix warning about NULL comparison. (Sakari)
> - intel-ipu3.h:
> remove redundant IPU3_ALIGN attribute (Sakari).
> fix up un-needed fields in struct ipu3_uapi_params (Sakari)
> re-order this to be 2nd in the patch set.
> - Makefile: remove Copyright header. (Sakari)
> - Internal fix:
> optimize shot-to-shot performance.
> update default white balance gains defined in ipu3-tables.c
>
> TODOs:
>
> - Documentation on ImgU driver programming interface to configure and enable
> ISP HW, which will include details on complete V4L2 Kernel driver interface
> and IO-Control parameters, except for the ISP internal algorithm and its
> parameters (which is Intel proprietary IP).
>
> - Review ipu3_css_pool_* group APIs usage.
>
> version 4:
> - Used V4L2_BUF_TYPE_META_OUTPUT for:
> - V4L2_META_FMT_IPU3_STAT_PARAMS
>
> - Used V4L2_BUF_TYPE_META_CAPTURE for:
> - V4L2_META_FMT_IPU3_STAT_3A
> - V4L2_META_FMT_IPU3_STAT_DVS
> - V4L2_META_FMT_IPU3_STAT_LACE
> - Supported v4l2 MPLANE format on video nodes.
> - ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
> - ipu3-mmu.c: Restructured the driver.
> - intel-ipu3.h: Added __padding qualifier for uapi definitions.
> - Internal fix: power and performance related issues.
> - Fixed v4l2-compliance test.
> - Fixed build failure for x86 with 32bit config.
>
> version 3:
> - ipu3-mmu.c and ipu3-dmamap.c:
> Tomasz Figa reworked both drivers and updated related files.
> - ipu2-abi.h:
> update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
> use __packed qualifier on structs suggested by Sakari Ailus.
> - ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
> remove pointer type in firmware blob structs.
> fix binary_header array in struct imgu_fw_header.
> fix calling ipu3_css_fw_show_binary() before proper checking.
> fix logic error for valid length checking of blob name.
> - ipu3-css-params.c/ipu3_css_scaler_get_exp():
> use lib helper suggested by Andy Shevchenko.
> - ipu3-v4l2.c/ipu3_videoc_querycap():
> fill device_caps fix suggested by Hans Verkuil.
> add VB2_DMABUF suggested by Tomasz Figa.
> - ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
> - ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
>
> version 2:
> This version cherry-picked firmware ABI change and other
> fix in order to bring the code up-to-date with our internal release.
>
> I will go over the review comments in v1 and address them in v3 and
> future update.
>
> version 1:
> - Initial submission
>
> --------------------------------------------------------------------------------
>
> v4l2-compliance test output:
>
> ./v4l2-compliance -m 0
>
> v4l2-compliance SHA: 7aa151889ffe89b1cd94a8198b0caba1a8c70398, 64 bits
>
> Compliance test for device /dev/media0:
>
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
Is there no revision field for the hardware? We could also use the SoC name
in the model if it's known. It might be that there is another SoC that
contains the same device but I don't see that as a problem really.
> Driver version : 4.19.0
>
> Required ioctls:
> test MEDIA_IOC_DEVICE_INFO: OK
>
> Allow for multiple opens:
> test second /dev/media0 open: OK
> test MEDIA_IOC_DEVICE_INFO: OK
> test for unlimited opens: OK
>
> Media Controller ioctls:
> test MEDIA_IOC_G_TOPOLOGY: OK
> Entities: 12 Interfaces: 12 Pads: 20 Links: 22
> test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
> test MEDIA_IOC_SETUP_LINK: OK
--
Kind regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-01 12:03 ` [PATCH v7 00/16] Intel IPU3 ImgU patchset Sakari Ailus
@ 2018-11-07 4:16 ` Bing Bu Cao
2018-11-09 1:28 ` Zhi, Yong
` (2 more replies)
0 siblings, 3 replies; 123+ messages in thread
From: Bing Bu Cao @ 2018-11-07 4:16 UTC (permalink / raw)
To: Sakari Ailus, Yong Zhi
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> Hi Yong,
>
> Thanks for the update!
>
> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
>> Hi,
>>
>> This series adds support for the Intel IPU3 (Image Processing Unit)
>> ImgU which is essentially a modern memory-to-memory ISP. It implements
>> raw Bayer to YUV image format conversion as well as a large number of
>> other pixel processing algorithms for improving the image quality.
>>
>> Meta data formats are defined for image statistics (3A, i.e. automatic
>> white balance, exposure and focus, histogram and local area contrast
>> enhancement) as well as for the pixel processing algorithm parameters.
>> The documentation for these formats is currently not included in the
>> patchset but will be added in a future version of this set.
>>
>> The algorithm parameters need to be considered specific to a given frame
>> and typically a large number of these parameters change on frame to frame
>> basis. Additionally, the parameters are highly structured (and not a flat
>> space of independent configuration primitives). They also reflect the
>> data structures used by the firmware and the hardware. On top of that,
>> the algorithms require highly specialized user space to make meaningful
>> use of them. For these reasons it has been chosen video buffers to pass
>> the parameters to the device.
>>
>> On individual patches:
>>
>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
>> image processors and HW accelerators.
>>
>> The 3A statistics and other firmware parameter computation related
>> functions are implemented in patch 11.
>>
>> All IPU3 pipeline default settings can be found in patch 10.
>>
>> To access DDR via ImgU's own memory space, IPU3 is also equipped with
>> its own MMU unit, the driver is implemented in patch 6.
>>
>> Patch 7 uses above driver for DMA mapping operation.
>>
>> The communication between IPU3 firmware and driver is implemented with circular
>> queues in patch 8.
>>
>> Patch 9 provide some utility functions and manage IPU3 fw download and
>> install.
>>
>> The firmware which is called ipu3-fw.bin can be downloaded from:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>>
>> Firmware ABI is defined in patches 4 and 5.
>>
>> Patches 12 and 13 are of the same file, the former contains all h/w programming
>> related code, the latter implements interface functions for access fw & hw
>> capabilities.
>>
>> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>>
>> <URL:https://patchwork.kernel.org/patch/9976295/>
> I've pushed the latest set here:
>
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
>
> You can just say the entire set depends on those going forward; the
> documentation is needed, too.
>
>> Patch 15 represents the top level that glues all of the other components together,
>> passing arguments between the components.
>>
>> Patch 16 is a recent effort to extend v6 for advanced camera features like
>> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>>
>> Link to user space implementation:
>>
>> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>>
>> ImgU media topology print:
>>
>> # media-ctl -d /dev/media0 -p
>> Media controller API version 4.19.0
>>
>> Media device information
>> ------------------------
>> driver ipu3-imgu
>> model ipu3-imgu
>> serial
>> bus info PCI:0000:00:05.0
>> hw revision 0x80862015
>> driver version 4.19.0
>>
>> Device topology
>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
>> type V4L2 subdev subtype Unknown flags 0
>> device node name /dev/v4l-subdev0
>> pad0: Sink
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> This doesn't seem right. Which formats can be enumerated from the pad?
>
>> crop:(0,0)/1920x1080
>> compose:(0,0)/1920x1080]
> Does the compose rectangle affect the scaling on all outputs?
Sakari, driver use crop and compose targets to help set input-feeder and BDS
output resolutions which are 2 key block of whole imaging pipeline, not the
actual ending output, but they will impact the final output.
>
>> <- "ipu3-imgu 0 input":0 []
> Are there links that have no useful link configuration? If so, you should
> set them enabled and immutable in the driver.
The enabled status of input pads is used to get which pipe that user is
trying to enable (ipu3_link_setup()), so it could not been set as immutable.
>
>> pad1: Sink
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> I'd suggest to use MEDIA_BUS_FMT_FIXED here.
>
>> <- "ipu3-imgu 0 parameters":0 []
>> pad2: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "ipu3-imgu 0 output":0 []
>> pad3: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "ipu3-imgu 0 viewfinder":0 []
> Are there other differences between output and viewfinder?
output and viewfinder are the main and secondary output of output system.
'main' output is not allowed to be scaled, only support crop. secondary
output 'viewfinder'
can support both cropping and scaling. User can select different nodes
to use
as preview and capture flexibly based on the actual use cases.
>
>> pad4: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "ipu3-imgu 0 3a stat":0 []
> FIXED here, too.
>
>> - entity 7: ipu3-imgu 1 (5 pads, 5 links)
>> type V4L2 subdev subtype Unknown flags 0
>> device node name /dev/v4l-subdev1
>> pad0: Sink
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
>> crop:(0,0)/1920x1080
>> compose:(0,0)/1920x1080]
>> <- "ipu3-imgu 1 input":0 []
>> pad1: Sink
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> <- "ipu3-imgu 1 parameters":0 []
>> pad2: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "ipu3-imgu 1 output":0 []
>> pad3: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "ipu3-imgu 1 viewfinder":0 []
>> pad4: Source
>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>> -> "ipu3-imgu 1 3a stat":0 []
> This is a minor matter but --- could you create the second sub-device after
> the video device nodes related to the first one have been already created?
> That'd make reading the output easier.
>
>> - entity 17: ipu3-imgu 0 input (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video0
>> pad0: Source
>> -> "ipu3-imgu 0":0 []
>>
>> - entity 23: ipu3-imgu 0 parameters (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video1
>> pad0: Source
>> -> "ipu3-imgu 0":1 []
>>
>> - entity 29: ipu3-imgu 0 output (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video2
>> pad0: Sink
>> <- "ipu3-imgu 0":2 []
>>
>> - entity 35: ipu3-imgu 0 viewfinder (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video3
>> pad0: Sink
>> <- "ipu3-imgu 0":3 []
>>
>> - entity 41: ipu3-imgu 0 3a stat (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video4
>> pad0: Sink
>> <- "ipu3-imgu 0":4 []
>>
>> - entity 47: ipu3-imgu 1 input (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video5
>> pad0: Source
>> -> "ipu3-imgu 1":0 []
>>
>> - entity 53: ipu3-imgu 1 parameters (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video6
>> pad0: Source
>> -> "ipu3-imgu 1":1 []
>>
>> - entity 59: ipu3-imgu 1 output (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video7
>> pad0: Sink
>> <- "ipu3-imgu 1":2 []
>>
>> - entity 65: ipu3-imgu 1 viewfinder (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video8
>> pad0: Sink
>> <- "ipu3-imgu 1":3 []
>>
>> - entity 71: ipu3-imgu 1 3a stat (1 pad, 1 link)
>> type Node subtype V4L flags 0
>> device node name /dev/video9
>> pad0: Sink
>> <- "ipu3-imgu 1":4 []
>>
>>
>> v4l2-compliance utility is built with Sakari's patches for meta data
>> output support(rebased):
>>
>> <URL:https://patchwork.linuxtv.org/patch/43370/>
>> <URL:https://patchwork.linuxtv.org/patch/43369/>
>>
>> The test (v4l2-compliance -m 0) passes without error, outputs are appended at
>> the end of revision history.
>>
>> Note:
>>
>> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be enabled
>> prior to the test.
>> 2. Stream tests are not performed since it requires pre-configuration for each case.
>>
>> ===========
>> = history =
>> ===========
>>
>> v7 update:
>>
>> 1. Add driver and uAPI documentation.
>>
>> Update based on v1 review from Tomasz, Hans, Sokari and Mauro:
>> https://patchwork.kernel.org/patch/10465663/
>> https://patchwork.kernel.org/patch/10465665/
>>
>> 2. Add dual pipe support which includes:
>> - Extend current IMGU device to contain 2 subdevs and two groups of video nodes.
>> - Add a v4l2 ctrl to allow user to specify the mode(video or still) of the pipe.
>>
>> 3. Kconfig
>> - Restrict build for X86 arch to fix build error for ia64/sparc.
>> (fatal error: asm/set_memory.h: No such file or directory)
>>
>> 4. ipu3-abi.h
>> - Change __u32 to u32.
>> - Use generic __attribute__((aligned(x))) format. (Mauro/Hans)
>> - Split abi to 2 patches, one for register defines, enums, the other for structs. (Tomasz)
>>
>> 5. ipu3-mmu.c
>> - Fix ipu3-mmu/dmamap exit functions. (Tomasz)
>> (Port from https://chromium-review.googlesource.com/1084522)
>> - Use free_page instead of kfree. (Tomasz)
>> - document struct ipu3_mmu_info.
>> - Fix copyright information.
>>
>> 6. ipu3-dmamap.c (Tomasz)
>> - Update APIs based on v6 review.
>> - Replace sizeof(struct page *) with sizeof(*pages).
>> - Remove un-needed (WARN_ON(!dev)) inside void *ipu3_dmamap_alloc().
>>
>> 7. ipu3.c (Tomasz)
>> - imgu_video_nodes_init()
>> Fix the missing call to ipu3_v4l2_unregister() in the error path of
>> imgu_dummybufs_preallocate().
>> - imgu_queue_buffers()
>> Evaluate loop condition explicitly for code clarity and simplicity.
>> FW requires all output buffers to be queued at start, so adjust the order of
>> buffer queuing accordingly. (bufix by Tianshu)
>> - imgu_isr_threaded()
>> Fix interrupt handler return value.
>> (Port from https://chromium-review.googlesource.com/1088539)
>> - Add back the buf_drain_wq from ("avoid sleep in wait_event condition")'
>> (Port from https://chromium-review.googlesource.com/875420)
>>
>> 8. ipu3-v4l2.c
>> - ipu3_v4l2_register(). (Tomasz)
>> Split media initialization and registration, also change media device
>> register/un-register order.
>>
>> - Fix v4l2-compliance fail on sub-devicef for VIDIOC_CREATE_BUFS and
>> VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT.
>>
>> 9. ipu3-css.c, ipu3-css.h, ipu3-css-fw.h, ipu3-abi.h
>> - Convert macros in structs to enums. (Tomasz)
>>
>> 10. ipu3-css-pool.c, ipu3-css-pool.h, ipu3.c
>> - Document the structs. (Hans/Maruo)
>>
>> 11. ipu3-css-params.c
>> - Fixup for noise reduction parameters processing. (bug fixing)
>>
>> version 6:
>>
>> - intel-ipu3.h uAPI
>> Move out the definitions not used by user space. (suggested by Sakari)
>> - ipu3-abi.h, ipu3-css-fw.h
>> Clean up the header files.
>> Remove enum type from ABI structs.
>> - ipu3-css.h and ipu3-css.c
>> Disable DVS support and remove related code.
>> - ipu3-v4l2.c
>> Fixes of v4l2_compliance test fails on ImgU sub-dev.
>> - ipu3-css-params.c
>> Refactor awb/awb_fr/af_ops_calc() functions. (Sakari)
>> - Build mmu and dmamap driver as part of ImgU ko module; (Sakari)
>> - Add "ipu3-imgu" prefix to media entity names; (Sakari)
>> - Fix indentation and white space; (Sakari)
>> - Rebase to kernel v4.16;
>> - Use SPDX license identifiers in all drivers; (Sakari)
>> - Internal fix and performance improvements such as:
>> Stop fw gracefully during stream off.
>> Enable irq only after start streaming to avoid unexpected interrupt.
>> Use spinlock to protect IPU3_CSS_QUEUES access.
>> Return NULL when dequeuing buffer before streaming.
>>
>> TODOs:
>> - Documentation on ImgU driver programming interface to configure and enable
>> ISP HW, which will include details on complete V4L2 Kernel driver interface
>> and IO-Control parameters, except for the ISP internal algorithm and its
>> parameters (which is Intel proprietary IP).
>>
>> version 5:
>> - ipu3-css-pool.c/ipu3_css_pool_check().
>> add handling of the framenum wrap around case in ipu3_css_pool_check().
>> - ipu3.c, ipu3-v4l2.c, ipu3.h
>> merge struct ipu3_mem2mem2_device into imgu_device and update the code
>> accordingly. (Suggested by Sakari)
>> - ipu3-mmu.c driver:
>> use __get_free_page() for page-aligned allocations (Tomasz).
>> optimize tlb invalidation by calling them at the end of map/unmap. (Tomasz).
>> remove dependency on iommu. (Sakari)
>> introduce few new functions from iommu.c.
>> - ipu3-dmamap.c driver
>> call mmu directly without IOMMU_SUPPORT (Sakari)
>> update dmamap APIs. (Suggested by Tomasz)
>> - ipu3_v4l2.c
>> move g/s_selection callback to V4l2 sub-device (Sakari)
>> remove colon from ImgU sub-device name. (Sakari)
>> - ipu3-css-params.c
>> fix indentation, 0-day scan warnings etc.
>> - ipu3-css.c
>> fix warning about NULL comparison. (Sakari)
>> - intel-ipu3.h:
>> remove redundant IPU3_ALIGN attribute (Sakari).
>> fix up un-needed fields in struct ipu3_uapi_params (Sakari)
>> re-order this to be 2nd in the patch set.
>> - Makefile: remove Copyright header. (Sakari)
>> - Internal fix:
>> optimize shot-to-shot performance.
>> update default white balance gains defined in ipu3-tables.c
>>
>> TODOs:
>>
>> - Documentation on ImgU driver programming interface to configure and enable
>> ISP HW, which will include details on complete V4L2 Kernel driver interface
>> and IO-Control parameters, except for the ISP internal algorithm and its
>> parameters (which is Intel proprietary IP).
>>
>> - Review ipu3_css_pool_* group APIs usage.
>>
>> version 4:
>> - Used V4L2_BUF_TYPE_META_OUTPUT for:
>> - V4L2_META_FMT_IPU3_STAT_PARAMS
>>
>> - Used V4L2_BUF_TYPE_META_CAPTURE for:
>> - V4L2_META_FMT_IPU3_STAT_3A
>> - V4L2_META_FMT_IPU3_STAT_DVS
>> - V4L2_META_FMT_IPU3_STAT_LACE
>> - Supported v4l2 MPLANE format on video nodes.
>> - ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
>> - ipu3-mmu.c: Restructured the driver.
>> - intel-ipu3.h: Added __padding qualifier for uapi definitions.
>> - Internal fix: power and performance related issues.
>> - Fixed v4l2-compliance test.
>> - Fixed build failure for x86 with 32bit config.
>>
>> version 3:
>> - ipu3-mmu.c and ipu3-dmamap.c:
>> Tomasz Figa reworked both drivers and updated related files.
>> - ipu2-abi.h:
>> update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
>> use __packed qualifier on structs suggested by Sakari Ailus.
>> - ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
>> remove pointer type in firmware blob structs.
>> fix binary_header array in struct imgu_fw_header.
>> fix calling ipu3_css_fw_show_binary() before proper checking.
>> fix logic error for valid length checking of blob name.
>> - ipu3-css-params.c/ipu3_css_scaler_get_exp():
>> use lib helper suggested by Andy Shevchenko.
>> - ipu3-v4l2.c/ipu3_videoc_querycap():
>> fill device_caps fix suggested by Hans Verkuil.
>> add VB2_DMABUF suggested by Tomasz Figa.
>> - ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
>> - ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
>>
>> version 2:
>> This version cherry-picked firmware ABI change and other
>> fix in order to bring the code up-to-date with our internal release.
>>
>> I will go over the review comments in v1 and address them in v3 and
>> future update.
>>
>> version 1:
>> - Initial submission
>>
>> --------------------------------------------------------------------------------
>>
>> v4l2-compliance test output:
>>
>> ./v4l2-compliance -m 0
>>
>> v4l2-compliance SHA: 7aa151889ffe89b1cd94a8198b0caba1a8c70398, 64 bits
>>
>> Compliance test for device /dev/media0:
>>
>> Media Driver Info:
>> Driver name : ipu3-imgu
>> Model : ipu3-imgu
>> Serial :
>> Bus info : PCI:0000:00:05.0
>> Media version : 4.19.0
>> Hardware revision: 0x80862015 (2156273685)
> Is there no revision field for the hardware? We could also use the SoC name
> in the model if it's known. It might be that there is another SoC that
> contains the same device but I don't see that as a problem really.
>
>> Driver version : 4.19.0
>>
>> Required ioctls:
>> test MEDIA_IOC_DEVICE_INFO: OK
>>
>> Allow for multiple opens:
>> test second /dev/media0 open: OK
>> test MEDIA_IOC_DEVICE_INFO: OK
>> test for unlimited opens: OK
>>
>> Media Controller ioctls:
>> test MEDIA_IOC_G_TOPOLOGY: OK
>> Entities: 12 Interfaces: 12 Pads: 20 Links: 22
>> test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
>> test MEDIA_IOC_SETUP_LINK: OK
^ permalink raw reply [flat|nested] 123+ messages in thread* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-07 4:16 ` Bing Bu Cao
@ 2018-11-09 1:28 ` Zhi, Yong
2018-11-09 11:28 ` Sakari Ailus
2018-11-09 10:09 ` Sakari Ailus
2018-11-29 23:07 ` Laurent Pinchart
2 siblings, 1 reply; 123+ messages in thread
From: Zhi, Yong @ 2018-11-09 1:28 UTC (permalink / raw)
To: Bing Bu Cao, Sakari Ailus
Cc: linux-media, tfiga, mchehab, hans.verkuil, laurent.pinchart,
Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka,
Qiu, Tian Shu, Cao, Bingbu
Hi, Sakari,
Thanks for your review and comments.
Bingbu has replied to some of your questions, so I will continue with the rest.
> -----Original Message-----
> From: Bing Bu Cao [mailto:bingbu.cao@linux.intel.com]
> Sent: Tuesday, November 6, 2018 10:17 PM
> To: Sakari Ailus <sakari.ailus@linux.intel.com>; Zhi, Yong
> <yong.zhi@intel.com>
> Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> mchehab@kernel.org; hans.verkuil@cisco.com;
> laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
>
> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> > Hi Yong,
> >
> > Thanks for the update!
> >
> > On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> >> Hi,
> >>
> >> This series adds support for the Intel IPU3 (Image Processing Unit)
> >> ImgU which is essentially a modern memory-to-memory ISP. It
> >> implements raw Bayer to YUV image format conversion as well as a
> >> large number of other pixel processing algorithms for improving the image
> quality.
> >>
> >> Meta data formats are defined for image statistics (3A, i.e.
> >> automatic white balance, exposure and focus, histogram and local area
> >> contrast
> >> enhancement) as well as for the pixel processing algorithm parameters.
> >> The documentation for these formats is currently not included in the
> >> patchset but will be added in a future version of this set.
> >>
> >> The algorithm parameters need to be considered specific to a given
> >> frame and typically a large number of these parameters change on
> >> frame to frame basis. Additionally, the parameters are highly
> >> structured (and not a flat space of independent configuration
> >> primitives). They also reflect the data structures used by the
> >> firmware and the hardware. On top of that, the algorithms require
> >> highly specialized user space to make meaningful use of them. For
> >> these reasons it has been chosen video buffers to pass the parameters to
> the device.
> >>
> >> On individual patches:
> >>
> >> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> >> image processors and HW accelerators.
> >>
> >> The 3A statistics and other firmware parameter computation related
> >> functions are implemented in patch 11.
> >>
> >> All IPU3 pipeline default settings can be found in patch 10.
> >>
> >> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> >> its own MMU unit, the driver is implemented in patch 6.
> >>
> >> Patch 7 uses above driver for DMA mapping operation.
> >>
> >> The communication between IPU3 firmware and driver is implemented
> >> with circular queues in patch 8.
> >>
> >> Patch 9 provide some utility functions and manage IPU3 fw download
> >> and install.
> >>
> >> The firmware which is called ipu3-fw.bin can be downloaded from:
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware
> >> .git (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
> >>
> >> Firmware ABI is defined in patches 4 and 5.
> >>
> >> Patches 12 and 13 are of the same file, the former contains all h/w
> >> programming related code, the latter implements interface functions
> >> for access fw & hw capabilities.
> >>
> >> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT
> work:
> >>
> >> <URL:https://patchwork.kernel.org/patch/9976295/>
> > I've pushed the latest set here:
> >
> > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
> >
> > You can just say the entire set depends on those going forward; the
> > documentation is needed, too.
> >
Ack.
> >> Patch 15 represents the top level that glues all of the other
> >> components together, passing arguments between the components.
> >>
> >> Patch 16 is a recent effort to extend v6 for advanced camera features
> >> like Continuous View Finder (CVF) and Snapshot During Video(SDV)
> support.
> >>
> >> Link to user space implementation:
> >>
> >> git clone
> >> https://chromium.googlesource.com/chromiumos/platform/arc-camera
> >>
> >> ImgU media topology print:
> >>
> >> # media-ctl -d /dev/media0 -p
> >> Media controller API version 4.19.0
> >>
> >> Media device information
> >> ------------------------
> >> driver ipu3-imgu
> >> model ipu3-imgu
> >> serial
> >> bus info PCI:0000:00:05.0
> >> hw revision 0x80862015
> >> driver version 4.19.0
> >>
> >> Device topology
> >> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >> type V4L2 subdev subtype Unknown flags 0
> >> device node name /dev/v4l-subdev0
> >> pad0: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> > This doesn't seem right. Which formats can be enumerated from the pad?
Supposed to be raw formats for the colorspace field.
> >
> >> crop:(0,0)/1920x1080
> >> compose:(0,0)/1920x1080]
> > Does the compose rectangle affect the scaling on all outputs?
> Sakari, driver use crop and compose targets to help set input-feeder and BDS
> output resolutions which are 2 key block of whole imaging pipeline, not the
> actual ending output, but they will impact the final output.
> >
> >> <- "ipu3-imgu 0 input":0 []
> > Are there links that have no useful link configuration? If so, you
> > should set them enabled and immutable in the driver.
> The enabled status of input pads is used to get which pipe that user is trying
> to enable (ipu3_link_setup()), so it could not been set as immutable.
> >
> >> pad1: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > I'd suggest to use MEDIA_BUS_FMT_FIXED here.
Make sense as the parameter size is fixed.
> >
> >> <- "ipu3-imgu 0 parameters":0 []
> >> pad2: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 output":0 []
> >> pad3: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 viewfinder":0 []
> > Are there other differences between output and viewfinder?
> output and viewfinder are the main and secondary output of output system.
> 'main' output is not allowed to be scaled, only support crop. secondary
> output 'viewfinder'
> can support both cropping and scaling. User can select different nodes to use
> as preview and capture flexibly based on the actual use cases.
> >
> >> pad4: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 3a stat":0 []
> > FIXED here, too.
Sure.
> >
> >> - entity 7: ipu3-imgu 1 (5 pads, 5 links)
> >> type V4L2 subdev subtype Unknown flags 0
> >> device node name /dev/v4l-subdev1
> >> pad0: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >> crop:(0,0)/1920x1080
> >> compose:(0,0)/1920x1080]
> >> <- "ipu3-imgu 1 input":0 []
> >> pad1: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> <- "ipu3-imgu 1 parameters":0 []
> >> pad2: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 1 output":0 []
> >> pad3: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 1 viewfinder":0 []
> >> pad4: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 1 3a stat":0 []
> > This is a minor matter but --- could you create the second sub-device
> > after the video device nodes related to the first one have been already
> created?
> > That'd make reading the output easier.
> >
This can be done in next update.
> >> - entity 17: ipu3-imgu 0 input (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video0
> >> pad0: Source
> >> -> "ipu3-imgu 0":0 []
> >>
> >> - entity 23: ipu3-imgu 0 parameters (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video1
> >> pad0: Source
> >> -> "ipu3-imgu 0":1 []
> >>
> >> - entity 29: ipu3-imgu 0 output (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video2
> >> pad0: Sink
> >> <- "ipu3-imgu 0":2 []
> >>
> >> - entity 35: ipu3-imgu 0 viewfinder (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video3
> >> pad0: Sink
> >> <- "ipu3-imgu 0":3 []
> >>
> >> - entity 41: ipu3-imgu 0 3a stat (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video4
> >> pad0: Sink
> >> <- "ipu3-imgu 0":4 []
> >>
> >> - entity 47: ipu3-imgu 1 input (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video5
> >> pad0: Source
> >> -> "ipu3-imgu 1":0 []
> >>
> >> - entity 53: ipu3-imgu 1 parameters (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video6
> >> pad0: Source
> >> -> "ipu3-imgu 1":1 []
> >>
> >> - entity 59: ipu3-imgu 1 output (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video7
> >> pad0: Sink
> >> <- "ipu3-imgu 1":2 []
> >>
> >> - entity 65: ipu3-imgu 1 viewfinder (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video8
> >> pad0: Sink
> >> <- "ipu3-imgu 1":3 []
> >>
> >> - entity 71: ipu3-imgu 1 3a stat (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video9
> >> pad0: Sink
> >> <- "ipu3-imgu 1":4 []
> >>
> >>
> >> v4l2-compliance utility is built with Sakari's patches for meta data
> >> output support(rebased):
> >>
> >> <URL:https://patchwork.linuxtv.org/patch/43370/>
> >> <URL:https://patchwork.linuxtv.org/patch/43369/>
> >>
> >> The test (v4l2-compliance -m 0) passes without error, outputs are
> >> appended at the end of revision history.
> >>
> >> Note:
> >>
> >> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be
> enabled
> >> prior to the test.
> >> 2. Stream tests are not performed since it requires pre-configuration for
> each case.
> >>
> >> ===========
> >> = history =
> >> ===========
> >>
> >> v7 update:
> >>
> >> 1. Add driver and uAPI documentation.
> >>
> >> Update based on v1 review from Tomasz, Hans, Sokari and Mauro:
> >> https://patchwork.kernel.org/patch/10465663/
> >> https://patchwork.kernel.org/patch/10465665/
> >>
> >> 2. Add dual pipe support which includes:
> >> - Extend current IMGU device to contain 2 subdevs and two groups of
> video nodes.
> >> - Add a v4l2 ctrl to allow user to specify the mode(video or still) of the
> pipe.
> >>
> >> 3. Kconfig
> >> - Restrict build for X86 arch to fix build error for ia64/sparc.
> >> (fatal error: asm/set_memory.h: No such file or directory)
> >>
> >> 4. ipu3-abi.h
> >> - Change __u32 to u32.
> >> - Use generic __attribute__((aligned(x))) format. (Mauro/Hans)
> >> - Split abi to 2 patches, one for register defines, enums, the other
> >> for structs. (Tomasz)
> >>
> >> 5. ipu3-mmu.c
> >> - Fix ipu3-mmu/dmamap exit functions. (Tomasz)
> >> (Port from https://chromium-review.googlesource.com/1084522)
> >> - Use free_page instead of kfree. (Tomasz)
> >> - document struct ipu3_mmu_info.
> >> - Fix copyright information.
> >>
> >> 6. ipu3-dmamap.c (Tomasz)
> >> - Update APIs based on v6 review.
> >> - Replace sizeof(struct page *) with sizeof(*pages).
> >> - Remove un-needed (WARN_ON(!dev)) inside void
> *ipu3_dmamap_alloc().
> >>
> >> 7. ipu3.c (Tomasz)
> >> - imgu_video_nodes_init()
> >> Fix the missing call to ipu3_v4l2_unregister() in the error path of
> >> imgu_dummybufs_preallocate().
> >> - imgu_queue_buffers()
> >> Evaluate loop condition explicitly for code clarity and simplicity.
> >> FW requires all output buffers to be queued at start, so adjust the order
> of
> >> buffer queuing accordingly. (bufix by Tianshu)
> >> - imgu_isr_threaded()
> >> Fix interrupt handler return value.
> >> (Port from https://chromium-review.googlesource.com/1088539)
> >> - Add back the buf_drain_wq from ("avoid sleep in wait_event
> condition")'
> >> (Port from https://chromium-review.googlesource.com/875420)
> >>
> >> 8. ipu3-v4l2.c
> >> - ipu3_v4l2_register(). (Tomasz)
> >> Split media initialization and registration, also change media device
> >> register/un-register order.
> >>
> >> - Fix v4l2-compliance fail on sub-devicef for VIDIOC_CREATE_BUFS and
> >> VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT.
> >>
> >> 9. ipu3-css.c, ipu3-css.h, ipu3-css-fw.h, ipu3-abi.h
> >> - Convert macros in structs to enums. (Tomasz)
> >>
> >> 10. ipu3-css-pool.c, ipu3-css-pool.h, ipu3.c
> >> - Document the structs. (Hans/Maruo)
> >>
> >> 11. ipu3-css-params.c
> >> - Fixup for noise reduction parameters processing. (bug fixing)
> >>
> >> version 6:
> >>
> >> - intel-ipu3.h uAPI
> >> Move out the definitions not used by user space. (suggested by
> >> Sakari)
> >> - ipu3-abi.h, ipu3-css-fw.h
> >> Clean up the header files.
> >> Remove enum type from ABI structs.
> >> - ipu3-css.h and ipu3-css.c
> >> Disable DVS support and remove related code.
> >> - ipu3-v4l2.c
> >> Fixes of v4l2_compliance test fails on ImgU sub-dev.
> >> - ipu3-css-params.c
> >> Refactor awb/awb_fr/af_ops_calc() functions. (Sakari)
> >> - Build mmu and dmamap driver as part of ImgU ko module; (Sakari)
> >> - Add "ipu3-imgu" prefix to media entity names; (Sakari)
> >> - Fix indentation and white space; (Sakari)
> >> - Rebase to kernel v4.16;
> >> - Use SPDX license identifiers in all drivers; (Sakari)
> >> - Internal fix and performance improvements such as:
> >> Stop fw gracefully during stream off.
> >> Enable irq only after start streaming to avoid unexpected interrupt.
> >> Use spinlock to protect IPU3_CSS_QUEUES access.
> >> Return NULL when dequeuing buffer before streaming.
> >>
> >> TODOs:
> >> - Documentation on ImgU driver programming interface to configure and
> enable
> >> ISP HW, which will include details on complete V4L2 Kernel driver
> interface
> >> and IO-Control parameters, except for the ISP internal algorithm and its
> >> parameters (which is Intel proprietary IP).
> >>
> >> version 5:
> >> - ipu3-css-pool.c/ipu3_css_pool_check().
> >> add handling of the framenum wrap around case in
> ipu3_css_pool_check().
> >> - ipu3.c, ipu3-v4l2.c, ipu3.h
> >> merge struct ipu3_mem2mem2_device into imgu_device and update the
> code
> >> accordingly. (Suggested by Sakari)
> >> - ipu3-mmu.c driver:
> >> use __get_free_page() for page-aligned allocations (Tomasz).
> >> optimize tlb invalidation by calling them at the end of map/unmap.
> (Tomasz).
> >> remove dependency on iommu. (Sakari)
> >> introduce few new functions from iommu.c.
> >> - ipu3-dmamap.c driver
> >> call mmu directly without IOMMU_SUPPORT (Sakari)
> >> update dmamap APIs. (Suggested by Tomasz)
> >> - ipu3_v4l2.c
> >> move g/s_selection callback to V4l2 sub-device (Sakari)
> >> remove colon from ImgU sub-device name. (Sakari)
> >> - ipu3-css-params.c
> >> fix indentation, 0-day scan warnings etc.
> >> - ipu3-css.c
> >> fix warning about NULL comparison. (Sakari)
> >> - intel-ipu3.h:
> >> remove redundant IPU3_ALIGN attribute (Sakari).
> >> fix up un-needed fields in struct ipu3_uapi_params (Sakari)
> >> re-order this to be 2nd in the patch set.
> >> - Makefile: remove Copyright header. (Sakari)
> >> - Internal fix:
> >> optimize shot-to-shot performance.
> >> update default white balance gains defined in ipu3-tables.c
> >>
> >> TODOs:
> >>
> >> - Documentation on ImgU driver programming interface to configure and
> enable
> >> ISP HW, which will include details on complete V4L2 Kernel driver
> interface
> >> and IO-Control parameters, except for the ISP internal algorithm and its
> >> parameters (which is Intel proprietary IP).
> >>
> >> - Review ipu3_css_pool_* group APIs usage.
> >>
> >> version 4:
> >> - Used V4L2_BUF_TYPE_META_OUTPUT for:
> >> - V4L2_META_FMT_IPU3_STAT_PARAMS
> >>
> >> - Used V4L2_BUF_TYPE_META_CAPTURE for:
> >> - V4L2_META_FMT_IPU3_STAT_3A
> >> - V4L2_META_FMT_IPU3_STAT_DVS
> >> - V4L2_META_FMT_IPU3_STAT_LACE
> >> - Supported v4l2 MPLANE format on video nodes.
> >> - ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA
> lib.
> >> - ipu3-mmu.c: Restructured the driver.
> >> - intel-ipu3.h: Added __padding qualifier for uapi definitions.
> >> - Internal fix: power and performance related issues.
> >> - Fixed v4l2-compliance test.
> >> - Fixed build failure for x86 with 32bit config.
> >>
> >> version 3:
> >> - ipu3-mmu.c and ipu3-dmamap.c:
> >> Tomasz Figa reworked both drivers and updated related files.
> >> - ipu2-abi.h:
> >> update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
> >> use __packed qualifier on structs suggested by Sakari Ailus.
> >> - ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
> >> remove pointer type in firmware blob structs.
> >> fix binary_header array in struct imgu_fw_header.
> >> fix calling ipu3_css_fw_show_binary() before proper checking.
> >> fix logic error for valid length checking of blob name.
> >> - ipu3-css-params.c/ipu3_css_scaler_get_exp():
> >> use lib helper suggested by Andy Shevchenko.
> >> - ipu3-v4l2.c/ipu3_videoc_querycap():
> >> fill device_caps fix suggested by Hans Verkuil.
> >> add VB2_DMABUF suggested by Tomasz Figa.
> >> - ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
> >> - ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
> >>
> >> version 2:
> >> This version cherry-picked firmware ABI change and other fix in order
> >> to bring the code up-to-date with our internal release.
> >>
> >> I will go over the review comments in v1 and address them in v3 and
> >> future update.
> >>
> >> version 1:
> >> - Initial submission
> >>
> >> ---------------------------------------------------------------------
> >> -----------
> >>
> >> v4l2-compliance test output:
> >>
> >> ./v4l2-compliance -m 0
> >>
> >> v4l2-compliance SHA: 7aa151889ffe89b1cd94a8198b0caba1a8c70398, 64
> >> bits
> >>
> >> Compliance test for device /dev/media0:
> >>
> >> Media Driver Info:
> >> Driver name : ipu3-imgu
> >> Model : ipu3-imgu
> >> Serial :
> >> Bus info : PCI:0000:00:05.0
> >> Media version : 4.19.0
> >> Hardware revision: 0x80862015 (2156273685)
> > Is there no revision field for the hardware? We could also use the SoC
> > name in the model if it's known. It might be that there is another SoC
> > that contains the same device but I don't see that as a problem really.
> >
For the first question, do you mean revision field of IPU3?
For the Model, do you suggest changing "ipu3-imgu" to "Intel(R) Core(TM) m3-7Y30 CPU ipu3-imgu"? I have not found a handy way to read SoC name yet.
Thanks,
Yong
> >> Driver version : 4.19.0
> >>
> >> Required ioctls:
> >> test MEDIA_IOC_DEVICE_INFO: OK
> >>
> >> Allow for multiple opens:
> >> test second /dev/media0 open: OK
> >> test MEDIA_IOC_DEVICE_INFO: OK
> >> test for unlimited opens: OK
> >>
> >> Media Controller ioctls:
> >> test MEDIA_IOC_G_TOPOLOGY: OK
> >> Entities: 12 Interfaces: 12 Pads: 20 Links: 22
> >> test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
> >> test MEDIA_IOC_SETUP_LINK: OK
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-09 1:28 ` Zhi, Yong
@ 2018-11-09 11:28 ` Sakari Ailus
0 siblings, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-11-09 11:28 UTC (permalink / raw)
To: Zhi, Yong
Cc: Bing Bu Cao, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, Mani, Rajmohan, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hi Yong,
On Fri, Nov 09, 2018 at 01:28:27AM +0000, Zhi, Yong wrote:
> Hi, Sakari,
>
> Thanks for your review and comments.
> Bingbu has replied to some of your questions, so I will continue with the rest.
>
> > -----Original Message-----
> > From: Bing Bu Cao [mailto:bingbu.cao@linux.intel.com]
> > Sent: Tuesday, November 6, 2018 10:17 PM
> > To: Sakari Ailus <sakari.ailus@linux.intel.com>; Zhi, Yong
> > <yong.zhi@intel.com>
> > Cc: linux-media@vger.kernel.org; tfiga@chromium.org;
> > mchehab@kernel.org; hans.verkuil@cisco.com;
> > laurent.pinchart@ideasonboard.com; Mani, Rajmohan
> > <rajmohan.mani@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>; Hu,
> > Jerry W <jerry.w.hu@intel.com>; Toivonen, Tuukka
> > <tuukka.toivonen@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> > Bingbu <bingbu.cao@intel.com>
> > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> >
> >
> > On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> > > Hi Yong,
> > >
> > > Thanks for the update!
> > >
> > > On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> > >> Hi,
> > >>
> > >> This series adds support for the Intel IPU3 (Image Processing Unit)
> > >> ImgU which is essentially a modern memory-to-memory ISP. It
> > >> implements raw Bayer to YUV image format conversion as well as a
> > >> large number of other pixel processing algorithms for improving the image
> > quality.
> > >>
> > >> Meta data formats are defined for image statistics (3A, i.e.
> > >> automatic white balance, exposure and focus, histogram and local area
> > >> contrast
> > >> enhancement) as well as for the pixel processing algorithm parameters.
> > >> The documentation for these formats is currently not included in the
> > >> patchset but will be added in a future version of this set.
> > >>
> > >> The algorithm parameters need to be considered specific to a given
> > >> frame and typically a large number of these parameters change on
> > >> frame to frame basis. Additionally, the parameters are highly
> > >> structured (and not a flat space of independent configuration
> > >> primitives). They also reflect the data structures used by the
> > >> firmware and the hardware. On top of that, the algorithms require
> > >> highly specialized user space to make meaningful use of them. For
> > >> these reasons it has been chosen video buffers to pass the parameters to
> > the device.
> > >>
> > >> On individual patches:
> > >>
> > >> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> > >> image processors and HW accelerators.
> > >>
> > >> The 3A statistics and other firmware parameter computation related
> > >> functions are implemented in patch 11.
> > >>
> > >> All IPU3 pipeline default settings can be found in patch 10.
> > >>
> > >> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> > >> its own MMU unit, the driver is implemented in patch 6.
> > >>
> > >> Patch 7 uses above driver for DMA mapping operation.
> > >>
> > >> The communication between IPU3 firmware and driver is implemented
> > >> with circular queues in patch 8.
> > >>
> > >> Patch 9 provide some utility functions and manage IPU3 fw download
> > >> and install.
> > >>
> > >> The firmware which is called ipu3-fw.bin can be downloaded from:
> > >>
> > >> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware
> > >> .git (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
> > >>
> > >> Firmware ABI is defined in patches 4 and 5.
> > >>
> > >> Patches 12 and 13 are of the same file, the former contains all h/w
> > >> programming related code, the latter implements interface functions
> > >> for access fw & hw capabilities.
> > >>
> > >> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT
> > work:
> > >>
> > >> <URL:https://patchwork.kernel.org/patch/9976295/>
> > > I've pushed the latest set here:
> > >
> > > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
> > >
> > > You can just say the entire set depends on those going forward; the
> > > documentation is needed, too.
> > >
>
> Ack.
>
> > >> Patch 15 represents the top level that glues all of the other
> > >> components together, passing arguments between the components.
> > >>
> > >> Patch 16 is a recent effort to extend v6 for advanced camera features
> > >> like Continuous View Finder (CVF) and Snapshot During Video(SDV)
> > support.
> > >>
> > >> Link to user space implementation:
> > >>
> > >> git clone
> > >> https://chromium.googlesource.com/chromiumos/platform/arc-camera
> > >>
> > >> ImgU media topology print:
> > >>
> > >> # media-ctl -d /dev/media0 -p
> > >> Media controller API version 4.19.0
> > >>
> > >> Media device information
> > >> ------------------------
> > >> driver ipu3-imgu
> > >> model ipu3-imgu
> > >> serial
> > >> bus info PCI:0000:00:05.0
> > >> hw revision 0x80862015
> > >> driver version 4.19.0
> > >>
> > >> Device topology
> > >> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> > >> type V4L2 subdev subtype Unknown flags 0
> > >> device node name /dev/v4l-subdev0
> > >> pad0: Sink
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> > > This doesn't seem right. Which formats can be enumerated from the pad?
>
> Supposed to be raw formats for the colorspace field.
I think you could even leave colorspace unchanged. Please also see my reply
to Bing Bu.
>
> > >
> > >> crop:(0,0)/1920x1080
> > >> compose:(0,0)/1920x1080]
> > > Does the compose rectangle affect the scaling on all outputs?
> > Sakari, driver use crop and compose targets to help set input-feeder and BDS
> > output resolutions which are 2 key block of whole imaging pipeline, not the
> > actual ending output, but they will impact the final output.
> > >
> > >> <- "ipu3-imgu 0 input":0 []
> > > Are there links that have no useful link configuration? If so, you
> > > should set them enabled and immutable in the driver.
> > The enabled status of input pads is used to get which pipe that user is trying
> > to enable (ipu3_link_setup()), so it could not been set as immutable.
> > >
> > >> pad1: Sink
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > > I'd suggest to use MEDIA_BUS_FMT_FIXED here.
>
> Make sense as the parameter size is fixed.
>
> > >
> > >> <- "ipu3-imgu 0 parameters":0 []
> > >> pad2: Source
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> -> "ipu3-imgu 0 output":0 []
> > >> pad3: Source
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> -> "ipu3-imgu 0 viewfinder":0 []
> > > Are there other differences between output and viewfinder?
> > output and viewfinder are the main and secondary output of output system.
> > 'main' output is not allowed to be scaled, only support crop. secondary
> > output 'viewfinder'
> > can support both cropping and scaling. User can select different nodes to use
> > as preview and capture flexibly based on the actual use cases.
> > >
> > >> pad4: Source
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> -> "ipu3-imgu 0 3a stat":0 []
> > > FIXED here, too.
>
> Sure.
>
> > >
> > >> - entity 7: ipu3-imgu 1 (5 pads, 5 links)
> > >> type V4L2 subdev subtype Unknown flags 0
> > >> device node name /dev/v4l-subdev1
> > >> pad0: Sink
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> > >> crop:(0,0)/1920x1080
> > >> compose:(0,0)/1920x1080]
> > >> <- "ipu3-imgu 1 input":0 []
> > >> pad1: Sink
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> <- "ipu3-imgu 1 parameters":0 []
> > >> pad2: Source
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> -> "ipu3-imgu 1 output":0 []
> > >> pad3: Source
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> -> "ipu3-imgu 1 viewfinder":0 []
> > >> pad4: Source
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > >> -> "ipu3-imgu 1 3a stat":0 []
> > > This is a minor matter but --- could you create the second sub-device
> > > after the video device nodes related to the first one have been already
> > created?
> > > That'd make reading the output easier.
> > >
>
> This can be done in next update.
Thanks!
...
> > >> v4l2-compliance SHA: 7aa151889ffe89b1cd94a8198b0caba1a8c70398, 64
> > >> bits
> > >>
> > >> Compliance test for device /dev/media0:
> > >>
> > >> Media Driver Info:
> > >> Driver name : ipu3-imgu
> > >> Model : ipu3-imgu
> > >> Serial :
> > >> Bus info : PCI:0000:00:05.0
> > >> Media version : 4.19.0
> > >> Hardware revision: 0x80862015 (2156273685)
> > > Is there no revision field for the hardware? We could also use the SoC
> > > name in the model if it's known. It might be that there is another SoC
> > > that contains the same device but I don't see that as a problem really.
> > >
>
> For the first question, do you mean revision field of IPU3?
> For the Model, do you suggest changing "ipu3-imgu" to "Intel(R) Core(TM)
> m3-7Y30 CPU ipu3-imgu"? I have not found a handy way to read SoC name
> yet.
I was thinking of something like "Kaby lake". If the IPU3 appears in other
SoCs it would likely have a different PCI ID.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-07 4:16 ` Bing Bu Cao
2018-11-09 1:28 ` Zhi, Yong
@ 2018-11-09 10:09 ` Sakari Ailus
2018-11-12 4:31 ` Bing Bu Cao
2018-11-29 23:09 ` Laurent Pinchart
2018-11-29 23:07 ` Laurent Pinchart
2 siblings, 2 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-11-09 10:09 UTC (permalink / raw)
To: Bing Bu Cao
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
Hi Bing Bu,
On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
>
> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> > Hi Yong,
> >
> > Thanks for the update!
> >
> > On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> >> Hi,
> >>
> >> This series adds support for the Intel IPU3 (Image Processing Unit)
> >> ImgU which is essentially a modern memory-to-memory ISP. It implements
> >> raw Bayer to YUV image format conversion as well as a large number of
> >> other pixel processing algorithms for improving the image quality.
> >>
> >> Meta data formats are defined for image statistics (3A, i.e. automatic
> >> white balance, exposure and focus, histogram and local area contrast
> >> enhancement) as well as for the pixel processing algorithm parameters.
> >> The documentation for these formats is currently not included in the
> >> patchset but will be added in a future version of this set.
> >>
> >> The algorithm parameters need to be considered specific to a given frame
> >> and typically a large number of these parameters change on frame to frame
> >> basis. Additionally, the parameters are highly structured (and not a flat
> >> space of independent configuration primitives). They also reflect the
> >> data structures used by the firmware and the hardware. On top of that,
> >> the algorithms require highly specialized user space to make meaningful
> >> use of them. For these reasons it has been chosen video buffers to pass
> >> the parameters to the device.
> >>
> >> On individual patches:
> >>
> >> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> >> image processors and HW accelerators.
> >>
> >> The 3A statistics and other firmware parameter computation related
> >> functions are implemented in patch 11.
> >>
> >> All IPU3 pipeline default settings can be found in patch 10.
> >>
> >> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> >> its own MMU unit, the driver is implemented in patch 6.
> >>
> >> Patch 7 uses above driver for DMA mapping operation.
> >>
> >> The communication between IPU3 firmware and driver is implemented with circular
> >> queues in patch 8.
> >>
> >> Patch 9 provide some utility functions and manage IPU3 fw download and
> >> install.
> >>
> >> The firmware which is called ipu3-fw.bin can be downloaded from:
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> >> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
> >>
> >> Firmware ABI is defined in patches 4 and 5.
> >>
> >> Patches 12 and 13 are of the same file, the former contains all h/w programming
> >> related code, the latter implements interface functions for access fw & hw
> >> capabilities.
> >>
> >> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
> >>
> >> <URL:https://patchwork.kernel.org/patch/9976295/>
> > I've pushed the latest set here:
> >
> > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
> >
> > You can just say the entire set depends on those going forward; the
> > documentation is needed, too.
> >
> >> Patch 15 represents the top level that glues all of the other components together,
> >> passing arguments between the components.
> >>
> >> Patch 16 is a recent effort to extend v6 for advanced camera features like
> >> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
> >>
> >> Link to user space implementation:
> >>
> >> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
> >>
> >> ImgU media topology print:
> >>
> >> # media-ctl -d /dev/media0 -p
> >> Media controller API version 4.19.0
> >>
> >> Media device information
> >> ------------------------
> >> driver ipu3-imgu
> >> model ipu3-imgu
> >> serial
> >> bus info PCI:0000:00:05.0
> >> hw revision 0x80862015
> >> driver version 4.19.0
> >>
> >> Device topology
> >> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >> type V4L2 subdev subtype Unknown flags 0
> >> device node name /dev/v4l-subdev0
> >> pad0: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> > This doesn't seem right. Which formats can be enumerated from the pad?
Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
format whereas the CAPTURE video nodes always have NV12. Can you confirm?
If the OUTPUT video node format selection has no effect on the rest of the
pipeline (device capabilities, which processing blocks are in use, CAPTURE
video nodes formats etc.), I think you could simply use the FIXED media bus
code for each pad. That would actually make sense: this device always works
from memory to memory, and thus does not really have a pixel data bus
external to the device which is what the media bus codes really are for.
> >
> >> crop:(0,0)/1920x1080
> >> compose:(0,0)/1920x1080]
> > Does the compose rectangle affect the scaling on all outputs?
> Sakari, driver use crop and compose targets to help set input-feeder and BDS
> output resolutions which are 2 key block of whole imaging pipeline, not the
> actual ending output, but they will impact the final output.
Ack. Thanks for the clarification.
> >
> >> <- "ipu3-imgu 0 input":0 []
> > Are there links that have no useful link configuration? If so, you should
> > set them enabled and immutable in the driver.
> The enabled status of input pads is used to get which pipe that user is
> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
But the rest of them could be, right?
> >
> >> pad1: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> > I'd suggest to use MEDIA_BUS_FMT_FIXED here.
> >
> >> <- "ipu3-imgu 0 parameters":0 []
> >> pad2: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 output":0 []
> >> pad3: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 viewfinder":0 []
> > Are there other differences between output and viewfinder?
> output and viewfinder are the main and secondary output of output system.
> 'main' output is not allowed to be scaled, only support crop. secondary
> output 'viewfinder'
> can support both cropping and scaling. User can select different nodes
> to use
> as preview and capture flexibly based on the actual use cases.
If there's scaling to be configured, I'd expect to see the COMPOSE target
supported.
--
Kind regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-09 10:09 ` Sakari Ailus
@ 2018-11-12 4:31 ` Bing Bu Cao
2018-11-13 10:31 ` Sakari Ailus
2018-11-29 23:09 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Bing Bu Cao @ 2018-11-12 4:31 UTC (permalink / raw)
To: Sakari Ailus
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
On 11/09/2018 06:09 PM, Sakari Ailus wrote:
> Hi Bing Bu,
>
> On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
>> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
>>> Hi Yong,
>>>
>>> Thanks for the update!
>>>
>>> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
>>>> Hi,
>>>>
>>>> This series adds support for the Intel IPU3 (Image Processing Unit)
>>>> ImgU which is essentially a modern memory-to-memory ISP. It implements
>>>> raw Bayer to YUV image format conversion as well as a large number of
>>>> other pixel processing algorithms for improving the image quality.
>>>>
>>>> Meta data formats are defined for image statistics (3A, i.e. automatic
>>>> white balance, exposure and focus, histogram and local area contrast
>>>> enhancement) as well as for the pixel processing algorithm parameters.
>>>> The documentation for these formats is currently not included in the
>>>> patchset but will be added in a future version of this set.
>>>>
>>>> The algorithm parameters need to be considered specific to a given frame
>>>> and typically a large number of these parameters change on frame to frame
>>>> basis. Additionally, the parameters are highly structured (and not a flat
>>>> space of independent configuration primitives). They also reflect the
>>>> data structures used by the firmware and the hardware. On top of that,
>>>> the algorithms require highly specialized user space to make meaningful
>>>> use of them. For these reasons it has been chosen video buffers to pass
>>>> the parameters to the device.
>>>>
>>>> On individual patches:
>>>>
>>>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
>>>> image processors and HW accelerators.
>>>>
>>>> The 3A statistics and other firmware parameter computation related
>>>> functions are implemented in patch 11.
>>>>
>>>> All IPU3 pipeline default settings can be found in patch 10.
>>>>
>>>> To access DDR via ImgU's own memory space, IPU3 is also equipped with
>>>> its own MMU unit, the driver is implemented in patch 6.
>>>>
>>>> Patch 7 uses above driver for DMA mapping operation.
>>>>
>>>> The communication between IPU3 firmware and driver is implemented with circular
>>>> queues in patch 8.
>>>>
>>>> Patch 9 provide some utility functions and manage IPU3 fw download and
>>>> install.
>>>>
>>>> The firmware which is called ipu3-fw.bin can be downloaded from:
>>>>
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
>>>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>>>>
>>>> Firmware ABI is defined in patches 4 and 5.
>>>>
>>>> Patches 12 and 13 are of the same file, the former contains all h/w programming
>>>> related code, the latter implements interface functions for access fw & hw
>>>> capabilities.
>>>>
>>>> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>>>>
>>>> <URL:https://patchwork.kernel.org/patch/9976295/>
>>> I've pushed the latest set here:
>>>
>>> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
>>>
>>> You can just say the entire set depends on those going forward; the
>>> documentation is needed, too.
>>>
>>>> Patch 15 represents the top level that glues all of the other components together,
>>>> passing arguments between the components.
>>>>
>>>> Patch 16 is a recent effort to extend v6 for advanced camera features like
>>>> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>>>>
>>>> Link to user space implementation:
>>>>
>>>> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>>>>
>>>> ImgU media topology print:
>>>>
>>>> # media-ctl -d /dev/media0 -p
>>>> Media controller API version 4.19.0
>>>>
>>>> Media device information
>>>> ------------------------
>>>> driver ipu3-imgu
>>>> model ipu3-imgu
>>>> serial
>>>> bus info PCI:0000:00:05.0
>>>> hw revision 0x80862015
>>>> driver version 4.19.0
>>>>
>>>> Device topology
>>>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
>>>> type V4L2 subdev subtype Unknown flags 0
>>>> device node name /dev/v4l-subdev0
>>>> pad0: Sink
>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
>>> This doesn't seem right. Which formats can be enumerated from the pad?
> Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
> format whereas the CAPTURE video nodes always have NV12. Can you confirm?
Hi, Sakari,
Yes, I think the pad_fmt should also be changed.
Yong, could you add some extra code for this and test? like:
static int ipu3_v4l2_node_setup(struct imgu_device *imgu, unsigned int pipe,
...
V4L2_PIX_FMT_NV12;
node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
}
+ if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+ node->pad_fmt.code = MEDIA_BUS_FMT_SGRBG10_1X10;
+
>
> If the OUTPUT video node format selection has no effect on the rest of the
> pipeline (device capabilities, which processing blocks are in use, CAPTURE
> video nodes formats etc.), I think you could simply use the FIXED media bus
> code for each pad. That would actually make sense: this device always works
> from memory to memory, and thus does not really have a pixel data bus
> external to the device which is what the media bus codes really are for.
>
>>>> crop:(0,0)/1920x1080
>>>> compose:(0,0)/1920x1080]
>>> Does the compose rectangle affect the scaling on all outputs?
>> Sakari, driver use crop and compose targets to help set input-feeder and BDS
>> output resolutions which are 2 key block of whole imaging pipeline, not the
>> actual ending output, but they will impact the final output.
> Ack. Thanks for the clarification.
>
>>>> <- "ipu3-imgu 0 input":0 []
>>> Are there links that have no useful link configuration? If so, you should
>>> set them enabled and immutable in the driver.
>> The enabled status of input pads is used to get which pipe that user is
>> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
> But the rest of them could be, right?
Yes.
>
>>>> pad1: Sink
>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>> I'd suggest to use MEDIA_BUS_FMT_FIXED here.
>>>
>>>> <- "ipu3-imgu 0 parameters":0 []
>>>> pad2: Source
>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>> -> "ipu3-imgu 0 output":0 []
>>>> pad3: Source
>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>> -> "ipu3-imgu 0 viewfinder":0 []
>>> Are there other differences between output and viewfinder?
>> output and viewfinder are the main and secondary output of output system.
>> 'main' output is not allowed to be scaled, only support crop. secondary
>> output 'viewfinder'
>> can support both cropping and scaling. User can select different nodes
>> to use
>> as preview and capture flexibly based on the actual use cases.
> If there's scaling to be configured, I'd expect to see the COMPOSE target
> supported.
Actually the viewfinder is the result of scaling, that means you can not
do more scaling.
The resolution of output and viewfinder should be fixed once the
pipeline is determined.
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-12 4:31 ` Bing Bu Cao
@ 2018-11-13 10:31 ` Sakari Ailus
2018-11-13 11:04 ` Bing Bu Cao
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-13 10:31 UTC (permalink / raw)
To: Bing Bu Cao
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
Hi Bing Bu,
On Mon, Nov 12, 2018 at 12:31:16PM +0800, Bing Bu Cao wrote:
>
>
> On 11/09/2018 06:09 PM, Sakari Ailus wrote:
> > Hi Bing Bu,
> >
> > On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
> >> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> >>> Hi Yong,
> >>>
> >>> Thanks for the update!
> >>>
> >>> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> >>>> Hi,
> >>>>
> >>>> This series adds support for the Intel IPU3 (Image Processing Unit)
> >>>> ImgU which is essentially a modern memory-to-memory ISP. It implements
> >>>> raw Bayer to YUV image format conversion as well as a large number of
> >>>> other pixel processing algorithms for improving the image quality.
> >>>>
> >>>> Meta data formats are defined for image statistics (3A, i.e. automatic
> >>>> white balance, exposure and focus, histogram and local area contrast
> >>>> enhancement) as well as for the pixel processing algorithm parameters.
> >>>> The documentation for these formats is currently not included in the
> >>>> patchset but will be added in a future version of this set.
> >>>>
> >>>> The algorithm parameters need to be considered specific to a given frame
> >>>> and typically a large number of these parameters change on frame to frame
> >>>> basis. Additionally, the parameters are highly structured (and not a flat
> >>>> space of independent configuration primitives). They also reflect the
> >>>> data structures used by the firmware and the hardware. On top of that,
> >>>> the algorithms require highly specialized user space to make meaningful
> >>>> use of them. For these reasons it has been chosen video buffers to pass
> >>>> the parameters to the device.
> >>>>
> >>>> On individual patches:
> >>>>
> >>>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> >>>> image processors and HW accelerators.
> >>>>
> >>>> The 3A statistics and other firmware parameter computation related
> >>>> functions are implemented in patch 11.
> >>>>
> >>>> All IPU3 pipeline default settings can be found in patch 10.
> >>>>
> >>>> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> >>>> its own MMU unit, the driver is implemented in patch 6.
> >>>>
> >>>> Patch 7 uses above driver for DMA mapping operation.
> >>>>
> >>>> The communication between IPU3 firmware and driver is implemented with circular
> >>>> queues in patch 8.
> >>>>
> >>>> Patch 9 provide some utility functions and manage IPU3 fw download and
> >>>> install.
> >>>>
> >>>> The firmware which is called ipu3-fw.bin can be downloaded from:
> >>>>
> >>>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> >>>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
> >>>>
> >>>> Firmware ABI is defined in patches 4 and 5.
> >>>>
> >>>> Patches 12 and 13 are of the same file, the former contains all h/w programming
> >>>> related code, the latter implements interface functions for access fw & hw
> >>>> capabilities.
> >>>>
> >>>> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
> >>>>
> >>>> <URL:https://patchwork.kernel.org/patch/9976295/>
> >>> I've pushed the latest set here:
> >>>
> >>> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
> >>>
> >>> You can just say the entire set depends on those going forward; the
> >>> documentation is needed, too.
> >>>
> >>>> Patch 15 represents the top level that glues all of the other components together,
> >>>> passing arguments between the components.
> >>>>
> >>>> Patch 16 is a recent effort to extend v6 for advanced camera features like
> >>>> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
> >>>>
> >>>> Link to user space implementation:
> >>>>
> >>>> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
> >>>>
> >>>> ImgU media topology print:
> >>>>
> >>>> # media-ctl -d /dev/media0 -p
> >>>> Media controller API version 4.19.0
> >>>>
> >>>> Media device information
> >>>> ------------------------
> >>>> driver ipu3-imgu
> >>>> model ipu3-imgu
> >>>> serial
> >>>> bus info PCI:0000:00:05.0
> >>>> hw revision 0x80862015
> >>>> driver version 4.19.0
> >>>>
> >>>> Device topology
> >>>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >>>> type V4L2 subdev subtype Unknown flags 0
> >>>> device node name /dev/v4l-subdev0
> >>>> pad0: Sink
> >>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >>> This doesn't seem right. Which formats can be enumerated from the pad?
> > Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
> > format whereas the CAPTURE video nodes always have NV12. Can you confirm?
> Hi, Sakari,
> Yes, I think the pad_fmt should also be changed.
> Yong, could you add some extra code for this and test? like:
>
> static int ipu3_v4l2_node_setup(struct imgu_device *imgu, unsigned int pipe,
> ...
> V4L2_PIX_FMT_NV12;
> node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
> }
>
> + if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
> + node->pad_fmt.code = MEDIA_BUS_FMT_SGRBG10_1X10;
> +
>
> >
> > If the OUTPUT video node format selection has no effect on the rest of the
> > pipeline (device capabilities, which processing blocks are in use, CAPTURE
> > video nodes formats etc.), I think you could simply use the FIXED media bus
> > code for each pad. That would actually make sense: this device always works
> > from memory to memory, and thus does not really have a pixel data bus
> > external to the device which is what the media bus codes really are for.
> >
> >>>> crop:(0,0)/1920x1080
> >>>> compose:(0,0)/1920x1080]
> >>> Does the compose rectangle affect the scaling on all outputs?
> >> Sakari, driver use crop and compose targets to help set input-feeder and BDS
> >> output resolutions which are 2 key block of whole imaging pipeline, not the
> >> actual ending output, but they will impact the final output.
> > Ack. Thanks for the clarification.
> >
> >>>> <- "ipu3-imgu 0 input":0 []
> >>> Are there links that have no useful link configuration? If so, you should
> >>> set them enabled and immutable in the driver.
> >> The enabled status of input pads is used to get which pipe that user is
> >> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
> > But the rest of them could be, right?
> Yes.
> >
> >>>> pad1: Sink
> >>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>> I'd suggest to use MEDIA_BUS_FMT_FIXED here.
> >>>
> >>>> <- "ipu3-imgu 0 parameters":0 []
> >>>> pad2: Source
> >>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>>> -> "ipu3-imgu 0 output":0 []
> >>>> pad3: Source
> >>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>>> -> "ipu3-imgu 0 viewfinder":0 []
> >>> Are there other differences between output and viewfinder?
> >> output and viewfinder are the main and secondary output of output system.
> >> 'main' output is not allowed to be scaled, only support crop. secondary
> >> output 'viewfinder'
> >> can support both cropping and scaling. User can select different nodes
> >> to use
> >> as preview and capture flexibly based on the actual use cases.
> > If there's scaling to be configured, I'd expect to see the COMPOSE target
> > supported.
> Actually the viewfinder is the result of scaling, that means you can not
> do more scaling.
How do you configure the scaling of the viewfinder currently?
> The resolution of output and viewfinder should be fixed once the
> pipeline is determined.
--
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-13 10:31 ` Sakari Ailus
@ 2018-11-13 11:04 ` Bing Bu Cao
2018-11-13 21:58 ` Sakari Ailus
0 siblings, 1 reply; 123+ messages in thread
From: Bing Bu Cao @ 2018-11-13 11:04 UTC (permalink / raw)
To: Sakari Ailus
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
On 11/13/2018 06:31 PM, Sakari Ailus wrote:
> Hi Bing Bu,
>
> On Mon, Nov 12, 2018 at 12:31:16PM +0800, Bing Bu Cao wrote:
>>
>> On 11/09/2018 06:09 PM, Sakari Ailus wrote:
>>> Hi Bing Bu,
>>>
>>> On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
>>>> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
>>>>> Hi Yong,
>>>>>
>>>>> Thanks for the update!
>>>>>
>>>>> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
>>>>>> Hi,
>>>>>>
>>>>>> This series adds support for the Intel IPU3 (Image Processing Unit)
>>>>>> ImgU which is essentially a modern memory-to-memory ISP. It implements
>>>>>> raw Bayer to YUV image format conversion as well as a large number of
>>>>>> other pixel processing algorithms for improving the image quality.
>>>>>>
>>>>>> Meta data formats are defined for image statistics (3A, i.e. automatic
>>>>>> white balance, exposure and focus, histogram and local area contrast
>>>>>> enhancement) as well as for the pixel processing algorithm parameters.
>>>>>> The documentation for these formats is currently not included in the
>>>>>> patchset but will be added in a future version of this set.
>>>>>>
>>>>>> The algorithm parameters need to be considered specific to a given frame
>>>>>> and typically a large number of these parameters change on frame to frame
>>>>>> basis. Additionally, the parameters are highly structured (and not a flat
>>>>>> space of independent configuration primitives). They also reflect the
>>>>>> data structures used by the firmware and the hardware. On top of that,
>>>>>> the algorithms require highly specialized user space to make meaningful
>>>>>> use of them. For these reasons it has been chosen video buffers to pass
>>>>>> the parameters to the device.
>>>>>>
>>>>>> On individual patches:
>>>>>>
>>>>>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
>>>>>> image processors and HW accelerators.
>>>>>>
>>>>>> The 3A statistics and other firmware parameter computation related
>>>>>> functions are implemented in patch 11.
>>>>>>
>>>>>> All IPU3 pipeline default settings can be found in patch 10.
>>>>>>
>>>>>> To access DDR via ImgU's own memory space, IPU3 is also equipped with
>>>>>> its own MMU unit, the driver is implemented in patch 6.
>>>>>>
>>>>>> Patch 7 uses above driver for DMA mapping operation.
>>>>>>
>>>>>> The communication between IPU3 firmware and driver is implemented with circular
>>>>>> queues in patch 8.
>>>>>>
>>>>>> Patch 9 provide some utility functions and manage IPU3 fw download and
>>>>>> install.
>>>>>>
>>>>>> The firmware which is called ipu3-fw.bin can be downloaded from:
>>>>>>
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
>>>>>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>>>>>>
>>>>>> Firmware ABI is defined in patches 4 and 5.
>>>>>>
>>>>>> Patches 12 and 13 are of the same file, the former contains all h/w programming
>>>>>> related code, the latter implements interface functions for access fw & hw
>>>>>> capabilities.
>>>>>>
>>>>>> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>>>>>>
>>>>>> <URL:https://patchwork.kernel.org/patch/9976295/>
>>>>> I've pushed the latest set here:
>>>>>
>>>>> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
>>>>>
>>>>> You can just say the entire set depends on those going forward; the
>>>>> documentation is needed, too.
>>>>>
>>>>>> Patch 15 represents the top level that glues all of the other components together,
>>>>>> passing arguments between the components.
>>>>>>
>>>>>> Patch 16 is a recent effort to extend v6 for advanced camera features like
>>>>>> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>>>>>>
>>>>>> Link to user space implementation:
>>>>>>
>>>>>> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>>>>>>
>>>>>> ImgU media topology print:
>>>>>>
>>>>>> # media-ctl -d /dev/media0 -p
>>>>>> Media controller API version 4.19.0
>>>>>>
>>>>>> Media device information
>>>>>> ------------------------
>>>>>> driver ipu3-imgu
>>>>>> model ipu3-imgu
>>>>>> serial
>>>>>> bus info PCI:0000:00:05.0
>>>>>> hw revision 0x80862015
>>>>>> driver version 4.19.0
>>>>>>
>>>>>> Device topology
>>>>>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
>>>>>> type V4L2 subdev subtype Unknown flags 0
>>>>>> device node name /dev/v4l-subdev0
>>>>>> pad0: Sink
>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
>>>>> This doesn't seem right. Which formats can be enumerated from the pad?
>>> Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
>>> format whereas the CAPTURE video nodes always have NV12. Can you confirm?
>> Hi, Sakari,
>> Yes, I think the pad_fmt should also be changed.
>> Yong, could you add some extra code for this and test? like:
>>
>> static int ipu3_v4l2_node_setup(struct imgu_device *imgu, unsigned int pipe,
>> ...
>> V4L2_PIX_FMT_NV12;
>> node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
>> }
>>
>> + if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
>> + node->pad_fmt.code = MEDIA_BUS_FMT_SGRBG10_1X10;
>> +
>>
>>> If the OUTPUT video node format selection has no effect on the rest of the
>>> pipeline (device capabilities, which processing blocks are in use, CAPTURE
>>> video nodes formats etc.), I think you could simply use the FIXED media bus
>>> code for each pad. That would actually make sense: this device always works
>>> from memory to memory, and thus does not really have a pixel data bus
>>> external to the device which is what the media bus codes really are for.
>>>
>>>>>> crop:(0,0)/1920x1080
>>>>>> compose:(0,0)/1920x1080]
>>>>> Does the compose rectangle affect the scaling on all outputs?
>>>> Sakari, driver use crop and compose targets to help set input-feeder and BDS
>>>> output resolutions which are 2 key block of whole imaging pipeline, not the
>>>> actual ending output, but they will impact the final output.
>>> Ack. Thanks for the clarification.
>>>
>>>>>> <- "ipu3-imgu 0 input":0 []
>>>>> Are there links that have no useful link configuration? If so, you should
>>>>> set them enabled and immutable in the driver.
>>>> The enabled status of input pads is used to get which pipe that user is
>>>> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
>>> But the rest of them could be, right?
>> Yes.
>>>>>> pad1: Sink
>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>> I'd suggest to use MEDIA_BUS_FMT_FIXED here.
>>>>>
>>>>>> <- "ipu3-imgu 0 parameters":0 []
>>>>>> pad2: Source
>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>>> -> "ipu3-imgu 0 output":0 []
>>>>>> pad3: Source
>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>>> -> "ipu3-imgu 0 viewfinder":0 []
>>>>> Are there other differences between output and viewfinder?
>>>> output and viewfinder are the main and secondary output of output system.
>>>> 'main' output is not allowed to be scaled, only support crop. secondary
>>>> output 'viewfinder'
>>>> can support both cropping and scaling. User can select different nodes
>>>> to use
>>>> as preview and capture flexibly based on the actual use cases.
>>> If there's scaling to be configured, I'd expect to see the COMPOSE target
>>> supported.
>> Actually the viewfinder is the result of scaling, that means you can not
>> do more scaling.
> How do you configure the scaling of the viewfinder currently?
We consider that the viewfinder as a secondary output, and set the format by
subdev set_fmt() directly and all pads formats will be used to find
binary and
build pipeline.
>
>> The resolution of output and viewfinder should be fixed once the
>> pipeline is determined.
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-13 11:04 ` Bing Bu Cao
@ 2018-11-13 21:58 ` Sakari Ailus
2018-11-14 7:02 ` Bing Bu Cao
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-13 21:58 UTC (permalink / raw)
To: Bing Bu Cao
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
On Tue, Nov 13, 2018 at 07:04:01PM +0800, Bing Bu Cao wrote:
>
>
> On 11/13/2018 06:31 PM, Sakari Ailus wrote:
> > Hi Bing Bu,
> >
> > On Mon, Nov 12, 2018 at 12:31:16PM +0800, Bing Bu Cao wrote:
> >>
> >> On 11/09/2018 06:09 PM, Sakari Ailus wrote:
> >>> Hi Bing Bu,
> >>>
> >>> On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
> >>>> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> >>>>> Hi Yong,
> >>>>>
> >>>>> Thanks for the update!
> >>>>>
> >>>>> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> This series adds support for the Intel IPU3 (Image Processing Unit)
> >>>>>> ImgU which is essentially a modern memory-to-memory ISP. It implements
> >>>>>> raw Bayer to YUV image format conversion as well as a large number of
> >>>>>> other pixel processing algorithms for improving the image quality.
> >>>>>>
> >>>>>> Meta data formats are defined for image statistics (3A, i.e. automatic
> >>>>>> white balance, exposure and focus, histogram and local area contrast
> >>>>>> enhancement) as well as for the pixel processing algorithm parameters.
> >>>>>> The documentation for these formats is currently not included in the
> >>>>>> patchset but will be added in a future version of this set.
> >>>>>>
> >>>>>> The algorithm parameters need to be considered specific to a given frame
> >>>>>> and typically a large number of these parameters change on frame to frame
> >>>>>> basis. Additionally, the parameters are highly structured (and not a flat
> >>>>>> space of independent configuration primitives). They also reflect the
> >>>>>> data structures used by the firmware and the hardware. On top of that,
> >>>>>> the algorithms require highly specialized user space to make meaningful
> >>>>>> use of them. For these reasons it has been chosen video buffers to pass
> >>>>>> the parameters to the device.
> >>>>>>
> >>>>>> On individual patches:
> >>>>>>
> >>>>>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> >>>>>> image processors and HW accelerators.
> >>>>>>
> >>>>>> The 3A statistics and other firmware parameter computation related
> >>>>>> functions are implemented in patch 11.
> >>>>>>
> >>>>>> All IPU3 pipeline default settings can be found in patch 10.
> >>>>>>
> >>>>>> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> >>>>>> its own MMU unit, the driver is implemented in patch 6.
> >>>>>>
> >>>>>> Patch 7 uses above driver for DMA mapping operation.
> >>>>>>
> >>>>>> The communication between IPU3 firmware and driver is implemented with circular
> >>>>>> queues in patch 8.
> >>>>>>
> >>>>>> Patch 9 provide some utility functions and manage IPU3 fw download and
> >>>>>> install.
> >>>>>>
> >>>>>> The firmware which is called ipu3-fw.bin can be downloaded from:
> >>>>>>
> >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> >>>>>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
> >>>>>>
> >>>>>> Firmware ABI is defined in patches 4 and 5.
> >>>>>>
> >>>>>> Patches 12 and 13 are of the same file, the former contains all h/w programming
> >>>>>> related code, the latter implements interface functions for access fw & hw
> >>>>>> capabilities.
> >>>>>>
> >>>>>> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
> >>>>>>
> >>>>>> <URL:https://patchwork.kernel.org/patch/9976295/>
> >>>>> I've pushed the latest set here:
> >>>>>
> >>>>> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
> >>>>>
> >>>>> You can just say the entire set depends on those going forward; the
> >>>>> documentation is needed, too.
> >>>>>
> >>>>>> Patch 15 represents the top level that glues all of the other components together,
> >>>>>> passing arguments between the components.
> >>>>>>
> >>>>>> Patch 16 is a recent effort to extend v6 for advanced camera features like
> >>>>>> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
> >>>>>>
> >>>>>> Link to user space implementation:
> >>>>>>
> >>>>>> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
> >>>>>>
> >>>>>> ImgU media topology print:
> >>>>>>
> >>>>>> # media-ctl -d /dev/media0 -p
> >>>>>> Media controller API version 4.19.0
> >>>>>>
> >>>>>> Media device information
> >>>>>> ------------------------
> >>>>>> driver ipu3-imgu
> >>>>>> model ipu3-imgu
> >>>>>> serial
> >>>>>> bus info PCI:0000:00:05.0
> >>>>>> hw revision 0x80862015
> >>>>>> driver version 4.19.0
> >>>>>>
> >>>>>> Device topology
> >>>>>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >>>>>> type V4L2 subdev subtype Unknown flags 0
> >>>>>> device node name /dev/v4l-subdev0
> >>>>>> pad0: Sink
> >>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >>>>> This doesn't seem right. Which formats can be enumerated from the pad?
> >>> Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
> >>> format whereas the CAPTURE video nodes always have NV12. Can you confirm?
> >> Hi, Sakari,
> >> Yes, I think the pad_fmt should also be changed.
> >> Yong, could you add some extra code for this and test? like:
> >>
> >> static int ipu3_v4l2_node_setup(struct imgu_device *imgu, unsigned int pipe,
> >> ...
> >> V4L2_PIX_FMT_NV12;
> >> node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
> >> }
> >>
> >> + if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
> >> + node->pad_fmt.code = MEDIA_BUS_FMT_SGRBG10_1X10;
> >> +
> >>
> >>> If the OUTPUT video node format selection has no effect on the rest of the
> >>> pipeline (device capabilities, which processing blocks are in use, CAPTURE
> >>> video nodes formats etc.), I think you could simply use the FIXED media bus
> >>> code for each pad. That would actually make sense: this device always works
> >>> from memory to memory, and thus does not really have a pixel data bus
> >>> external to the device which is what the media bus codes really are for.
> >>>
> >>>>>> crop:(0,0)/1920x1080
> >>>>>> compose:(0,0)/1920x1080]
> >>>>> Does the compose rectangle affect the scaling on all outputs?
> >>>> Sakari, driver use crop and compose targets to help set input-feeder and BDS
> >>>> output resolutions which are 2 key block of whole imaging pipeline, not the
> >>>> actual ending output, but they will impact the final output.
> >>> Ack. Thanks for the clarification.
> >>>
> >>>>>> <- "ipu3-imgu 0 input":0 []
> >>>>> Are there links that have no useful link configuration? If so, you should
> >>>>> set them enabled and immutable in the driver.
> >>>> The enabled status of input pads is used to get which pipe that user is
> >>>> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
> >>> But the rest of them could be, right?
> >> Yes.
> >>>>>> pad1: Sink
> >>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>>>> I'd suggest to use MEDIA_BUS_FMT_FIXED here.
> >>>>>
> >>>>>> <- "ipu3-imgu 0 parameters":0 []
> >>>>>> pad2: Source
> >>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>>>>> -> "ipu3-imgu 0 output":0 []
> >>>>>> pad3: Source
> >>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>>>>> -> "ipu3-imgu 0 viewfinder":0 []
> >>>>> Are there other differences between output and viewfinder?
> >>>> output and viewfinder are the main and secondary output of output system.
> >>>> 'main' output is not allowed to be scaled, only support crop. secondary
> >>>> output 'viewfinder'
> >>>> can support both cropping and scaling. User can select different nodes
> >>>> to use
> >>>> as preview and capture flexibly based on the actual use cases.
> >>> If there's scaling to be configured, I'd expect to see the COMPOSE target
> >>> supported.
> >> Actually the viewfinder is the result of scaling, that means you can not
> >> do more scaling.
> > How do you configure the scaling of the viewfinder currently?
> We consider that the viewfinder as a secondary output, and set the format by
> subdev set_fmt() directly and all pads formats will be used to find
> binary and
> build pipeline.
Ok.
Could you instead use the compose target to configure the scaling? Setting
the format on the source pad would have no effect.
--
Kind regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-13 21:58 ` Sakari Ailus
@ 2018-11-14 7:02 ` Bing Bu Cao
0 siblings, 0 replies; 123+ messages in thread
From: Bing Bu Cao @ 2018-11-14 7:02 UTC (permalink / raw)
To: Sakari Ailus
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
On 11/14/2018 05:58 AM, Sakari Ailus wrote:
> On Tue, Nov 13, 2018 at 07:04:01PM +0800, Bing Bu Cao wrote:
>>
>> On 11/13/2018 06:31 PM, Sakari Ailus wrote:
>>> Hi Bing Bu,
>>>
>>> On Mon, Nov 12, 2018 at 12:31:16PM +0800, Bing Bu Cao wrote:
>>>> On 11/09/2018 06:09 PM, Sakari Ailus wrote:
>>>>> Hi Bing Bu,
>>>>>
>>>>> On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
>>>>>> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
>>>>>>> Hi Yong,
>>>>>>>
>>>>>>> Thanks for the update!
>>>>>>>
>>>>>>> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This series adds support for the Intel IPU3 (Image Processing Unit)
>>>>>>>> ImgU which is essentially a modern memory-to-memory ISP. It implements
>>>>>>>> raw Bayer to YUV image format conversion as well as a large number of
>>>>>>>> other pixel processing algorithms for improving the image quality.
>>>>>>>>
>>>>>>>> Meta data formats are defined for image statistics (3A, i.e. automatic
>>>>>>>> white balance, exposure and focus, histogram and local area contrast
>>>>>>>> enhancement) as well as for the pixel processing algorithm parameters.
>>>>>>>> The documentation for these formats is currently not included in the
>>>>>>>> patchset but will be added in a future version of this set.
>>>>>>>>
>>>>>>>> The algorithm parameters need to be considered specific to a given frame
>>>>>>>> and typically a large number of these parameters change on frame to frame
>>>>>>>> basis. Additionally, the parameters are highly structured (and not a flat
>>>>>>>> space of independent configuration primitives). They also reflect the
>>>>>>>> data structures used by the firmware and the hardware. On top of that,
>>>>>>>> the algorithms require highly specialized user space to make meaningful
>>>>>>>> use of them. For these reasons it has been chosen video buffers to pass
>>>>>>>> the parameters to the device.
>>>>>>>>
>>>>>>>> On individual patches:
>>>>>>>>
>>>>>>>> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
>>>>>>>> image processors and HW accelerators.
>>>>>>>>
>>>>>>>> The 3A statistics and other firmware parameter computation related
>>>>>>>> functions are implemented in patch 11.
>>>>>>>>
>>>>>>>> All IPU3 pipeline default settings can be found in patch 10.
>>>>>>>>
>>>>>>>> To access DDR via ImgU's own memory space, IPU3 is also equipped with
>>>>>>>> its own MMU unit, the driver is implemented in patch 6.
>>>>>>>>
>>>>>>>> Patch 7 uses above driver for DMA mapping operation.
>>>>>>>>
>>>>>>>> The communication between IPU3 firmware and driver is implemented with circular
>>>>>>>> queues in patch 8.
>>>>>>>>
>>>>>>>> Patch 9 provide some utility functions and manage IPU3 fw download and
>>>>>>>> install.
>>>>>>>>
>>>>>>>> The firmware which is called ipu3-fw.bin can be downloaded from:
>>>>>>>>
>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
>>>>>>>> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>>>>>>>>
>>>>>>>> Firmware ABI is defined in patches 4 and 5.
>>>>>>>>
>>>>>>>> Patches 12 and 13 are of the same file, the former contains all h/w programming
>>>>>>>> related code, the latter implements interface functions for access fw & hw
>>>>>>>> capabilities.
>>>>>>>>
>>>>>>>> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>>>>>>>>
>>>>>>>> <URL:https://patchwork.kernel.org/patch/9976295/>
>>>>>>> I've pushed the latest set here:
>>>>>>>
>>>>>>> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=meta-output>
>>>>>>>
>>>>>>> You can just say the entire set depends on those going forward; the
>>>>>>> documentation is needed, too.
>>>>>>>
>>>>>>>> Patch 15 represents the top level that glues all of the other components together,
>>>>>>>> passing arguments between the components.
>>>>>>>>
>>>>>>>> Patch 16 is a recent effort to extend v6 for advanced camera features like
>>>>>>>> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>>>>>>>>
>>>>>>>> Link to user space implementation:
>>>>>>>>
>>>>>>>> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>>>>>>>>
>>>>>>>> ImgU media topology print:
>>>>>>>>
>>>>>>>> # media-ctl -d /dev/media0 -p
>>>>>>>> Media controller API version 4.19.0
>>>>>>>>
>>>>>>>> Media device information
>>>>>>>> ------------------------
>>>>>>>> driver ipu3-imgu
>>>>>>>> model ipu3-imgu
>>>>>>>> serial
>>>>>>>> bus info PCI:0000:00:05.0
>>>>>>>> hw revision 0x80862015
>>>>>>>> driver version 4.19.0
>>>>>>>>
>>>>>>>> Device topology
>>>>>>>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
>>>>>>>> type V4L2 subdev subtype Unknown flags 0
>>>>>>>> device node name /dev/v4l-subdev0
>>>>>>>> pad0: Sink
>>>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
>>>>>>> This doesn't seem right. Which formats can be enumerated from the pad?
>>>>> Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
>>>>> format whereas the CAPTURE video nodes always have NV12. Can you confirm?
>>>> Hi, Sakari,
>>>> Yes, I think the pad_fmt should also be changed.
>>>> Yong, could you add some extra code for this and test? like:
>>>>
>>>> static int ipu3_v4l2_node_setup(struct imgu_device *imgu, unsigned int pipe,
>>>> ...
>>>> V4L2_PIX_FMT_NV12;
>>>> node->vdev_fmt.fmt.pix_mp = def_pix_fmt;
>>>> }
>>>>
>>>> + if (node->vdev_fmt.type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
>>>> + node->pad_fmt.code = MEDIA_BUS_FMT_SGRBG10_1X10;
>>>> +
>>>>
>>>>> If the OUTPUT video node format selection has no effect on the rest of the
>>>>> pipeline (device capabilities, which processing blocks are in use, CAPTURE
>>>>> video nodes formats etc.), I think you could simply use the FIXED media bus
>>>>> code for each pad. That would actually make sense: this device always works
>>>>> from memory to memory, and thus does not really have a pixel data bus
>>>>> external to the device which is what the media bus codes really are for.
>>>>>
>>>>>>>> crop:(0,0)/1920x1080
>>>>>>>> compose:(0,0)/1920x1080]
>>>>>>> Does the compose rectangle affect the scaling on all outputs?
>>>>>> Sakari, driver use crop and compose targets to help set input-feeder and BDS
>>>>>> output resolutions which are 2 key block of whole imaging pipeline, not the
>>>>>> actual ending output, but they will impact the final output.
>>>>> Ack. Thanks for the clarification.
>>>>>
>>>>>>>> <- "ipu3-imgu 0 input":0 []
>>>>>>> Are there links that have no useful link configuration? If so, you should
>>>>>>> set them enabled and immutable in the driver.
>>>>>> The enabled status of input pads is used to get which pipe that user is
>>>>>> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
>>>>> But the rest of them could be, right?
>>>> Yes.
>>>>>>>> pad1: Sink
>>>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>>>> I'd suggest to use MEDIA_BUS_FMT_FIXED here.
>>>>>>>
>>>>>>>> <- "ipu3-imgu 0 parameters":0 []
>>>>>>>> pad2: Source
>>>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>>>>> -> "ipu3-imgu 0 output":0 []
>>>>>>>> pad3: Source
>>>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>>>>> -> "ipu3-imgu 0 viewfinder":0 []
>>>>>>> Are there other differences between output and viewfinder?
>>>>>> output and viewfinder are the main and secondary output of output system.
>>>>>> 'main' output is not allowed to be scaled, only support crop. secondary
>>>>>> output 'viewfinder'
>>>>>> can support both cropping and scaling. User can select different nodes
>>>>>> to use
>>>>>> as preview and capture flexibly based on the actual use cases.
>>>>> If there's scaling to be configured, I'd expect to see the COMPOSE target
>>>>> supported.
>>>> Actually the viewfinder is the result of scaling, that means you can not
>>>> do more scaling.
>>> How do you configure the scaling of the viewfinder currently?
>> We consider that the viewfinder as a secondary output, and set the format by
>> subdev set_fmt() directly and all pads formats will be used to find
>> binary and
>> build pipeline.
> Ok.
>
> Could you instead use the compose target to configure the scaling? Setting
> the format on the source pad would have no effect.
Hi, Sakari,
For the secondary output (viewfinder), it support both cropping and
scaling, in order
to keep the aspect ratio, system will do [crop --> compose], sounds like
it should
have 2 selection targets, but firmware/driver did not provide clear
interfaces to allow
user to configure the cropping, driver calculate the scaler and cropping
parameters
based on the output-system input and output resolution to keep aspect ratio
and field of view. I think it is more succinct to set the actual output
by set format
instead of using selection targets.
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-09 10:09 ` Sakari Ailus
2018-11-12 4:31 ` Bing Bu Cao
@ 2018-11-29 23:09 ` Laurent Pinchart
2018-11-30 13:37 ` Sakari Ailus
1 sibling, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-11-29 23:09 UTC (permalink / raw)
To: Sakari Ailus
Cc: Bing Bu Cao, Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Sakari,
On Friday, 9 November 2018 12:09:54 EET Sakari Ailus wrote:
> On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
> > On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> >> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
[snip]
> >>> ImgU media topology print:
> >>>
> >>> # media-ctl -d /dev/media0 -p
> >>> Media controller API version 4.19.0
> >>>
> >>> Media device information
> >>> ------------------------
> >>> driver ipu3-imgu
> >>> model ipu3-imgu
> >>> serial
> >>> bus info PCI:0000:00:05.0
> >>> hw revision 0x80862015
> >>> driver version 4.19.0
> >>>
> >>> Device topology
> >>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >>> type V4L2 subdev subtype Unknown flags 0
> >>> device node name /dev/v4l-subdev0
> >>> pad0: Sink
> >>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >>
> >> This doesn't seem right. Which formats can be enumerated from the pad?
>
> Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
> format whereas the CAPTURE video nodes always have NV12. Can you confirm?
>
> If the OUTPUT video node format selection has no effect on the rest of the
> pipeline (device capabilities, which processing blocks are in use, CAPTURE
> video nodes formats etc.), I think you could simply use the FIXED media bus
> code for each pad. That would actually make sense: this device always works
> from memory to memory, and thus does not really have a pixel data bus
> external to the device which is what the media bus codes really are for.
Isn't the Bayer variant useful information to configure debayering ? I would
expect it to be passed through the format on pad 0.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-29 23:09 ` Laurent Pinchart
@ 2018-11-30 13:37 ` Sakari Ailus
0 siblings, 0 replies; 123+ messages in thread
From: Sakari Ailus @ 2018-11-30 13:37 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Bing Bu Cao, Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Laurent,
On Fri, Nov 30, 2018 at 01:09:37AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
>
>
> On Friday, 9 November 2018 12:09:54 EET Sakari Ailus wrote:
> > On Wed, Nov 07, 2018 at 12:16:47PM +0800, Bing Bu Cao wrote:
> > > On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> > >> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
>
> [snip]
>
> > >>> ImgU media topology print:
> > >>>
> > >>> # media-ctl -d /dev/media0 -p
> > >>> Media controller API version 4.19.0
> > >>>
> > >>> Media device information
> > >>> ------------------------
> > >>> driver ipu3-imgu
> > >>> model ipu3-imgu
> > >>> serial
> > >>> bus info PCI:0000:00:05.0
> > >>> hw revision 0x80862015
> > >>> driver version 4.19.0
> > >>>
> > >>> Device topology
> > >>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> > >>> type V4L2 subdev subtype Unknown flags 0
> > >>> device node name /dev/v4l-subdev0
> > >>> pad0: Sink
> > >>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> > >>
> > >> This doesn't seem right. Which formats can be enumerated from the pad?
> >
> > Looking at the code, the OUTPUT video nodes have 10-bit GRBG (or a variant)
> > format whereas the CAPTURE video nodes always have NV12. Can you confirm?
> >
> > If the OUTPUT video node format selection has no effect on the rest of the
> > pipeline (device capabilities, which processing blocks are in use, CAPTURE
> > video nodes formats etc.), I think you could simply use the FIXED media bus
> > code for each pad. That would actually make sense: this device always works
> > from memory to memory, and thus does not really have a pixel data bus
> > external to the device which is what the media bus codes really are for.
>
> Isn't the Bayer variant useful information to configure debayering ? I would
> expect it to be passed through the format on pad 0.
That's already configured on the video node. The FIXED media bus code is
intended for links where there's nothing to configure --- which is the case
here.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-07 4:16 ` Bing Bu Cao
2018-11-09 1:28 ` Zhi, Yong
2018-11-09 10:09 ` Sakari Ailus
@ 2018-11-29 23:07 ` Laurent Pinchart
2018-12-03 9:51 ` Sakari Ailus
2 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-11-29 23:07 UTC (permalink / raw)
To: Bing Bu Cao
Cc: Sakari Ailus, Yong Zhi, linux-media, tfiga, mchehab,
hans.verkuil, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
Hello Bing,
On Wednesday, 7 November 2018 06:16:47 EET Bing Bu Cao wrote:
> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> > On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
[snip]
> >> ImgU media topology print:
> >>
> >> # media-ctl -d /dev/media0 -p
> >> Media controller API version 4.19.0
> >>
> >> Media device information
> >> ------------------------
> >> driver ipu3-imgu
> >> model ipu3-imgu
> >> serial
> >> bus info PCI:0000:00:05.0
> >> hw revision 0x80862015
> >> driver version 4.19.0
> >>
> >> Device topology
> >> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >> type V4L2 subdev subtype Unknown flags 0
> >> device node name /dev/v4l-subdev0
> >> pad0: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >
> > This doesn't seem right. Which formats can be enumerated from the pad?
> >
> >> crop:(0,0)/1920x1080
> >> compose:(0,0)/1920x1080]
> >
> > Does the compose rectangle affect the scaling on all outputs?
>
> Sakari, driver use crop and compose targets to help set input-feeder and BDS
> output resolutions which are 2 key block of whole imaging pipeline, not the
> actual ending output, but they will impact the final output.
>
> >> <- "ipu3-imgu 0 input":0 []
> >
> > Are there links that have no useful link configuration? If so, you should
> > set them enabled and immutable in the driver.
>
> The enabled status of input pads is used to get which pipe that user is
> trying to enable (ipu3_link_setup()), so it could not been set as immutable.
Each pipe needs an input in order to operate, so from that point of view the
input is mandatory. Why can't we make this link immutable, and use the stream
state (VIDIOC_STREAMON/VIDIOC_STREAMOFF) to enable/disable the pipes ?
> >> pad1: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >
> > I'd suggest to use MEDIA_BUS_FMT_FIXED here.
> >
> >> <- "ipu3-imgu 0 parameters":0 []
> >>
> >> pad2: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 output":0 []
> >>
> >> pad3: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 viewfinder":0 []
> >
> > Are there other differences between output and viewfinder?
>
> output and viewfinder are the main and secondary output of output system.
> 'main' output is not allowed to be scaled, only support crop. secondary
> output 'viewfinder' can support both cropping and scaling. User can select
> different nodes to use as preview and capture flexibly based on the actual
> use cases.
This is very useful information, thank you. Could it be added to the
documentation (patch 02/16) with a block diagram ?
> >> pad4: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 0 3a stat":0 []
> >
> > FIXED here, too.
> >
> >> - entity 7: ipu3-imgu 1 (5 pads, 5 links)
> >> type V4L2 subdev subtype Unknown flags 0
> >> device node name /dev/v4l-subdev1
> >>
> >> pad0: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >> crop:(0,0)/1920x1080
> >> compose:(0,0)/1920x1080]
> >> <- "ipu3-imgu 1 input":0 []
> >>
> >> pad1: Sink
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> <- "ipu3-imgu 1 parameters":0 []
> >>
> >> pad2: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 1 output":0 []
> >>
> >> pad3: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 1 viewfinder":0 []
> >>
> >> pad4: Source
> >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >> -> "ipu3-imgu 1 3a stat":0 []
> >
> > This is a minor matter but --- could you create the second sub-device
> > after the video device nodes related to the first one have been already
> > created? That'd make reading the output easier.
> >
> >> - entity 17: ipu3-imgu 0 input (1 pad, 1 link)
> >>
> >> type Node subtype V4L flags 0
> >> device node name /dev/video0
> >>
> >> pad0: Source
> >> -> "ipu3-imgu 0":0 []
> >>
> >> - entity 23: ipu3-imgu 0 parameters (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video1
> >>
> >> pad0: Source
> >> -> "ipu3-imgu 0":1 []
> >>
> >> - entity 29: ipu3-imgu 0 output (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video2
> >>
> >> pad0: Sink
> >> <- "ipu3-imgu 0":2 []
> >>
> >> - entity 35: ipu3-imgu 0 viewfinder (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video3
> >>
> >> pad0: Sink
> >> <- "ipu3-imgu 0":3 []
> >>
> >> - entity 41: ipu3-imgu 0 3a stat (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video4
> >>
> >> pad0: Sink
> >> <- "ipu3-imgu 0":4 []
> >>
> >> - entity 47: ipu3-imgu 1 input (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video5
> >>
> >> pad0: Source
> >> -> "ipu3-imgu 1":0 []
> >>
> >> - entity 53: ipu3-imgu 1 parameters (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video6
> >>
> >> pad0: Source
> >> -> "ipu3-imgu 1":1 []
> >>
> >> - entity 59: ipu3-imgu 1 output (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video7
> >>
> >> pad0: Sink
> >> <- "ipu3-imgu 1":2 []
> >>
> >> - entity 65: ipu3-imgu 1 viewfinder (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video8
> >>
> >> pad0: Sink
> >> <- "ipu3-imgu 1":3 []
> >>
> >> - entity 71: ipu3-imgu 1 3a stat (1 pad, 1 link)
> >> type Node subtype V4L flags 0
> >> device node name /dev/video9
> >>
> >> pad0: Sink
> >> <- "ipu3-imgu 1":4 []
[snip]
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-29 23:07 ` Laurent Pinchart
@ 2018-12-03 9:51 ` Sakari Ailus
2018-12-03 12:34 ` Laurent Pinchart
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-12-03 9:51 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Bing Bu Cao, Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Laurent,
On Fri, Nov 30, 2018 at 01:07:53AM +0200, Laurent Pinchart wrote:
> Hello Bing,
>
> On Wednesday, 7 November 2018 06:16:47 EET Bing Bu Cao wrote:
> > On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> > > On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
>
> [snip]
>
> > >> ImgU media topology print:
> > >>
> > >> # media-ctl -d /dev/media0 -p
> > >> Media controller API version 4.19.0
> > >>
> > >> Media device information
> > >> ------------------------
> > >> driver ipu3-imgu
> > >> model ipu3-imgu
> > >> serial
> > >> bus info PCI:0000:00:05.0
> > >> hw revision 0x80862015
> > >> driver version 4.19.0
> > >>
> > >> Device topology
> > >> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> > >> type V4L2 subdev subtype Unknown flags 0
> > >> device node name /dev/v4l-subdev0
> > >> pad0: Sink
> > >> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> > >
> > > This doesn't seem right. Which formats can be enumerated from the pad?
> > >
> > >> crop:(0,0)/1920x1080
> > >> compose:(0,0)/1920x1080]
> > >
> > > Does the compose rectangle affect the scaling on all outputs?
> >
> > Sakari, driver use crop and compose targets to help set input-feeder and BDS
> > output resolutions which are 2 key block of whole imaging pipeline, not the
> > actual ending output, but they will impact the final output.
> >
> > >> <- "ipu3-imgu 0 input":0 []
> > >
> > > Are there links that have no useful link configuration? If so, you should
> > > set them enabled and immutable in the driver.
> >
> > The enabled status of input pads is used to get which pipe that user is
> > trying to enable (ipu3_link_setup()), so it could not been set as immutable.
>
> Each pipe needs an input in order to operate, so from that point of view the
> input is mandatory. Why can't we make this link immutable, and use the stream
> state (VIDIOC_STREAMON/VIDIOC_STREAMOFF) to enable/disable the pipes ?
There are only two options (AFAIK) in choosing the firmware, and by
configuring the links this is better visible to the user: the links the
state of which can be changed are not immutable. The driver can also obtain
the explicit pipeline configuration, which makes the implementation more
simple.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-03 9:51 ` Sakari Ailus
@ 2018-12-03 12:34 ` Laurent Pinchart
0 siblings, 0 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-03 12:34 UTC (permalink / raw)
To: Sakari Ailus
Cc: Bing Bu Cao, Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hi Sakari,
On Monday, 3 December 2018 11:51:01 EET Sakari Ailus wrote:
> On Fri, Nov 30, 2018 at 01:07:53AM +0200, Laurent Pinchart wrote:
> > On Wednesday, 7 November 2018 06:16:47 EET Bing Bu Cao wrote:
> >> On 11/01/2018 08:03 PM, Sakari Ailus wrote:
> >>> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> >
> > [snip]
> >
> >>>> ImgU media topology print:
> >>>>
> >>>> # media-ctl -d /dev/media0 -p
> >>>> Media controller API version 4.19.0
> >>>>
> >>>> Media device information
> >>>> ------------------------
> >>>> driver ipu3-imgu
> >>>> model ipu3-imgu
> >>>> serial
> >>>> bus info PCI:0000:00:05.0
> >>>> hw revision 0x80862015
> >>>> driver version 4.19.0
> >>>>
> >>>> Device topology
> >>>> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> >>>> type V4L2 subdev subtype Unknown flags 0
> >>>> device node name /dev/v4l-subdev0
> >>>> pad0: Sink
> >>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> >>>
> >>> This doesn't seem right. Which formats can be enumerated from the pad?
> >>>
> >>>> crop:(0,0)/1920x1080
> >>>> compose:(0,0)/1920x1080]
> >>>
> >>> Does the compose rectangle affect the scaling on all outputs?
> >>
> >> Sakari, driver use crop and compose targets to help set input-feeder and
> >> BDS output resolutions which are 2 key block of whole imaging pipeline,
> >> not the actual ending output, but they will impact the final output.
> >>
> >>>> <- "ipu3-imgu 0 input":0 []
> >>>
> >>> Are there links that have no useful link configuration? If so, you
> >>> should set them enabled and immutable in the driver.
> >>
> >> The enabled status of input pads is used to get which pipe that user is
> >> trying to enable (ipu3_link_setup()), so it could not been set as
> >> immutable.
> >
> > Each pipe needs an input in order to operate, so from that point of view
> > the input is mandatory. Why can't we make this link immutable, and use
> > the stream state (VIDIOC_STREAMON/VIDIOC_STREAMOFF) to enable/disable the
> > pipes ?
>
> There are only two options (AFAIK) in choosing the firmware, and by
> configuring the links this is better visible to the user: the links the
> state of which can be changed are not immutable. The driver can also obtain
> the explicit pipeline configuration, which makes the implementation more
> simple.
Do you mean that different firmwares are loaded based on link configuration ?
Does it also mean that once we start using the first pipeline the
configuration of the second pipeline can't be changed anymore ? If so, what's
the reason for such a limitation ?
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (15 preceding siblings ...)
2018-11-01 12:03 ` [PATCH v7 00/16] Intel IPU3 ImgU patchset Sakari Ailus
@ 2018-11-14 0:25 ` jacopo mondi
2018-11-14 7:40 ` Sakari Ailus
2018-11-29 14:43 ` Laurent Pinchart
17 siblings, 1 reply; 123+ messages in thread
From: jacopo mondi @ 2018-11-14 0:25 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
[-- Attachment #1: Type: text/plain, Size: 64908 bytes --]
On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> Hi,
>
> This series adds support for the Intel IPU3 (Image Processing Unit)
> ImgU which is essentially a modern memory-to-memory ISP. It implements
> raw Bayer to YUV image format conversion as well as a large number of
> other pixel processing algorithms for improving the image quality.
>
> Meta data formats are defined for image statistics (3A, i.e. automatic
> white balance, exposure and focus, histogram and local area contrast
> enhancement) as well as for the pixel processing algorithm parameters.
> The documentation for these formats is currently not included in the
> patchset but will be added in a future version of this set.
>
> The algorithm parameters need to be considered specific to a given frame
> and typically a large number of these parameters change on frame to frame
> basis. Additionally, the parameters are highly structured (and not a flat
> space of independent configuration primitives). They also reflect the
> data structures used by the firmware and the hardware. On top of that,
> the algorithms require highly specialized user space to make meaningful
> use of them. For these reasons it has been chosen video buffers to pass
> the parameters to the device.
>
> On individual patches:
>
> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> image processors and HW accelerators.
>
> The 3A statistics and other firmware parameter computation related
> functions are implemented in patch 11.
>
> All IPU3 pipeline default settings can be found in patch 10.
>
Seems to me that patch 10 didn't make it to the mailing list, am I
wrong?
I'm pointing it out as the same happened on your v6.
Thanks
j
> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> its own MMU unit, the driver is implemented in patch 6.
>
> Patch 7 uses above driver for DMA mapping operation.
>
> The communication between IPU3 firmware and driver is implemented with circular
> queues in patch 8.
>
> Patch 9 provide some utility functions and manage IPU3 fw download and
> install.
>
> The firmware which is called ipu3-fw.bin can be downloaded from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>
> Firmware ABI is defined in patches 4 and 5.
>
> Patches 12 and 13 are of the same file, the former contains all h/w programming
> related code, the latter implements interface functions for access fw & hw
> capabilities.
>
> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>
> <URL:https://patchwork.kernel.org/patch/9976295/>
>
> Patch 15 represents the top level that glues all of the other components together,
> passing arguments between the components.
>
> Patch 16 is a recent effort to extend v6 for advanced camera features like
> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>
> Link to user space implementation:
>
> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>
> ImgU media topology print:
>
> # media-ctl -d /dev/media0 -p
> Media controller API version 4.19.0
>
> Media device information
> ------------------------
> driver ipu3-imgu
> model ipu3-imgu
> serial
> bus info PCI:0000:00:05.0
> hw revision 0x80862015
> driver version 4.19.0
>
> Device topology
> - entity 1: ipu3-imgu 0 (5 pads, 5 links)
> type V4L2 subdev subtype Unknown flags 0
> device node name /dev/v4l-subdev0
> pad0: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> crop:(0,0)/1920x1080
> compose:(0,0)/1920x1080]
> <- "ipu3-imgu 0 input":0 []
> pad1: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> <- "ipu3-imgu 0 parameters":0 []
> pad2: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 0 output":0 []
> pad3: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 0 viewfinder":0 []
> pad4: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 0 3a stat":0 []
>
> - entity 7: ipu3-imgu 1 (5 pads, 5 links)
> type V4L2 subdev subtype Unknown flags 0
> device node name /dev/v4l-subdev1
> pad0: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown
> crop:(0,0)/1920x1080
> compose:(0,0)/1920x1080]
> <- "ipu3-imgu 1 input":0 []
> pad1: Sink
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> <- "ipu3-imgu 1 parameters":0 []
> pad2: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 1 output":0 []
> pad3: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 1 viewfinder":0 []
> pad4: Source
> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> -> "ipu3-imgu 1 3a stat":0 []
>
> - entity 17: ipu3-imgu 0 input (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video0
> pad0: Source
> -> "ipu3-imgu 0":0 []
>
> - entity 23: ipu3-imgu 0 parameters (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video1
> pad0: Source
> -> "ipu3-imgu 0":1 []
>
> - entity 29: ipu3-imgu 0 output (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video2
> pad0: Sink
> <- "ipu3-imgu 0":2 []
>
> - entity 35: ipu3-imgu 0 viewfinder (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video3
> pad0: Sink
> <- "ipu3-imgu 0":3 []
>
> - entity 41: ipu3-imgu 0 3a stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video4
> pad0: Sink
> <- "ipu3-imgu 0":4 []
>
> - entity 47: ipu3-imgu 1 input (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video5
> pad0: Source
> -> "ipu3-imgu 1":0 []
>
> - entity 53: ipu3-imgu 1 parameters (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video6
> pad0: Source
> -> "ipu3-imgu 1":1 []
>
> - entity 59: ipu3-imgu 1 output (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video7
> pad0: Sink
> <- "ipu3-imgu 1":2 []
>
> - entity 65: ipu3-imgu 1 viewfinder (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video8
> pad0: Sink
> <- "ipu3-imgu 1":3 []
>
> - entity 71: ipu3-imgu 1 3a stat (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video9
> pad0: Sink
> <- "ipu3-imgu 1":4 []
>
>
> v4l2-compliance utility is built with Sakari's patches for meta data
> output support(rebased):
>
> <URL:https://patchwork.linuxtv.org/patch/43370/>
> <URL:https://patchwork.linuxtv.org/patch/43369/>
>
> The test (v4l2-compliance -m 0) passes without error, outputs are appended at
> the end of revision history.
>
> Note:
>
> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be enabled
> prior to the test.
> 2. Stream tests are not performed since it requires pre-configuration for each case.
>
> ===========
> = history =
> ===========
>
> v7 update:
>
> 1. Add driver and uAPI documentation.
>
> Update based on v1 review from Tomasz, Hans, Sokari and Mauro:
> https://patchwork.kernel.org/patch/10465663/
> https://patchwork.kernel.org/patch/10465665/
>
> 2. Add dual pipe support which includes:
> - Extend current IMGU device to contain 2 subdevs and two groups of video nodes.
> - Add a v4l2 ctrl to allow user to specify the mode(video or still) of the pipe.
>
> 3. Kconfig
> - Restrict build for X86 arch to fix build error for ia64/sparc.
> (fatal error: asm/set_memory.h: No such file or directory)
>
> 4. ipu3-abi.h
> - Change __u32 to u32.
> - Use generic __attribute__((aligned(x))) format. (Mauro/Hans)
> - Split abi to 2 patches, one for register defines, enums, the other for structs. (Tomasz)
>
> 5. ipu3-mmu.c
> - Fix ipu3-mmu/dmamap exit functions. (Tomasz)
> (Port from https://chromium-review.googlesource.com/1084522)
> - Use free_page instead of kfree. (Tomasz)
> - document struct ipu3_mmu_info.
> - Fix copyright information.
>
> 6. ipu3-dmamap.c (Tomasz)
> - Update APIs based on v6 review.
> - Replace sizeof(struct page *) with sizeof(*pages).
> - Remove un-needed (WARN_ON(!dev)) inside void *ipu3_dmamap_alloc().
>
> 7. ipu3.c (Tomasz)
> - imgu_video_nodes_init()
> Fix the missing call to ipu3_v4l2_unregister() in the error path of
> imgu_dummybufs_preallocate().
> - imgu_queue_buffers()
> Evaluate loop condition explicitly for code clarity and simplicity.
> FW requires all output buffers to be queued at start, so adjust the order of
> buffer queuing accordingly. (bufix by Tianshu)
> - imgu_isr_threaded()
> Fix interrupt handler return value.
> (Port from https://chromium-review.googlesource.com/1088539)
> - Add back the buf_drain_wq from ("avoid sleep in wait_event condition")'
> (Port from https://chromium-review.googlesource.com/875420)
>
> 8. ipu3-v4l2.c
> - ipu3_v4l2_register(). (Tomasz)
> Split media initialization and registration, also change media device
> register/un-register order.
>
> - Fix v4l2-compliance fail on sub-devicef for VIDIOC_CREATE_BUFS and
> VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT.
>
> 9. ipu3-css.c, ipu3-css.h, ipu3-css-fw.h, ipu3-abi.h
> - Convert macros in structs to enums. (Tomasz)
>
> 10. ipu3-css-pool.c, ipu3-css-pool.h, ipu3.c
> - Document the structs. (Hans/Maruo)
>
> 11. ipu3-css-params.c
> - Fixup for noise reduction parameters processing. (bug fixing)
>
> version 6:
>
> - intel-ipu3.h uAPI
> Move out the definitions not used by user space. (suggested by Sakari)
> - ipu3-abi.h, ipu3-css-fw.h
> Clean up the header files.
> Remove enum type from ABI structs.
> - ipu3-css.h and ipu3-css.c
> Disable DVS support and remove related code.
> - ipu3-v4l2.c
> Fixes of v4l2_compliance test fails on ImgU sub-dev.
> - ipu3-css-params.c
> Refactor awb/awb_fr/af_ops_calc() functions. (Sakari)
> - Build mmu and dmamap driver as part of ImgU ko module; (Sakari)
> - Add "ipu3-imgu" prefix to media entity names; (Sakari)
> - Fix indentation and white space; (Sakari)
> - Rebase to kernel v4.16;
> - Use SPDX license identifiers in all drivers; (Sakari)
> - Internal fix and performance improvements such as:
> Stop fw gracefully during stream off.
> Enable irq only after start streaming to avoid unexpected interrupt.
> Use spinlock to protect IPU3_CSS_QUEUES access.
> Return NULL when dequeuing buffer before streaming.
>
> TODOs:
> - Documentation on ImgU driver programming interface to configure and enable
> ISP HW, which will include details on complete V4L2 Kernel driver interface
> and IO-Control parameters, except for the ISP internal algorithm and its
> parameters (which is Intel proprietary IP).
>
> version 5:
> - ipu3-css-pool.c/ipu3_css_pool_check().
> add handling of the framenum wrap around case in ipu3_css_pool_check().
> - ipu3.c, ipu3-v4l2.c, ipu3.h
> merge struct ipu3_mem2mem2_device into imgu_device and update the code
> accordingly. (Suggested by Sakari)
> - ipu3-mmu.c driver:
> use __get_free_page() for page-aligned allocations (Tomasz).
> optimize tlb invalidation by calling them at the end of map/unmap. (Tomasz).
> remove dependency on iommu. (Sakari)
> introduce few new functions from iommu.c.
> - ipu3-dmamap.c driver
> call mmu directly without IOMMU_SUPPORT (Sakari)
> update dmamap APIs. (Suggested by Tomasz)
> - ipu3_v4l2.c
> move g/s_selection callback to V4l2 sub-device (Sakari)
> remove colon from ImgU sub-device name. (Sakari)
> - ipu3-css-params.c
> fix indentation, 0-day scan warnings etc.
> - ipu3-css.c
> fix warning about NULL comparison. (Sakari)
> - intel-ipu3.h:
> remove redundant IPU3_ALIGN attribute (Sakari).
> fix up un-needed fields in struct ipu3_uapi_params (Sakari)
> re-order this to be 2nd in the patch set.
> - Makefile: remove Copyright header. (Sakari)
> - Internal fix:
> optimize shot-to-shot performance.
> update default white balance gains defined in ipu3-tables.c
>
> TODOs:
>
> - Documentation on ImgU driver programming interface to configure and enable
> ISP HW, which will include details on complete V4L2 Kernel driver interface
> and IO-Control parameters, except for the ISP internal algorithm and its
> parameters (which is Intel proprietary IP).
>
> - Review ipu3_css_pool_* group APIs usage.
>
> version 4:
> - Used V4L2_BUF_TYPE_META_OUTPUT for:
> - V4L2_META_FMT_IPU3_STAT_PARAMS
>
> - Used V4L2_BUF_TYPE_META_CAPTURE for:
> - V4L2_META_FMT_IPU3_STAT_3A
> - V4L2_META_FMT_IPU3_STAT_DVS
> - V4L2_META_FMT_IPU3_STAT_LACE
> - Supported v4l2 MPLANE format on video nodes.
> - ipu3-dmamap.c: Removed dma ops and dependencies on IOMMU_DMA lib.
> - ipu3-mmu.c: Restructured the driver.
> - intel-ipu3.h: Added __padding qualifier for uapi definitions.
> - Internal fix: power and performance related issues.
> - Fixed v4l2-compliance test.
> - Fixed build failure for x86 with 32bit config.
>
> version 3:
> - ipu3-mmu.c and ipu3-dmamap.c:
> Tomasz Figa reworked both drivers and updated related files.
> - ipu2-abi.h:
> update imgu_abi_binary_info ABI to support latest ipu3-fw.bin.
> use __packed qualifier on structs suggested by Sakari Ailus.
> - ipu3-css-fw.c/ipu3-css-fw.h: following fix were suggested by Tomasz Figa:
> remove pointer type in firmware blob structs.
> fix binary_header array in struct imgu_fw_header.
> fix calling ipu3_css_fw_show_binary() before proper checking.
> fix logic error for valid length checking of blob name.
> - ipu3-css-params.c/ipu3_css_scaler_get_exp():
> use lib helper suggested by Andy Shevchenko.
> - ipu3-v4l2.c/ipu3_videoc_querycap():
> fill device_caps fix suggested by Hans Verkuil.
> add VB2_DMABUF suggested by Tomasz Figa.
> - ipu3-css.c: increase IMGU freq from 300MHZ to 450MHZ (internal fix)
> - ipu3.c: use vb2_dma_sg_memop for the time being(internal fix).
>
> version 2:
> This version cherry-picked firmware ABI change and other
> fix in order to bring the code up-to-date with our internal release.
>
> I will go over the review comments in v1 and address them in v3 and
> future update.
>
> version 1:
> - Initial submission
>
> --------------------------------------------------------------------------------
>
> v4l2-compliance test output:
>
> ./v4l2-compliance -m 0
>
> v4l2-compliance SHA: 7aa151889ffe89b1cd94a8198b0caba1a8c70398, 64 bits
>
> Compliance test for device /dev/media0:
>
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
>
> Required ioctls:
> test MEDIA_IOC_DEVICE_INFO: OK
>
> Allow for multiple opens:
> test second /dev/media0 open: OK
> test MEDIA_IOC_DEVICE_INFO: OK
> test for unlimited opens: OK
>
> Media Controller ioctls:
> test MEDIA_IOC_G_TOPOLOGY: OK
> Entities: 12 Interfaces: 12 Pads: 20 Links: 22
> test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
> test MEDIA_IOC_SETUP_LINK: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/v4l-subdev0:
>
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x0300000d
> Type : V4L Sub-Device
> Entity Info:
> ID : 0x00000001 (1)
> Name : ipu3-imgu 0
> Function : Video Statistics
> Pad 0x01000002 : 0: Sink
> Link 0x02000015: from remote pad 0x1000012 of entity 'ipu3-imgu 0 input': Data, Enabled
> Pad 0x01000003 : 1: Sink
> Link 0x0200001b: from remote pad 0x1000018 of entity 'ipu3-imgu 0 parameters': Data
> Pad 0x01000004 : 2: Source
> Link 0x02000021: to remote pad 0x100001e of entity 'ipu3-imgu 0 output': Data, Enabled
> Pad 0x01000005 : 3: Source
> Link 0x02000027: to remote pad 0x1000024 of entity 'ipu3-imgu 0 viewfinder': Data, Enabled
> Pad 0x01000006 : 4: Source
> Link 0x0200002d: to remote pad 0x100002a of entity 'ipu3-imgu 0 3a stat': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
>
> Allow for multiple opens:
> test second /dev/v4l-subdev0 open: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Sub-Device ioctls (Sink Pad 0):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Sink Pad 1):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Source Pad 2):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Source Pad 3):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Source Pad 4):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
> test VIDIOC_QUERYCTRL: OK
> test VIDIOC_G/S_CTRL: OK
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 1 Private Controls: 1
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK (Not Supported)
> test VIDIOC_TRY_FMT: OK (Not Supported)
> test VIDIOC_S_FMT: OK (Not Supported)
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
> test VIDIOC_EXPBUF: OK (Not Supported)
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/v4l-subdev1:
>
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x0300000f
> Type : V4L Sub-Device
> Entity Info:
> ID : 0x00000007 (7)
> Name : ipu3-imgu 1
> Function : Video Statistics
> Pad 0x01000008 : 0: Sink
> Link 0x02000033: from remote pad 0x1000030 of entity 'ipu3-imgu 1 input': Data, Enabled
> Pad 0x01000009 : 1: Sink
> Link 0x02000039: from remote pad 0x1000036 of entity 'ipu3-imgu 1 parameters': Data
> Pad 0x0100000a : 2: Source
> Link 0x0200003f: to remote pad 0x100003c of entity 'ipu3-imgu 1 output': Data, Enabled
> Pad 0x0100000b : 3: Source
> Link 0x02000045: to remote pad 0x1000042 of entity 'ipu3-imgu 1 viewfinder': Data, Enabled
> Pad 0x0100000c : 4: Source
> Link 0x0200004b: to remote pad 0x1000048 of entity 'ipu3-imgu 1 3a stat': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
>
> Allow for multiple opens:
> test second /dev/v4l-subdev1 open: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Sub-Device ioctls (Sink Pad 0):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Sink Pad 1):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Source Pad 2):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Source Pad 3):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Sub-Device ioctls (Source Pad 4):
> test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Try VIDIOC_SUBDEV_G/S_FMT: OK
> test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK (Not Supported)
> test Active VIDIOC_SUBDEV_G/S_FMT: OK
> test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
> test VIDIOC_QUERYCTRL: OK
> test VIDIOC_G/S_CTRL: OK
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 1 Private Controls: 1
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK (Not Supported)
> test VIDIOC_TRY_FMT: OK (Not Supported)
> test VIDIOC_S_FMT: OK (Not Supported)
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
> test VIDIOC_EXPBUF: OK (Not Supported)
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video0:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:input
> Driver version : 4.19.0
> Capabilities : 0x84202000
> Video Output Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04202000
> Video Output Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000013
> Type : V4L Video
> Entity Info:
> ID : 0x00000011 (17)
> Name : ipu3-imgu 0 input
> Function : V4L2 I/O
> Pad 0x01000012 : 0: Source
> Link 0x02000015: to remote pad 0x1000002 of entity 'ipu3-imgu 0': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video0 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 1 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Output 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Output 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls (Output 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Output 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video1:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:parameters
> Driver version : 4.19.0
> Capabilities : 0x8c200000
> Metadata Output
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x0c200000
> Metadata Output
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000019
> Type : V4L Video
> Entity Info:
> ID : 0x00000017 (23)
> Name : ipu3-imgu 0 parameters
> Function : V4L2 I/O
> Pad 0x01000018 : 0: Source
> Link 0x0200001b: to remote pad 0x1000003 of entity 'ipu3-imgu 0': Data
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video1 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video2:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:output
> Driver version : 4.19.0
> Capabilities : 0x84201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x0300001f
> Type : V4L Video
> Entity Info:
> ID : 0x0000001d (29)
> Name : ipu3-imgu 0 output
> Function : V4L2 I/O
> Pad 0x0100001e : 0: Sink
> Link 0x02000021: from remote pad 0x1000004 of entity 'ipu3-imgu 0': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video2 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 1 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Input 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Input 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls (Input 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Input 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video3:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:viewfinder
> Driver version : 4.19.0
> Capabilities : 0x84201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000025
> Type : V4L Video
> Entity Info:
> ID : 0x00000023 (35)
> Name : ipu3-imgu 0 viewfinder
> Function : V4L2 I/O
> Pad 0x01000024 : 0: Sink
> Link 0x02000027: from remote pad 0x1000005 of entity 'ipu3-imgu 0': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video3 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 1 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Input 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Input 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls (Input 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Input 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video4:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:3a stat
> Driver version : 4.19.0
> Capabilities : 0x84a00000
> Metadata Capture
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04a00000
> Metadata Capture
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x0300002b
> Type : V4L Video
> Entity Info:
> ID : 0x00000029 (41)
> Name : ipu3-imgu 0 3a stat
> Function : V4L2 I/O
> Pad 0x0100002a : 0: Sink
> Link 0x0200002d: from remote pad 0x1000006 of entity 'ipu3-imgu 0': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video4 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video5:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:input
> Driver version : 4.19.0
> Capabilities : 0x84202000
> Video Output Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04202000
> Video Output Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000031
> Type : V4L Video
> Entity Info:
> ID : 0x0000002f (47)
> Name : ipu3-imgu 1 input
> Function : V4L2 I/O
> Pad 0x01000030 : 0: Source
> Link 0x02000033: to remote pad 0x1000008 of entity 'ipu3-imgu 1': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video5 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 1 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Output 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Output 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls (Output 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Output 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video6:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:parameters
> Driver version : 4.19.0
> Capabilities : 0x8c200000
> Metadata Output
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x0c200000
> Metadata Output
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000037
> Type : V4L Video
> Entity Info:
> ID : 0x00000035 (53)
> Name : ipu3-imgu 1 parameters
> Function : V4L2 I/O
> Pad 0x01000036 : 0: Source
> Link 0x02000039: to remote pad 0x1000009 of entity 'ipu3-imgu 1': Data
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video6 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video7:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:output
> Driver version : 4.19.0
> Capabilities : 0x84201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x0300003d
> Type : V4L Video
> Entity Info:
> ID : 0x0000003b (59)
> Name : ipu3-imgu 1 output
> Function : V4L2 I/O
> Pad 0x0100003c : 0: Sink
> Link 0x0200003f: from remote pad 0x100000a of entity 'ipu3-imgu 1': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video7 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 1 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Input 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Input 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls (Input 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Input 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video8:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:viewfinder
> Driver version : 4.19.0
> Capabilities : 0x84201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04201000
> Video Capture Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000043
> Type : V4L Video
> Entity Info:
> ID : 0x00000041 (65)
> Name : ipu3-imgu 1 viewfinder
> Function : V4L2 I/O
> Pad 0x01000042 : 0: Sink
> Link 0x02000045: from remote pad 0x100000b of entity 'ipu3-imgu 1': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video8 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 1 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Input 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls (Input 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK
>
> Codec ioctls (Input 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Input 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> --------------------------------------------------------------------------------
> Compliance test for device /dev/video9:
>
> Driver Info:
> Driver name : ipu3-imgu
> Card type : ipu3-imgu
> Bus info : PCI:3a stat
> Driver version : 4.19.0
> Capabilities : 0x84a00000
> Metadata Capture
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04a00000
> Metadata Capture
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : ipu3-imgu
> Model : ipu3-imgu
> Serial :
> Bus info : PCI:0000:00:05.0
> Media version : 4.19.0
> Hardware revision: 0x80862015 (2156273685)
> Driver version : 4.19.0
> Interface Info:
> ID : 0x03000049
> Type : V4L Video
> Entity Info:
> ID : 0x00000047 (71)
> Name : ipu3-imgu 1 3a stat
> Function : V4L2 I/O
> Pad 0x01000048 : 0: Sink
> Link 0x0200004b: from remote pad 0x100000c of entity 'ipu3-imgu 1': Data, Enabled
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video9 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> test VIDIOC_QUERYCTRL: OK (Not Supported)
> test VIDIOC_G/S_CTRL: OK (Not Supported)
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 0 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
>
> Total: 597, Succeeded: 597, Failed: 0, Warnings: 0
>
> --------------------------------------------------------------------------------
>
> Thanks,
>
> Yong
>
> Cao,Bing Bu (1):
> intel-ipu3: add dual pipe support
>
> Rajmohan Mani (1):
> doc-rst: Add Intel IPU3 documentation
>
> Tomasz Figa (2):
> intel-ipu3: mmu: Implement driver
> intel-ipu3: Implement DMA mapping functions
>
> Yong Zhi (12):
> v4l: Add Intel IPU3 meta buffer formats
> v4l: Add Intel IPU3 meta data uAPI
> intel-ipu3: abi: Add register definitions and enum
> intel-ipu3: abi: Add structs
> intel-ipu3: css: Add dma buff pool utility functions
> intel-ipu3: css: Add support for firmware management
> intel-ipu3: css: Add static settings for image pipeline
> intel-ipu3: css: Compute and program ccs
> intel-ipu3: css: Initialize css hardware
> intel-ipu3: Add css pipeline programming
> intel-ipu3: Add v4l2 driver based on media framework
> intel-ipu3: Add imgu top level pci device driver
>
> Documentation/media/uapi/v4l/meta-formats.rst | 1 +
> .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 181 +
> Documentation/media/v4l-drivers/index.rst | 1 +
> Documentation/media/v4l-drivers/ipu3.rst | 326 +
> drivers/media/pci/intel/ipu3/Kconfig | 16 +
> drivers/media/pci/intel/ipu3/Makefile | 12 +
> drivers/media/pci/intel/ipu3/ipu3-abi.h | 2011 ++++
> drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 265 +
> drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 188 +
> drivers/media/pci/intel/ipu3/ipu3-css-params.c | 2936 ++++++
> drivers/media/pci/intel/ipu3/ipu3-css-params.h | 28 +
> drivers/media/pci/intel/ipu3/ipu3-css-pool.c | 136 +
> drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 56 +
> drivers/media/pci/intel/ipu3/ipu3-css.c | 2407 +++++
> drivers/media/pci/intel/ipu3/ipu3-css.h | 215 +
> drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 270 +
> drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 22 +
> drivers/media/pci/intel/ipu3/ipu3-mmu.c | 560 ++
> drivers/media/pci/intel/ipu3/ipu3-mmu.h | 35 +
> drivers/media/pci/intel/ipu3/ipu3-tables.c | 9609 ++++++++++++++++++++
> drivers/media/pci/intel/ipu3/ipu3-tables.h | 66 +
> drivers/media/pci/intel/ipu3/ipu3-v4l2.c | 1436 +++
> drivers/media/pci/intel/ipu3/ipu3.c | 830 ++
> drivers/media/pci/intel/ipu3/ipu3.h | 169 +
> drivers/media/v4l2-core/v4l2-ioctl.c | 2 +
> include/uapi/linux/intel-ipu3.h | 2827 ++++++
> include/uapi/linux/videodev2.h | 4 +
> 27 files changed, 24609 insertions(+)
> create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> create mode 100644 Documentation/media/v4l-drivers/ipu3.rst
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-params.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-tables.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-v4l2.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
> create mode 100644 include/uapi/linux/intel-ipu3.h
>
> --
> 2.7.4
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-14 0:25 ` jacopo mondi
@ 2018-11-14 7:40 ` Sakari Ailus
2018-11-18 0:12 ` jacopo mondi
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2018-11-14 7:40 UTC (permalink / raw)
To: jacopo mondi
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
Hi Jacopo,
On Wed, Nov 14, 2018 at 01:25:11AM +0100, jacopo mondi wrote:
> On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> > Hi,
> >
> > This series adds support for the Intel IPU3 (Image Processing Unit)
> > ImgU which is essentially a modern memory-to-memory ISP. It implements
> > raw Bayer to YUV image format conversion as well as a large number of
> > other pixel processing algorithms for improving the image quality.
> >
> > Meta data formats are defined for image statistics (3A, i.e. automatic
> > white balance, exposure and focus, histogram and local area contrast
> > enhancement) as well as for the pixel processing algorithm parameters.
> > The documentation for these formats is currently not included in the
> > patchset but will be added in a future version of this set.
> >
> > The algorithm parameters need to be considered specific to a given frame
> > and typically a large number of these parameters change on frame to frame
> > basis. Additionally, the parameters are highly structured (and not a flat
> > space of independent configuration primitives). They also reflect the
> > data structures used by the firmware and the hardware. On top of that,
> > the algorithms require highly specialized user space to make meaningful
> > use of them. For these reasons it has been chosen video buffers to pass
> > the parameters to the device.
> >
> > On individual patches:
> >
> > The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> > image processors and HW accelerators.
> >
> > The 3A statistics and other firmware parameter computation related
> > functions are implemented in patch 11.
> >
> > All IPU3 pipeline default settings can be found in patch 10.
> >
>
> Seems to me that patch 10 didn't make it to the mailing list, am I
> wrong?
>
> I'm pointing it out as the same happened on your v6.
Thanks for pointing this out. I've uploaded the entire set here:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=ipu3-v7>
including the 10th patch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/commit/?h=ipu3-v7&id=41e2f0d114dbc195efed079202d22748ddedbe83>
It's too big to get through the list server. :-(
Luckily, it's mostly tables so there's not much to review there. Default
settings, effectively.
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-14 7:40 ` Sakari Ailus
@ 2018-11-18 0:12 ` jacopo mondi
0 siblings, 0 replies; 123+ messages in thread
From: jacopo mondi @ 2018-11-18 0:12 UTC (permalink / raw)
To: Sakari Ailus
Cc: Yong Zhi, linux-media, tfiga, mchehab, hans.verkuil,
laurent.pinchart, rajmohan.mani, jian.xu.zheng, jerry.w.hu,
tuukka.toivonen, tian.shu.qiu, bingbu.cao
[-- Attachment #1: Type: text/plain, Size: 2571 bytes --]
Hi Sakari,
On Wed, Nov 14, 2018 at 09:40:50AM +0200, Sakari Ailus wrote:
> Hi Jacopo,
>
> On Wed, Nov 14, 2018 at 01:25:11AM +0100, jacopo mondi wrote:
> > On Mon, Oct 29, 2018 at 03:22:54PM -0700, Yong Zhi wrote:
> > > Hi,
> > >
> > > This series adds support for the Intel IPU3 (Image Processing Unit)
> > > ImgU which is essentially a modern memory-to-memory ISP. It implements
> > > raw Bayer to YUV image format conversion as well as a large number of
> > > other pixel processing algorithms for improving the image quality.
> > >
> > > Meta data formats are defined for image statistics (3A, i.e. automatic
> > > white balance, exposure and focus, histogram and local area contrast
> > > enhancement) as well as for the pixel processing algorithm parameters.
> > > The documentation for these formats is currently not included in the
> > > patchset but will be added in a future version of this set.
> > >
> > > The algorithm parameters need to be considered specific to a given frame
> > > and typically a large number of these parameters change on frame to frame
> > > basis. Additionally, the parameters are highly structured (and not a flat
> > > space of independent configuration primitives). They also reflect the
> > > data structures used by the firmware and the hardware. On top of that,
> > > the algorithms require highly specialized user space to make meaningful
> > > use of them. For these reasons it has been chosen video buffers to pass
> > > the parameters to the device.
> > >
> > > On individual patches:
> > >
> > > The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> > > image processors and HW accelerators.
> > >
> > > The 3A statistics and other firmware parameter computation related
> > > functions are implemented in patch 11.
> > >
> > > All IPU3 pipeline default settings can be found in patch 10.
> > >
> >
> > Seems to me that patch 10 didn't make it to the mailing list, am I
> > wrong?
> >
> > I'm pointing it out as the same happened on your v6.
>
> Thanks for pointing this out. I've uploaded the entire set here:
>
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=ipu3-v7>
>
> including the 10th patch:
>
> <URL:https://git.linuxtv.org/sailus/media_tree.git/commit/?h=ipu3-v7&id=41e2f0d114dbc195efed079202d22748ddedbe83>
>
> It's too big to get through the list server. :-(
>
> Luckily, it's mostly tables so there's not much to review there. Default
> settings, effectively.
Thanks, I've now received all patches.
Thanks
j
>
> --
> Regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-10-29 22:22 [PATCH v7 00/16] Intel IPU3 ImgU patchset Yong Zhi
` (16 preceding siblings ...)
2018-11-14 0:25 ` jacopo mondi
@ 2018-11-29 14:43 ` Laurent Pinchart
2018-11-29 19:51 ` Tomasz Figa
17 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-11-29 14:43 UTC (permalink / raw)
To: Yong Zhi
Cc: linux-media, sakari.ailus, tfiga, mchehab, hans.verkuil,
rajmohan.mani, jian.xu.zheng, jerry.w.hu, tuukka.toivonen,
tian.shu.qiu, bingbu.cao
Hello Yong,
On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
> Hi,
>
> This series adds support for the Intel IPU3 (Image Processing Unit)
> ImgU which is essentially a modern memory-to-memory ISP. It implements
> raw Bayer to YUV image format conversion as well as a large number of
> other pixel processing algorithms for improving the image quality.
>
> Meta data formats are defined for image statistics (3A, i.e. automatic
> white balance, exposure and focus, histogram and local area contrast
> enhancement) as well as for the pixel processing algorithm parameters.
> The documentation for these formats is currently not included in the
> patchset but will be added in a future version of this set.
>
> The algorithm parameters need to be considered specific to a given frame
> and typically a large number of these parameters change on frame to frame
> basis. Additionally, the parameters are highly structured (and not a flat
> space of independent configuration primitives). They also reflect the
> data structures used by the firmware and the hardware. On top of that,
> the algorithms require highly specialized user space to make meaningful
> use of them. For these reasons it has been chosen video buffers to pass
> the parameters to the device.
>
> On individual patches:
>
> The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> image processors and HW accelerators.
>
> The 3A statistics and other firmware parameter computation related
> functions are implemented in patch 11.
>
> All IPU3 pipeline default settings can be found in patch 10.
>
> To access DDR via ImgU's own memory space, IPU3 is also equipped with
> its own MMU unit, the driver is implemented in patch 6.
>
> Patch 7 uses above driver for DMA mapping operation.
>
> The communication between IPU3 firmware and driver is implemented with
> circular queues in patch 8.
>
> Patch 9 provide some utility functions and manage IPU3 fw download and
> install.
>
> The firmware which is called ipu3-fw.bin can be downloaded from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
>
> Firmware ABI is defined in patches 4 and 5.
>
> Patches 12 and 13 are of the same file, the former contains all h/w
> programming related code, the latter implements interface functions for
> access fw & hw capabilities.
>
> Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
>
> <URL:https://patchwork.kernel.org/patch/9976295/>
>
> Patch 15 represents the top level that glues all of the other components
> together, passing arguments between the components.
>
> Patch 16 is a recent effort to extend v6 for advanced camera features like
> Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
>
> Link to user space implementation:
>
> git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
>
> ImgU media topology print:
[snip]
> v4l2-compliance utility is built with Sakari's patches for meta data
> output support(rebased):
>
> <URL:https://patchwork.linuxtv.org/patch/43370/>
> <URL:https://patchwork.linuxtv.org/patch/43369/>
>
> The test (v4l2-compliance -m 0) passes without error, outputs are appended
> at the end of revision history.
>
> Note:
>
> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be enabled
> prior to the test.
> 2. Stream tests are not performed since it requires pre-configuration for
> each case.
And that's a bit of an issue. I've tested the driver with a small script based
on media-ctl to configure links and yavta to interface with the video nodes,
and got the following oops:
[ 136.927788] divide error: 0000 [#1] PREEMPT SMP PTI
[ 136.927801] CPU: 2 PID: 2069 Comm: yavta Not tainted 4.20.0-rc1+ #9
[ 136.927806] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
[ 136.927820] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu]
[ 136.927825] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd
ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 <f7> fb
0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2
[ 136.927830] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202
[ 136.927835] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
ffff9af2c3e353c0
[ 136.927839] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
ffff9af2c3e353c0
[ 136.927843] RBP: 0000000000000001 R08: 0000000000000000 R09:
ffff9af2c0b83880
[ 136.927846] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
00000000000003a0
[ 136.927849] R13: 0000000000025a0a R14: 0000000000000000 R15:
0000000000000000
[ 136.927854] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000) knlGS:
0000000000000000
[ 136.927858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 136.927862] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
00000000003606e0
[ 136.927865] Call Trace:
[ 136.927884] ? __accumulate_pelt_segments+0x29/0x3a
[ 136.927892] ? __switch_to_asm+0x40/0x70
[ 136.927899] ? alloc_vmap_area+0x78/0x2f6
[ 136.927903] ? __switch_to_asm+0x40/0x70
[ 136.927907] ? __switch_to_asm+0x34/0x70
[ 136.927911] ? __switch_to_asm+0x40/0x70
[ 136.927915] ? __switch_to_asm+0x34/0x70
[ 136.927923] ? __inc_numa_state+0x28/0x70
[ 136.927929] ? preempt_latency_start+0x1e/0x3d
[ 136.927936] ? get_page_from_freelist+0x821/0xb62
[ 136.927943] ? slab_pre_alloc_hook+0x12/0x3b
[ 136.927948] ? kmem_cache_alloc_node_trace+0xf6/0x108
[ 136.927954] ? alloc_vmap_area+0x78/0x2f6
[ 136.927965] ipu3_css_cfg_acc+0xa0/0x1b5f [ipu3_imgu]
[ 136.927981] ipu3_css_set_parameters+0x286/0x6e7 [ipu3_imgu]
[ 136.927995] ipu3_css_start_streaming+0x1230/0x130a [ipu3_imgu]
[ 136.928010] imgu_s_stream+0x104/0x2f7 [ipu3_imgu]
[ 136.928022] ipu3_vb2_start_streaming+0x168/0x1bd [ipu3_imgu]
[ 136.928034] vb2_start_streaming+0x6c/0xf2 [videobuf2_common]
[ 136.928044] vb2_core_streamon+0xcf/0x109 [videobuf2_common]
[ 136.928061] __video_do_ioctl+0x239/0x388 [videodev]
[ 136.928081] video_usercopy+0x25d/0x47a [videodev]
[ 136.928097] ? copy_overflow+0x14/0x14 [videodev]
[ 136.928115] v4l2_ioctl+0x4d/0x58 [videodev]
[ 136.928123] vfs_ioctl+0x1b/0x28
[ 136.928130] do_vfs_ioctl+0x4de/0x566
[ 136.928139] ksys_ioctl+0x50/0x70
[ 136.928146] __x64_sys_ioctl+0x16/0x19
[ 136.928152] do_syscall_64+0x4d/0x5a
[ 136.928158] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 136.928164] RIP: 0033:0x7f1ec9a84f47
[ 136.928169] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d
01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
[ 136.928173] RSP: 002b:00007ffe279e6188 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 136.928178] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
00007f1ec9a84f47
[ 136.928181] RDX: 00007ffe279e6194 RSI: 0000000040045612 RDI:
0000000000000003
[ 136.928184] RBP: 0000000000000000 R08: 00007f1ec776d000 R09:
0000000000000000
[ 136.928188] R10: 0000000000000020 R11: 0000000000000246 R12:
00007ffe279e6360
[ 136.928191] R13: 0000000000000004 R14: 00007ffe279e6360 R15:
00007ffe279e8826
[ 136.928198] Modules linked in: ccm zram arc4 iwlmvm mac80211 intel_rapl
x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi cfg80211 hid_multitouch
ipu3_imgu ipu3_cio2 8250_dw videobuf2_dma_sg videobuf2_memops videobuf2_v4l2
processor_thermal_device intel_soc_dts_iosf videobuf2_common ov5670 ov13858
dw9714 v4l2_fwnode v4l2_common videodev media at24 cros_ec_lpcs cros_ec_core
int3403_thermal int340x_thermal_zone int3400_thermal acpi_thermal_rel
chromeos_pstore mac_hid autofs4 usbhid mmc_block hid_generic i915 sdhci_pci
video cqhci i2c_algo_bit sdhci drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops drm drm_panel_orientation_quirks i2c_hid hid
[ 136.928273] ---[ end trace 4ec6c2ce09e06d9d ]---
[ 136.928288] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu]
[ 136.928293] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd
ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 <f7> fb
0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2
[ 136.928297] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202
[ 136.928302] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
ffff9af2c3e353c0
[ 136.928307] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
ffff9af2c3e353c0
[ 136.928311] RBP: 0000000000000001 R08: 0000000000000000 R09:
ffff9af2c0b83880
[ 136.928320] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
00000000000003a0
[ 136.928324] R13: 0000000000025a0a R14: 0000000000000000 R15:
0000000000000000
[ 136.928330] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000) knlGS:
0000000000000000
[ 136.928349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 136.928364] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
00000000003606e0
The script can be found at https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000040.html.
I may be doing something wrong (and I probably am), but in any case, the
driver shouldn't crash. Could you please have a look ?
Please note that, as I couldn't get the IMGU to process frames, the part of
the script that converts output files to PPM is untested and thus probably
incorrect.
[snip]
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-29 14:43 ` Laurent Pinchart
@ 2018-11-29 19:51 ` Tomasz Figa
2018-11-29 22:54 ` Laurent Pinchart
0 siblings, 1 reply; 123+ messages in thread
From: Tomasz Figa @ 2018-11-29 19:51 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Yong Zhi, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Mani, Rajmohan, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu,
Cao Bing Bu
On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hello Yong,
>
> On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
> > Hi,
> >
> > This series adds support for the Intel IPU3 (Image Processing Unit)
> > ImgU which is essentially a modern memory-to-memory ISP. It implements
> > raw Bayer to YUV image format conversion as well as a large number of
> > other pixel processing algorithms for improving the image quality.
> >
> > Meta data formats are defined for image statistics (3A, i.e. automatic
> > white balance, exposure and focus, histogram and local area contrast
> > enhancement) as well as for the pixel processing algorithm parameters.
> > The documentation for these formats is currently not included in the
> > patchset but will be added in a future version of this set.
> >
> > The algorithm parameters need to be considered specific to a given frame
> > and typically a large number of these parameters change on frame to frame
> > basis. Additionally, the parameters are highly structured (and not a flat
> > space of independent configuration primitives). They also reflect the
> > data structures used by the firmware and the hardware. On top of that,
> > the algorithms require highly specialized user space to make meaningful
> > use of them. For these reasons it has been chosen video buffers to pass
> > the parameters to the device.
> >
> > On individual patches:
> >
> > The heart of ImgU is the CSS, or Camera Subsystem, which contains the
> > image processors and HW accelerators.
> >
> > The 3A statistics and other firmware parameter computation related
> > functions are implemented in patch 11.
> >
> > All IPU3 pipeline default settings can be found in patch 10.
> >
> > To access DDR via ImgU's own memory space, IPU3 is also equipped with
> > its own MMU unit, the driver is implemented in patch 6.
> >
> > Patch 7 uses above driver for DMA mapping operation.
> >
> > The communication between IPU3 firmware and driver is implemented with
> > circular queues in patch 8.
> >
> > Patch 9 provide some utility functions and manage IPU3 fw download and
> > install.
> >
> > The firmware which is called ipu3-fw.bin can be downloaded from:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> > (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f)
> >
> > Firmware ABI is defined in patches 4 and 5.
> >
> > Patches 12 and 13 are of the same file, the former contains all h/w
> > programming related code, the latter implements interface functions for
> > access fw & hw capabilities.
> >
> > Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work:
> >
> > <URL:https://patchwork.kernel.org/patch/9976295/>
> >
> > Patch 15 represents the top level that glues all of the other components
> > together, passing arguments between the components.
> >
> > Patch 16 is a recent effort to extend v6 for advanced camera features like
> > Continuous View Finder (CVF) and Snapshot During Video(SDV) support.
> >
> > Link to user space implementation:
> >
> > git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera
> >
> > ImgU media topology print:
>
> [snip]
>
> > v4l2-compliance utility is built with Sakari's patches for meta data
> > output support(rebased):
> >
> > <URL:https://patchwork.linuxtv.org/patch/43370/>
> > <URL:https://patchwork.linuxtv.org/patch/43369/>
> >
> > The test (v4l2-compliance -m 0) passes without error, outputs are appended
> > at the end of revision history.
> >
> > Note:
> >
> > 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be enabled
> > prior to the test.
> > 2. Stream tests are not performed since it requires pre-configuration for
> > each case.
>
> And that's a bit of an issue. I've tested the driver with a small script based
> on media-ctl to configure links and yavta to interface with the video nodes,
> and got the following oops:
>
> [ 136.927788] divide error: 0000 [#1] PREEMPT SMP PTI
> [ 136.927801] CPU: 2 PID: 2069 Comm: yavta Not tainted 4.20.0-rc1+ #9
> [ 136.927806] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 136.927820] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu]
> [ 136.927825] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd
> ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 <f7> fb
> 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2
> [ 136.927830] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202
> [ 136.927835] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> ffff9af2c3e353c0
> [ 136.927839] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> ffff9af2c3e353c0
> [ 136.927843] RBP: 0000000000000001 R08: 0000000000000000 R09:
> ffff9af2c0b83880
> [ 136.927846] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> 00000000000003a0
> [ 136.927849] R13: 0000000000025a0a R14: 0000000000000000 R15:
> 0000000000000000
> [ 136.927854] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000) knlGS:
> 0000000000000000
> [ 136.927858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 136.927862] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> 00000000003606e0
> [ 136.927865] Call Trace:
> [ 136.927884] ? __accumulate_pelt_segments+0x29/0x3a
> [ 136.927892] ? __switch_to_asm+0x40/0x70
> [ 136.927899] ? alloc_vmap_area+0x78/0x2f6
> [ 136.927903] ? __switch_to_asm+0x40/0x70
> [ 136.927907] ? __switch_to_asm+0x34/0x70
> [ 136.927911] ? __switch_to_asm+0x40/0x70
> [ 136.927915] ? __switch_to_asm+0x34/0x70
> [ 136.927923] ? __inc_numa_state+0x28/0x70
> [ 136.927929] ? preempt_latency_start+0x1e/0x3d
> [ 136.927936] ? get_page_from_freelist+0x821/0xb62
> [ 136.927943] ? slab_pre_alloc_hook+0x12/0x3b
> [ 136.927948] ? kmem_cache_alloc_node_trace+0xf6/0x108
> [ 136.927954] ? alloc_vmap_area+0x78/0x2f6
Is it just me or the backtrace above doesn't seem to make sense? I
don't see any allocations inside ipu3_css_cfg_acc().
> [ 136.927965] ipu3_css_cfg_acc+0xa0/0x1b5f [ipu3_imgu]
> [ 136.927981] ipu3_css_set_parameters+0x286/0x6e7 [ipu3_imgu]
> [ 136.927995] ipu3_css_start_streaming+0x1230/0x130a [ipu3_imgu]
> [ 136.928010] imgu_s_stream+0x104/0x2f7 [ipu3_imgu]
> [ 136.928022] ipu3_vb2_start_streaming+0x168/0x1bd [ipu3_imgu]
> [ 136.928034] vb2_start_streaming+0x6c/0xf2 [videobuf2_common]
> [ 136.928044] vb2_core_streamon+0xcf/0x109 [videobuf2_common]
> [ 136.928061] __video_do_ioctl+0x239/0x388 [videodev]
> [ 136.928081] video_usercopy+0x25d/0x47a [videodev]
> [ 136.928097] ? copy_overflow+0x14/0x14 [videodev]
> [ 136.928115] v4l2_ioctl+0x4d/0x58 [videodev]
> [ 136.928123] vfs_ioctl+0x1b/0x28
> [ 136.928130] do_vfs_ioctl+0x4de/0x566
> [ 136.928139] ksys_ioctl+0x50/0x70
> [ 136.928146] __x64_sys_ioctl+0x16/0x19
> [ 136.928152] do_syscall_64+0x4d/0x5a
> [ 136.928158] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 136.928164] RIP: 0033:0x7f1ec9a84f47
> [ 136.928169] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7
> c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d
> 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 136.928173] RSP: 002b:00007ffe279e6188 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000010
> [ 136.928178] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> 00007f1ec9a84f47
> [ 136.928181] RDX: 00007ffe279e6194 RSI: 0000000040045612 RDI:
> 0000000000000003
> [ 136.928184] RBP: 0000000000000000 R08: 00007f1ec776d000 R09:
> 0000000000000000
> [ 136.928188] R10: 0000000000000020 R11: 0000000000000246 R12:
> 00007ffe279e6360
> [ 136.928191] R13: 0000000000000004 R14: 00007ffe279e6360 R15:
> 00007ffe279e8826
> [ 136.928198] Modules linked in: ccm zram arc4 iwlmvm mac80211 intel_rapl
> x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi cfg80211 hid_multitouch
> ipu3_imgu ipu3_cio2 8250_dw videobuf2_dma_sg videobuf2_memops videobuf2_v4l2
> processor_thermal_device intel_soc_dts_iosf videobuf2_common ov5670 ov13858
> dw9714 v4l2_fwnode v4l2_common videodev media at24 cros_ec_lpcs cros_ec_core
> int3403_thermal int340x_thermal_zone int3400_thermal acpi_thermal_rel
> chromeos_pstore mac_hid autofs4 usbhid mmc_block hid_generic i915 sdhci_pci
> video cqhci i2c_algo_bit sdhci drm_kms_helper syscopyarea sysfillrect
> sysimgblt fb_sys_fops drm drm_panel_orientation_quirks i2c_hid hid
> [ 136.928273] ---[ end trace 4ec6c2ce09e06d9d ]---
> [ 136.928288] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu]
> [ 136.928293] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd
> ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 <f7> fb
> 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2
> [ 136.928297] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202
> [ 136.928302] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> ffff9af2c3e353c0
> [ 136.928307] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> ffff9af2c3e353c0
> [ 136.928311] RBP: 0000000000000001 R08: 0000000000000000 R09:
> ffff9af2c0b83880
> [ 136.928320] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> 00000000000003a0
> [ 136.928324] R13: 0000000000025a0a R14: 0000000000000000 R15:
> 0000000000000000
> [ 136.928330] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000) knlGS:
> 0000000000000000
> [ 136.928349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 136.928364] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> 00000000003606e0
>
> The script can be found at https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000040.html.
>
> I may be doing something wrong (and I probably am), but in any case, the
> driver shouldn't crash. Could you please have a look ?
It looks like the driver doesn't have the default state initialized
correctly somewhere and it ends up using 0 as the divisor in some
calculation? Something to fix indeed.
Best regards,
Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-29 19:51 ` Tomasz Figa
@ 2018-11-29 22:54 ` Laurent Pinchart
2018-11-29 22:58 ` Mani, Rajmohan
2018-12-04 16:07 ` Mani, Rajmohan
0 siblings, 2 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-11-29 22:54 UTC (permalink / raw)
To: Tomasz Figa
Cc: Yong Zhi, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Mani, Rajmohan, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu,
Cao Bing Bu
Hi Tomasz,
On Thursday, 29 November 2018 21:51:32 EET Tomasz Figa wrote:
> On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote:
> > On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
[snip]
> >> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be
> >> enabled prior to the test.
> >> 2. Stream tests are not performed since it requires pre-configuration
> >> for each case.
> >
> > And that's a bit of an issue. I've tested the driver with a small script
> > based on media-ctl to configure links and yavta to interface with the
> > video nodes, and got the following oops:
> >
> > [ 136.927788] divide error: 0000 [#1] PREEMPT SMP PTI
> > [ 136.927801] CPU: 2 PID: 2069 Comm: yavta Not tainted 4.20.0-rc1+ #9
> > [ 136.927806] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 136.927820] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu]
> > [ 136.927825] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00
> > fd ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99
> > <f7> fb 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2
> > [ 136.927830] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202
> > [ 136.927835] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> > ffff9af2c3e353c0
> > [ 136.927839] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> > ffff9af2c3e353c0
> > [ 136.927843] RBP: 0000000000000001 R08: 0000000000000000 R09:
> > ffff9af2c0b83880
> > [ 136.927846] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> > 00000000000003a0
> > [ 136.927849] R13: 0000000000025a0a R14: 0000000000000000 R15:
> > 0000000000000000
> > [ 136.927854] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000)
> > knlGS:
> > 0000000000000000
> > [ 136.927858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 136.927862] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> > 00000000003606e0
> > [ 136.927865] Call Trace:
> > [ 136.927884] ? __accumulate_pelt_segments+0x29/0x3a
> > [ 136.927892] ? __switch_to_asm+0x40/0x70
> > [ 136.927899] ? alloc_vmap_area+0x78/0x2f6
> > [ 136.927903] ? __switch_to_asm+0x40/0x70
> > [ 136.927907] ? __switch_to_asm+0x34/0x70
> > [ 136.927911] ? __switch_to_asm+0x40/0x70
> > [ 136.927915] ? __switch_to_asm+0x34/0x70
> > [ 136.927923] ? __inc_numa_state+0x28/0x70
> > [ 136.927929] ? preempt_latency_start+0x1e/0x3d
> > [ 136.927936] ? get_page_from_freelist+0x821/0xb62
> > [ 136.927943] ? slab_pre_alloc_hook+0x12/0x3b
> > [ 136.927948] ? kmem_cache_alloc_node_trace+0xf6/0x108
> > [ 136.927954] ? alloc_vmap_area+0x78/0x2f6
>
> Is it just me or the backtrace above doesn't seem to make sense? I
> don't see any allocations inside ipu3_css_cfg_acc().
I suppose that's why it's prefixed with '?' :-)
> > [ 136.927965] ipu3_css_cfg_acc+0xa0/0x1b5f [ipu3_imgu]
> > [ 136.927981] ipu3_css_set_parameters+0x286/0x6e7 [ipu3_imgu]
> > [ 136.927995] ipu3_css_start_streaming+0x1230/0x130a [ipu3_imgu]
> > [ 136.928010] imgu_s_stream+0x104/0x2f7 [ipu3_imgu]
> > [ 136.928022] ipu3_vb2_start_streaming+0x168/0x1bd [ipu3_imgu]
> > [ 136.928034] vb2_start_streaming+0x6c/0xf2 [videobuf2_common]
> > [ 136.928044] vb2_core_streamon+0xcf/0x109 [videobuf2_common]
> > [ 136.928061] __video_do_ioctl+0x239/0x388 [videodev]
> > [ 136.928081] video_usercopy+0x25d/0x47a [videodev]
> > [ 136.928097] ? copy_overflow+0x14/0x14 [videodev]
> > [ 136.928115] v4l2_ioctl+0x4d/0x58 [videodev]
> > [ 136.928123] vfs_ioctl+0x1b/0x28
> > [ 136.928130] do_vfs_ioctl+0x4de/0x566
> > [ 136.928139] ksys_ioctl+0x50/0x70
> > [ 136.928146] __x64_sys_ioctl+0x16/0x19
> > [ 136.928152] do_syscall_64+0x4d/0x5a
> > [ 136.928158] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 136.928164] RIP: 0033:0x7f1ec9a84f47
> > [ 136.928169] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 136.928173] RSP: 002b:00007ffe279e6188 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000010
> > [ 136.928178] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > 00007f1ec9a84f47
> > [ 136.928181] RDX: 00007ffe279e6194 RSI: 0000000040045612 RDI:
> > 0000000000000003
> > [ 136.928184] RBP: 0000000000000000 R08: 00007f1ec776d000 R09:
> > 0000000000000000
> > [ 136.928188] R10: 0000000000000020 R11: 0000000000000246 R12:
> > 00007ffe279e6360
> > [ 136.928191] R13: 0000000000000004 R14: 00007ffe279e6360 R15:
> > 00007ffe279e8826
> > [ 136.928198] Modules linked in: ccm zram arc4 iwlmvm mac80211 intel_rapl
> > x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi cfg80211
> > hid_multitouch ipu3_imgu ipu3_cio2 8250_dw videobuf2_dma_sg
> > videobuf2_memops videobuf2_v4l2 processor_thermal_device
> > intel_soc_dts_iosf videobuf2_common ov5670 ov13858 dw9714 v4l2_fwnode
> > v4l2_common videodev media at24 cros_ec_lpcs cros_ec_core int3403_thermal
> > int340x_thermal_zone int3400_thermal acpi_thermal_rel chromeos_pstore
> > mac_hid autofs4 usbhid mmc_block hid_generic i915 sdhci_pci video cqhci
> > i2c_algo_bit sdhci drm_kms_helper syscopyarea sysfillrect sysimgblt
> > fb_sys_fops drm drm_panel_orientation_quirks i2c_hid hid [ 136.928273]
> > ---[ end trace 4ec6c2ce09e06d9d ]---
> > [ 136.928288] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu]
> > [ 136.928293] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00
> > fd ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99
> > <f7> fb 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2
> > [ 136.928297] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202
> > [ 136.928302] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> > ffff9af2c3e353c0
> > [ 136.928307] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> > ffff9af2c3e353c0
> > [ 136.928311] RBP: 0000000000000001 R08: 0000000000000000 R09:
> > ffff9af2c0b83880
> > [ 136.928320] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> > 00000000000003a0
> > [ 136.928324] R13: 0000000000025a0a R14: 0000000000000000 R15:
> > 0000000000000000
> > [ 136.928330] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000)
> > knlGS:
> > 0000000000000000
> > [ 136.928349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 136.928364] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> > 00000000003606e0
> >
> > The script can be found at
> > https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/00004
> > 0.html.
> >
> > I may be doing something wrong (and I probably am), but in any case, the
> > driver shouldn't crash. Could you please have a look ?
>
> It looks like the driver doesn't have the default state initialized
> correctly somewhere and it ends up using 0 as the divisor in some
> calculation? Something to fix indeed.
That's probably the case. I'll trust Intel to fix that in v8 :-)
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-29 22:54 ` Laurent Pinchart
@ 2018-11-29 22:58 ` Mani, Rajmohan
2018-12-04 16:07 ` Mani, Rajmohan
1 sibling, 0 replies; 123+ messages in thread
From: Mani, Rajmohan @ 2018-11-29 22:58 UTC (permalink / raw)
To: Laurent Pinchart, Tomasz Figa
Cc: Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hi Laurent, Tomasz,
Thanks for the reviews.
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hi Tomasz,
>
> On Thursday, 29 November 2018 21:51:32 EET Tomasz Figa wrote:
> > On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote:
> > > On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
>
> [snip]
>
> > >> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to
> > >> be enabled prior to the test.
> > >> 2. Stream tests are not performed since it requires
> > >> pre-configuration for each case.
> > >
> > > And that's a bit of an issue. I've tested the driver with a small
> > > script based on media-ctl to configure links and yavta to interface
> > > with the video nodes, and got the following oops:
> > >
> > > [ 136.927788] divide error: 0000 [#1] PREEMPT SMP PTI [
> > > 136.927801] CPU: 2 PID: 2069 Comm: yavta Not tainted 4.20.0-rc1+ #9
> > > [ 136.927806] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018 [
> > > 136.927820] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu] [
> > > 136.927825] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28
> > > 00 fd ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00
> > > 00 99 <f7> fb 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01
> > > 19 d2 [ 136.927830] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202 [
> > > 136.927835] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> > > ffff9af2c3e353c0
> > > [ 136.927839] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> > > ffff9af2c3e353c0
> > > [ 136.927843] RBP: 0000000000000001 R08: 0000000000000000 R09:
> > > ffff9af2c0b83880
> > > [ 136.927846] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> > > 00000000000003a0
> > > [ 136.927849] R13: 0000000000025a0a R14: 0000000000000000 R15:
> > > 0000000000000000
> > > [ 136.927854] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000)
> > > knlGS:
> > > 0000000000000000
> > > [ 136.927858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > > 136.927862] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> > > 00000000003606e0
> > > [ 136.927865] Call Trace:
> > > [ 136.927884] ? __accumulate_pelt_segments+0x29/0x3a
> > > [ 136.927892] ? __switch_to_asm+0x40/0x70 [ 136.927899] ?
> > > alloc_vmap_area+0x78/0x2f6 [ 136.927903] ?
> > > __switch_to_asm+0x40/0x70 [ 136.927907] ?
> > > __switch_to_asm+0x34/0x70 [ 136.927911] ?
> > > __switch_to_asm+0x40/0x70 [ 136.927915] ?
> > > __switch_to_asm+0x34/0x70 [ 136.927923] ?
> > > __inc_numa_state+0x28/0x70 [ 136.927929] ?
> > > preempt_latency_start+0x1e/0x3d [ 136.927936] ?
> > > get_page_from_freelist+0x821/0xb62
> > > [ 136.927943] ? slab_pre_alloc_hook+0x12/0x3b [ 136.927948] ?
> > > kmem_cache_alloc_node_trace+0xf6/0x108
> > > [ 136.927954] ? alloc_vmap_area+0x78/0x2f6
> >
> > Is it just me or the backtrace above doesn't seem to make sense? I
> > don't see any allocations inside ipu3_css_cfg_acc().
>
> I suppose that's why it's prefixed with '?' :-)
>
> > > [ 136.927965] ipu3_css_cfg_acc+0xa0/0x1b5f [ipu3_imgu] [
> > > 136.927981] ipu3_css_set_parameters+0x286/0x6e7 [ipu3_imgu] [
> > > 136.927995] ipu3_css_start_streaming+0x1230/0x130a [ipu3_imgu] [
> > > 136.928010] imgu_s_stream+0x104/0x2f7 [ipu3_imgu] [ 136.928022]
> > > ipu3_vb2_start_streaming+0x168/0x1bd [ipu3_imgu] [ 136.928034]
> > > vb2_start_streaming+0x6c/0xf2 [videobuf2_common] [ 136.928044]
> > > vb2_core_streamon+0xcf/0x109 [videobuf2_common] [ 136.928061]
> > > __video_do_ioctl+0x239/0x388 [videodev] [ 136.928081]
> > > video_usercopy+0x25d/0x47a [videodev] [ 136.928097] ?
> > > copy_overflow+0x14/0x14 [videodev] [ 136.928115]
> > > v4l2_ioctl+0x4d/0x58 [videodev] [ 136.928123] vfs_ioctl+0x1b/0x28
> > > [ 136.928130] do_vfs_ioctl+0x4de/0x566 [ 136.928139]
> > > ksys_ioctl+0x50/0x70 [ 136.928146] __x64_sys_ioctl+0x16/0x19 [
> > > 136.928152] do_syscall_64+0x4d/0x5a [ 136.928158]
> > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [ 136.928164] RIP: 0033:0x7f1ec9a84f47 [ 136.928169] Code: 00 00
> > > 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f
> > > 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01
> > > 48 [ 136.928173] RSP: 002b:00007ffe279e6188 EFLAGS: 00000246
> ORIG_RAX:
> > > 0000000000000010
> > > [ 136.928178] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > > 00007f1ec9a84f47
> > > [ 136.928181] RDX: 00007ffe279e6194 RSI: 0000000040045612 RDI:
> > > 0000000000000003
> > > [ 136.928184] RBP: 0000000000000000 R08: 00007f1ec776d000 R09:
> > > 0000000000000000
> > > [ 136.928188] R10: 0000000000000020 R11: 0000000000000246 R12:
> > > 00007ffe279e6360
> > > [ 136.928191] R13: 0000000000000004 R14: 00007ffe279e6360 R15:
> > > 00007ffe279e8826
> > > [ 136.928198] Modules linked in: ccm zram arc4 iwlmvm mac80211
> > > intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi
> > > cfg80211 hid_multitouch ipu3_imgu ipu3_cio2 8250_dw
> videobuf2_dma_sg
> > > videobuf2_memops videobuf2_v4l2 processor_thermal_device
> > > intel_soc_dts_iosf videobuf2_common ov5670 ov13858 dw9714
> > > v4l2_fwnode v4l2_common videodev media at24 cros_ec_lpcs
> > > cros_ec_core int3403_thermal int340x_thermal_zone int3400_thermal
> > > acpi_thermal_rel chromeos_pstore mac_hid autofs4 usbhid mmc_block
> > > hid_generic i915 sdhci_pci video cqhci i2c_algo_bit sdhci
> > > drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
> > > drm_panel_orientation_quirks i2c_hid hid [ 136.928273] ---[ end
> > > trace 4ec6c2ce09e06d9d ]--- [ 136.928288] RIP:
> > > 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu] [ 136.928293] Code:
> > > 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd ff ff 81 64
> > > 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 <f7> fb 0f
> > > af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2 [
> > > 136.928297] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202 [ 136.928302]
> RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> > > ffff9af2c3e353c0
> > > [ 136.928307] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> > > ffff9af2c3e353c0
> > > [ 136.928311] RBP: 0000000000000001 R08: 0000000000000000 R09:
> > > ffff9af2c0b83880
> > > [ 136.928320] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> > > 00000000000003a0
> > > [ 136.928324] R13: 0000000000025a0a R14: 0000000000000000 R15:
> > > 0000000000000000
> > > [ 136.928330] FS: 00007f1eca167700(0000) GS:ffff8c19fab00000(0000)
> > > knlGS:
> > > 0000000000000000
> > > [ 136.928349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > > 136.928364] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> > > 00000000003606e0
> > >
> > > The script can be found at
> > > https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/
> > > 00004
> > > 0.html.
> > >
> > > I may be doing something wrong (and I probably am), but in any case,
> > > the driver shouldn't crash. Could you please have a look ?
> >
> > It looks like the driver doesn't have the default state initialized
> > correctly somewhere and it ends up using 0 as the divisor in some
> > calculation? Something to fix indeed.
>
> That's probably the case. I'll trust Intel to fix that in v8 :-)
>
Ack.
> --
> Regards,
>
> Laurent Pinchart
>
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-11-29 22:54 ` Laurent Pinchart
2018-11-29 22:58 ` Mani, Rajmohan
@ 2018-12-04 16:07 ` Mani, Rajmohan
2018-12-04 16:42 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2018-12-04 16:07 UTC (permalink / raw)
To: Laurent Pinchart, Tomasz Figa
Cc: Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hi Laurent, Tomasz,
>
> Thanks for the reviews.
>
> > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> >
> > Hi Tomasz,
> >
> > On Thursday, 29 November 2018 21:51:32 EET Tomasz Figa wrote:
> > > On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote:
> > > > On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
> >
> > [snip]
> >
> > > >> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to
> > > >> be enabled prior to the test.
> > > >> 2. Stream tests are not performed since it requires
> > > >> pre-configuration for each case.
> > > >
> > > > And that's a bit of an issue. I've tested the driver with a small
> > > > script based on media-ctl to configure links and yavta to
> > > > interface with the video nodes, and got the following oops:
> > > >
> > > > [ 136.927788] divide error: 0000 [#1] PREEMPT SMP PTI [
> > > > 136.927801] CPU: 2 PID: 2069 Comm: yavta Not tainted 4.20.0-rc1+
> > > > #9 [ 136.927806] Hardware name: HP Soraka/Soraka, BIOS
> > > > 08/30/2018 [ 136.927820] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14
> > > > [ipu3_imgu] [ 136.927825] Code: 89 44 24 28 42 8b 44 86 6c f7 54
> > > > 24 04 81 64 24 28
> > > > 00 fd ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03
> > > > 00
> > > > 00 99 <f7> fb 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd
> > > > 01
> > > > 19 d2 [ 136.927830] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202 [
> > > > 136.927835] RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> > > > ffff9af2c3e353c0
> > > > [ 136.927839] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> > > > ffff9af2c3e353c0
> > > > [ 136.927843] RBP: 0000000000000001 R08: 0000000000000000 R09:
> > > > ffff9af2c0b83880
> > > > [ 136.927846] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> > > > 00000000000003a0
> > > > [ 136.927849] R13: 0000000000025a0a R14: 0000000000000000 R15:
> > > > 0000000000000000
> > > > [ 136.927854] FS: 00007f1eca167700(0000)
> > > > GS:ffff8c19fab00000(0000)
> > > > knlGS:
> > > > 0000000000000000
> > > > [ 136.927858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > > > 136.927862] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> > > > 00000000003606e0
> > > > [ 136.927865] Call Trace:
> > > > [ 136.927884] ? __accumulate_pelt_segments+0x29/0x3a
> > > > [ 136.927892] ? __switch_to_asm+0x40/0x70 [ 136.927899] ?
> > > > alloc_vmap_area+0x78/0x2f6 [ 136.927903] ?
> > > > __switch_to_asm+0x40/0x70 [ 136.927907] ?
> > > > __switch_to_asm+0x34/0x70 [ 136.927911] ?
> > > > __switch_to_asm+0x40/0x70 [ 136.927915] ?
> > > > __switch_to_asm+0x34/0x70 [ 136.927923] ?
> > > > __inc_numa_state+0x28/0x70 [ 136.927929] ?
> > > > preempt_latency_start+0x1e/0x3d [ 136.927936] ?
> > > > get_page_from_freelist+0x821/0xb62
> > > > [ 136.927943] ? slab_pre_alloc_hook+0x12/0x3b [ 136.927948] ?
> > > > kmem_cache_alloc_node_trace+0xf6/0x108
> > > > [ 136.927954] ? alloc_vmap_area+0x78/0x2f6
> > >
> > > Is it just me or the backtrace above doesn't seem to make sense? I
> > > don't see any allocations inside ipu3_css_cfg_acc().
> >
> > I suppose that's why it's prefixed with '?' :-)
> >
> > > > [ 136.927965] ipu3_css_cfg_acc+0xa0/0x1b5f [ipu3_imgu] [
> > > > 136.927981] ipu3_css_set_parameters+0x286/0x6e7 [ipu3_imgu] [
> > > > 136.927995] ipu3_css_start_streaming+0x1230/0x130a [ipu3_imgu] [
> > > > 136.928010] imgu_s_stream+0x104/0x2f7 [ipu3_imgu] [ 136.928022]
> > > > ipu3_vb2_start_streaming+0x168/0x1bd [ipu3_imgu] [ 136.928034]
> > > > vb2_start_streaming+0x6c/0xf2 [videobuf2_common] [ 136.928044]
> > > > vb2_core_streamon+0xcf/0x109 [videobuf2_common] [ 136.928061]
> > > > __video_do_ioctl+0x239/0x388 [videodev] [ 136.928081]
> > > > video_usercopy+0x25d/0x47a [videodev] [ 136.928097] ?
> > > > copy_overflow+0x14/0x14 [videodev] [ 136.928115]
> > > > v4l2_ioctl+0x4d/0x58 [videodev] [ 136.928123]
> > > > vfs_ioctl+0x1b/0x28 [ 136.928130] do_vfs_ioctl+0x4de/0x566 [
> > > > 136.928139]
> > > > ksys_ioctl+0x50/0x70 [ 136.928146] __x64_sys_ioctl+0x16/0x19 [
> > > > 136.928152] do_syscall_64+0x4d/0x5a [ 136.928158]
> > > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 136.928164] RIP: 0033:0x7f1ec9a84f47 [ 136.928169] Code: 00 00
> > > > 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00
> > > > 0f
> > > > 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89
> > > > 01
> > > > 48 [ 136.928173] RSP: 002b:00007ffe279e6188 EFLAGS: 00000246
> > ORIG_RAX:
> > > > 0000000000000010
> > > > [ 136.928178] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > > > 00007f1ec9a84f47
> > > > [ 136.928181] RDX: 00007ffe279e6194 RSI: 0000000040045612 RDI:
> > > > 0000000000000003
> > > > [ 136.928184] RBP: 0000000000000000 R08: 00007f1ec776d000 R09:
> > > > 0000000000000000
> > > > [ 136.928188] R10: 0000000000000020 R11: 0000000000000246 R12:
> > > > 00007ffe279e6360
> > > > [ 136.928191] R13: 0000000000000004 R14: 00007ffe279e6360 R15:
> > > > 00007ffe279e8826
> > > > [ 136.928198] Modules linked in: ccm zram arc4 iwlmvm mac80211
> > > > intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp iwlwifi
> > > > cfg80211 hid_multitouch ipu3_imgu ipu3_cio2 8250_dw
> > videobuf2_dma_sg
> > > > videobuf2_memops videobuf2_v4l2 processor_thermal_device
> > > > intel_soc_dts_iosf videobuf2_common ov5670 ov13858 dw9714
> > > > v4l2_fwnode v4l2_common videodev media at24 cros_ec_lpcs
> > > > cros_ec_core int3403_thermal int340x_thermal_zone int3400_thermal
> > > > acpi_thermal_rel chromeos_pstore mac_hid autofs4 usbhid mmc_block
> > > > hid_generic i915 sdhci_pci video cqhci i2c_algo_bit sdhci
> > > > drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
> > > > drm_panel_orientation_quirks i2c_hid hid [ 136.928273] ---[ end
> > > > trace 4ec6c2ce09e06d9d ]--- [ 136.928288] RIP:
> > > > 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu] [ 136.928293] Code:
> > > > 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd ff ff 81
> > > > 64
> > > > 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 <f7> fb
> > > > 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2 [
> > > > 136.928297] RSP: 0018:ffff9af2c0b837c8 EFLAGS: 00010202 [
> > > > 136.928302]
> > RAX: 00000000ffffffff RBX: 0000000000000000 RCX:
> > > > ffff9af2c3e353c0
> > > > [ 136.928307] RDX: 00000000ffffffff RSI: ffff9af2c0b838e0 RDI:
> > > > ffff9af2c3e353c0
> > > > [ 136.928311] RBP: 0000000000000001 R08: 0000000000000000 R09:
> > > > ffff9af2c0b83880
> > > > [ 136.928320] R10: ffff9af2c3e353c0 R11: ffff9af2c3e357c0 R12:
> > > > 00000000000003a0
> > > > [ 136.928324] R13: 0000000000025a0a R14: 0000000000000000 R15:
> > > > 0000000000000000
> > > > [ 136.928330] FS: 00007f1eca167700(0000)
> > > > GS:ffff8c19fab00000(0000)
> > > > knlGS:
> > > > 0000000000000000
> > > > [ 136.928349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > > > 136.928364] CR2: 00007f1ec776c000 CR3: 00000001312a4003 CR4:
> > > > 00000000003606e0
> > > >
> > > > The script can be found at
> > > > https://lists.libcamera.org/pipermail/libcamera-devel/2018-Novembe
> > > > r/
> > > > 00004
> > > > 0.html.
> > > >
> > > > I may be doing something wrong (and I probably am), but in any
> > > > case, the driver shouldn't crash. Could you please have a look ?
> > >
> > > It looks like the driver doesn't have the default state initialized
> > > correctly somewhere and it ends up using 0 as the divisor in some
> > > calculation? Something to fix indeed.
> >
> > That's probably the case. I'll trust Intel to fix that in v8 :-)
> >
>
> Ack.
>
Thanks for catching this.
I was able to reproduce this error and I see that error handling
is missing, leading to the panic.
https://git.linuxtv.org/sailus/media_tree.git/tree/drivers/media
/pci/intel/ipu3/ipu3-css-params.c?h=ipu3-v7&id=
19cee7329ca2d0156043cac6afcde535e93310af#n433
is where the -EINVAL is returned.
Setting the return type as int for the following function and all
its callers to use the return value properly to error out, makes
the panic go away.
ipu3_css_osys_calc_frame_and_stripe_params()
Will include the fix in v8.
Thanks for catching this.
Raj
> > --
> > Regards,
> >
> > Laurent Pinchart
> >
> >
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-04 16:07 ` Mani, Rajmohan
@ 2018-12-04 16:42 ` Laurent Pinchart
2018-12-04 16:53 ` Mani, Rajmohan
2018-12-05 0:30 ` Mani, Rajmohan
0 siblings, 2 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-04 16:42 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hi Rajmohan,
On Tuesday, 4 December 2018 18:07:16 EET Mani, Rajmohan wrote:
> >> On Thursday, 29 November 2018 21:51:32 EET Tomasz Figa wrote:
> >>> On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote:
> >>>> On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
> >>
> >> [snip]
> >>
> >>>>> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to
> >>>>> be enabled prior to the test.
> >>>>> 2. Stream tests are not performed since it requires
> >>>>> pre-configuration for each case.
> >>>>
> >>>> And that's a bit of an issue. I've tested the driver with a small
> >>>> script based on media-ctl to configure links and yavta to
> >>>> interface with the video nodes, and got the following oops:
[snip]
> >>>> The script can be found at
> >>>> https://lists.libcamera.org/pipermail/libcamera-devel/2018-Novembe
> >>>> r/000040.html.
> >>>>
> >>>> I may be doing something wrong (and I probably am), but in any
> >>>> case, the driver shouldn't crash. Could you please have a look ?
> >>>
> >>> It looks like the driver doesn't have the default state initialized
> >>> correctly somewhere and it ends up using 0 as the divisor in some
> >>> calculation? Something to fix indeed.
> >>
> >> That's probably the case. I'll trust Intel to fix that in v8 :-)
> >
> > Ack.
>
> Thanks for catching this.
> I was able to reproduce this error and I see that error handling
> is missing, leading to the panic.
>
> https://git.linuxtv.org/sailus/media_tree.git/tree/drivers/media
> /pci/intel/ipu3/ipu3-css-params.c?h=ipu3-v7&id=
> 19cee7329ca2d0156043cac6afcde535e93310af#n433
>
> is where the -EINVAL is returned.
>
> Setting the return type as int for the following function and all
> its callers to use the return value properly to error out, makes
> the panic go away.
I assume that I will still not be able to process frames through the pipe
then, as I'll get an error :-) Could you tell me why the configuration is
incorrect and how I can fix it ?
> ipu3_css_osys_calc_frame_and_stripe_params()
>
> Will include the fix in v8.
Thank you.
> Thanks for catching this.
You're welcome.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-04 16:42 ` Laurent Pinchart
@ 2018-12-04 16:53 ` Mani, Rajmohan
2018-12-05 0:30 ` Mani, Rajmohan
1 sibling, 0 replies; 123+ messages in thread
From: Mani, Rajmohan @ 2018-12-04 16:53 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Tomasz Figa, Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hi Laurent,
> -----Original Message-----
> From: Laurent Pinchart [mailto:laurent.pinchart@ideasonboard.com]
> Sent: Tuesday, December 04, 2018 8:42 AM
> To: Mani, Rajmohan <rajmohan.mani@intel.com>
> Cc: Tomasz Figa <tfiga@chromium.org>; Zhi, Yong <yong.zhi@intel.com>;
> Linux Media Mailing List <linux-media@vger.kernel.org>; Sakari Ailus
> <sakari.ailus@linux.intel.com>; Mauro Carvalho Chehab
> <mchehab@kernel.org>; Hans Verkuil <hans.verkuil@cisco.com>; Zheng, Jian
> Xu <jian.xu.zheng@intel.com>; Hu, Jerry W <jerry.w.hu@intel.com>;
> Toivonen, Tuukka <tuukka.toivonen@intel.com>; Qiu, Tian Shu
> <tian.shu.qiu@intel.com>; Cao, Bingbu <bingbu.cao@intel.com>
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hi Rajmohan,
>
> On Tuesday, 4 December 2018 18:07:16 EET Mani, Rajmohan wrote:
> > >> On Thursday, 29 November 2018 21:51:32 EET Tomasz Figa wrote:
> > >>> On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote:
> > >>>> On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
> > >>
> > >> [snip]
> > >>
> > >>>>> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need
> > >>>>> to be enabled prior to the test.
> > >>>>> 2. Stream tests are not performed since it requires
> > >>>>> pre-configuration for each case.
> > >>>>
> > >>>> And that's a bit of an issue. I've tested the driver with a small
> > >>>> script based on media-ctl to configure links and yavta to
> > >>>> interface with the video nodes, and got the following oops:
>
> [snip]
>
> > >>>> The script can be found at
> > >>>> https://lists.libcamera.org/pipermail/libcamera-devel/2018-Novemb
> > >>>> e
> > >>>> r/000040.html.
> > >>>>
> > >>>> I may be doing something wrong (and I probably am), but in any
> > >>>> case, the driver shouldn't crash. Could you please have a look ?
> > >>>
> > >>> It looks like the driver doesn't have the default state
> > >>> initialized correctly somewhere and it ends up using 0 as the
> > >>> divisor in some calculation? Something to fix indeed.
> > >>
> > >> That's probably the case. I'll trust Intel to fix that in v8 :-)
> > >
> > > Ack.
> >
> > Thanks for catching this.
> > I was able to reproduce this error and I see that error handling is
> > missing, leading to the panic.
> >
> > https://git.linuxtv.org/sailus/media_tree.git/tree/drivers/media
> > /pci/intel/ipu3/ipu3-css-params.c?h=ipu3-v7&id=
> > 19cee7329ca2d0156043cac6afcde535e93310af#n433
> >
> > is where the -EINVAL is returned.
> >
> > Setting the return type as int for the following function and all its
> > callers to use the return value properly to error out, makes the panic
> > go away.
>
> I assume that I will still not be able to process frames through the pipe then, as
> I'll get an error :-) Could you tell me why the configuration is incorrect and how
> I can fix it ?
>
:)
Let me look into this more and get back.
Thanks for your patience.
Raj
> > ipu3_css_osys_calc_frame_and_stripe_params()
> >
> > Will include the fix in v8.
>
> Thank you.
>
> > Thanks for catching this.
>
> You're welcome.
>
> --
> Regards,
>
> Laurent Pinchart
>
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-04 16:42 ` Laurent Pinchart
2018-12-04 16:53 ` Mani, Rajmohan
@ 2018-12-05 0:30 ` Mani, Rajmohan
2018-12-11 13:34 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2018-12-05 0:30 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Tomasz Figa, Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hi Laurent,
> > Hi Rajmohan,
> >
> > On Tuesday, 4 December 2018 18:07:16 EET Mani, Rajmohan wrote:
> > > >> On Thursday, 29 November 2018 21:51:32 EET Tomasz Figa wrote:
> > > >>> On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote:
> > > >>>> On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote:
> > > >>
> > > >> [snip]
> > > >>
> > > >>>>> 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need
> > > >>>>> to be enabled prior to the test.
> > > >>>>> 2. Stream tests are not performed since it requires
> > > >>>>> pre-configuration for each case.
> > > >>>>
> > > >>>> And that's a bit of an issue. I've tested the driver with a
> > > >>>> small script based on media-ctl to configure links and yavta to
> > > >>>> interface with the video nodes, and got the following oops:
> >
> > [snip]
> >
> > > >>>> The script can be found at
> > > >>>> https://lists.libcamera.org/pipermail/libcamera-devel/2018-Nove
> > > >>>> mb
> > > >>>> e
> > > >>>> r/000040.html.
> > > >>>>
> > > >>>> I may be doing something wrong (and I probably am), but in any
> > > >>>> case, the driver shouldn't crash. Could you please have a look ?
> > > >>>
> > > >>> It looks like the driver doesn't have the default state
> > > >>> initialized correctly somewhere and it ends up using 0 as the
> > > >>> divisor in some calculation? Something to fix indeed.
> > > >>
> > > >> That's probably the case. I'll trust Intel to fix that in v8 :-)
> > > >
> > > > Ack.
> > >
> > > Thanks for catching this.
> > > I was able to reproduce this error and I see that error handling is
> > > missing, leading to the panic.
> > >
> > > https://git.linuxtv.org/sailus/media_tree.git/tree/drivers/media
> > > /pci/intel/ipu3/ipu3-css-params.c?h=ipu3-v7&id=
> > > 19cee7329ca2d0156043cac6afcde535e93310af#n433
> > >
> > > is where the -EINVAL is returned.
> > >
> > > Setting the return type as int for the following function and all
> > > its callers to use the return value properly to error out, makes the
> > > panic go away.
> >
> > I assume that I will still not be able to process frames through the
> > pipe then, as I'll get an error :-) Could you tell me why the
> > configuration is incorrect and how I can fix it ?
> >
>
> :)
> Let me look into this more and get back.
> Thanks for your patience.
I can see a couple of steps missing in the script below.
(https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000040.html)
From patch 02 of this v7 series "doc-rst: Add Intel IPU3 documentation", under section
"Configuring ImgU V4L2 subdev for image processing"...
1. The pipe mode needs to be configured for the V4L2 subdev.
Also the pipe mode of the corresponding V4L2 subdev should be set as
desired (e.g 0 for video mode or 1 for still mode) through the control
id 0x009819a1 as below.
e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
2. ImgU pipeline needs to be configured for image processing as below.
RAW bayer frames go through the following ISP pipeline HW blocks to
have the processed image output to the DDR memory.
RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
Geometric Distortion Correction (GDC) -> DDR
The ImgU V4L2 subdev has to be configured with the supported
resolutions in all the above HW blocks, for a given input resolution.
For a given supported resolution for an input frame, the Input Feeder,
Bayer Down Scaling and GDC blocks should be configured with the
supported resolutions. This information can be obtained by looking at
the following IPU3 ISP configuration table for ov5670 sensor.
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
/graph_settings_ov5670.xml
For the ov5670 example, for an input frame with a resolution of
2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
2560x1920 respectively.
The following steps prepare the ImgU ISP pipeline for the image processing.
1. The ImgU V4L2 subdev data format should be set by using the
VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
2. The ImgU V4L2 subdev cropping should be set by using the
VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
target, using the input feeder height and width.
3. The ImgU V4L2 subdev composing should be set by using the
VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
target, using the BDS height and width.
Once these 2 steps are done, the raw bayer frames can be input to the
ImgU V4L2 subdev for processing.
> Raj
>
> > > ipu3_css_osys_calc_frame_and_stripe_params()
> > >
> > > Will include the fix in v8.
> >
> > Thank you.
> >
> > > Thanks for catching this.
> >
> > You're welcome.
> >
> > --
> > Regards,
> >
> > Laurent Pinchart
> >
> >
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-05 0:30 ` Mani, Rajmohan
@ 2018-12-11 13:34 ` Laurent Pinchart
2018-12-11 13:43 ` Laurent Pinchart
0 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-11 13:34 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
[-- Attachment #1: Type: text/plain, Size: 4159 bytes --]
Hi Rajmohan,
On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
[snip]
> I can see a couple of steps missing in the script below.
> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000040.
> html)
>
> From patch 02 of this v7 series "doc-rst: Add Intel IPU3 documentation",
> under section "Configuring ImgU V4L2 subdev for image processing"...
>
> 1. The pipe mode needs to be configured for the V4L2 subdev.
>
> Also the pipe mode of the corresponding V4L2 subdev should be set as
> desired (e.g 0 for video mode or 1 for still mode) through the control
> id 0x009819a1 as below.
>
> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
I assume the control takes a valid default value ? It's better to set it
explicitly anyway, so I'll do so.
> 2. ImgU pipeline needs to be configured for image processing as below.
>
> RAW bayer frames go through the following ISP pipeline HW blocks to
> have the processed image output to the DDR memory.
>
> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> Geometric Distortion Correction (GDC) -> DDR
>
> The ImgU V4L2 subdev has to be configured with the supported
> resolutions in all the above HW blocks, for a given input resolution.
>
> For a given supported resolution for an input frame, the Input Feeder,
> Bayer Down Scaling and GDC blocks should be configured with the
> supported resolutions. This information can be obtained by looking at
> the following IPU3 ISP configuration table for ov5670 sensor.
>
> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/maste
> r /baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
> /graph_settings_ov5670.xml
>
> For the ov5670 example, for an input frame with a resolution of
> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> 2560x1920 respectively.
How is the GDC output resolution computed from the input resolution ? Does the
GDC always consume 32 columns and 22 lines ?
> The following steps prepare the ImgU ISP pipeline for the image processing.
>
> 1. The ImgU V4L2 subdev data format should be set by using the
> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
If I understand things correctly, the GDC resolution is the pipeline output
resolution. Why is it configured on pad 0 ?
> 2. The ImgU V4L2 subdev cropping should be set by using the
> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> target, using the input feeder height and width.
>
> 3. The ImgU V4L2 subdev composing should be set by using the
> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> target, using the BDS height and width.
>
> Once these 2 steps are done, the raw bayer frames can be input to the
> ImgU V4L2 subdev for processing.
Do I need to capture from both the output and viewfinder nodes ? How are they
related to the IF -> BDS -> GDC pipeline, are they both fed from the GDC
output ? If so, how does the viewfinder scaler fit in that picture ?
I have tried the above configuration with the IPU3 v8 driver, and while the
kernel doesn't crash, no images get processed. The userspace processes wait
forever for buffers to be ready. I then configured pad 2 to 2560x1920 and pad
3 to 1920x1080, and managed to capture images \o/
There's one problem though: during capture, or very soon after it, the machine
locks up completely. I suspect a memory corruption, as when it doesn't log
immediately commands such as dmesg will not produce any output and just block,
until the system freezes soon after (especially when moving the mouse).
I would still call this an improvement to some extent, but there's definitely
room for more improvements :-)
To reproduce the issue, you can run the ipu3-process.sh script (attached to
this e-mail) with the following arguments:
$ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in the
IPU3-specific Bayer format (for a total of 6469632 bytes).
--
Regards,
Laurent Pinchart
[-- Attachment #2: ipu3-process.sh --]
[-- Type: application/x-shellscript, Size: 4846 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-11 13:34 ` Laurent Pinchart
@ 2018-12-11 13:43 ` Laurent Pinchart
2018-12-11 14:20 ` Laurent Pinchart
2018-12-12 4:55 ` Bingbu Cao
0 siblings, 2 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-11 13:43 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
Hello Rajmohan,
On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
>
> [snip]
>
> > I can see a couple of steps missing in the script below.
> > (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/00004
> > 0. html)
> >
> > From patch 02 of this v7 series "doc-rst: Add Intel IPU3 documentation",
> > under section "Configuring ImgU V4L2 subdev for image processing"...
> >
> > 1. The pipe mode needs to be configured for the V4L2 subdev.
> >
> > Also the pipe mode of the corresponding V4L2 subdev should be set as
> > desired (e.g 0 for video mode or 1 for still mode) through the control
> > id 0x009819a1 as below.
> >
> > e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
>
> I assume the control takes a valid default value ? It's better to set it
> explicitly anyway, so I'll do so.
>
> > 2. ImgU pipeline needs to be configured for image processing as below.
> >
> > RAW bayer frames go through the following ISP pipeline HW blocks to
> > have the processed image output to the DDR memory.
> >
> > RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > Geometric Distortion Correction (GDC) -> DDR
> >
> > The ImgU V4L2 subdev has to be configured with the supported
> > resolutions in all the above HW blocks, for a given input resolution.
> >
> > For a given supported resolution for an input frame, the Input Feeder,
> > Bayer Down Scaling and GDC blocks should be configured with the
> > supported resolutions. This information can be obtained by looking at
> > the following IPU3 ISP configuration table for ov5670 sensor.
> >
> > https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/mas
> > te r /baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
> > /graph_settings_ov5670.xml
> >
> > For the ov5670 example, for an input frame with a resolution of
> > 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > 2560x1920 respectively.
>
> How is the GDC output resolution computed from the input resolution ? Does
> the GDC always consume 32 columns and 22 lines ?
>
> > The following steps prepare the ImgU ISP pipeline for the image
> > processing.
> >
> > 1. The ImgU V4L2 subdev data format should be set by using the
> > VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > above.
>
> If I understand things correctly, the GDC resolution is the pipeline output
> resolution. Why is it configured on pad 0 ?
>
> > 2. The ImgU V4L2 subdev cropping should be set by using the
> > VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > target, using the input feeder height and width.
> >
> > 3. The ImgU V4L2 subdev composing should be set by using the
> > VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > target, using the BDS height and width.
> >
> > Once these 2 steps are done, the raw bayer frames can be input to the
> > ImgU V4L2 subdev for processing.
>
> Do I need to capture from both the output and viewfinder nodes ? How are
> they related to the IF -> BDS -> GDC pipeline, are they both fed from the
> GDC output ? If so, how does the viewfinder scaler fit in that picture ?
>
> I have tried the above configuration with the IPU3 v8 driver, and while the
> kernel doesn't crash, no images get processed. The userspace processes wait
> forever for buffers to be ready. I then configured pad 2 to 2560x1920 and
> pad 3 to 1920x1080, and managed to capture images \o/
>
> There's one problem though: during capture, or very soon after it, the
> machine locks up completely. I suspect a memory corruption, as when it
> doesn't log immediately commands such as dmesg will not produce any output
> and just block, until the system freezes soon after (especially when moving
> the mouse).
>
> I would still call this an improvement to some extent, but there's
> definitely room for more improvements :-)
>
> To reproduce the issue, you can run the ipu3-process.sh script (attached to
> this e-mail) with the following arguments:
>
> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
>
> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in the
> IPU3-specific Bayer format (for a total of 6469632 bytes).
I managed to get the dmesg output, and it doesn't look pretty.
[ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
[ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm mac80211
iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev at24
media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
drm_panel_orientation_quirks i2c_hid hid
[ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
4.20.0-rc6+ #2
[ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
[ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
[ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3 48
0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07 <0f> 0b
5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
[ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
[ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
000000000000000c
[ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
00000000ffffffff
[ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
ffff8f5cfaba16f0
[ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
ffff8f5cf58f0028
[ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
ffff8f5cf58f04e8
[ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000) knlGS:
0000000000000000
[ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
00000000003606e0
[ 571.217301] Call Trace:
[ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
[ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
[ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
[ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
[ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
[ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
[ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
[ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
[ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
[ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
[ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
[ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
[ 571.217480] vfs_ioctl+0x1e/0x2b
[ 571.217486] do_vfs_ioctl+0x531/0x559
[ 571.217494] ? vfs_write+0xd1/0xdf
[ 571.217500] ksys_ioctl+0x50/0x70
[ 571.217506] __x64_sys_ioctl+0x16/0x19
[ 571.217512] do_syscall_64+0x53/0x60
[ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 571.217524] RIP: 0033:0x7f85cf9b9f47
[ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d
01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
[ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f85cf9b9f47
[ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
0000000000000003
[ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
00007f85d009c700
[ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
000055f4c4dc0b06
[ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
00007ffc59057825
[ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
[ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-11 13:43 ` Laurent Pinchart
@ 2018-12-11 14:20 ` Laurent Pinchart
2018-12-16 7:26 ` Laurent Pinchart
[not found] ` <6F87890CF0F5204F892DEA1EF0D77A599B31FAF4@fmsmsx122.amr.corp.intel.com>
2018-12-12 4:55 ` Bingbu Cao
1 sibling, 2 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-11 14:20 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu
Hello again,
On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> >
> > [snip]
> >
> > > I can see a couple of steps missing in the script below.
> > > (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000
> > > 040.html)
> > >
> > > From patch 02 of this v7 series "doc-rst: Add Intel IPU3 documentation",
> > > under section "Configuring ImgU V4L2 subdev for image processing"...
> > >
> > > 1. The pipe mode needs to be configured for the V4L2 subdev.
> > >
> > > Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > desired (e.g 0 for video mode or 1 for still mode) through the control
> > > id 0x009819a1 as below.
> > >
> > > e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> >
> > I assume the control takes a valid default value ? It's better to set it
> > explicitly anyway, so I'll do so.
> >
> > > 2. ImgU pipeline needs to be configured for image processing as below.
> > >
> > > RAW bayer frames go through the following ISP pipeline HW blocks to
> > > have the processed image output to the DDR memory.
> > >
> > > RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > Geometric Distortion Correction (GDC) -> DDR
> > >
> > > The ImgU V4L2 subdev has to be configured with the supported
> > > resolutions in all the above HW blocks, for a given input resolution.
> > >
> > > For a given supported resolution for an input frame, the Input Feeder,
> > > Bayer Down Scaling and GDC blocks should be configured with the
> > > supported resolutions. This information can be obtained by looking at
> > > the following IPU3 ISP configuration table for ov5670 sensor.
> > >
> > > https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/m
> > > aster/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/
> > > gcss/graph_settings_ov5670.xml
> > >
> > > For the ov5670 example, for an input frame with a resolution of
> > > 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > > resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > > 2560x1920 respectively.
> >
> > How is the GDC output resolution computed from the input resolution ? Does
> > the GDC always consume 32 columns and 22 lines ?
> >
> > > The following steps prepare the ImgU ISP pipeline for the image
> > > processing.
> > >
> > > 1. The ImgU V4L2 subdev data format should be set by using the
> > > VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > > above.
> >
> > If I understand things correctly, the GDC resolution is the pipeline
> > output resolution. Why is it configured on pad 0 ?
> >
> > > 2. The ImgU V4L2 subdev cropping should be set by using the
> > > VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > > target, using the input feeder height and width.
> > >
> > > 3. The ImgU V4L2 subdev composing should be set by using the
> > > VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > > target, using the BDS height and width.
> > >
> > > Once these 2 steps are done, the raw bayer frames can be input to the
> > > ImgU V4L2 subdev for processing.
> >
> > Do I need to capture from both the output and viewfinder nodes ? How are
> > they related to the IF -> BDS -> GDC pipeline, are they both fed from the
> > GDC output ? If so, how does the viewfinder scaler fit in that picture ?
> >
> > I have tried the above configuration with the IPU3 v8 driver, and while
> > the kernel doesn't crash, no images get processed. The userspace processes
> > wait forever for buffers to be ready. I then configured pad 2 to 2560x1920
> > and pad 3 to 1920x1080, and managed to capture images \o/
> >
> > There's one problem though: during capture, or very soon after it, the
> > machine locks up completely. I suspect a memory corruption, as when it
> > doesn't log immediately commands such as dmesg will not produce any output
> > and just block, until the system freezes soon after (especially when
> > moving the mouse).
> >
> > I would still call this an improvement to some extent, but there's
> > definitely room for more improvements :-)
> >
> > To reproduce the issue, you can run the ipu3-process.sh script (attached
> > to this e-mail) with the following arguments:
> >
> > $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
This should have read
$ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
Without the --vf argument no images are processed.
It seems that the Intel mail server blocked the mail that contained the
script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
> > frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in the
> > IPU3-specific Bayer format (for a total of 6469632 bytes).
>
> I managed to get the dmesg output, and it doesn't look pretty.
>
> [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm mac80211
> iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
> 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
> intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev
> at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
> int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> drm_panel_orientation_quirks i2c_hid hid
> [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> 4.20.0-rc6+ #2
> [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3
> 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> 000000000000000c
> [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> 00000000ffffffff
> [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> ffff8f5cfaba16f0
> [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> ffff8f5cf58f0028
> [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> ffff8f5cf58f04e8
> [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000) knlGS:
> 0000000000000000
> [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> 00000000003606e0
> [ 571.217301] Call Trace:
> [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> [ 571.217480] vfs_ioctl+0x1e/0x2b
> [ 571.217486] do_vfs_ioctl+0x531/0x559
> [ 571.217494] ? vfs_write+0xd1/0xdf
> [ 571.217500] ksys_ioctl+0x50/0x70
> [ 571.217506] __x64_sys_ioctl+0x16/0x19
> [ 571.217512] do_syscall_64+0x53/0x60
> [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000010
> [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> 00007f85cf9b9f47
> [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> 0000000000000003
> [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> 00007f85d009c700
> [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> 000055f4c4dc0b06
> [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> 00007ffc59057825
> [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
And after fixing another issue in the capture script (which was setting the
format on the ImgU subdev pad 3 to 2560x1920 but capture in 1920x1080), I now
get plenty of the following messages:
[ 221.366131] BUG: Bad page state in process yavta pfn:14a4ff
[ 221.366134] page:ffffde5d45293fc0 count:-1 mapcount:0 mapping:
0000000000000000 index:0x0
[ 221.366137] flags: 0x200000000000000()
[ 221.366140] raw: 0200000000000000 dead000000000100 dead000000000200
0000000000000000
[ 221.366143] raw: 0000000000000000 0000000000000000 ffffffffffffffff
0000000000000000
[ 221.366145] page dumped because: nonzero _refcount
[ 221.366147] Modules linked in: asix usbnet mii zram arc4 iwlmvm intel_rapl
x86_pkg_temp_thermal intel_powerclamp coretemp mac80211 iwlwifi cfg80211
hid_multitouch 8250_dw ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
videobuf2_memops videobuf2_v4l2 processor_thermal_device videobuf2_common
intel_soc_dts_iosf ov13858 ov5670 dw9714 v4l2_fwnode v4l2_common videodev
media at24 cros_ec_lpcs cros_ec_core int3403_thermal int340x_thermal_zone
chromeos_pstore mac_hid int3400_thermal acpi_thermal_rel autofs4 usbhid
mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops sdhci_pci cqhci sdhci drm
drm_panel_orientation_quirks i2c_hid hid
[ 221.366172] CPU: 3 PID: 1022 Comm: yavta Tainted: G B WC
4.20.0-rc6+ #2
[ 221.366173] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
[ 221.366173] Call Trace:
[ 221.366176] dump_stack+0x46/0x59
[ 221.366179] bad_page+0xf2/0x10c
[ 221.366182] free_pages_check+0x78/0x81
[ 221.366186] free_pcppages_bulk+0xa6/0x236
[ 221.366190] free_unref_page+0x4b/0x53
[ 221.366193] vb2_dma_sg_put+0x95/0xb5 [videobuf2_dma_sg]
[ 221.366197] __vb2_buf_mem_free+0x3a/0x6e [videobuf2_common]
[ 221.366202] __vb2_queue_free+0xe3/0x1be [videobuf2_common]
[ 221.366207] vb2_core_reqbufs+0xe9/0x2cc [videobuf2_common]
[ 221.366212] vb2_ioctl_reqbufs+0x78/0x9e [videobuf2_v4l2]
[ 221.366220] __video_do_ioctl+0x258/0x38e [videodev]
[ 221.366229] video_usercopy+0x25f/0x4e5 [videodev]
[ 221.366237] ? copy_overflow+0x14/0x14 [videodev]
[ 221.366240] ? unmap_region+0xe0/0x10a
[ 221.366250] v4l2_ioctl+0x4d/0x58 [videodev]
[ 221.366253] vfs_ioctl+0x1e/0x2b
[ 221.366255] do_vfs_ioctl+0x531/0x559
[ 221.366260] ksys_ioctl+0x50/0x70
[ 221.366263] __x64_sys_ioctl+0x16/0x19
[ 221.366266] do_syscall_64+0x53/0x60
[ 221.366269] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 221.366270] RIP: 0033:0x7fbe39f6af47
[ 221.366272] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d
01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
[ 221.366273] RSP: 002b:00007fff05638e68 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 221.366275] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
00007fbe39f6af47
[ 221.366279] RDX: 00007fff05638f90 RSI: 00000000c0145608 RDI:
0000000000000003
[ 221.366283] RBP: 0000000000000004 R08: 0000000000000000 R09:
0000000000000045
[ 221.366287] R10: 0000000000000557 R11: 0000000000000246 R12:
000055c83bd76750
[ 221.366290] R13: 000055c83b6b26a0 R14: 0000000000000001 R15:
00007fff0563a825
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-11 14:20 ` Laurent Pinchart
@ 2018-12-16 7:26 ` Laurent Pinchart
2018-12-20 22:25 ` Laurent Pinchart
[not found] ` <6F87890CF0F5204F892DEA1EF0D77A599B31FAF4@fmsmsx122.amr.corp.intel.com>
1 sibling, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-16 7:26 UTC (permalink / raw)
To: Zhi, Yong
Cc: Mani, Rajmohan, Tomasz Figa, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
Hello Yong,
Could you please have a look at the crash reported below ?
On Tuesday, 11 December 2018 16:20:43 EET Laurent Pinchart wrote:
> On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> > On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> >> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> >>
> >> [snip]
> >>
> >>> I can see a couple of steps missing in the script below.
> >>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
> >>> 00040.html)
> >>>
> >>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> >>> documentation",
> >>> under section "Configuring ImgU V4L2 subdev for image processing"...
> >>>
> >>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> >>>
> >>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> >>> desired (e.g 0 for video mode or 1 for still mode) through the control
> >>> id 0x009819a1 as below.
> >>>
> >>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> >>
> >> I assume the control takes a valid default value ? It's better to set it
> >> explicitly anyway, so I'll do so.
> >>
> >>> 2. ImgU pipeline needs to be configured for image processing as below.
> >>>
> >>> RAW bayer frames go through the following ISP pipeline HW blocks to
> >>> have the processed image output to the DDR memory.
> >>>
> >>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >>> Geometric Distortion Correction (GDC) -> DDR
> >>>
> >>> The ImgU V4L2 subdev has to be configured with the supported
> >>> resolutions in all the above HW blocks, for a given input resolution.
> >>>
> >>> For a given supported resolution for an input frame, the Input Feeder,
> >>> Bayer Down Scaling and GDC blocks should be configured with the
> >>> supported resolutions. This information can be obtained by looking at
> >>> the following IPU3 ISP configuration table for ov5670 sensor.
> >>>
> >>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> >>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/
> >>> gcss/graph_settings_ov5670.xml
> >>>
> >>> For the ov5670 example, for an input frame with a resolution of
> >>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> >>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> >>> 2560x1920 respectively.
> >>
> >> How is the GDC output resolution computed from the input resolution ?
> >> Does the GDC always consume 32 columns and 22 lines ?
> >>
> >>> The following steps prepare the ImgU ISP pipeline for the image
> >>> processing.
> >>>
> >>> 1. The ImgU V4L2 subdev data format should be set by using the
> >>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> >>> above.
> >>
> >> If I understand things correctly, the GDC resolution is the pipeline
> >> output resolution. Why is it configured on pad 0 ?
> >>
> >>> 2. The ImgU V4L2 subdev cropping should be set by using the
> >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> >>> target, using the input feeder height and width.
> >>>
> >>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >>> target, using the BDS height and width.
> >>>
> >>> Once these 2 steps are done, the raw bayer frames can be input to the
> >>> ImgU V4L2 subdev for processing.
> >>
> >> Do I need to capture from both the output and viewfinder nodes ? How are
> >> they related to the IF -> BDS -> GDC pipeline, are they both fed from
> >> the GDC output ? If so, how does the viewfinder scaler fit in that
> >> picture ?
> >>
> >> I have tried the above configuration with the IPU3 v8 driver, and while
> >> the kernel doesn't crash, no images get processed. The userspace
> >> processes wait forever for buffers to be ready. I then configured pad 2
> >> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
> >>
> >> There's one problem though: during capture, or very soon after it, the
> >> machine locks up completely. I suspect a memory corruption, as when it
> >> doesn't log immediately commands such as dmesg will not produce any
> >> output and just block, until the system freezes soon after (especially
> >> when moving the mouse).
> >>
> >> I would still call this an improvement to some extent, but there's
> >> definitely room for more improvements :-)
> >>
> >> To reproduce the issue, you can run the ipu3-process.sh script (attached
> >> to this e-mail) with the following arguments:
> >>
> >> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
>
> This should have read
>
> $ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
>
> Without the --vf argument no images are processed.
>
> It seems that the Intel mail server blocked the mail that contained the
> script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
>
> >> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
> >> the IPU3-specific Bayer format (for a total of 6469632 bytes).
> >
> > I managed to get the dmesg output, and it doesn't look pretty.
> >
> > [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> > libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> > ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > mac80211
> > iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
> > 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
> > intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev
> > at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
> > int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> > drm_panel_orientation_quirks i2c_hid hid
> > [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> > 4.20.0-rc6+ #2
> > [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3
> > 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> > <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> > [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> > [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> > 000000000000000c
> > [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> > 00000000ffffffff
> > [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> > ffff8f5cfaba16f0
> > [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> > ffff8f5cf58f0028
> > [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> > ffff8f5cf58f04e8
> > [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
> > knlGS:
> > 0000000000000000
> > [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> > 00000000003606e0
> > [ 571.217301] Call Trace:
> > [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> > [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> > [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> > [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> > [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> > [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> > [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> > [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> > [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> > [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> > [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> > [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> > [ 571.217480] vfs_ioctl+0x1e/0x2b
> > [ 571.217486] do_vfs_ioctl+0x531/0x559
> > [ 571.217494] ? vfs_write+0xd1/0xdf
> > [ 571.217500] ksys_ioctl+0x50/0x70
> > [ 571.217506] __x64_sys_ioctl+0x16/0x19
> > [ 571.217512] do_syscall_64+0x53/0x60
> > [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> > [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000010
> > [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> > 00007f85cf9b9f47
> > [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> > 0000000000000003
> > [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> > 00007f85d009c700
> > [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> > 000055f4c4dc0b06
> > [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> > 00007ffc59057825
> > [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> > [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
>
> And after fixing another issue in the capture script (which was setting the
> format on the ImgU subdev pad 3 to 2560x1920 but capture in 1920x1080), I
> now get plenty of the following messages:
>
> [ 221.366131] BUG: Bad page state in process yavta pfn:14a4ff
> [ 221.366134] page:ffffde5d45293fc0 count:-1 mapcount:0 mapping:
> 0000000000000000 index:0x0
> [ 221.366137] flags: 0x200000000000000()
> [ 221.366140] raw: 0200000000000000 dead000000000100 dead000000000200
> 0000000000000000
> [ 221.366143] raw: 0000000000000000 0000000000000000 ffffffffffffffff
> 0000000000000000
> [ 221.366145] page dumped because: nonzero _refcount
> [ 221.366147] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp mac80211 iwlwifi
> cfg80211 hid_multitouch 8250_dw ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> videobuf2_memops videobuf2_v4l2 processor_thermal_device videobuf2_common
> intel_soc_dts_iosf ov13858 ov5670 dw9714 v4l2_fwnode v4l2_common videodev
> media at24 cros_ec_lpcs cros_ec_core int3403_thermal int340x_thermal_zone
> chromeos_pstore mac_hid int3400_thermal acpi_thermal_rel autofs4 usbhid
> mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> sysfillrect sysimgblt fb_sys_fops sdhci_pci cqhci sdhci drm
> drm_panel_orientation_quirks i2c_hid hid
> [ 221.366172] CPU: 3 PID: 1022 Comm: yavta Tainted: G B WC
> 4.20.0-rc6+ #2
> [ 221.366173] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 221.366173] Call Trace:
> [ 221.366176] dump_stack+0x46/0x59
> [ 221.366179] bad_page+0xf2/0x10c
> [ 221.366182] free_pages_check+0x78/0x81
> [ 221.366186] free_pcppages_bulk+0xa6/0x236
> [ 221.366190] free_unref_page+0x4b/0x53
> [ 221.366193] vb2_dma_sg_put+0x95/0xb5 [videobuf2_dma_sg]
> [ 221.366197] __vb2_buf_mem_free+0x3a/0x6e [videobuf2_common]
> [ 221.366202] __vb2_queue_free+0xe3/0x1be [videobuf2_common]
> [ 221.366207] vb2_core_reqbufs+0xe9/0x2cc [videobuf2_common]
> [ 221.366212] vb2_ioctl_reqbufs+0x78/0x9e [videobuf2_v4l2]
> [ 221.366220] __video_do_ioctl+0x258/0x38e [videodev]
> [ 221.366229] video_usercopy+0x25f/0x4e5 [videodev]
> [ 221.366237] ? copy_overflow+0x14/0x14 [videodev]
> [ 221.366240] ? unmap_region+0xe0/0x10a
> [ 221.366250] v4l2_ioctl+0x4d/0x58 [videodev]
> [ 221.366253] vfs_ioctl+0x1e/0x2b
> [ 221.366255] do_vfs_ioctl+0x531/0x559
> [ 221.366260] ksys_ioctl+0x50/0x70
> [ 221.366263] __x64_sys_ioctl+0x16/0x19
> [ 221.366266] do_syscall_64+0x53/0x60
> [ 221.366269] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 221.366270] RIP: 0033:0x7fbe39f6af47
> [ 221.366272] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 221.366273] RSP: 002b:00007fff05638e68 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000010
> [ 221.366275] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> 00007fbe39f6af47
> [ 221.366279] RDX: 00007fff05638f90 RSI: 00000000c0145608 RDI:
> 0000000000000003
> [ 221.366283] RBP: 0000000000000004 R08: 0000000000000000 R09:
> 0000000000000045
> [ 221.366287] R10: 0000000000000557 R11: 0000000000000246 R12:
> 000055c83bd76750
> [ 221.366290] R13: 000055c83b6b26a0 R14: 0000000000000001 R15:
> 00007fff0563a825
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-16 7:26 ` Laurent Pinchart
@ 2018-12-20 22:25 ` Laurent Pinchart
2018-12-21 3:04 ` Tomasz Figa
0 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-20 22:25 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Zhi, Yong, Mani, Rajmohan, Tomasz Figa, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
Hellon
On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> Hello Yong,
>
> Could you please have a look at the crash reported below ?
A bit more information to help you debugging this. I've enabled KASAN in the
kernel configuration, and get the following use-after-free reports.
[ 166.332920] ==================================================================
[ 166.332937] BUG: KASAN: use-after-free in __cached_rbnode_delete_update+0x36/0x202
[ 166.332944] Read of size 8 at addr ffff888133823718 by task yavta/1305
[ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C 4.20.0-rc6+ #3
[ 166.332958] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
[ 166.332959] Call Trace:
[ 166.332967] dump_stack+0x5b/0x81
[ 166.332974] print_address_description+0x65/0x227
[ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
[ 166.332983] kasan_report+0x247/0x285
[ 166.332989] __cached_rbnode_delete_update+0x36/0x202
[ 166.332995] private_free_iova+0x57/0x6d
[ 166.332999] __free_iova+0x23/0x31
[ 166.333011] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
[ 166.333022] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 166.333032] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
[ 166.333043] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
[ 166.333056] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
[ 166.333067] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
[ 166.333079] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
[ 166.333088] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
[ 166.333096] ? __mutex_lock_interruptible_slowpath+0xf/0xf
[ 166.333104] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
[ 166.333123] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.333142] ? copy_overflow+0x14/0x14 [videodev]
[ 166.333147] ? slab_free_freelist_hook+0x46/0x94
[ 166.333151] ? kfree+0x107/0x1a0
[ 166.333169] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.333187] ? copy_overflow+0x14/0x14 [videodev]
[ 166.333203] ? v4l_enumstd+0x49/0x49 [videodev]
[ 166.333207] ? __wake_up_common+0x342/0x342
[ 166.333215] ? atomic_long_add_return+0x15/0x24
[ 166.333219] ? ldsem_up_read+0x15/0x29
[ 166.333223] ? tty_write+0x4c6/0x4d8
[ 166.333227] ? n_tty_receive_char_special+0x1152/0x1152
[ 166.333244] ? video_usercopy+0x8ae/0x8ae [videodev]
[ 166.333260] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.333266] vfs_ioctl+0x76/0x89
[ 166.333271] do_vfs_ioctl+0xb33/0xb7e
[ 166.333275] ? __switch_to_asm+0x40/0x70
[ 166.333279] ? __switch_to_asm+0x40/0x70
[ 166.333282] ? __switch_to_asm+0x34/0x70
[ 166.333286] ? __switch_to_asm+0x40/0x70
[ 166.333290] ? ioctl_preallocate+0x174/0x174
[ 166.333294] ? __switch_to+0x71c/0xb00
[ 166.333299] ? compat_start_thread+0x6b/0x6b
[ 166.333302] ? __switch_to_asm+0x34/0x70
[ 166.333305] ? __switch_to_asm+0x40/0x70
[ 166.333309] ? mmdrop+0x12/0x23
[ 166.333313] ? finish_task_switch+0x34d/0x3de
[ 166.333319] ? __schedule+0x1004/0x1045
[ 166.333325] ? firmware_map_remove+0x119/0x119
[ 166.333330] ksys_ioctl+0x50/0x70
[ 166.333335] __x64_sys_ioctl+0x82/0x89
[ 166.333340] do_syscall_64+0xa0/0xd2
[ 166.333345] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.333349] RIP: 0033:0x7f2481541f47
[ 166.333354] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
[ 166.333357] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 166.333362] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
[ 166.333364] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
[ 166.333367] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
[ 166.333369] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
[ 166.333372] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
[ 166.333383] Allocated by task 1305:
[ 166.333389] kasan_kmalloc+0x8a/0x98
[ 166.333392] slab_post_alloc_hook+0x31/0x51
[ 166.333396] kmem_cache_alloc+0xd7/0x174
[ 166.333399] alloc_iova+0x24/0x2ea
[ 166.333407] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
[ 166.333415] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
[ 166.333424] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
[ 166.333433] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
[ 166.333442] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
[ 166.333449] vb2_start_streaming+0x164/0x33b [videobuf2_common]
[ 166.333455] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
[ 166.333471] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.333487] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.333501] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.333505] vfs_ioctl+0x76/0x89
[ 166.333508] do_vfs_ioctl+0xb33/0xb7e
[ 166.333511] ksys_ioctl+0x50/0x70
[ 166.333514] __x64_sys_ioctl+0x82/0x89
[ 166.333518] do_syscall_64+0xa0/0xd2
[ 166.333521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.333526] Freed by task 1301:
[ 166.333532] __kasan_slab_free+0xfa/0x11c
[ 166.333535] slab_free_freelist_hook+0x46/0x94
[ 166.333538] kmem_cache_free+0x7b/0x172
[ 166.333542] __free_iova+0x23/0x31
[ 166.333550] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
[ 166.333557] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 166.333566] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
[ 166.333574] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
[ 166.333584] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
[ 166.333593] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
[ 166.333599] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
[ 166.333606] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
[ 166.333621] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.333637] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.333652] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.333655] vfs_ioctl+0x76/0x89
[ 166.333658] do_vfs_ioctl+0xb33/0xb7e
[ 166.333662] ksys_ioctl+0x50/0x70
[ 166.333665] __x64_sys_ioctl+0x82/0x89
[ 166.333668] do_syscall_64+0xa0/0xd2
[ 166.333671] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.333678] The buggy address belongs to the object at ffff888133823700
which belongs to the cache iommu_iova of size 40
[ 166.333685] The buggy address is located 24 bytes inside of
40-byte region [ffff888133823700, ffff888133823728)
[ 166.333690] The buggy address belongs to the page:
[ 166.333696] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
[ 166.333703] flags: 0x200000000010200(slab|head)
[ 166.333710] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
[ 166.333717] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
[ 166.333720] page dumped because: kasan: bad access detected
[ 166.333726] Memory state around the buggy address:
[ 166.333732] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.333737] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.333742] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
[ 166.333745] ^
[ 166.333750] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.333755] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.333759] ==================================================================
[ 166.333762] Disabling lock debugging due to kernel taint
[ 166.333764] ==================================================================
[ 166.333770] BUG: KASAN: double-free or invalid-free in kmem_cache_free+0x7b/0x172
[ 166.333780] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
[ 166.333782] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
[ 166.333783] Call Trace:
[ 166.333789] dump_stack+0x5b/0x81
[ 166.333795] print_address_description+0x65/0x227
[ 166.333799] ? kmem_cache_free+0x7b/0x172
[ 166.333803] kasan_report_invalid_free+0x67/0xa0
[ 166.333807] ? kmem_cache_free+0x7b/0x172
[ 166.333812] __kasan_slab_free+0x86/0x11c
[ 166.333817] slab_free_freelist_hook+0x46/0x94
[ 166.333822] kmem_cache_free+0x7b/0x172
[ 166.333826] ? __free_iova+0x23/0x31
[ 166.333831] __free_iova+0x23/0x31
[ 166.333840] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
[ 166.333851] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 166.333861] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
[ 166.333872] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
[ 166.333885] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
[ 166.333896] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
[ 166.333908] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
[ 166.333917] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
[ 166.333923] ? __mutex_lock_interruptible_slowpath+0xf/0xf
[ 166.333932] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
[ 166.333950] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.333970] ? copy_overflow+0x14/0x14 [videodev]
[ 166.333974] ? slab_free_freelist_hook+0x46/0x94
[ 166.333979] ? kfree+0x107/0x1a0
[ 166.333997] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.334015] ? copy_overflow+0x14/0x14 [videodev]
[ 166.334031] ? v4l_enumstd+0x49/0x49 [videodev]
[ 166.334035] ? __wake_up_common+0x342/0x342
[ 166.334042] ? atomic_long_add_return+0x15/0x24
[ 166.334046] ? ldsem_up_read+0x15/0x29
[ 166.334050] ? tty_write+0x4c6/0x4d8
[ 166.334054] ? n_tty_receive_char_special+0x1152/0x1152
[ 166.334071] ? video_usercopy+0x8ae/0x8ae [videodev]
[ 166.334087] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.334092] vfs_ioctl+0x76/0x89
[ 166.334097] do_vfs_ioctl+0xb33/0xb7e
[ 166.334101] ? __switch_to_asm+0x40/0x70
[ 166.334105] ? __switch_to_asm+0x40/0x70
[ 166.334108] ? __switch_to_asm+0x34/0x70
[ 166.334111] ? __switch_to_asm+0x40/0x70
[ 166.334116] ? ioctl_preallocate+0x174/0x174
[ 166.334120] ? __switch_to+0x71c/0xb00
[ 166.334124] ? compat_start_thread+0x6b/0x6b
[ 166.334127] ? __switch_to_asm+0x34/0x70
[ 166.334130] ? __switch_to_asm+0x40/0x70
[ 166.334134] ? mmdrop+0x12/0x23
[ 166.334137] ? finish_task_switch+0x34d/0x3de
[ 166.334143] ? __schedule+0x1004/0x1045
[ 166.334148] ? firmware_map_remove+0x119/0x119
[ 166.334153] ksys_ioctl+0x50/0x70
[ 166.334158] __x64_sys_ioctl+0x82/0x89
[ 166.334163] do_syscall_64+0xa0/0xd2
[ 166.334167] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.334171] RIP: 0033:0x7f2481541f47
[ 166.334175] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
[ 166.334177] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 166.334181] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
[ 166.334184] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
[ 166.334186] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
[ 166.334189] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
[ 166.334191] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
[ 166.334201] Allocated by task 1305:
[ 166.334207] kasan_kmalloc+0x8a/0x98
[ 166.334210] slab_post_alloc_hook+0x31/0x51
[ 166.334213] kmem_cache_alloc+0xd7/0x174
[ 166.334216] alloc_iova+0x24/0x2ea
[ 166.334225] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
[ 166.334233] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
[ 166.334241] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
[ 166.334250] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
[ 166.334259] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
[ 166.334266] vb2_start_streaming+0x164/0x33b [videobuf2_common]
[ 166.334273] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
[ 166.334288] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.334304] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.334319] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.334322] vfs_ioctl+0x76/0x89
[ 166.334325] do_vfs_ioctl+0xb33/0xb7e
[ 166.334328] ksys_ioctl+0x50/0x70
[ 166.334332] __x64_sys_ioctl+0x82/0x89
[ 166.334335] do_syscall_64+0xa0/0xd2
[ 166.334338] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.334343] Freed by task 1301:
[ 166.334349] __kasan_slab_free+0xfa/0x11c
[ 166.334352] slab_free_freelist_hook+0x46/0x94
[ 166.334355] kmem_cache_free+0x7b/0x172
[ 166.334359] __free_iova+0x23/0x31
[ 166.334367] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
[ 166.334375] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 166.334383] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
[ 166.334392] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
[ 166.334401] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
[ 166.334410] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
[ 166.334416] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
[ 166.334423] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
[ 166.334438] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.334454] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.334469] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.334472] vfs_ioctl+0x76/0x89
[ 166.334475] do_vfs_ioctl+0xb33/0xb7e
[ 166.334479] ksys_ioctl+0x50/0x70
[ 166.334482] __x64_sys_ioctl+0x82/0x89
[ 166.334485] do_syscall_64+0xa0/0xd2
[ 166.334488] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.334494] The buggy address belongs to the object at ffff888133823700
which belongs to the cache iommu_iova of size 40
[ 166.334501] The buggy address is located 0 bytes inside of
40-byte region [ffff888133823700, ffff888133823728)
[ 166.334506] The buggy address belongs to the page:
[ 166.334511] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
[ 166.334517] flags: 0x200000000010200(slab|head)
[ 166.334524] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
[ 166.334530] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
[ 166.334533] page dumped because: kasan: bad access detected
[ 166.334539] Memory state around the buggy address:
[ 166.334544] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.334549] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.334554] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
[ 166.334558] ^
[ 166.334562] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.334567] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.334571] ==================================================================
[ 166.340377] ==================================================================
[ 166.340388] BUG: KASAN: double-free or invalid-free in kfree+0x107/0x1a0
[ 166.340399] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
[ 166.340401] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
[ 166.340403] Call Trace:
[ 166.340410] dump_stack+0x5b/0x81
[ 166.340416] print_address_description+0x65/0x227
[ 166.340420] ? kfree+0x107/0x1a0
[ 166.340425] kasan_report_invalid_free+0x67/0xa0
[ 166.340428] ? kfree+0x107/0x1a0
[ 166.340433] __kasan_slab_free+0x86/0x11c
[ 166.340438] slab_free_freelist_hook+0x46/0x94
[ 166.340443] kfree+0x107/0x1a0
[ 166.340454] ? ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
[ 166.340464] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
[ 166.340475] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 166.340485] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
[ 166.340495] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
[ 166.340509] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
[ 166.340520] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
[ 166.340531] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
[ 166.340541] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
[ 166.340548] ? __mutex_lock_interruptible_slowpath+0xf/0xf
[ 166.340557] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
[ 166.340575] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.340595] ? copy_overflow+0x14/0x14 [videodev]
[ 166.340600] ? slab_free_freelist_hook+0x46/0x94
[ 166.340604] ? kfree+0x107/0x1a0
[ 166.340622] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.340640] ? copy_overflow+0x14/0x14 [videodev]
[ 166.340657] ? v4l_enumstd+0x49/0x49 [videodev]
[ 166.340660] ? __wake_up_common+0x342/0x342
[ 166.340668] ? atomic_long_add_return+0x15/0x24
[ 166.340672] ? ldsem_up_read+0x15/0x29
[ 166.340677] ? tty_write+0x4c6/0x4d8
[ 166.340681] ? n_tty_receive_char_special+0x1152/0x1152
[ 166.340698] ? video_usercopy+0x8ae/0x8ae [videodev]
[ 166.340714] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.340720] vfs_ioctl+0x76/0x89
[ 166.340725] do_vfs_ioctl+0xb33/0xb7e
[ 166.340729] ? __switch_to_asm+0x40/0x70
[ 166.340733] ? __switch_to_asm+0x40/0x70
[ 166.340736] ? __switch_to_asm+0x34/0x70
[ 166.340739] ? __switch_to_asm+0x40/0x70
[ 166.340743] ? ioctl_preallocate+0x174/0x174
[ 166.340748] ? __switch_to+0x71c/0xb00
[ 166.340752] ? compat_start_thread+0x6b/0x6b
[ 166.340756] ? __switch_to_asm+0x34/0x70
[ 166.340759] ? __switch_to_asm+0x40/0x70
[ 166.340762] ? mmdrop+0x12/0x23
[ 166.340766] ? finish_task_switch+0x34d/0x3de
[ 166.340772] ? __schedule+0x1004/0x1045
[ 166.340777] ? firmware_map_remove+0x119/0x119
[ 166.340782] ksys_ioctl+0x50/0x70
[ 166.340788] __x64_sys_ioctl+0x82/0x89
[ 166.340793] do_syscall_64+0xa0/0xd2
[ 166.340797] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.340802] RIP: 0033:0x7f2481541f47
[ 166.340806] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
[ 166.340809] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 166.340813] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
[ 166.340816] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
[ 166.340819] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
[ 166.340821] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
[ 166.340824] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
[ 166.340834] Allocated by task 1305:
[ 166.340840] kasan_kmalloc+0x8a/0x98
[ 166.340844] __kmalloc_node+0x193/0x1ba
[ 166.340848] kvmalloc_node+0x44/0x6d
[ 166.340856] ipu3_dmamap_alloc+0x1c9/0x83f [ipu3_imgu]
[ 166.340864] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
[ 166.340873] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
[ 166.340882] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
[ 166.340891] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
[ 166.340897] vb2_start_streaming+0x164/0x33b [videobuf2_common]
[ 166.340904] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
[ 166.340920] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.340935] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.340950] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.340954] vfs_ioctl+0x76/0x89
[ 166.340957] do_vfs_ioctl+0xb33/0xb7e
[ 166.340960] ksys_ioctl+0x50/0x70
[ 166.340963] __x64_sys_ioctl+0x82/0x89
[ 166.340966] do_syscall_64+0xa0/0xd2
[ 166.340969] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.340974] Freed by task 1301:
[ 166.340980] __kasan_slab_free+0xfa/0x11c
[ 166.340983] slab_free_freelist_hook+0x46/0x94
[ 166.340986] kfree+0x107/0x1a0
[ 166.340994] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
[ 166.341002] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
[ 166.341010] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
[ 166.341019] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
[ 166.341028] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
[ 166.341037] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
[ 166.341043] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
[ 166.341050] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
[ 166.341066] __video_do_ioctl+0x625/0x887 [videodev]
[ 166.341081] video_usercopy+0x3a3/0x8ae [videodev]
[ 166.341096] v4l2_ioctl+0xb7/0xc5 [videodev]
[ 166.341100] vfs_ioctl+0x76/0x89
[ 166.341103] do_vfs_ioctl+0xb33/0xb7e
[ 166.341106] ksys_ioctl+0x50/0x70
[ 166.341109] __x64_sys_ioctl+0x82/0x89
[ 166.341112] do_syscall_64+0xa0/0xd2
[ 166.341116] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 166.341122] The buggy address belongs to the object at ffff88811d228440
which belongs to the cache kmalloc-8 of size 8
[ 166.341129] The buggy address is located 0 bytes inside of
8-byte region [ffff88811d228440, ffff88811d228448)
[ 166.341134] The buggy address belongs to the page:
[ 166.341140] page:ffffea0004748a00 count:1 mapcount:0 mapping:ffff88815a80c340 index:0xffff88811d228f80 compound_mapcount: 0
[ 166.341146] flags: 0x200000000010200(slab|head)
[ 166.341153] raw: 0200000000010200 ffffea000564b288 ffffea00049ef708 ffff88815a80c340
[ 166.341159] raw: ffff88811d228f80 0000000000160013 00000001ffffffff 0000000000000000
[ 166.341163] page dumped because: kasan: bad access detected
[ 166.341169] Memory state around the buggy address:
[ 166.341174] ffff88811d228300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.341179] ffff88811d228380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.341184] >ffff88811d228400: fc fc fc fc fc fc fc fc fb fc fc fc fc fc fc fc
[ 166.341188] ^
[ 166.341192] ffff88811d228480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.341197] ffff88811d228500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 166.341201] ==================================================================
> On Tuesday, 11 December 2018 16:20:43 EET Laurent Pinchart wrote:
> > On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> > > On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > >> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > >>
> > >> [snip]
> > >>
> > >>> I can see a couple of steps missing in the script below.
> > >>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
> > >>> 00040.html)
> > >>>
> > >>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > >>> documentation",
> > >>> under section "Configuring ImgU V4L2 subdev for image processing"...
> > >>>
> > >>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > >>>
> > >>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > >>> desired (e.g 0 for video mode or 1 for still mode) through the control
> > >>> id 0x009819a1 as below.
> > >>>
> > >>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > >>
> > >> I assume the control takes a valid default value ? It's better to set
> > >> it
> > >> explicitly anyway, so I'll do so.
> > >>
> > >>> 2. ImgU pipeline needs to be configured for image processing as below.
> > >>>
> > >>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > >>> have the processed image output to the DDR memory.
> > >>>
> > >>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > >>> Geometric Distortion Correction (GDC) -> DDR
> > >>>
> > >>> The ImgU V4L2 subdev has to be configured with the supported
> > >>> resolutions in all the above HW blocks, for a given input resolution.
> > >>>
> > >>> For a given supported resolution for an input frame, the Input Feeder,
> > >>> Bayer Down Scaling and GDC blocks should be configured with the
> > >>> supported resolutions. This information can be obtained by looking at
> > >>> the following IPU3 ISP configuration table for ov5670 sensor.
> > >>>
> > >>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> > >>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files
> > >>> /
> > >>> gcss/graph_settings_ov5670.xml
> > >>>
> > >>> For the ov5670 example, for an input frame with a resolution of
> > >>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > >>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > >>> 2560x1920 respectively.
> > >>
> > >> How is the GDC output resolution computed from the input resolution ?
> > >> Does the GDC always consume 32 columns and 22 lines ?
> > >>
> > >>> The following steps prepare the ImgU ISP pipeline for the image
> > >>> processing.
> > >>>
> > >>> 1. The ImgU V4L2 subdev data format should be set by using the
> > >>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > >>> above.
> > >>
> > >> If I understand things correctly, the GDC resolution is the pipeline
> > >> output resolution. Why is it configured on pad 0 ?
> > >>
> > >>> 2. The ImgU V4L2 subdev cropping should be set by using the
> > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > >>> target, using the input feeder height and width.
> > >>>
> > >>> 3. The ImgU V4L2 subdev composing should be set by using the
> > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > >>> target, using the BDS height and width.
> > >>>
> > >>> Once these 2 steps are done, the raw bayer frames can be input to the
> > >>> ImgU V4L2 subdev for processing.
> > >>
> > >> Do I need to capture from both the output and viewfinder nodes ? How
> > >> are
> > >> they related to the IF -> BDS -> GDC pipeline, are they both fed from
> > >> the GDC output ? If so, how does the viewfinder scaler fit in that
> > >> picture ?
> > >>
> > >> I have tried the above configuration with the IPU3 v8 driver, and while
> > >> the kernel doesn't crash, no images get processed. The userspace
> > >> processes wait forever for buffers to be ready. I then configured pad 2
> > >> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
> > >>
> > >> There's one problem though: during capture, or very soon after it, the
> > >> machine locks up completely. I suspect a memory corruption, as when it
> > >> doesn't log immediately commands such as dmesg will not produce any
> > >> output and just block, until the system freezes soon after (especially
> > >> when moving the mouse).
> > >>
> > >> I would still call this an improvement to some extent, but there's
> > >> definitely room for more improvements :-)
> > >>
> > >> To reproduce the issue, you can run the ipu3-process.sh script
> > >> (attached
> > >> to this e-mail) with the following arguments:
> > >>
> > >> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
> >
> > This should have read
> >
> > $ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
> >
> > Without the --vf argument no images are processed.
> >
> > It seems that the Intel mail server blocked the mail that contained the
> > script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
> >
> > >> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
> > >> the IPU3-specific Bayer format (for a total of 6469632 bytes).
> > >
> > > I managed to get the dmesg output, and it doesn't look pretty.
> > >
> > > [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> > > libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> > > ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > mac80211
> > > iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp
> > > cfg80211
> > > 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > videobuf2_memops videobuf2_v4l2 videobuf2_common
> > > processor_thermal_device
> > > intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common
> > > videodev
> > > at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs
> > > cros_ec_core
> > > int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> > > drm_panel_orientation_quirks i2c_hid hid
> > > [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> > > 4.20.0-rc6+ #2
> > > [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc
> > > f3
> > > 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> > > <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> > > [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> > > [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> > > 000000000000000c
> > > [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> > > 00000000ffffffff
> > > [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> > > ffff8f5cfaba16f0
> > > [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> > > ffff8f5cf58f0028
> > > [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> > > ffff8f5cf58f04e8
> > > [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
> > > knlGS:
> > > 0000000000000000
> > > [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> > > 00000000003606e0
> > > [ 571.217301] Call Trace:
> > > [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> > > [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> > > [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> > > [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> > > [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> > > [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> > > [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> > > [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> > > [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> > > [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> > > [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> > > [ 571.217480] vfs_ioctl+0x1e/0x2b
> > > [ 571.217486] do_vfs_ioctl+0x531/0x559
> > > [ 571.217494] ? vfs_write+0xd1/0xdf
> > > [ 571.217500] ksys_ioctl+0x50/0x70
> > > [ 571.217506] __x64_sys_ioctl+0x16/0x19
> > > [ 571.217512] do_syscall_64+0x53/0x60
> > > [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> > > [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00
> > > 48
> > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> > > 0000000000000010
> > > [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> > > 00007f85cf9b9f47
> > > [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> > > 0000000000000003
> > > [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> > > 00007f85d009c700
> > > [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> > > 000055f4c4dc0b06
> > > [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> > > 00007ffc59057825
> > > [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> > > [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
> >
> > And after fixing another issue in the capture script (which was setting
> > the
> > format on the ImgU subdev pad 3 to 2560x1920 but capture in 1920x1080), I
> > now get plenty of the following messages:
> >
> > [ 221.366131] BUG: Bad page state in process yavta pfn:14a4ff
> > [ 221.366134] page:ffffde5d45293fc0 count:-1 mapcount:0 mapping:
> > 0000000000000000 index:0x0
> > [ 221.366137] flags: 0x200000000000000()
> > [ 221.366140] raw: 0200000000000000 dead000000000100 dead000000000200
> > 0000000000000000
> > [ 221.366143] raw: 0000000000000000 0000000000000000 ffffffffffffffff
> > 0000000000000000
> > [ 221.366145] page dumped because: nonzero _refcount
> > [ 221.366147] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp mac80211 iwlwifi
> > cfg80211 hid_multitouch 8250_dw ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > videobuf2_memops videobuf2_v4l2 processor_thermal_device videobuf2_common
> > intel_soc_dts_iosf ov13858 ov5670 dw9714 v4l2_fwnode v4l2_common videodev
> > media at24 cros_ec_lpcs cros_ec_core int3403_thermal int340x_thermal_zone
> > chromeos_pstore mac_hid int3400_thermal acpi_thermal_rel autofs4 usbhid
> > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > sysfillrect sysimgblt fb_sys_fops sdhci_pci cqhci sdhci drm
> > drm_panel_orientation_quirks i2c_hid hid
> > [ 221.366172] CPU: 3 PID: 1022 Comm: yavta Tainted: G B WC
> > 4.20.0-rc6+ #2
> > [ 221.366173] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 221.366173] Call Trace:
> > [ 221.366176] dump_stack+0x46/0x59
> > [ 221.366179] bad_page+0xf2/0x10c
> > [ 221.366182] free_pages_check+0x78/0x81
> > [ 221.366186] free_pcppages_bulk+0xa6/0x236
> > [ 221.366190] free_unref_page+0x4b/0x53
> > [ 221.366193] vb2_dma_sg_put+0x95/0xb5 [videobuf2_dma_sg]
> > [ 221.366197] __vb2_buf_mem_free+0x3a/0x6e [videobuf2_common]
> > [ 221.366202] __vb2_queue_free+0xe3/0x1be [videobuf2_common]
> > [ 221.366207] vb2_core_reqbufs+0xe9/0x2cc [videobuf2_common]
> > [ 221.366212] vb2_ioctl_reqbufs+0x78/0x9e [videobuf2_v4l2]
> > [ 221.366220] __video_do_ioctl+0x258/0x38e [videodev]
> > [ 221.366229] video_usercopy+0x25f/0x4e5 [videodev]
> > [ 221.366237] ? copy_overflow+0x14/0x14 [videodev]
> > [ 221.366240] ? unmap_region+0xe0/0x10a
> > [ 221.366250] v4l2_ioctl+0x4d/0x58 [videodev]
> > [ 221.366253] vfs_ioctl+0x1e/0x2b
> > [ 221.366255] do_vfs_ioctl+0x531/0x559
> > [ 221.366260] ksys_ioctl+0x50/0x70
> > [ 221.366263] __x64_sys_ioctl+0x16/0x19
> > [ 221.366266] do_syscall_64+0x53/0x60
> > [ 221.366269] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 221.366270] RIP: 0033:0x7fbe39f6af47
> > [ 221.366272] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 221.366273] RSP: 002b:00007fff05638e68 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000010
> > [ 221.366275] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > 00007fbe39f6af47
> > [ 221.366279] RDX: 00007fff05638f90 RSI: 00000000c0145608 RDI:
> > 0000000000000003
> > [ 221.366283] RBP: 0000000000000004 R08: 0000000000000000 R09:
> > 0000000000000045
> > [ 221.366287] R10: 0000000000000557 R11: 0000000000000246 R12:
> > 000055c83bd76750
> > [ 221.366290] R13: 000055c83b6b26a0 R14: 0000000000000001 R15:
> > 00007fff0563a825
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-20 22:25 ` Laurent Pinchart
@ 2018-12-21 3:04 ` Tomasz Figa
2019-01-08 6:54 ` Tomasz Figa
0 siblings, 1 reply; 123+ messages in thread
From: Tomasz Figa @ 2018-12-21 3:04 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Yong Zhi, Mani, Rajmohan, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W,
Toivonen, Tuukka, Qiu, Tian Shu, Cao Bing Bu
On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hellon
>
> On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > Hello Yong,
> >
> > Could you please have a look at the crash reported below ?
>
> A bit more information to help you debugging this. I've enabled KASAN in the
> kernel configuration, and get the following use-after-free reports.
>
> [ 166.332920] ==================================================================
> [ 166.332937] BUG: KASAN: use-after-free in __cached_rbnode_delete_update+0x36/0x202
> [ 166.332944] Read of size 8 at addr ffff888133823718 by task yavta/1305
>
> [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C 4.20.0-rc6+ #3
> [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 166.332959] Call Trace:
> [ 166.332967] dump_stack+0x5b/0x81
> [ 166.332974] print_address_description+0x65/0x227
> [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> [ 166.332983] kasan_report+0x247/0x285
> [ 166.332989] __cached_rbnode_delete_update+0x36/0x202
> [ 166.332995] private_free_iova+0x57/0x6d
> [ 166.332999] __free_iova+0x23/0x31
> [ 166.333011] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
Thanks Laurent, I think this is a very good hint. It looks like we're
basically freeing and already freed IOVA and corrupting some allocator
state?
> [ 166.333022] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 166.333032] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> [ 166.333043] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> [ 166.333056] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> [ 166.333067] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> [ 166.333079] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> [ 166.333088] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> [ 166.333096] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> [ 166.333104] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> [ 166.333123] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.333142] ? copy_overflow+0x14/0x14 [videodev]
> [ 166.333147] ? slab_free_freelist_hook+0x46/0x94
> [ 166.333151] ? kfree+0x107/0x1a0
> [ 166.333169] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.333187] ? copy_overflow+0x14/0x14 [videodev]
> [ 166.333203] ? v4l_enumstd+0x49/0x49 [videodev]
> [ 166.333207] ? __wake_up_common+0x342/0x342
> [ 166.333215] ? atomic_long_add_return+0x15/0x24
> [ 166.333219] ? ldsem_up_read+0x15/0x29
> [ 166.333223] ? tty_write+0x4c6/0x4d8
> [ 166.333227] ? n_tty_receive_char_special+0x1152/0x1152
> [ 166.333244] ? video_usercopy+0x8ae/0x8ae [videodev]
> [ 166.333260] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.333266] vfs_ioctl+0x76/0x89
> [ 166.333271] do_vfs_ioctl+0xb33/0xb7e
> [ 166.333275] ? __switch_to_asm+0x40/0x70
> [ 166.333279] ? __switch_to_asm+0x40/0x70
> [ 166.333282] ? __switch_to_asm+0x34/0x70
> [ 166.333286] ? __switch_to_asm+0x40/0x70
> [ 166.333290] ? ioctl_preallocate+0x174/0x174
> [ 166.333294] ? __switch_to+0x71c/0xb00
> [ 166.333299] ? compat_start_thread+0x6b/0x6b
> [ 166.333302] ? __switch_to_asm+0x34/0x70
> [ 166.333305] ? __switch_to_asm+0x40/0x70
> [ 166.333309] ? mmdrop+0x12/0x23
> [ 166.333313] ? finish_task_switch+0x34d/0x3de
> [ 166.333319] ? __schedule+0x1004/0x1045
> [ 166.333325] ? firmware_map_remove+0x119/0x119
> [ 166.333330] ksys_ioctl+0x50/0x70
> [ 166.333335] __x64_sys_ioctl+0x82/0x89
> [ 166.333340] do_syscall_64+0xa0/0xd2
> [ 166.333345] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 166.333349] RIP: 0033:0x7f2481541f47
> [ 166.333354] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 166.333357] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [ 166.333362] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> [ 166.333364] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> [ 166.333367] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> [ 166.333369] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> [ 166.333372] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
>
> [ 166.333383] Allocated by task 1305:
> [ 166.333389] kasan_kmalloc+0x8a/0x98
> [ 166.333392] slab_post_alloc_hook+0x31/0x51
> [ 166.333396] kmem_cache_alloc+0xd7/0x174
> [ 166.333399] alloc_iova+0x24/0x2ea
> [ 166.333407] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
> [ 166.333415] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> [ 166.333424] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> [ 166.333433] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> [ 166.333442] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> [ 166.333449] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> [ 166.333455] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> [ 166.333471] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.333487] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.333501] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.333505] vfs_ioctl+0x76/0x89
> [ 166.333508] do_vfs_ioctl+0xb33/0xb7e
> [ 166.333511] ksys_ioctl+0x50/0x70
> [ 166.333514] __x64_sys_ioctl+0x82/0x89
> [ 166.333518] do_syscall_64+0xa0/0xd2
> [ 166.333521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [ 166.333526] Freed by task 1301:
> [ 166.333532] __kasan_slab_free+0xfa/0x11c
> [ 166.333535] slab_free_freelist_hook+0x46/0x94
> [ 166.333538] kmem_cache_free+0x7b/0x172
> [ 166.333542] __free_iova+0x23/0x31
> [ 166.333550] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> [ 166.333557] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 166.333566] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> [ 166.333574] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> [ 166.333584] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> [ 166.333593] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> [ 166.333599] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> [ 166.333606] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> [ 166.333621] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.333637] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.333652] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.333655] vfs_ioctl+0x76/0x89
> [ 166.333658] do_vfs_ioctl+0xb33/0xb7e
> [ 166.333662] ksys_ioctl+0x50/0x70
> [ 166.333665] __x64_sys_ioctl+0x82/0x89
> [ 166.333668] do_syscall_64+0xa0/0xd2
> [ 166.333671] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [ 166.333678] The buggy address belongs to the object at ffff888133823700
> which belongs to the cache iommu_iova of size 40
> [ 166.333685] The buggy address is located 24 bytes inside of
> 40-byte region [ffff888133823700, ffff888133823728)
> [ 166.333690] The buggy address belongs to the page:
> [ 166.333696] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
> [ 166.333703] flags: 0x200000000010200(slab|head)
> [ 166.333710] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
> [ 166.333717] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
> [ 166.333720] page dumped because: kasan: bad access detected
>
> [ 166.333726] Memory state around the buggy address:
> [ 166.333732] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.333737] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.333742] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
> [ 166.333745] ^
> [ 166.333750] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.333755] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.333759] ==================================================================
> [ 166.333762] Disabling lock debugging due to kernel taint
> [ 166.333764] ==================================================================
> [ 166.333770] BUG: KASAN: double-free or invalid-free in kmem_cache_free+0x7b/0x172
>
> [ 166.333780] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
> [ 166.333782] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 166.333783] Call Trace:
> [ 166.333789] dump_stack+0x5b/0x81
> [ 166.333795] print_address_description+0x65/0x227
> [ 166.333799] ? kmem_cache_free+0x7b/0x172
> [ 166.333803] kasan_report_invalid_free+0x67/0xa0
> [ 166.333807] ? kmem_cache_free+0x7b/0x172
> [ 166.333812] __kasan_slab_free+0x86/0x11c
> [ 166.333817] slab_free_freelist_hook+0x46/0x94
> [ 166.333822] kmem_cache_free+0x7b/0x172
> [ 166.333826] ? __free_iova+0x23/0x31
> [ 166.333831] __free_iova+0x23/0x31
> [ 166.333840] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> [ 166.333851] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 166.333861] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> [ 166.333872] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> [ 166.333885] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> [ 166.333896] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> [ 166.333908] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> [ 166.333917] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> [ 166.333923] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> [ 166.333932] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> [ 166.333950] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.333970] ? copy_overflow+0x14/0x14 [videodev]
> [ 166.333974] ? slab_free_freelist_hook+0x46/0x94
> [ 166.333979] ? kfree+0x107/0x1a0
> [ 166.333997] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.334015] ? copy_overflow+0x14/0x14 [videodev]
> [ 166.334031] ? v4l_enumstd+0x49/0x49 [videodev]
> [ 166.334035] ? __wake_up_common+0x342/0x342
> [ 166.334042] ? atomic_long_add_return+0x15/0x24
> [ 166.334046] ? ldsem_up_read+0x15/0x29
> [ 166.334050] ? tty_write+0x4c6/0x4d8
> [ 166.334054] ? n_tty_receive_char_special+0x1152/0x1152
> [ 166.334071] ? video_usercopy+0x8ae/0x8ae [videodev]
> [ 166.334087] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.334092] vfs_ioctl+0x76/0x89
> [ 166.334097] do_vfs_ioctl+0xb33/0xb7e
> [ 166.334101] ? __switch_to_asm+0x40/0x70
> [ 166.334105] ? __switch_to_asm+0x40/0x70
> [ 166.334108] ? __switch_to_asm+0x34/0x70
> [ 166.334111] ? __switch_to_asm+0x40/0x70
> [ 166.334116] ? ioctl_preallocate+0x174/0x174
> [ 166.334120] ? __switch_to+0x71c/0xb00
> [ 166.334124] ? compat_start_thread+0x6b/0x6b
> [ 166.334127] ? __switch_to_asm+0x34/0x70
> [ 166.334130] ? __switch_to_asm+0x40/0x70
> [ 166.334134] ? mmdrop+0x12/0x23
> [ 166.334137] ? finish_task_switch+0x34d/0x3de
> [ 166.334143] ? __schedule+0x1004/0x1045
> [ 166.334148] ? firmware_map_remove+0x119/0x119
> [ 166.334153] ksys_ioctl+0x50/0x70
> [ 166.334158] __x64_sys_ioctl+0x82/0x89
> [ 166.334163] do_syscall_64+0xa0/0xd2
> [ 166.334167] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 166.334171] RIP: 0033:0x7f2481541f47
> [ 166.334175] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 166.334177] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [ 166.334181] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> [ 166.334184] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> [ 166.334186] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> [ 166.334189] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> [ 166.334191] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
>
> [ 166.334201] Allocated by task 1305:
> [ 166.334207] kasan_kmalloc+0x8a/0x98
> [ 166.334210] slab_post_alloc_hook+0x31/0x51
> [ 166.334213] kmem_cache_alloc+0xd7/0x174
> [ 166.334216] alloc_iova+0x24/0x2ea
> [ 166.334225] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
> [ 166.334233] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> [ 166.334241] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> [ 166.334250] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> [ 166.334259] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> [ 166.334266] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> [ 166.334273] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> [ 166.334288] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.334304] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.334319] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.334322] vfs_ioctl+0x76/0x89
> [ 166.334325] do_vfs_ioctl+0xb33/0xb7e
> [ 166.334328] ksys_ioctl+0x50/0x70
> [ 166.334332] __x64_sys_ioctl+0x82/0x89
> [ 166.334335] do_syscall_64+0xa0/0xd2
> [ 166.334338] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [ 166.334343] Freed by task 1301:
> [ 166.334349] __kasan_slab_free+0xfa/0x11c
> [ 166.334352] slab_free_freelist_hook+0x46/0x94
> [ 166.334355] kmem_cache_free+0x7b/0x172
> [ 166.334359] __free_iova+0x23/0x31
> [ 166.334367] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> [ 166.334375] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 166.334383] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> [ 166.334392] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> [ 166.334401] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> [ 166.334410] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> [ 166.334416] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> [ 166.334423] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> [ 166.334438] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.334454] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.334469] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.334472] vfs_ioctl+0x76/0x89
> [ 166.334475] do_vfs_ioctl+0xb33/0xb7e
> [ 166.334479] ksys_ioctl+0x50/0x70
> [ 166.334482] __x64_sys_ioctl+0x82/0x89
> [ 166.334485] do_syscall_64+0xa0/0xd2
> [ 166.334488] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [ 166.334494] The buggy address belongs to the object at ffff888133823700
> which belongs to the cache iommu_iova of size 40
> [ 166.334501] The buggy address is located 0 bytes inside of
> 40-byte region [ffff888133823700, ffff888133823728)
> [ 166.334506] The buggy address belongs to the page:
> [ 166.334511] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
> [ 166.334517] flags: 0x200000000010200(slab|head)
> [ 166.334524] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
> [ 166.334530] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
> [ 166.334533] page dumped because: kasan: bad access detected
>
> [ 166.334539] Memory state around the buggy address:
> [ 166.334544] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.334549] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.334554] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
> [ 166.334558] ^
> [ 166.334562] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.334567] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.334571] ==================================================================
> [ 166.340377] ==================================================================
> [ 166.340388] BUG: KASAN: double-free or invalid-free in kfree+0x107/0x1a0
>
> [ 166.340399] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
> [ 166.340401] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 166.340403] Call Trace:
> [ 166.340410] dump_stack+0x5b/0x81
> [ 166.340416] print_address_description+0x65/0x227
> [ 166.340420] ? kfree+0x107/0x1a0
> [ 166.340425] kasan_report_invalid_free+0x67/0xa0
> [ 166.340428] ? kfree+0x107/0x1a0
> [ 166.340433] __kasan_slab_free+0x86/0x11c
> [ 166.340438] slab_free_freelist_hook+0x46/0x94
> [ 166.340443] kfree+0x107/0x1a0
> [ 166.340454] ? ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> [ 166.340464] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> [ 166.340475] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 166.340485] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> [ 166.340495] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> [ 166.340509] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> [ 166.340520] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> [ 166.340531] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> [ 166.340541] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> [ 166.340548] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> [ 166.340557] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> [ 166.340575] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.340595] ? copy_overflow+0x14/0x14 [videodev]
> [ 166.340600] ? slab_free_freelist_hook+0x46/0x94
> [ 166.340604] ? kfree+0x107/0x1a0
> [ 166.340622] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.340640] ? copy_overflow+0x14/0x14 [videodev]
> [ 166.340657] ? v4l_enumstd+0x49/0x49 [videodev]
> [ 166.340660] ? __wake_up_common+0x342/0x342
> [ 166.340668] ? atomic_long_add_return+0x15/0x24
> [ 166.340672] ? ldsem_up_read+0x15/0x29
> [ 166.340677] ? tty_write+0x4c6/0x4d8
> [ 166.340681] ? n_tty_receive_char_special+0x1152/0x1152
> [ 166.340698] ? video_usercopy+0x8ae/0x8ae [videodev]
> [ 166.340714] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.340720] vfs_ioctl+0x76/0x89
> [ 166.340725] do_vfs_ioctl+0xb33/0xb7e
> [ 166.340729] ? __switch_to_asm+0x40/0x70
> [ 166.340733] ? __switch_to_asm+0x40/0x70
> [ 166.340736] ? __switch_to_asm+0x34/0x70
> [ 166.340739] ? __switch_to_asm+0x40/0x70
> [ 166.340743] ? ioctl_preallocate+0x174/0x174
> [ 166.340748] ? __switch_to+0x71c/0xb00
> [ 166.340752] ? compat_start_thread+0x6b/0x6b
> [ 166.340756] ? __switch_to_asm+0x34/0x70
> [ 166.340759] ? __switch_to_asm+0x40/0x70
> [ 166.340762] ? mmdrop+0x12/0x23
> [ 166.340766] ? finish_task_switch+0x34d/0x3de
> [ 166.340772] ? __schedule+0x1004/0x1045
> [ 166.340777] ? firmware_map_remove+0x119/0x119
> [ 166.340782] ksys_ioctl+0x50/0x70
> [ 166.340788] __x64_sys_ioctl+0x82/0x89
> [ 166.340793] do_syscall_64+0xa0/0xd2
> [ 166.340797] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 166.340802] RIP: 0033:0x7f2481541f47
> [ 166.340806] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 166.340809] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [ 166.340813] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> [ 166.340816] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> [ 166.340819] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> [ 166.340821] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> [ 166.340824] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
>
> [ 166.340834] Allocated by task 1305:
> [ 166.340840] kasan_kmalloc+0x8a/0x98
> [ 166.340844] __kmalloc_node+0x193/0x1ba
> [ 166.340848] kvmalloc_node+0x44/0x6d
> [ 166.340856] ipu3_dmamap_alloc+0x1c9/0x83f [ipu3_imgu]
> [ 166.340864] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> [ 166.340873] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> [ 166.340882] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> [ 166.340891] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> [ 166.340897] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> [ 166.340904] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> [ 166.340920] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.340935] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.340950] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.340954] vfs_ioctl+0x76/0x89
> [ 166.340957] do_vfs_ioctl+0xb33/0xb7e
> [ 166.340960] ksys_ioctl+0x50/0x70
> [ 166.340963] __x64_sys_ioctl+0x82/0x89
> [ 166.340966] do_syscall_64+0xa0/0xd2
> [ 166.340969] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [ 166.340974] Freed by task 1301:
> [ 166.340980] __kasan_slab_free+0xfa/0x11c
> [ 166.340983] slab_free_freelist_hook+0x46/0x94
> [ 166.340986] kfree+0x107/0x1a0
> [ 166.340994] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> [ 166.341002] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 166.341010] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> [ 166.341019] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> [ 166.341028] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> [ 166.341037] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> [ 166.341043] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> [ 166.341050] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> [ 166.341066] __video_do_ioctl+0x625/0x887 [videodev]
> [ 166.341081] video_usercopy+0x3a3/0x8ae [videodev]
> [ 166.341096] v4l2_ioctl+0xb7/0xc5 [videodev]
> [ 166.341100] vfs_ioctl+0x76/0x89
> [ 166.341103] do_vfs_ioctl+0xb33/0xb7e
> [ 166.341106] ksys_ioctl+0x50/0x70
> [ 166.341109] __x64_sys_ioctl+0x82/0x89
> [ 166.341112] do_syscall_64+0xa0/0xd2
> [ 166.341116] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [ 166.341122] The buggy address belongs to the object at ffff88811d228440
> which belongs to the cache kmalloc-8 of size 8
> [ 166.341129] The buggy address is located 0 bytes inside of
> 8-byte region [ffff88811d228440, ffff88811d228448)
> [ 166.341134] The buggy address belongs to the page:
> [ 166.341140] page:ffffea0004748a00 count:1 mapcount:0 mapping:ffff88815a80c340 index:0xffff88811d228f80 compound_mapcount: 0
> [ 166.341146] flags: 0x200000000010200(slab|head)
> [ 166.341153] raw: 0200000000010200 ffffea000564b288 ffffea00049ef708 ffff88815a80c340
> [ 166.341159] raw: ffff88811d228f80 0000000000160013 00000001ffffffff 0000000000000000
> [ 166.341163] page dumped because: kasan: bad access detected
>
> [ 166.341169] Memory state around the buggy address:
> [ 166.341174] ffff88811d228300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.341179] ffff88811d228380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.341184] >ffff88811d228400: fc fc fc fc fc fc fc fc fb fc fc fc fc fc fc fc
> [ 166.341188] ^
> [ 166.341192] ffff88811d228480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.341197] ffff88811d228500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> [ 166.341201] ==================================================================
>
> > On Tuesday, 11 December 2018 16:20:43 EET Laurent Pinchart wrote:
> > > On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> > > > On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > >> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > >>
> > > >> [snip]
> > > >>
> > > >>> I can see a couple of steps missing in the script below.
> > > >>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
> > > >>> 00040.html)
> > > >>>
> > > >>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > >>> documentation",
> > > >>> under section "Configuring ImgU V4L2 subdev for image processing"...
> > > >>>
> > > >>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > >>>
> > > >>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > >>> desired (e.g 0 for video mode or 1 for still mode) through the control
> > > >>> id 0x009819a1 as below.
> > > >>>
> > > >>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > >>
> > > >> I assume the control takes a valid default value ? It's better to set
> > > >> it
> > > >> explicitly anyway, so I'll do so.
> > > >>
> > > >>> 2. ImgU pipeline needs to be configured for image processing as below.
> > > >>>
> > > >>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > >>> have the processed image output to the DDR memory.
> > > >>>
> > > >>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > >>> Geometric Distortion Correction (GDC) -> DDR
> > > >>>
> > > >>> The ImgU V4L2 subdev has to be configured with the supported
> > > >>> resolutions in all the above HW blocks, for a given input resolution.
> > > >>>
> > > >>> For a given supported resolution for an input frame, the Input Feeder,
> > > >>> Bayer Down Scaling and GDC blocks should be configured with the
> > > >>> supported resolutions. This information can be obtained by looking at
> > > >>> the following IPU3 ISP configuration table for ov5670 sensor.
> > > >>>
> > > >>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> > > >>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files
> > > >>> /
> > > >>> gcss/graph_settings_ov5670.xml
> > > >>>
> > > >>> For the ov5670 example, for an input frame with a resolution of
> > > >>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > > >>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > > >>> 2560x1920 respectively.
> > > >>
> > > >> How is the GDC output resolution computed from the input resolution ?
> > > >> Does the GDC always consume 32 columns and 22 lines ?
> > > >>
> > > >>> The following steps prepare the ImgU ISP pipeline for the image
> > > >>> processing.
> > > >>>
> > > >>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > >>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > > >>> above.
> > > >>
> > > >> If I understand things correctly, the GDC resolution is the pipeline
> > > >> output resolution. Why is it configured on pad 0 ?
> > > >>
> > > >>> 2. The ImgU V4L2 subdev cropping should be set by using the
> > > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > > >>> target, using the input feeder height and width.
> > > >>>
> > > >>> 3. The ImgU V4L2 subdev composing should be set by using the
> > > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > > >>> target, using the BDS height and width.
> > > >>>
> > > >>> Once these 2 steps are done, the raw bayer frames can be input to the
> > > >>> ImgU V4L2 subdev for processing.
> > > >>
> > > >> Do I need to capture from both the output and viewfinder nodes ? How
> > > >> are
> > > >> they related to the IF -> BDS -> GDC pipeline, are they both fed from
> > > >> the GDC output ? If so, how does the viewfinder scaler fit in that
> > > >> picture ?
> > > >>
> > > >> I have tried the above configuration with the IPU3 v8 driver, and while
> > > >> the kernel doesn't crash, no images get processed. The userspace
> > > >> processes wait forever for buffers to be ready. I then configured pad 2
> > > >> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
> > > >>
> > > >> There's one problem though: during capture, or very soon after it, the
> > > >> machine locks up completely. I suspect a memory corruption, as when it
> > > >> doesn't log immediately commands such as dmesg will not produce any
> > > >> output and just block, until the system freezes soon after (especially
> > > >> when moving the mouse).
> > > >>
> > > >> I would still call this an improvement to some extent, but there's
> > > >> definitely room for more improvements :-)
> > > >>
> > > >> To reproduce the issue, you can run the ipu3-process.sh script
> > > >> (attached
> > > >> to this e-mail) with the following arguments:
> > > >>
> > > >> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
> > >
> > > This should have read
> > >
> > > $ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
> > >
> > > Without the --vf argument no images are processed.
> > >
> > > It seems that the Intel mail server blocked the mail that contained the
> > > script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
> > >
> > > >> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
> > > >> the IPU3-specific Bayer format (for a total of 6469632 bytes).
> > > >
> > > > I managed to get the dmesg output, and it doesn't look pretty.
> > > >
> > > > [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> > > > libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> > > > ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > > [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > > mac80211
> > > > iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp
> > > > cfg80211
> > > > 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > > videobuf2_memops videobuf2_v4l2 videobuf2_common
> > > > processor_thermal_device
> > > > intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common
> > > > videodev
> > > > at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs
> > > > cros_ec_core
> > > > int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> > > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > > sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> > > > drm_panel_orientation_quirks i2c_hid hid
> > > > [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> > > > 4.20.0-rc6+ #2
> > > > [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > > [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > > [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc
> > > > f3
> > > > 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> > > > <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> > > > [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> > > > [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> > > > 000000000000000c
> > > > [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> > > > 00000000ffffffff
> > > > [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> > > > ffff8f5cfaba16f0
> > > > [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> > > > ffff8f5cf58f0028
> > > > [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> > > > ffff8f5cf58f04e8
> > > > [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
> > > > knlGS:
> > > > 0000000000000000
> > > > [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> > > > 00000000003606e0
> > > > [ 571.217301] Call Trace:
> > > > [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> > > > [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > > [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> > > > [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> > > > [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> > > > [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> > > > [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> > > > [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> > > > [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> > > > [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> > > > [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> > > > [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> > > > [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> > > > [ 571.217480] vfs_ioctl+0x1e/0x2b
> > > > [ 571.217486] do_vfs_ioctl+0x531/0x559
> > > > [ 571.217494] ? vfs_write+0xd1/0xdf
> > > > [ 571.217500] ksys_ioctl+0x50/0x70
> > > > [ 571.217506] __x64_sys_ioctl+0x16/0x19
> > > > [ 571.217512] do_syscall_64+0x53/0x60
> > > > [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> > > > [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00
> > > > 48
> > > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > > [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> > > > 0000000000000010
> > > > [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> > > > 00007f85cf9b9f47
> > > > [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> > > > 0000000000000003
> > > > [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> > > > 00007f85d009c700
> > > > [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> > > > 000055f4c4dc0b06
> > > > [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> > > > 00007ffc59057825
> > > > [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> > > > [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
> > >
> > > And after fixing another issue in the capture script (which was setting
> > > the
> > > format on the ImgU subdev pad 3 to 2560x1920 but capture in 1920x1080), I
> > > now get plenty of the following messages:
> > >
> > > [ 221.366131] BUG: Bad page state in process yavta pfn:14a4ff
> > > [ 221.366134] page:ffffde5d45293fc0 count:-1 mapcount:0 mapping:
> > > 0000000000000000 index:0x0
> > > [ 221.366137] flags: 0x200000000000000()
> > > [ 221.366140] raw: 0200000000000000 dead000000000100 dead000000000200
> > > 0000000000000000
> > > [ 221.366143] raw: 0000000000000000 0000000000000000 ffffffffffffffff
> > > 0000000000000000
> > > [ 221.366145] page dumped because: nonzero _refcount
> > > [ 221.366147] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp mac80211 iwlwifi
> > > cfg80211 hid_multitouch 8250_dw ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > videobuf2_memops videobuf2_v4l2 processor_thermal_device videobuf2_common
> > > intel_soc_dts_iosf ov13858 ov5670 dw9714 v4l2_fwnode v4l2_common videodev
> > > media at24 cros_ec_lpcs cros_ec_core int3403_thermal int340x_thermal_zone
> > > chromeos_pstore mac_hid int3400_thermal acpi_thermal_rel autofs4 usbhid
> > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > sysfillrect sysimgblt fb_sys_fops sdhci_pci cqhci sdhci drm
> > > drm_panel_orientation_quirks i2c_hid hid
> > > [ 221.366172] CPU: 3 PID: 1022 Comm: yavta Tainted: G B WC
> > > 4.20.0-rc6+ #2
> > > [ 221.366173] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > [ 221.366173] Call Trace:
> > > [ 221.366176] dump_stack+0x46/0x59
> > > [ 221.366179] bad_page+0xf2/0x10c
> > > [ 221.366182] free_pages_check+0x78/0x81
> > > [ 221.366186] free_pcppages_bulk+0xa6/0x236
> > > [ 221.366190] free_unref_page+0x4b/0x53
> > > [ 221.366193] vb2_dma_sg_put+0x95/0xb5 [videobuf2_dma_sg]
> > > [ 221.366197] __vb2_buf_mem_free+0x3a/0x6e [videobuf2_common]
> > > [ 221.366202] __vb2_queue_free+0xe3/0x1be [videobuf2_common]
> > > [ 221.366207] vb2_core_reqbufs+0xe9/0x2cc [videobuf2_common]
> > > [ 221.366212] vb2_ioctl_reqbufs+0x78/0x9e [videobuf2_v4l2]
> > > [ 221.366220] __video_do_ioctl+0x258/0x38e [videodev]
> > > [ 221.366229] video_usercopy+0x25f/0x4e5 [videodev]
> > > [ 221.366237] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 221.366240] ? unmap_region+0xe0/0x10a
> > > [ 221.366250] v4l2_ioctl+0x4d/0x58 [videodev]
> > > [ 221.366253] vfs_ioctl+0x1e/0x2b
> > > [ 221.366255] do_vfs_ioctl+0x531/0x559
> > > [ 221.366260] ksys_ioctl+0x50/0x70
> > > [ 221.366263] __x64_sys_ioctl+0x16/0x19
> > > [ 221.366266] do_syscall_64+0x53/0x60
> > > [ 221.366269] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [ 221.366270] RIP: 0033:0x7fbe39f6af47
> > > [ 221.366272] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > [ 221.366273] RSP: 002b:00007fff05638e68 EFLAGS: 00000246 ORIG_RAX:
> > > 0000000000000010
> > > [ 221.366275] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > > 00007fbe39f6af47
> > > [ 221.366279] RDX: 00007fff05638f90 RSI: 00000000c0145608 RDI:
> > > 0000000000000003
> > > [ 221.366283] RBP: 0000000000000004 R08: 0000000000000000 R09:
> > > 0000000000000045
> > > [ 221.366287] R10: 0000000000000557 R11: 0000000000000246 R12:
> > > 000055c83bd76750
> > > [ 221.366290] R13: 000055c83b6b26a0 R14: 0000000000000001 R15:
> > > 00007fff0563a825
>
> --
> Regards,
>
> Laurent Pinchart
>
>
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-21 3:04 ` Tomasz Figa
@ 2019-01-08 6:54 ` Tomasz Figa
2019-01-09 16:40 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Tomasz Figa @ 2019-01-08 6:54 UTC (permalink / raw)
To: Yong Zhi, Mani, Rajmohan, Qiu, Tian Shu, Cao Bing Bu
Cc: Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
Hi Raj, Yong, Bingbu, Tianshu,
On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa <tfiga@chromium.org> wrote:
>
> On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> >
> > Hellon
> >
> > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > > Hello Yong,
> > >
> > > Could you please have a look at the crash reported below ?
> >
> > A bit more information to help you debugging this. I've enabled KASAN in the
> > kernel configuration, and get the following use-after-free reports.
> >
> > [ 166.332920] ==================================================================
> > [ 166.332937] BUG: KASAN: use-after-free in __cached_rbnode_delete_update+0x36/0x202
> > [ 166.332944] Read of size 8 at addr ffff888133823718 by task yavta/1305
> >
> > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C 4.20.0-rc6+ #3
> > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 166.332959] Call Trace:
> > [ 166.332967] dump_stack+0x5b/0x81
> > [ 166.332974] print_address_description+0x65/0x227
> > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > [ 166.332983] kasan_report+0x247/0x285
> > [ 166.332989] __cached_rbnode_delete_update+0x36/0x202
> > [ 166.332995] private_free_iova+0x57/0x6d
> > [ 166.332999] __free_iova+0x23/0x31
> > [ 166.333011] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
>
> Thanks Laurent, I think this is a very good hint. It looks like we're
> basically freeing and already freed IOVA and corrupting some allocator
> state?
Did you have any luck in reproducing and fixing this double free issue?
Best regards,
Tomasz
>
> > [ 166.333022] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 166.333032] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > [ 166.333043] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > [ 166.333056] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > [ 166.333067] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> > [ 166.333079] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > [ 166.333088] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > [ 166.333096] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> > [ 166.333104] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > [ 166.333123] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.333142] ? copy_overflow+0x14/0x14 [videodev]
> > [ 166.333147] ? slab_free_freelist_hook+0x46/0x94
> > [ 166.333151] ? kfree+0x107/0x1a0
> > [ 166.333169] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.333187] ? copy_overflow+0x14/0x14 [videodev]
> > [ 166.333203] ? v4l_enumstd+0x49/0x49 [videodev]
> > [ 166.333207] ? __wake_up_common+0x342/0x342
> > [ 166.333215] ? atomic_long_add_return+0x15/0x24
> > [ 166.333219] ? ldsem_up_read+0x15/0x29
> > [ 166.333223] ? tty_write+0x4c6/0x4d8
> > [ 166.333227] ? n_tty_receive_char_special+0x1152/0x1152
> > [ 166.333244] ? video_usercopy+0x8ae/0x8ae [videodev]
> > [ 166.333260] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.333266] vfs_ioctl+0x76/0x89
> > [ 166.333271] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.333275] ? __switch_to_asm+0x40/0x70
> > [ 166.333279] ? __switch_to_asm+0x40/0x70
> > [ 166.333282] ? __switch_to_asm+0x34/0x70
> > [ 166.333286] ? __switch_to_asm+0x40/0x70
> > [ 166.333290] ? ioctl_preallocate+0x174/0x174
> > [ 166.333294] ? __switch_to+0x71c/0xb00
> > [ 166.333299] ? compat_start_thread+0x6b/0x6b
> > [ 166.333302] ? __switch_to_asm+0x34/0x70
> > [ 166.333305] ? __switch_to_asm+0x40/0x70
> > [ 166.333309] ? mmdrop+0x12/0x23
> > [ 166.333313] ? finish_task_switch+0x34d/0x3de
> > [ 166.333319] ? __schedule+0x1004/0x1045
> > [ 166.333325] ? firmware_map_remove+0x119/0x119
> > [ 166.333330] ksys_ioctl+0x50/0x70
> > [ 166.333335] __x64_sys_ioctl+0x82/0x89
> > [ 166.333340] do_syscall_64+0xa0/0xd2
> > [ 166.333345] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 166.333349] RIP: 0033:0x7f2481541f47
> > [ 166.333354] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 166.333357] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > [ 166.333362] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> > [ 166.333364] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> > [ 166.333367] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> > [ 166.333369] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> > [ 166.333372] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> >
> > [ 166.333383] Allocated by task 1305:
> > [ 166.333389] kasan_kmalloc+0x8a/0x98
> > [ 166.333392] slab_post_alloc_hook+0x31/0x51
> > [ 166.333396] kmem_cache_alloc+0xd7/0x174
> > [ 166.333399] alloc_iova+0x24/0x2ea
> > [ 166.333407] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
> > [ 166.333415] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> > [ 166.333424] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> > [ 166.333433] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> > [ 166.333442] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> > [ 166.333449] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> > [ 166.333455] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> > [ 166.333471] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.333487] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.333501] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.333505] vfs_ioctl+0x76/0x89
> > [ 166.333508] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.333511] ksys_ioctl+0x50/0x70
> > [ 166.333514] __x64_sys_ioctl+0x82/0x89
> > [ 166.333518] do_syscall_64+0xa0/0xd2
> > [ 166.333521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > [ 166.333526] Freed by task 1301:
> > [ 166.333532] __kasan_slab_free+0xfa/0x11c
> > [ 166.333535] slab_free_freelist_hook+0x46/0x94
> > [ 166.333538] kmem_cache_free+0x7b/0x172
> > [ 166.333542] __free_iova+0x23/0x31
> > [ 166.333550] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > [ 166.333557] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 166.333566] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > [ 166.333574] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > [ 166.333584] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > [ 166.333593] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > [ 166.333599] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > [ 166.333606] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > [ 166.333621] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.333637] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.333652] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.333655] vfs_ioctl+0x76/0x89
> > [ 166.333658] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.333662] ksys_ioctl+0x50/0x70
> > [ 166.333665] __x64_sys_ioctl+0x82/0x89
> > [ 166.333668] do_syscall_64+0xa0/0xd2
> > [ 166.333671] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > [ 166.333678] The buggy address belongs to the object at ffff888133823700
> > which belongs to the cache iommu_iova of size 40
> > [ 166.333685] The buggy address is located 24 bytes inside of
> > 40-byte region [ffff888133823700, ffff888133823728)
> > [ 166.333690] The buggy address belongs to the page:
> > [ 166.333696] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
> > [ 166.333703] flags: 0x200000000010200(slab|head)
> > [ 166.333710] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
> > [ 166.333717] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
> > [ 166.333720] page dumped because: kasan: bad access detected
> >
> > [ 166.333726] Memory state around the buggy address:
> > [ 166.333732] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.333737] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.333742] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.333745] ^
> > [ 166.333750] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.333755] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.333759] ==================================================================
> > [ 166.333762] Disabling lock debugging due to kernel taint
> > [ 166.333764] ==================================================================
> > [ 166.333770] BUG: KASAN: double-free or invalid-free in kmem_cache_free+0x7b/0x172
> >
> > [ 166.333780] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
> > [ 166.333782] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 166.333783] Call Trace:
> > [ 166.333789] dump_stack+0x5b/0x81
> > [ 166.333795] print_address_description+0x65/0x227
> > [ 166.333799] ? kmem_cache_free+0x7b/0x172
> > [ 166.333803] kasan_report_invalid_free+0x67/0xa0
> > [ 166.333807] ? kmem_cache_free+0x7b/0x172
> > [ 166.333812] __kasan_slab_free+0x86/0x11c
> > [ 166.333817] slab_free_freelist_hook+0x46/0x94
> > [ 166.333822] kmem_cache_free+0x7b/0x172
> > [ 166.333826] ? __free_iova+0x23/0x31
> > [ 166.333831] __free_iova+0x23/0x31
> > [ 166.333840] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > [ 166.333851] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 166.333861] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > [ 166.333872] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > [ 166.333885] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > [ 166.333896] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> > [ 166.333908] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > [ 166.333917] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > [ 166.333923] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> > [ 166.333932] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > [ 166.333950] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.333970] ? copy_overflow+0x14/0x14 [videodev]
> > [ 166.333974] ? slab_free_freelist_hook+0x46/0x94
> > [ 166.333979] ? kfree+0x107/0x1a0
> > [ 166.333997] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.334015] ? copy_overflow+0x14/0x14 [videodev]
> > [ 166.334031] ? v4l_enumstd+0x49/0x49 [videodev]
> > [ 166.334035] ? __wake_up_common+0x342/0x342
> > [ 166.334042] ? atomic_long_add_return+0x15/0x24
> > [ 166.334046] ? ldsem_up_read+0x15/0x29
> > [ 166.334050] ? tty_write+0x4c6/0x4d8
> > [ 166.334054] ? n_tty_receive_char_special+0x1152/0x1152
> > [ 166.334071] ? video_usercopy+0x8ae/0x8ae [videodev]
> > [ 166.334087] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.334092] vfs_ioctl+0x76/0x89
> > [ 166.334097] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.334101] ? __switch_to_asm+0x40/0x70
> > [ 166.334105] ? __switch_to_asm+0x40/0x70
> > [ 166.334108] ? __switch_to_asm+0x34/0x70
> > [ 166.334111] ? __switch_to_asm+0x40/0x70
> > [ 166.334116] ? ioctl_preallocate+0x174/0x174
> > [ 166.334120] ? __switch_to+0x71c/0xb00
> > [ 166.334124] ? compat_start_thread+0x6b/0x6b
> > [ 166.334127] ? __switch_to_asm+0x34/0x70
> > [ 166.334130] ? __switch_to_asm+0x40/0x70
> > [ 166.334134] ? mmdrop+0x12/0x23
> > [ 166.334137] ? finish_task_switch+0x34d/0x3de
> > [ 166.334143] ? __schedule+0x1004/0x1045
> > [ 166.334148] ? firmware_map_remove+0x119/0x119
> > [ 166.334153] ksys_ioctl+0x50/0x70
> > [ 166.334158] __x64_sys_ioctl+0x82/0x89
> > [ 166.334163] do_syscall_64+0xa0/0xd2
> > [ 166.334167] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 166.334171] RIP: 0033:0x7f2481541f47
> > [ 166.334175] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 166.334177] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > [ 166.334181] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> > [ 166.334184] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> > [ 166.334186] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> > [ 166.334189] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> > [ 166.334191] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> >
> > [ 166.334201] Allocated by task 1305:
> > [ 166.334207] kasan_kmalloc+0x8a/0x98
> > [ 166.334210] slab_post_alloc_hook+0x31/0x51
> > [ 166.334213] kmem_cache_alloc+0xd7/0x174
> > [ 166.334216] alloc_iova+0x24/0x2ea
> > [ 166.334225] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
> > [ 166.334233] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> > [ 166.334241] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> > [ 166.334250] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> > [ 166.334259] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> > [ 166.334266] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> > [ 166.334273] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> > [ 166.334288] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.334304] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.334319] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.334322] vfs_ioctl+0x76/0x89
> > [ 166.334325] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.334328] ksys_ioctl+0x50/0x70
> > [ 166.334332] __x64_sys_ioctl+0x82/0x89
> > [ 166.334335] do_syscall_64+0xa0/0xd2
> > [ 166.334338] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > [ 166.334343] Freed by task 1301:
> > [ 166.334349] __kasan_slab_free+0xfa/0x11c
> > [ 166.334352] slab_free_freelist_hook+0x46/0x94
> > [ 166.334355] kmem_cache_free+0x7b/0x172
> > [ 166.334359] __free_iova+0x23/0x31
> > [ 166.334367] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > [ 166.334375] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 166.334383] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > [ 166.334392] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > [ 166.334401] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > [ 166.334410] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > [ 166.334416] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > [ 166.334423] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > [ 166.334438] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.334454] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.334469] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.334472] vfs_ioctl+0x76/0x89
> > [ 166.334475] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.334479] ksys_ioctl+0x50/0x70
> > [ 166.334482] __x64_sys_ioctl+0x82/0x89
> > [ 166.334485] do_syscall_64+0xa0/0xd2
> > [ 166.334488] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > [ 166.334494] The buggy address belongs to the object at ffff888133823700
> > which belongs to the cache iommu_iova of size 40
> > [ 166.334501] The buggy address is located 0 bytes inside of
> > 40-byte region [ffff888133823700, ffff888133823728)
> > [ 166.334506] The buggy address belongs to the page:
> > [ 166.334511] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
> > [ 166.334517] flags: 0x200000000010200(slab|head)
> > [ 166.334524] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
> > [ 166.334530] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
> > [ 166.334533] page dumped because: kasan: bad access detected
> >
> > [ 166.334539] Memory state around the buggy address:
> > [ 166.334544] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.334549] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.334554] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.334558] ^
> > [ 166.334562] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.334567] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.334571] ==================================================================
> > [ 166.340377] ==================================================================
> > [ 166.340388] BUG: KASAN: double-free or invalid-free in kfree+0x107/0x1a0
> >
> > [ 166.340399] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
> > [ 166.340401] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 166.340403] Call Trace:
> > [ 166.340410] dump_stack+0x5b/0x81
> > [ 166.340416] print_address_description+0x65/0x227
> > [ 166.340420] ? kfree+0x107/0x1a0
> > [ 166.340425] kasan_report_invalid_free+0x67/0xa0
> > [ 166.340428] ? kfree+0x107/0x1a0
> > [ 166.340433] __kasan_slab_free+0x86/0x11c
> > [ 166.340438] slab_free_freelist_hook+0x46/0x94
> > [ 166.340443] kfree+0x107/0x1a0
> > [ 166.340454] ? ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> > [ 166.340464] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> > [ 166.340475] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 166.340485] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > [ 166.340495] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > [ 166.340509] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > [ 166.340520] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> > [ 166.340531] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > [ 166.340541] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > [ 166.340548] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> > [ 166.340557] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > [ 166.340575] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.340595] ? copy_overflow+0x14/0x14 [videodev]
> > [ 166.340600] ? slab_free_freelist_hook+0x46/0x94
> > [ 166.340604] ? kfree+0x107/0x1a0
> > [ 166.340622] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.340640] ? copy_overflow+0x14/0x14 [videodev]
> > [ 166.340657] ? v4l_enumstd+0x49/0x49 [videodev]
> > [ 166.340660] ? __wake_up_common+0x342/0x342
> > [ 166.340668] ? atomic_long_add_return+0x15/0x24
> > [ 166.340672] ? ldsem_up_read+0x15/0x29
> > [ 166.340677] ? tty_write+0x4c6/0x4d8
> > [ 166.340681] ? n_tty_receive_char_special+0x1152/0x1152
> > [ 166.340698] ? video_usercopy+0x8ae/0x8ae [videodev]
> > [ 166.340714] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.340720] vfs_ioctl+0x76/0x89
> > [ 166.340725] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.340729] ? __switch_to_asm+0x40/0x70
> > [ 166.340733] ? __switch_to_asm+0x40/0x70
> > [ 166.340736] ? __switch_to_asm+0x34/0x70
> > [ 166.340739] ? __switch_to_asm+0x40/0x70
> > [ 166.340743] ? ioctl_preallocate+0x174/0x174
> > [ 166.340748] ? __switch_to+0x71c/0xb00
> > [ 166.340752] ? compat_start_thread+0x6b/0x6b
> > [ 166.340756] ? __switch_to_asm+0x34/0x70
> > [ 166.340759] ? __switch_to_asm+0x40/0x70
> > [ 166.340762] ? mmdrop+0x12/0x23
> > [ 166.340766] ? finish_task_switch+0x34d/0x3de
> > [ 166.340772] ? __schedule+0x1004/0x1045
> > [ 166.340777] ? firmware_map_remove+0x119/0x119
> > [ 166.340782] ksys_ioctl+0x50/0x70
> > [ 166.340788] __x64_sys_ioctl+0x82/0x89
> > [ 166.340793] do_syscall_64+0xa0/0xd2
> > [ 166.340797] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 166.340802] RIP: 0033:0x7f2481541f47
> > [ 166.340806] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 166.340809] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > [ 166.340813] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> > [ 166.340816] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> > [ 166.340819] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> > [ 166.340821] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> > [ 166.340824] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> >
> > [ 166.340834] Allocated by task 1305:
> > [ 166.340840] kasan_kmalloc+0x8a/0x98
> > [ 166.340844] __kmalloc_node+0x193/0x1ba
> > [ 166.340848] kvmalloc_node+0x44/0x6d
> > [ 166.340856] ipu3_dmamap_alloc+0x1c9/0x83f [ipu3_imgu]
> > [ 166.340864] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> > [ 166.340873] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> > [ 166.340882] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> > [ 166.340891] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> > [ 166.340897] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> > [ 166.340904] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> > [ 166.340920] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.340935] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.340950] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.340954] vfs_ioctl+0x76/0x89
> > [ 166.340957] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.340960] ksys_ioctl+0x50/0x70
> > [ 166.340963] __x64_sys_ioctl+0x82/0x89
> > [ 166.340966] do_syscall_64+0xa0/0xd2
> > [ 166.340969] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > [ 166.340974] Freed by task 1301:
> > [ 166.340980] __kasan_slab_free+0xfa/0x11c
> > [ 166.340983] slab_free_freelist_hook+0x46/0x94
> > [ 166.340986] kfree+0x107/0x1a0
> > [ 166.340994] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> > [ 166.341002] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 166.341010] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > [ 166.341019] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > [ 166.341028] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > [ 166.341037] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > [ 166.341043] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > [ 166.341050] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > [ 166.341066] __video_do_ioctl+0x625/0x887 [videodev]
> > [ 166.341081] video_usercopy+0x3a3/0x8ae [videodev]
> > [ 166.341096] v4l2_ioctl+0xb7/0xc5 [videodev]
> > [ 166.341100] vfs_ioctl+0x76/0x89
> > [ 166.341103] do_vfs_ioctl+0xb33/0xb7e
> > [ 166.341106] ksys_ioctl+0x50/0x70
> > [ 166.341109] __x64_sys_ioctl+0x82/0x89
> > [ 166.341112] do_syscall_64+0xa0/0xd2
> > [ 166.341116] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > [ 166.341122] The buggy address belongs to the object at ffff88811d228440
> > which belongs to the cache kmalloc-8 of size 8
> > [ 166.341129] The buggy address is located 0 bytes inside of
> > 8-byte region [ffff88811d228440, ffff88811d228448)
> > [ 166.341134] The buggy address belongs to the page:
> > [ 166.341140] page:ffffea0004748a00 count:1 mapcount:0 mapping:ffff88815a80c340 index:0xffff88811d228f80 compound_mapcount: 0
> > [ 166.341146] flags: 0x200000000010200(slab|head)
> > [ 166.341153] raw: 0200000000010200 ffffea000564b288 ffffea00049ef708 ffff88815a80c340
> > [ 166.341159] raw: ffff88811d228f80 0000000000160013 00000001ffffffff 0000000000000000
> > [ 166.341163] page dumped because: kasan: bad access detected
> >
> > [ 166.341169] Memory state around the buggy address:
> > [ 166.341174] ffff88811d228300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.341179] ffff88811d228380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.341184] >ffff88811d228400: fc fc fc fc fc fc fc fc fb fc fc fc fc fc fc fc
> > [ 166.341188] ^
> > [ 166.341192] ffff88811d228480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.341197] ffff88811d228500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > [ 166.341201] ==================================================================
> >
> > > On Tuesday, 11 December 2018 16:20:43 EET Laurent Pinchart wrote:
> > > > On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> > > > > On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > > >> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > > >>
> > > > >> [snip]
> > > > >>
> > > > >>> I can see a couple of steps missing in the script below.
> > > > >>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
> > > > >>> 00040.html)
> > > > >>>
> > > > >>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > > >>> documentation",
> > > > >>> under section "Configuring ImgU V4L2 subdev for image processing"...
> > > > >>>
> > > > >>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > > >>>
> > > > >>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > > >>> desired (e.g 0 for video mode or 1 for still mode) through the control
> > > > >>> id 0x009819a1 as below.
> > > > >>>
> > > > >>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > > >>
> > > > >> I assume the control takes a valid default value ? It's better to set
> > > > >> it
> > > > >> explicitly anyway, so I'll do so.
> > > > >>
> > > > >>> 2. ImgU pipeline needs to be configured for image processing as below.
> > > > >>>
> > > > >>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > > >>> have the processed image output to the DDR memory.
> > > > >>>
> > > > >>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > > >>> Geometric Distortion Correction (GDC) -> DDR
> > > > >>>
> > > > >>> The ImgU V4L2 subdev has to be configured with the supported
> > > > >>> resolutions in all the above HW blocks, for a given input resolution.
> > > > >>>
> > > > >>> For a given supported resolution for an input frame, the Input Feeder,
> > > > >>> Bayer Down Scaling and GDC blocks should be configured with the
> > > > >>> supported resolutions. This information can be obtained by looking at
> > > > >>> the following IPU3 ISP configuration table for ov5670 sensor.
> > > > >>>
> > > > >>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> > > > >>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files
> > > > >>> /
> > > > >>> gcss/graph_settings_ov5670.xml
> > > > >>>
> > > > >>> For the ov5670 example, for an input frame with a resolution of
> > > > >>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > > > >>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > > > >>> 2560x1920 respectively.
> > > > >>
> > > > >> How is the GDC output resolution computed from the input resolution ?
> > > > >> Does the GDC always consume 32 columns and 22 lines ?
> > > > >>
> > > > >>> The following steps prepare the ImgU ISP pipeline for the image
> > > > >>> processing.
> > > > >>>
> > > > >>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > > >>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > > > >>> above.
> > > > >>
> > > > >> If I understand things correctly, the GDC resolution is the pipeline
> > > > >> output resolution. Why is it configured on pad 0 ?
> > > > >>
> > > > >>> 2. The ImgU V4L2 subdev cropping should be set by using the
> > > > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > > > >>> target, using the input feeder height and width.
> > > > >>>
> > > > >>> 3. The ImgU V4L2 subdev composing should be set by using the
> > > > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > > > >>> target, using the BDS height and width.
> > > > >>>
> > > > >>> Once these 2 steps are done, the raw bayer frames can be input to the
> > > > >>> ImgU V4L2 subdev for processing.
> > > > >>
> > > > >> Do I need to capture from both the output and viewfinder nodes ? How
> > > > >> are
> > > > >> they related to the IF -> BDS -> GDC pipeline, are they both fed from
> > > > >> the GDC output ? If so, how does the viewfinder scaler fit in that
> > > > >> picture ?
> > > > >>
> > > > >> I have tried the above configuration with the IPU3 v8 driver, and while
> > > > >> the kernel doesn't crash, no images get processed. The userspace
> > > > >> processes wait forever for buffers to be ready. I then configured pad 2
> > > > >> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
> > > > >>
> > > > >> There's one problem though: during capture, or very soon after it, the
> > > > >> machine locks up completely. I suspect a memory corruption, as when it
> > > > >> doesn't log immediately commands such as dmesg will not produce any
> > > > >> output and just block, until the system freezes soon after (especially
> > > > >> when moving the mouse).
> > > > >>
> > > > >> I would still call this an improvement to some extent, but there's
> > > > >> definitely room for more improvements :-)
> > > > >>
> > > > >> To reproduce the issue, you can run the ipu3-process.sh script
> > > > >> (attached
> > > > >> to this e-mail) with the following arguments:
> > > > >>
> > > > >> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
> > > >
> > > > This should have read
> > > >
> > > > $ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
> > > >
> > > > Without the --vf argument no images are processed.
> > > >
> > > > It seems that the Intel mail server blocked the mail that contained the
> > > > script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
> > > >
> > > > >> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
> > > > >> the IPU3-specific Bayer format (for a total of 6469632 bytes).
> > > > >
> > > > > I managed to get the dmesg output, and it doesn't look pretty.
> > > > >
> > > > > [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> > > > > libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> > > > > ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > > > [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > > > mac80211
> > > > > iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp
> > > > > cfg80211
> > > > > 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > > > videobuf2_memops videobuf2_v4l2 videobuf2_common
> > > > > processor_thermal_device
> > > > > intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common
> > > > > videodev
> > > > > at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs
> > > > > cros_ec_core
> > > > > int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> > > > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > > > sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> > > > > drm_panel_orientation_quirks i2c_hid hid
> > > > > [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> > > > > 4.20.0-rc6+ #2
> > > > > [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > > > [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > > > [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc
> > > > > f3
> > > > > 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> > > > > <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> > > > > [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> > > > > [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> > > > > 000000000000000c
> > > > > [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> > > > > 00000000ffffffff
> > > > > [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> > > > > ffff8f5cfaba16f0
> > > > > [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> > > > > ffff8f5cf58f0028
> > > > > [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> > > > > ffff8f5cf58f04e8
> > > > > [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
> > > > > knlGS:
> > > > > 0000000000000000
> > > > > [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> > > > > 00000000003606e0
> > > > > [ 571.217301] Call Trace:
> > > > > [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> > > > > [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > > > [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> > > > > [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> > > > > [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> > > > > [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> > > > > [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> > > > > [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> > > > > [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> > > > > [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> > > > > [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> > > > > [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> > > > > [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> > > > > [ 571.217480] vfs_ioctl+0x1e/0x2b
> > > > > [ 571.217486] do_vfs_ioctl+0x531/0x559
> > > > > [ 571.217494] ? vfs_write+0xd1/0xdf
> > > > > [ 571.217500] ksys_ioctl+0x50/0x70
> > > > > [ 571.217506] __x64_sys_ioctl+0x16/0x19
> > > > > [ 571.217512] do_syscall_64+0x53/0x60
> > > > > [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > > [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> > > > > [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00
> > > > > 48
> > > > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > > > [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> > > > > 0000000000000010
> > > > > [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> > > > > 00007f85cf9b9f47
> > > > > [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> > > > > 0000000000000003
> > > > > [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> > > > > 00007f85d009c700
> > > > > [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> > > > > 000055f4c4dc0b06
> > > > > [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> > > > > 00007ffc59057825
> > > > > [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> > > > > [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
> > > >
> > > > And after fixing another issue in the capture script (which was setting
> > > > the
> > > > format on the ImgU subdev pad 3 to 2560x1920 but capture in 1920x1080), I
> > > > now get plenty of the following messages:
> > > >
> > > > [ 221.366131] BUG: Bad page state in process yavta pfn:14a4ff
> > > > [ 221.366134] page:ffffde5d45293fc0 count:-1 mapcount:0 mapping:
> > > > 0000000000000000 index:0x0
> > > > [ 221.366137] flags: 0x200000000000000()
> > > > [ 221.366140] raw: 0200000000000000 dead000000000100 dead000000000200
> > > > 0000000000000000
> > > > [ 221.366143] raw: 0000000000000000 0000000000000000 ffffffffffffffff
> > > > 0000000000000000
> > > > [ 221.366145] page dumped because: nonzero _refcount
> > > > [ 221.366147] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > > intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp mac80211 iwlwifi
> > > > cfg80211 hid_multitouch 8250_dw ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > > videobuf2_memops videobuf2_v4l2 processor_thermal_device videobuf2_common
> > > > intel_soc_dts_iosf ov13858 ov5670 dw9714 v4l2_fwnode v4l2_common videodev
> > > > media at24 cros_ec_lpcs cros_ec_core int3403_thermal int340x_thermal_zone
> > > > chromeos_pstore mac_hid int3400_thermal acpi_thermal_rel autofs4 usbhid
> > > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > > sysfillrect sysimgblt fb_sys_fops sdhci_pci cqhci sdhci drm
> > > > drm_panel_orientation_quirks i2c_hid hid
> > > > [ 221.366172] CPU: 3 PID: 1022 Comm: yavta Tainted: G B WC
> > > > 4.20.0-rc6+ #2
> > > > [ 221.366173] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > > [ 221.366173] Call Trace:
> > > > [ 221.366176] dump_stack+0x46/0x59
> > > > [ 221.366179] bad_page+0xf2/0x10c
> > > > [ 221.366182] free_pages_check+0x78/0x81
> > > > [ 221.366186] free_pcppages_bulk+0xa6/0x236
> > > > [ 221.366190] free_unref_page+0x4b/0x53
> > > > [ 221.366193] vb2_dma_sg_put+0x95/0xb5 [videobuf2_dma_sg]
> > > > [ 221.366197] __vb2_buf_mem_free+0x3a/0x6e [videobuf2_common]
> > > > [ 221.366202] __vb2_queue_free+0xe3/0x1be [videobuf2_common]
> > > > [ 221.366207] vb2_core_reqbufs+0xe9/0x2cc [videobuf2_common]
> > > > [ 221.366212] vb2_ioctl_reqbufs+0x78/0x9e [videobuf2_v4l2]
> > > > [ 221.366220] __video_do_ioctl+0x258/0x38e [videodev]
> > > > [ 221.366229] video_usercopy+0x25f/0x4e5 [videodev]
> > > > [ 221.366237] ? copy_overflow+0x14/0x14 [videodev]
> > > > [ 221.366240] ? unmap_region+0xe0/0x10a
> > > > [ 221.366250] v4l2_ioctl+0x4d/0x58 [videodev]
> > > > [ 221.366253] vfs_ioctl+0x1e/0x2b
> > > > [ 221.366255] do_vfs_ioctl+0x531/0x559
> > > > [ 221.366260] ksys_ioctl+0x50/0x70
> > > > [ 221.366263] __x64_sys_ioctl+0x16/0x19
> > > > [ 221.366266] do_syscall_64+0x53/0x60
> > > > [ 221.366269] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 221.366270] RIP: 0033:0x7fbe39f6af47
> > > > [ 221.366272] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > > [ 221.366273] RSP: 002b:00007fff05638e68 EFLAGS: 00000246 ORIG_RAX:
> > > > 0000000000000010
> > > > [ 221.366275] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > > > 00007fbe39f6af47
> > > > [ 221.366279] RDX: 00007fff05638f90 RSI: 00000000c0145608 RDI:
> > > > 0000000000000003
> > > > [ 221.366283] RBP: 0000000000000004 R08: 0000000000000000 R09:
> > > > 0000000000000045
> > > > [ 221.366287] R10: 0000000000000557 R11: 0000000000000246 R12:
> > > > 000055c83bd76750
> > > > [ 221.366290] R13: 000055c83b6b26a0 R14: 0000000000000001 R15:
> > > > 00007fff0563a825
> >
> > --
> > Regards,
> >
> > Laurent Pinchart
> >
> >
> >
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-08 6:54 ` Tomasz Figa
@ 2019-01-09 16:40 ` Jacopo Mondi
2019-01-09 17:00 ` Mani, Rajmohan
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-01-09 16:40 UTC (permalink / raw)
To: Tomasz Figa
Cc: Yong Zhi, Mani, Rajmohan, Qiu, Tian Shu, Cao Bing Bu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
[-- Attachment #1: Type: text/plain, Size: 40255 bytes --]
Hello,
On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> Hi Raj, Yong, Bingbu, Tianshu,
>
> On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa <tfiga@chromium.org> wrote:
> >
> > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > <laurent.pinchart@ideasonboard.com> wrote:
> > >
> > > Hellon
> > >
> > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > > > Hello Yong,
> > > >
> > > > Could you please have a look at the crash reported below ?
> > >
> > > A bit more information to help you debugging this. I've enabled KASAN in the
> > > kernel configuration, and get the following use-after-free reports.
I tested as well using the ipu-process.sh script shared by Laurent,
with the following command line:
./ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
and I got a very similar trace available at:
https://paste.debian.net/hidden/5855e15a/
Please note I have been able to process a set of images (with KASAN
enabled the machine does not freeze) but the kernel log gets flooded
and it is not possible to process any other frame after this.
The issue is currently quite annoying and it's a blocker for libcamera
development on IPU3. Please let me know if I can support with more
testing.
Thanks
j
> > >
> > > [ 166.332920] ==================================================================
> > > [ 166.332937] BUG: KASAN: use-after-free in __cached_rbnode_delete_update+0x36/0x202
> > > [ 166.332944] Read of size 8 at addr ffff888133823718 by task yavta/1305
> > >
> > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C 4.20.0-rc6+ #3
> > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > [ 166.332959] Call Trace:
> > > [ 166.332967] dump_stack+0x5b/0x81
> > > [ 166.332974] print_address_description+0x65/0x227
> > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > [ 166.332983] kasan_report+0x247/0x285
> > > [ 166.332989] __cached_rbnode_delete_update+0x36/0x202
> > > [ 166.332995] private_free_iova+0x57/0x6d
> > > [ 166.332999] __free_iova+0x23/0x31
> > > [ 166.333011] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> >
> > Thanks Laurent, I think this is a very good hint. It looks like we're
> > basically freeing and already freed IOVA and corrupting some allocator
> > state?
>
> Did you have any luck in reproducing and fixing this double free issue?
>
> Best regards,
> Tomasz
>
> >
> > > [ 166.333022] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 166.333032] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > > [ 166.333043] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > > [ 166.333056] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > > [ 166.333067] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> > > [ 166.333079] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > > [ 166.333088] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > > [ 166.333096] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> > > [ 166.333104] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > > [ 166.333123] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.333142] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 166.333147] ? slab_free_freelist_hook+0x46/0x94
> > > [ 166.333151] ? kfree+0x107/0x1a0
> > > [ 166.333169] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.333187] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 166.333203] ? v4l_enumstd+0x49/0x49 [videodev]
> > > [ 166.333207] ? __wake_up_common+0x342/0x342
> > > [ 166.333215] ? atomic_long_add_return+0x15/0x24
> > > [ 166.333219] ? ldsem_up_read+0x15/0x29
> > > [ 166.333223] ? tty_write+0x4c6/0x4d8
> > > [ 166.333227] ? n_tty_receive_char_special+0x1152/0x1152
> > > [ 166.333244] ? video_usercopy+0x8ae/0x8ae [videodev]
> > > [ 166.333260] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.333266] vfs_ioctl+0x76/0x89
> > > [ 166.333271] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.333275] ? __switch_to_asm+0x40/0x70
> > > [ 166.333279] ? __switch_to_asm+0x40/0x70
> > > [ 166.333282] ? __switch_to_asm+0x34/0x70
> > > [ 166.333286] ? __switch_to_asm+0x40/0x70
> > > [ 166.333290] ? ioctl_preallocate+0x174/0x174
> > > [ 166.333294] ? __switch_to+0x71c/0xb00
> > > [ 166.333299] ? compat_start_thread+0x6b/0x6b
> > > [ 166.333302] ? __switch_to_asm+0x34/0x70
> > > [ 166.333305] ? __switch_to_asm+0x40/0x70
> > > [ 166.333309] ? mmdrop+0x12/0x23
> > > [ 166.333313] ? finish_task_switch+0x34d/0x3de
> > > [ 166.333319] ? __schedule+0x1004/0x1045
> > > [ 166.333325] ? firmware_map_remove+0x119/0x119
> > > [ 166.333330] ksys_ioctl+0x50/0x70
> > > [ 166.333335] __x64_sys_ioctl+0x82/0x89
> > > [ 166.333340] do_syscall_64+0xa0/0xd2
> > > [ 166.333345] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [ 166.333349] RIP: 0033:0x7f2481541f47
> > > [ 166.333354] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > [ 166.333357] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > > [ 166.333362] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> > > [ 166.333364] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> > > [ 166.333367] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> > > [ 166.333369] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> > > [ 166.333372] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > >
> > > [ 166.333383] Allocated by task 1305:
> > > [ 166.333389] kasan_kmalloc+0x8a/0x98
> > > [ 166.333392] slab_post_alloc_hook+0x31/0x51
> > > [ 166.333396] kmem_cache_alloc+0xd7/0x174
> > > [ 166.333399] alloc_iova+0x24/0x2ea
> > > [ 166.333407] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
> > > [ 166.333415] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> > > [ 166.333424] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> > > [ 166.333433] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> > > [ 166.333442] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> > > [ 166.333449] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> > > [ 166.333455] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> > > [ 166.333471] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.333487] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.333501] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.333505] vfs_ioctl+0x76/0x89
> > > [ 166.333508] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.333511] ksys_ioctl+0x50/0x70
> > > [ 166.333514] __x64_sys_ioctl+0x82/0x89
> > > [ 166.333518] do_syscall_64+0xa0/0xd2
> > > [ 166.333521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > [ 166.333526] Freed by task 1301:
> > > [ 166.333532] __kasan_slab_free+0xfa/0x11c
> > > [ 166.333535] slab_free_freelist_hook+0x46/0x94
> > > [ 166.333538] kmem_cache_free+0x7b/0x172
> > > [ 166.333542] __free_iova+0x23/0x31
> > > [ 166.333550] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > [ 166.333557] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 166.333566] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > > [ 166.333574] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > > [ 166.333584] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > > [ 166.333593] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > > [ 166.333599] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > > [ 166.333606] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > > [ 166.333621] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.333637] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.333652] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.333655] vfs_ioctl+0x76/0x89
> > > [ 166.333658] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.333662] ksys_ioctl+0x50/0x70
> > > [ 166.333665] __x64_sys_ioctl+0x82/0x89
> > > [ 166.333668] do_syscall_64+0xa0/0xd2
> > > [ 166.333671] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > [ 166.333678] The buggy address belongs to the object at ffff888133823700
> > > which belongs to the cache iommu_iova of size 40
> > > [ 166.333685] The buggy address is located 24 bytes inside of
> > > 40-byte region [ffff888133823700, ffff888133823728)
> > > [ 166.333690] The buggy address belongs to the page:
> > > [ 166.333696] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
> > > [ 166.333703] flags: 0x200000000010200(slab|head)
> > > [ 166.333710] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
> > > [ 166.333717] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
> > > [ 166.333720] page dumped because: kasan: bad access detected
> > >
> > > [ 166.333726] Memory state around the buggy address:
> > > [ 166.333732] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.333737] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.333742] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.333745] ^
> > > [ 166.333750] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.333755] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.333759] ==================================================================
> > > [ 166.333762] Disabling lock debugging due to kernel taint
> > > [ 166.333764] ==================================================================
> > > [ 166.333770] BUG: KASAN: double-free or invalid-free in kmem_cache_free+0x7b/0x172
> > >
> > > [ 166.333780] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
> > > [ 166.333782] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > [ 166.333783] Call Trace:
> > > [ 166.333789] dump_stack+0x5b/0x81
> > > [ 166.333795] print_address_description+0x65/0x227
> > > [ 166.333799] ? kmem_cache_free+0x7b/0x172
> > > [ 166.333803] kasan_report_invalid_free+0x67/0xa0
> > > [ 166.333807] ? kmem_cache_free+0x7b/0x172
> > > [ 166.333812] __kasan_slab_free+0x86/0x11c
> > > [ 166.333817] slab_free_freelist_hook+0x46/0x94
> > > [ 166.333822] kmem_cache_free+0x7b/0x172
> > > [ 166.333826] ? __free_iova+0x23/0x31
> > > [ 166.333831] __free_iova+0x23/0x31
> > > [ 166.333840] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > [ 166.333851] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 166.333861] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > > [ 166.333872] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > > [ 166.333885] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > > [ 166.333896] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> > > [ 166.333908] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > > [ 166.333917] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > > [ 166.333923] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> > > [ 166.333932] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > > [ 166.333950] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.333970] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 166.333974] ? slab_free_freelist_hook+0x46/0x94
> > > [ 166.333979] ? kfree+0x107/0x1a0
> > > [ 166.333997] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.334015] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 166.334031] ? v4l_enumstd+0x49/0x49 [videodev]
> > > [ 166.334035] ? __wake_up_common+0x342/0x342
> > > [ 166.334042] ? atomic_long_add_return+0x15/0x24
> > > [ 166.334046] ? ldsem_up_read+0x15/0x29
> > > [ 166.334050] ? tty_write+0x4c6/0x4d8
> > > [ 166.334054] ? n_tty_receive_char_special+0x1152/0x1152
> > > [ 166.334071] ? video_usercopy+0x8ae/0x8ae [videodev]
> > > [ 166.334087] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.334092] vfs_ioctl+0x76/0x89
> > > [ 166.334097] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.334101] ? __switch_to_asm+0x40/0x70
> > > [ 166.334105] ? __switch_to_asm+0x40/0x70
> > > [ 166.334108] ? __switch_to_asm+0x34/0x70
> > > [ 166.334111] ? __switch_to_asm+0x40/0x70
> > > [ 166.334116] ? ioctl_preallocate+0x174/0x174
> > > [ 166.334120] ? __switch_to+0x71c/0xb00
> > > [ 166.334124] ? compat_start_thread+0x6b/0x6b
> > > [ 166.334127] ? __switch_to_asm+0x34/0x70
> > > [ 166.334130] ? __switch_to_asm+0x40/0x70
> > > [ 166.334134] ? mmdrop+0x12/0x23
> > > [ 166.334137] ? finish_task_switch+0x34d/0x3de
> > > [ 166.334143] ? __schedule+0x1004/0x1045
> > > [ 166.334148] ? firmware_map_remove+0x119/0x119
> > > [ 166.334153] ksys_ioctl+0x50/0x70
> > > [ 166.334158] __x64_sys_ioctl+0x82/0x89
> > > [ 166.334163] do_syscall_64+0xa0/0xd2
> > > [ 166.334167] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [ 166.334171] RIP: 0033:0x7f2481541f47
> > > [ 166.334175] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > [ 166.334177] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > > [ 166.334181] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> > > [ 166.334184] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> > > [ 166.334186] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> > > [ 166.334189] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> > > [ 166.334191] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > >
> > > [ 166.334201] Allocated by task 1305:
> > > [ 166.334207] kasan_kmalloc+0x8a/0x98
> > > [ 166.334210] slab_post_alloc_hook+0x31/0x51
> > > [ 166.334213] kmem_cache_alloc+0xd7/0x174
> > > [ 166.334216] alloc_iova+0x24/0x2ea
> > > [ 166.334225] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu]
> > > [ 166.334233] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> > > [ 166.334241] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> > > [ 166.334250] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> > > [ 166.334259] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> > > [ 166.334266] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> > > [ 166.334273] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> > > [ 166.334288] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.334304] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.334319] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.334322] vfs_ioctl+0x76/0x89
> > > [ 166.334325] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.334328] ksys_ioctl+0x50/0x70
> > > [ 166.334332] __x64_sys_ioctl+0x82/0x89
> > > [ 166.334335] do_syscall_64+0xa0/0xd2
> > > [ 166.334338] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > [ 166.334343] Freed by task 1301:
> > > [ 166.334349] __kasan_slab_free+0xfa/0x11c
> > > [ 166.334352] slab_free_freelist_hook+0x46/0x94
> > > [ 166.334355] kmem_cache_free+0x7b/0x172
> > > [ 166.334359] __free_iova+0x23/0x31
> > > [ 166.334367] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > [ 166.334375] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 166.334383] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > > [ 166.334392] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > > [ 166.334401] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > > [ 166.334410] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > > [ 166.334416] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > > [ 166.334423] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > > [ 166.334438] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.334454] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.334469] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.334472] vfs_ioctl+0x76/0x89
> > > [ 166.334475] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.334479] ksys_ioctl+0x50/0x70
> > > [ 166.334482] __x64_sys_ioctl+0x82/0x89
> > > [ 166.334485] do_syscall_64+0xa0/0xd2
> > > [ 166.334488] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > [ 166.334494] The buggy address belongs to the object at ffff888133823700
> > > which belongs to the cache iommu_iova of size 40
> > > [ 166.334501] The buggy address is located 0 bytes inside of
> > > 40-byte region [ffff888133823700, ffff888133823728)
> > > [ 166.334506] The buggy address belongs to the page:
> > > [ 166.334511] page:ffffea0004ce0880 count:1 mapcount:0 mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0
> > > [ 166.334517] flags: 0x200000000010200(slab|head)
> > > [ 166.334524] raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70 ffff8881519e8640
> > > [ 166.334530] raw: 0000000000000000 0000000000120012 00000001ffffffff 0000000000000000
> > > [ 166.334533] page dumped because: kasan: bad access detected
> > >
> > > [ 166.334539] Memory state around the buggy address:
> > > [ 166.334544] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.334549] ffff888133823680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.334554] >ffff888133823700: fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.334558] ^
> > > [ 166.334562] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.334567] ffff888133823800: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.334571] ==================================================================
> > > [ 166.340377] ==================================================================
> > > [ 166.340388] BUG: KASAN: double-free or invalid-free in kfree+0x107/0x1a0
> > >
> > > [ 166.340399] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-rc6+ #3
> > > [ 166.340401] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > [ 166.340403] Call Trace:
> > > [ 166.340410] dump_stack+0x5b/0x81
> > > [ 166.340416] print_address_description+0x65/0x227
> > > [ 166.340420] ? kfree+0x107/0x1a0
> > > [ 166.340425] kasan_report_invalid_free+0x67/0xa0
> > > [ 166.340428] ? kfree+0x107/0x1a0
> > > [ 166.340433] __kasan_slab_free+0x86/0x11c
> > > [ 166.340438] slab_free_freelist_hook+0x46/0x94
> > > [ 166.340443] kfree+0x107/0x1a0
> > > [ 166.340454] ? ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> > > [ 166.340464] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> > > [ 166.340475] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 166.340485] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > > [ 166.340495] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > > [ 166.340509] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > > [ 166.340520] ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu]
> > > [ 166.340531] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > > [ 166.340541] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > > [ 166.340548] ? __mutex_lock_interruptible_slowpath+0xf/0xf
> > > [ 166.340557] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > > [ 166.340575] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.340595] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 166.340600] ? slab_free_freelist_hook+0x46/0x94
> > > [ 166.340604] ? kfree+0x107/0x1a0
> > > [ 166.340622] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.340640] ? copy_overflow+0x14/0x14 [videodev]
> > > [ 166.340657] ? v4l_enumstd+0x49/0x49 [videodev]
> > > [ 166.340660] ? __wake_up_common+0x342/0x342
> > > [ 166.340668] ? atomic_long_add_return+0x15/0x24
> > > [ 166.340672] ? ldsem_up_read+0x15/0x29
> > > [ 166.340677] ? tty_write+0x4c6/0x4d8
> > > [ 166.340681] ? n_tty_receive_char_special+0x1152/0x1152
> > > [ 166.340698] ? video_usercopy+0x8ae/0x8ae [videodev]
> > > [ 166.340714] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.340720] vfs_ioctl+0x76/0x89
> > > [ 166.340725] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.340729] ? __switch_to_asm+0x40/0x70
> > > [ 166.340733] ? __switch_to_asm+0x40/0x70
> > > [ 166.340736] ? __switch_to_asm+0x34/0x70
> > > [ 166.340739] ? __switch_to_asm+0x40/0x70
> > > [ 166.340743] ? ioctl_preallocate+0x174/0x174
> > > [ 166.340748] ? __switch_to+0x71c/0xb00
> > > [ 166.340752] ? compat_start_thread+0x6b/0x6b
> > > [ 166.340756] ? __switch_to_asm+0x34/0x70
> > > [ 166.340759] ? __switch_to_asm+0x40/0x70
> > > [ 166.340762] ? mmdrop+0x12/0x23
> > > [ 166.340766] ? finish_task_switch+0x34d/0x3de
> > > [ 166.340772] ? __schedule+0x1004/0x1045
> > > [ 166.340777] ? firmware_map_remove+0x119/0x119
> > > [ 166.340782] ksys_ioctl+0x50/0x70
> > > [ 166.340788] __x64_sys_ioctl+0x82/0x89
> > > [ 166.340793] do_syscall_64+0xa0/0xd2
> > > [ 166.340797] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > [ 166.340802] RIP: 0033:0x7f2481541f47
> > > [ 166.340806] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > [ 166.340809] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > > [ 166.340813] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2481541f47
> > > [ 166.340816] RDX: 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003
> > > [ 166.340819] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09: 00007f2481c24700
> > > [ 166.340821] R10: 0000000000000020 R11: 0000000000000246 R12: 0000555f1c494b06
> > > [ 166.340824] R13: 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > >
> > > [ 166.340834] Allocated by task 1305:
> > > [ 166.340840] kasan_kmalloc+0x8a/0x98
> > > [ 166.340844] __kmalloc_node+0x193/0x1ba
> > > [ 166.340848] kvmalloc_node+0x44/0x6d
> > > [ 166.340856] ipu3_dmamap_alloc+0x1c9/0x83f [ipu3_imgu]
> > > [ 166.340864] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu]
> > > [ 166.340873] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu]
> > > [ 166.340882] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu]
> > > [ 166.340891] ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu]
> > > [ 166.340897] vb2_start_streaming+0x164/0x33b [videobuf2_common]
> > > [ 166.340904] vb2_core_streamon+0x1a1/0x208 [videobuf2_common]
> > > [ 166.340920] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.340935] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.340950] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.340954] vfs_ioctl+0x76/0x89
> > > [ 166.340957] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.340960] ksys_ioctl+0x50/0x70
> > > [ 166.340963] __x64_sys_ioctl+0x82/0x89
> > > [ 166.340966] do_syscall_64+0xa0/0xd2
> > > [ 166.340969] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > [ 166.340974] Freed by task 1301:
> > > [ 166.340980] __kasan_slab_free+0xfa/0x11c
> > > [ 166.340983] slab_free_freelist_hook+0x46/0x94
> > > [ 166.340986] kfree+0x107/0x1a0
> > > [ 166.340994] ipu3_dmamap_free+0x17b/0x1d6 [ipu3_imgu]
> > > [ 166.341002] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > [ 166.341010] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu]
> > > [ 166.341019] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu]
> > > [ 166.341028] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu]
> > > [ 166.341037] ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu]
> > > [ 166.341043] __vb2_queue_cancel+0xa8/0x705 [videobuf2_common]
> > > [ 166.341050] vb2_core_streamoff+0x68/0xf8 [videobuf2_common]
> > > [ 166.341066] __video_do_ioctl+0x625/0x887 [videodev]
> > > [ 166.341081] video_usercopy+0x3a3/0x8ae [videodev]
> > > [ 166.341096] v4l2_ioctl+0xb7/0xc5 [videodev]
> > > [ 166.341100] vfs_ioctl+0x76/0x89
> > > [ 166.341103] do_vfs_ioctl+0xb33/0xb7e
> > > [ 166.341106] ksys_ioctl+0x50/0x70
> > > [ 166.341109] __x64_sys_ioctl+0x82/0x89
> > > [ 166.341112] do_syscall_64+0xa0/0xd2
> > > [ 166.341116] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >
> > > [ 166.341122] The buggy address belongs to the object at ffff88811d228440
> > > which belongs to the cache kmalloc-8 of size 8
> > > [ 166.341129] The buggy address is located 0 bytes inside of
> > > 8-byte region [ffff88811d228440, ffff88811d228448)
> > > [ 166.341134] The buggy address belongs to the page:
> > > [ 166.341140] page:ffffea0004748a00 count:1 mapcount:0 mapping:ffff88815a80c340 index:0xffff88811d228f80 compound_mapcount: 0
> > > [ 166.341146] flags: 0x200000000010200(slab|head)
> > > [ 166.341153] raw: 0200000000010200 ffffea000564b288 ffffea00049ef708 ffff88815a80c340
> > > [ 166.341159] raw: ffff88811d228f80 0000000000160013 00000001ffffffff 0000000000000000
> > > [ 166.341163] page dumped because: kasan: bad access detected
> > >
> > > [ 166.341169] Memory state around the buggy address:
> > > [ 166.341174] ffff88811d228300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.341179] ffff88811d228380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.341184] >ffff88811d228400: fc fc fc fc fc fc fc fc fb fc fc fc fc fc fc fc
> > > [ 166.341188] ^
> > > [ 166.341192] ffff88811d228480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.341197] ffff88811d228500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> > > [ 166.341201] ==================================================================
> > >
> > > > On Tuesday, 11 December 2018 16:20:43 EET Laurent Pinchart wrote:
> > > > > On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> > > > > > On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > > > >> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > > > >>
> > > > > >> [snip]
> > > > > >>
> > > > > >>> I can see a couple of steps missing in the script below.
> > > > > >>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
> > > > > >>> 00040.html)
> > > > > >>>
> > > > > >>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > > > >>> documentation",
> > > > > >>> under section "Configuring ImgU V4L2 subdev for image processing"...
> > > > > >>>
> > > > > >>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > > > >>>
> > > > > >>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > > > >>> desired (e.g 0 for video mode or 1 for still mode) through the control
> > > > > >>> id 0x009819a1 as below.
> > > > > >>>
> > > > > >>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > > > >>
> > > > > >> I assume the control takes a valid default value ? It's better to set
> > > > > >> it
> > > > > >> explicitly anyway, so I'll do so.
> > > > > >>
> > > > > >>> 2. ImgU pipeline needs to be configured for image processing as below.
> > > > > >>>
> > > > > >>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > > > >>> have the processed image output to the DDR memory.
> > > > > >>>
> > > > > >>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > > > >>> Geometric Distortion Correction (GDC) -> DDR
> > > > > >>>
> > > > > >>> The ImgU V4L2 subdev has to be configured with the supported
> > > > > >>> resolutions in all the above HW blocks, for a given input resolution.
> > > > > >>>
> > > > > >>> For a given supported resolution for an input frame, the Input Feeder,
> > > > > >>> Bayer Down Scaling and GDC blocks should be configured with the
> > > > > >>> supported resolutions. This information can be obtained by looking at
> > > > > >>> the following IPU3 ISP configuration table for ov5670 sensor.
> > > > > >>>
> > > > > >>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> > > > > >>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files
> > > > > >>> /
> > > > > >>> gcss/graph_settings_ov5670.xml
> > > > > >>>
> > > > > >>> For the ov5670 example, for an input frame with a resolution of
> > > > > >>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > > > > >>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > > > > >>> 2560x1920 respectively.
> > > > > >>
> > > > > >> How is the GDC output resolution computed from the input resolution ?
> > > > > >> Does the GDC always consume 32 columns and 22 lines ?
> > > > > >>
> > > > > >>> The following steps prepare the ImgU ISP pipeline for the image
> > > > > >>> processing.
> > > > > >>>
> > > > > >>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > > > >>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> > > > > >>> above.
> > > > > >>
> > > > > >> If I understand things correctly, the GDC resolution is the pipeline
> > > > > >> output resolution. Why is it configured on pad 0 ?
> > > > > >>
> > > > > >>> 2. The ImgU V4L2 subdev cropping should be set by using the
> > > > > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > > > > >>> target, using the input feeder height and width.
> > > > > >>>
> > > > > >>> 3. The ImgU V4L2 subdev composing should be set by using the
> > > > > >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > > > > >>> target, using the BDS height and width.
> > > > > >>>
> > > > > >>> Once these 2 steps are done, the raw bayer frames can be input to the
> > > > > >>> ImgU V4L2 subdev for processing.
> > > > > >>
> > > > > >> Do I need to capture from both the output and viewfinder nodes ? How
> > > > > >> are
> > > > > >> they related to the IF -> BDS -> GDC pipeline, are they both fed from
> > > > > >> the GDC output ? If so, how does the viewfinder scaler fit in that
> > > > > >> picture ?
> > > > > >>
> > > > > >> I have tried the above configuration with the IPU3 v8 driver, and while
> > > > > >> the kernel doesn't crash, no images get processed. The userspace
> > > > > >> processes wait forever for buffers to be ready. I then configured pad 2
> > > > > >> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
> > > > > >>
> > > > > >> There's one problem though: during capture, or very soon after it, the
> > > > > >> machine locks up completely. I suspect a memory corruption, as when it
> > > > > >> doesn't log immediately commands such as dmesg will not produce any
> > > > > >> output and just block, until the system freezes soon after (especially
> > > > > >> when moving the mouse).
> > > > > >>
> > > > > >> I would still call this an improvement to some extent, but there's
> > > > > >> definitely room for more improvements :-)
> > > > > >>
> > > > > >> To reproduce the issue, you can run the ipu3-process.sh script
> > > > > >> (attached
> > > > > >> to this e-mail) with the following arguments:
> > > > > >>
> > > > > >> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
> > > > >
> > > > > This should have read
> > > > >
> > > > > $ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
> > > > >
> > > > > Without the --vf argument no images are processed.
> > > > >
> > > > > It seems that the Intel mail server blocked the mail that contained the
> > > > > script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
> > > > >
> > > > > >> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
> > > > > >> the IPU3-specific Bayer format (for a total of 6469632 bytes).
> > > > > >
> > > > > > I managed to get the dmesg output, and it doesn't look pretty.
> > > > > >
> > > > > > [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> > > > > > libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> > > > > > ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > > > > [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > > > > mac80211
> > > > > > iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp
> > > > > > cfg80211
> > > > > > 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > > > > videobuf2_memops videobuf2_v4l2 videobuf2_common
> > > > > > processor_thermal_device
> > > > > > intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common
> > > > > > videodev
> > > > > > at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs
> > > > > > cros_ec_core
> > > > > > int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> > > > > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > > > > sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> > > > > > drm_panel_orientation_quirks i2c_hid hid
> > > > > > [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> > > > > > 4.20.0-rc6+ #2
> > > > > > [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > > > > [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > > > > > [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc
> > > > > > f3
> > > > > > 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> > > > > > <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> > > > > > [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> > > > > > [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> > > > > > 000000000000000c
> > > > > > [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> > > > > > 00000000ffffffff
> > > > > > [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> > > > > > ffff8f5cfaba16f0
> > > > > > [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> > > > > > ffff8f5cf58f0028
> > > > > > [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> > > > > > ffff8f5cf58f04e8
> > > > > > [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
> > > > > > knlGS:
> > > > > > 0000000000000000
> > > > > > [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> > > > > > 00000000003606e0
> > > > > > [ 571.217301] Call Trace:
> > > > > > [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> > > > > > [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > > > > > [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> > > > > > [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> > > > > > [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> > > > > > [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> > > > > > [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> > > > > > [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> > > > > > [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> > > > > > [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> > > > > > [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> > > > > > [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> > > > > > [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> > > > > > [ 571.217480] vfs_ioctl+0x1e/0x2b
> > > > > > [ 571.217486] do_vfs_ioctl+0x531/0x559
> > > > > > [ 571.217494] ? vfs_write+0xd1/0xdf
> > > > > > [ 571.217500] ksys_ioctl+0x50/0x70
> > > > > > [ 571.217506] __x64_sys_ioctl+0x16/0x19
> > > > > > [ 571.217512] do_syscall_64+0x53/0x60
> > > > > > [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > > > [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> > > > > > [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00
> > > > > > 48
> > > > > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > > > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > > > > [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> > > > > > 0000000000000010
> > > > > > [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> > > > > > 00007f85cf9b9f47
> > > > > > [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> > > > > > 0000000000000003
> > > > > > [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> > > > > > 00007f85d009c700
> > > > > > [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> > > > > > 000055f4c4dc0b06
> > > > > > [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> > > > > > 00007ffc59057825
> > > > > > [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> > > > > > [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
> > > > >
> > > > > And after fixing another issue in the capture script (which was setting
> > > > > the
> > > > > format on the ImgU subdev pad 3 to 2560x1920 but capture in 1920x1080), I
> > > > > now get plenty of the following messages:
> > > > >
> > > > > [ 221.366131] BUG: Bad page state in process yavta pfn:14a4ff
> > > > > [ 221.366134] page:ffffde5d45293fc0 count:-1 mapcount:0 mapping:
> > > > > 0000000000000000 index:0x0
> > > > > [ 221.366137] flags: 0x200000000000000()
> > > > > [ 221.366140] raw: 0200000000000000 dead000000000100 dead000000000200
> > > > > 0000000000000000
> > > > > [ 221.366143] raw: 0000000000000000 0000000000000000 ffffffffffffffff
> > > > > 0000000000000000
> > > > > [ 221.366145] page dumped because: nonzero _refcount
> > > > > [ 221.366147] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > > > > intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp mac80211 iwlwifi
> > > > > cfg80211 hid_multitouch 8250_dw ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > > > > videobuf2_memops videobuf2_v4l2 processor_thermal_device videobuf2_common
> > > > > intel_soc_dts_iosf ov13858 ov5670 dw9714 v4l2_fwnode v4l2_common videodev
> > > > > media at24 cros_ec_lpcs cros_ec_core int3403_thermal int340x_thermal_zone
> > > > > chromeos_pstore mac_hid int3400_thermal acpi_thermal_rel autofs4 usbhid
> > > > > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > > > > sysfillrect sysimgblt fb_sys_fops sdhci_pci cqhci sdhci drm
> > > > > drm_panel_orientation_quirks i2c_hid hid
> > > > > [ 221.366172] CPU: 3 PID: 1022 Comm: yavta Tainted: G B WC
> > > > > 4.20.0-rc6+ #2
> > > > > [ 221.366173] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > > > > [ 221.366173] Call Trace:
> > > > > [ 221.366176] dump_stack+0x46/0x59
> > > > > [ 221.366179] bad_page+0xf2/0x10c
> > > > > [ 221.366182] free_pages_check+0x78/0x81
> > > > > [ 221.366186] free_pcppages_bulk+0xa6/0x236
> > > > > [ 221.366190] free_unref_page+0x4b/0x53
> > > > > [ 221.366193] vb2_dma_sg_put+0x95/0xb5 [videobuf2_dma_sg]
> > > > > [ 221.366197] __vb2_buf_mem_free+0x3a/0x6e [videobuf2_common]
> > > > > [ 221.366202] __vb2_queue_free+0xe3/0x1be [videobuf2_common]
> > > > > [ 221.366207] vb2_core_reqbufs+0xe9/0x2cc [videobuf2_common]
> > > > > [ 221.366212] vb2_ioctl_reqbufs+0x78/0x9e [videobuf2_v4l2]
> > > > > [ 221.366220] __video_do_ioctl+0x258/0x38e [videodev]
> > > > > [ 221.366229] video_usercopy+0x25f/0x4e5 [videodev]
> > > > > [ 221.366237] ? copy_overflow+0x14/0x14 [videodev]
> > > > > [ 221.366240] ? unmap_region+0xe0/0x10a
> > > > > [ 221.366250] v4l2_ioctl+0x4d/0x58 [videodev]
> > > > > [ 221.366253] vfs_ioctl+0x1e/0x2b
> > > > > [ 221.366255] do_vfs_ioctl+0x531/0x559
> > > > > [ 221.366260] ksys_ioctl+0x50/0x70
> > > > > [ 221.366263] __x64_sys_ioctl+0x16/0x19
> > > > > [ 221.366266] do_syscall_64+0x53/0x60
> > > > > [ 221.366269] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > > [ 221.366270] RIP: 0033:0x7fbe39f6af47
> > > > > [ 221.366272] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > > > > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > > > > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > > > > [ 221.366273] RSP: 002b:00007fff05638e68 EFLAGS: 00000246 ORIG_RAX:
> > > > > 0000000000000010
> > > > > [ 221.366275] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
> > > > > 00007fbe39f6af47
> > > > > [ 221.366279] RDX: 00007fff05638f90 RSI: 00000000c0145608 RDI:
> > > > > 0000000000000003
> > > > > [ 221.366283] RBP: 0000000000000004 R08: 0000000000000000 R09:
> > > > > 0000000000000045
> > > > > [ 221.366287] R10: 0000000000000557 R11: 0000000000000246 R12:
> > > > > 000055c83bd76750
> > > > > [ 221.366290] R13: 000055c83b6b26a0 R14: 0000000000000001 R15:
> > > > > 00007fff0563a825
> > >
> > > --
> > > Regards,
> > >
> > > Laurent Pinchart
> > >
> > >
> > >
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 16:40 ` Jacopo Mondi
@ 2019-01-09 17:00 ` Mani, Rajmohan
2019-01-09 17:25 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2019-01-09 17:00 UTC (permalink / raw)
To: Jacopo Mondi, Tomasz Figa
Cc: Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu, Laurent Pinchart,
Linux Media Mailing List, Sakari Ailus, Mauro Carvalho Chehab,
Hans Verkuil, Hu, Jerry W, Toivonen, Tuukka
Hi Laurent, Tomasz, Jacopo,
> -----Original Message-----
> From: Jacopo Mondi [mailto:jacopo@jmondi.org]
> Sent: Wednesday, January 09, 2019 8:41 AM
> To: Tomasz Figa <tfiga@chromium.org>
> Cc: Zhi, Yong <yong.zhi@intel.com>; Mani, Rajmohan
> <rajmohan.mani@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> Bingbu <bingbu.cao@intel.com>; Laurent Pinchart
> <laurent.pinchart@ideasonboard.com>; Linux Media Mailing List <linux-
> media@vger.kernel.org>; Sakari Ailus <sakari.ailus@linux.intel.com>; Mauro
> Carvalho Chehab <mchehab@kernel.org>; Hans Verkuil
> <hans.verkuil@cisco.com>; Hu, Jerry W <jerry.w.hu@intel.com>; Toivonen,
> Tuukka <tuukka.toivonen@intel.com>
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hello,
>
> On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > Hi Raj, Yong, Bingbu, Tianshu,
> >
> > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > >
> > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > <laurent.pinchart@ideasonboard.com> wrote:
> > > >
> > > > Hellon
> > > >
> > > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > > > > Hello Yong,
> > > > >
> > > > > Could you please have a look at the crash reported below ?
> > > >
> > > > A bit more information to help you debugging this. I've enabled
> > > > KASAN in the kernel configuration, and get the following use-after-free
> reports.
>
> I tested as well using the ipu-process.sh script shared by Laurent, with the
> following command line:
> ./ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
>
> and I got a very similar trace available at:
> https://paste.debian.net/hidden/5855e15a/
>
> Please note I have been able to process a set of images (with KASAN enabled
> the machine does not freeze) but the kernel log gets flooded and it is not
> possible to process any other frame after this.
>
> The issue is currently quite annoying and it's a blocker for libcamera
> development on IPU3. Please let me know if I can support with more testing.
>
> Thanks
> j
>
> > > >
> > > > [ 166.332920]
> > > >
> ================================================================
> ==
> > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > __cached_rbnode_delete_update+0x36/0x202
> > > > [ 166.332944] Read of size 8 at addr ffff888133823718 by task
> > > > yavta/1305
> > > >
> > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C 4.20.0-
> rc6+ #3
> > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018 [
> > > > 166.332959] Call Trace:
> > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > print_address_description+0x65/0x227
> > > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > __cached_rbnode_delete_update+0x36/0x202
> > > > [ 166.332995] private_free_iova+0x57/0x6d [ 166.332999]
> > > > __free_iova+0x23/0x31 [ 166.333011] ipu3_dmamap_free+0x118/0x1d6
> > > > [ipu3_imgu]
> > >
> > > Thanks Laurent, I think this is a very good hint. It looks like
> > > we're basically freeing and already freed IOVA and corrupting some
> > > allocator state?
> >
> > Did you have any luck in reproducing and fixing this double free issue?
> >
This issue is either hard to reproduce or comes with different signatures with
the updated yavta (that now supports meta output) with the 4.4 kernel that
I have been using.
I am switching to 4.20-rc6 for better reproducibility.
Enabling KASAN also results in storage space issues on my Chrome device.
Will enable this just for ImgU to get ahead and get back with more updates.
> > Best regards,
> > Tomasz
> >
> > >
> > > > [ 166.333022] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu] [
> > > > 166.333032] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu] [
> > > > 166.333043] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu] [
> > > > 166.333056] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu] [ 166.333067]
> > > > ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu] [ 166.333079]
> > > > ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu] [ 166.333088]
> > > > __vb2_queue_cancel+0xa8/0x705 [videobuf2_common] [ 166.333096] ?
> > > > __mutex_lock_interruptible_slowpath+0xf/0xf
> > > > [ 166.333104] vb2_core_streamoff+0x68/0xf8 [videobuf2_common] [
> > > > 166.333123] __video_do_ioctl+0x625/0x887 [videodev] [
> > > > 166.333142] ? copy_overflow+0x14/0x14 [videodev] [ 166.333147]
> > > > ? slab_free_freelist_hook+0x46/0x94 [ 166.333151] ?
> > > > kfree+0x107/0x1a0 [ 166.333169] video_usercopy+0x3a3/0x8ae
> > > > [videodev] [ 166.333187] ? copy_overflow+0x14/0x14 [videodev] [
> > > > 166.333203] ? v4l_enumstd+0x49/0x49 [videodev] [ 166.333207] ?
> > > > __wake_up_common+0x342/0x342 [ 166.333215] ?
> > > > atomic_long_add_return+0x15/0x24 [ 166.333219] ?
> > > > ldsem_up_read+0x15/0x29 [ 166.333223] ? tty_write+0x4c6/0x4d8 [
> > > > 166.333227] ? n_tty_receive_char_special+0x1152/0x1152
> > > > [ 166.333244] ? video_usercopy+0x8ae/0x8ae [videodev] [
> > > > 166.333260] v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.333266]
> > > > vfs_ioctl+0x76/0x89 [ 166.333271] do_vfs_ioctl+0xb33/0xb7e [
> > > > 166.333275] ? __switch_to_asm+0x40/0x70 [ 166.333279] ?
> > > > __switch_to_asm+0x40/0x70 [ 166.333282] ?
> > > > __switch_to_asm+0x34/0x70 [ 166.333286] ?
> > > > __switch_to_asm+0x40/0x70 [ 166.333290] ?
> > > > ioctl_preallocate+0x174/0x174 [ 166.333294] ?
> > > > __switch_to+0x71c/0xb00 [ 166.333299] ?
> > > > compat_start_thread+0x6b/0x6b [ 166.333302] ?
> > > > __switch_to_asm+0x34/0x70 [ 166.333305] ?
> > > > __switch_to_asm+0x40/0x70 [ 166.333309] ? mmdrop+0x12/0x23 [
> > > > 166.333313] ? finish_task_switch+0x34d/0x3de [ 166.333319] ?
> > > > __schedule+0x1004/0x1045 [ 166.333325] ?
> > > > firmware_map_remove+0x119/0x119 [ 166.333330]
> > > > ksys_ioctl+0x50/0x70 [ 166.333335] __x64_sys_ioctl+0x82/0x89 [
> > > > 166.333340] do_syscall_64+0xa0/0xd2 [ 166.333345]
> > > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 166.333349] RIP: 0033:0x7f2481541f47 [ 166.333354] Code: 00 00
> > > > 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff
> > > > c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01
> > > > f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48 [
> > > > 166.333357] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX:
> > > > 0000000000000010 [ 166.333362] RAX: ffffffffffffffda RBX:
> > > > 0000000000000000 RCX: 00007f2481541f47 [ 166.333364] RDX:
> > > > 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003 [
> > > > 166.333367] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09:
> > > > 00007f2481c24700 [ 166.333369] R10: 0000000000000020 R11:
> > > > 0000000000000246 R12: 0000555f1c494b06 [ 166.333372] R13:
> > > > 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > > >
> > > > [ 166.333383] Allocated by task 1305:
> > > > [ 166.333389] kasan_kmalloc+0x8a/0x98 [ 166.333392]
> > > > slab_post_alloc_hook+0x31/0x51 [ 166.333396]
> > > > kmem_cache_alloc+0xd7/0x174 [ 166.333399] alloc_iova+0x24/0x2ea
> > > > [ 166.333407] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu] [
> > > > 166.333415] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu] [
> > > > 166.333424] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu] [
> > > > 166.333433] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu] [ 166.333442]
> > > > ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu] [ 166.333449]
> > > > vb2_start_streaming+0x164/0x33b [videobuf2_common] [ 166.333455]
> > > > vb2_core_streamon+0x1a1/0x208 [videobuf2_common] [ 166.333471]
> > > > __video_do_ioctl+0x625/0x887 [videodev] [ 166.333487]
> > > > video_usercopy+0x3a3/0x8ae [videodev] [ 166.333501]
> > > > v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.333505]
> > > > vfs_ioctl+0x76/0x89 [ 166.333508] do_vfs_ioctl+0xb33/0xb7e [
> > > > 166.333511] ksys_ioctl+0x50/0x70 [ 166.333514]
> > > > __x64_sys_ioctl+0x82/0x89 [ 166.333518] do_syscall_64+0xa0/0xd2
> > > > [ 166.333521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > >
> > > > [ 166.333526] Freed by task 1301:
> > > > [ 166.333532] __kasan_slab_free+0xfa/0x11c [ 166.333535]
> > > > slab_free_freelist_hook+0x46/0x94 [ 166.333538]
> > > > kmem_cache_free+0x7b/0x172 [ 166.333542] __free_iova+0x23/0x31 [
> > > > 166.333550] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu] [
> > > > 166.333557] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu] [
> > > > 166.333566] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu] [
> > > > 166.333574] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu] [
> > > > 166.333584] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu] [ 166.333593]
> > > > ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu] [ 166.333599]
> > > > __vb2_queue_cancel+0xa8/0x705 [videobuf2_common] [ 166.333606]
> > > > vb2_core_streamoff+0x68/0xf8 [videobuf2_common] [ 166.333621]
> > > > __video_do_ioctl+0x625/0x887 [videodev] [ 166.333637]
> > > > video_usercopy+0x3a3/0x8ae [videodev] [ 166.333652]
> > > > v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.333655]
> > > > vfs_ioctl+0x76/0x89 [ 166.333658] do_vfs_ioctl+0xb33/0xb7e [
> > > > 166.333662] ksys_ioctl+0x50/0x70 [ 166.333665]
> > > > __x64_sys_ioctl+0x82/0x89 [ 166.333668] do_syscall_64+0xa0/0xd2
> > > > [ 166.333671] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > >
> > > > [ 166.333678] The buggy address belongs to the object at
> ffff888133823700
> > > > which belongs to the cache iommu_iova of size 40 [
> > > > 166.333685] The buggy address is located 24 bytes inside of
> > > > 40-byte region [ffff888133823700,
> > > > ffff888133823728) [ 166.333690] The buggy address belongs to the page:
> > > > [ 166.333696] page:ffffea0004ce0880 count:1 mapcount:0
> > > > mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0 [
> > > > 166.333703] flags: 0x200000000010200(slab|head) [ 166.333710]
> > > > raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70
> > > > ffff8881519e8640 [ 166.333717] raw: 0000000000000000
> > > > 0000000000120012 00000001ffffffff 0000000000000000 [ 166.333720]
> > > > page dumped because: kasan: bad access detected
> > > >
> > > > [ 166.333726] Memory state around the buggy address:
> > > > [ 166.333732] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc
> > > > fc fc fc fc fc [ 166.333737] ffff888133823680: fc fc fc fc fc fc
> > > > fc fc fc fc fc fc fc fc fc fc [ 166.333742] >ffff888133823700: fb fb fb fb fb fc fc
> fc fc fc fc fc fc fc fc fc
> > > > [ 166.333745] ^
> > > > [ 166.333750] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc
> > > > fc fc fc fc fc [ 166.333755] ffff888133823800: fc fc fc fc fc fc
> > > > fc fc fc fc fc fc fc fc fc fc [ 166.333759]
> > > >
> ================================================================
> ==
> > > > [ 166.333762] Disabling lock debugging due to kernel taint [
> > > > 166.333764]
> > > >
> ================================================================
> ==
> > > > [ 166.333770] BUG: KASAN: double-free or invalid-free in
> > > > kmem_cache_free+0x7b/0x172
> > > >
> > > > [ 166.333780] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-
> rc6+ #3
> > > > [ 166.333782] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018 [
> > > > 166.333783] Call Trace:
> > > > [ 166.333789] dump_stack+0x5b/0x81 [ 166.333795]
> > > > print_address_description+0x65/0x227
> > > > [ 166.333799] ? kmem_cache_free+0x7b/0x172 [ 166.333803]
> > > > kasan_report_invalid_free+0x67/0xa0
> > > > [ 166.333807] ? kmem_cache_free+0x7b/0x172 [ 166.333812]
> > > > __kasan_slab_free+0x86/0x11c [ 166.333817]
> > > > slab_free_freelist_hook+0x46/0x94 [ 166.333822]
> > > > kmem_cache_free+0x7b/0x172 [ 166.333826] ? __free_iova+0x23/0x31
> > > > [ 166.333831] __free_iova+0x23/0x31 [ 166.333840]
> > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu] [ 166.333851]
> > > > ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu] [ 166.333861]
> > > > ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu] [ 166.333872]
> > > > ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu] [ 166.333885]
> > > > imgu_s_stream+0xc0/0x6c0 [ipu3_imgu] [ 166.333896] ?
> > > > ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu] [ 166.333908]
> > > > ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu] [ 166.333917]
> > > > __vb2_queue_cancel+0xa8/0x705 [videobuf2_common] [ 166.333923] ?
> > > > __mutex_lock_interruptible_slowpath+0xf/0xf
> > > > [ 166.333932] vb2_core_streamoff+0x68/0xf8 [videobuf2_common] [
> > > > 166.333950] __video_do_ioctl+0x625/0x887 [videodev] [
> > > > 166.333970] ? copy_overflow+0x14/0x14 [videodev] [ 166.333974]
> > > > ? slab_free_freelist_hook+0x46/0x94 [ 166.333979] ?
> > > > kfree+0x107/0x1a0 [ 166.333997] video_usercopy+0x3a3/0x8ae
> > > > [videodev] [ 166.334015] ? copy_overflow+0x14/0x14 [videodev] [
> > > > 166.334031] ? v4l_enumstd+0x49/0x49 [videodev] [ 166.334035] ?
> > > > __wake_up_common+0x342/0x342 [ 166.334042] ?
> > > > atomic_long_add_return+0x15/0x24 [ 166.334046] ?
> > > > ldsem_up_read+0x15/0x29 [ 166.334050] ? tty_write+0x4c6/0x4d8 [
> > > > 166.334054] ? n_tty_receive_char_special+0x1152/0x1152
> > > > [ 166.334071] ? video_usercopy+0x8ae/0x8ae [videodev] [
> > > > 166.334087] v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.334092]
> > > > vfs_ioctl+0x76/0x89 [ 166.334097] do_vfs_ioctl+0xb33/0xb7e [
> > > > 166.334101] ? __switch_to_asm+0x40/0x70 [ 166.334105] ?
> > > > __switch_to_asm+0x40/0x70 [ 166.334108] ?
> > > > __switch_to_asm+0x34/0x70 [ 166.334111] ?
> > > > __switch_to_asm+0x40/0x70 [ 166.334116] ?
> > > > ioctl_preallocate+0x174/0x174 [ 166.334120] ?
> > > > __switch_to+0x71c/0xb00 [ 166.334124] ?
> > > > compat_start_thread+0x6b/0x6b [ 166.334127] ?
> > > > __switch_to_asm+0x34/0x70 [ 166.334130] ?
> > > > __switch_to_asm+0x40/0x70 [ 166.334134] ? mmdrop+0x12/0x23 [
> > > > 166.334137] ? finish_task_switch+0x34d/0x3de [ 166.334143] ?
> > > > __schedule+0x1004/0x1045 [ 166.334148] ?
> > > > firmware_map_remove+0x119/0x119 [ 166.334153]
> > > > ksys_ioctl+0x50/0x70 [ 166.334158] __x64_sys_ioctl+0x82/0x89 [
> > > > 166.334163] do_syscall_64+0xa0/0xd2 [ 166.334167]
> > > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 166.334171] RIP: 0033:0x7f2481541f47 [ 166.334175] Code: 00 00
> > > > 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff
> > > > c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01
> > > > f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48 [
> > > > 166.334177] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX:
> > > > 0000000000000010 [ 166.334181] RAX: ffffffffffffffda RBX:
> > > > 0000000000000000 RCX: 00007f2481541f47 [ 166.334184] RDX:
> > > > 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003 [
> > > > 166.334186] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09:
> > > > 00007f2481c24700 [ 166.334189] R10: 0000000000000020 R11:
> > > > 0000000000000246 R12: 0000555f1c494b06 [ 166.334191] R13:
> > > > 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > > >
[snip]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 17:00 ` Mani, Rajmohan
@ 2019-01-09 17:25 ` Jacopo Mondi
2019-01-09 18:01 ` Mani, Rajmohan
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-01-09 17:25 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
[-- Attachment #1.1: Type: text/plain, Size: 16557 bytes --]
Hello Raj,
On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> Hi Laurent, Tomasz, Jacopo,
>
> > -----Original Message-----
> > From: Jacopo Mondi [mailto:jacopo@jmondi.org]
> > Sent: Wednesday, January 09, 2019 8:41 AM
> > To: Tomasz Figa <tfiga@chromium.org>
> > Cc: Zhi, Yong <yong.zhi@intel.com>; Mani, Rajmohan
> > <rajmohan.mani@intel.com>; Qiu, Tian Shu <tian.shu.qiu@intel.com>; Cao,
> > Bingbu <bingbu.cao@intel.com>; Laurent Pinchart
> > <laurent.pinchart@ideasonboard.com>; Linux Media Mailing List <linux-
> > media@vger.kernel.org>; Sakari Ailus <sakari.ailus@linux.intel.com>; Mauro
> > Carvalho Chehab <mchehab@kernel.org>; Hans Verkuil
> > <hans.verkuil@cisco.com>; Hu, Jerry W <jerry.w.hu@intel.com>; Toivonen,
> > Tuukka <tuukka.toivonen@intel.com>
> > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> >
> > Hello,
> >
> > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > Hi Raj, Yong, Bingbu, Tianshu,
> > >
> > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > > >
> > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > >
> > > > > Hellon
> > > > >
> > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > > > > > Hello Yong,
> > > > > >
> > > > > > Could you please have a look at the crash reported below ?
> > > > >
> > > > > A bit more information to help you debugging this. I've enabled
> > > > > KASAN in the kernel configuration, and get the following use-after-free
> > reports.
> >
> > I tested as well using the ipu-process.sh script shared by Laurent, with the
> > following command line:
> > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
> >
> > and I got a very similar trace available at:
> > https://paste.debian.net/hidden/5855e15a/
> >
> > Please note I have been able to process a set of images (with KASAN enabled
> > the machine does not freeze) but the kernel log gets flooded and it is not
> > possible to process any other frame after this.
> >
> > The issue is currently quite annoying and it's a blocker for libcamera
> > development on IPU3. Please let me know if I can support with more testing.
> >
> > Thanks
> > j
> >
> > > > >
> > > > > [ 166.332920]
> > > > >
> > ================================================================
> > ==
> > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > [ 166.332944] Read of size 8 at addr ffff888133823718 by task
> > > > > yavta/1305
> > > > >
> > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C 4.20.0-
> > rc6+ #3
> > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018 [
> > > > > 166.332959] Call Trace:
> > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > print_address_description+0x65/0x227
> > > > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > [ 166.332995] private_free_iova+0x57/0x6d [ 166.332999]
> > > > > __free_iova+0x23/0x31 [ 166.333011] ipu3_dmamap_free+0x118/0x1d6
> > > > > [ipu3_imgu]
> > > >
> > > > Thanks Laurent, I think this is a very good hint. It looks like
> > > > we're basically freeing and already freed IOVA and corrupting some
> > > > allocator state?
> > >
> > > Did you have any luck in reproducing and fixing this double free issue?
> > >
>
> This issue is either hard to reproduce or comes with different signatures with
> the updated yavta (that now supports meta output) with the 4.4 kernel that
> I have been using.
> I am switching to 4.20-rc6 for better reproducibility.
> Enabling KASAN also results in storage space issues on my Chrome device.
> Will enable this just for ImgU to get ahead and get back with more updates.
>
Thanks for testing this.
For your informations I'm using the following branch, from Sakari's
tree: git://linuxtv.org/sailus/media_tree.git ipu3
Although it appears that the media tree master branch has everything
that is there, with a few additional patches on top. I should move to
use media tree master as well...
I have here attached 2 configuration files for v4.20-rc5 I am using on
Soraka, in case they might help you. One has KASAN enabled with an
increased kernel log size, the other one is the one we use for daily
development.
Also, please make sure to use (the most) recent media-ctl and yavta
utilities, as the ones provided by most distros are usually not recent
enough to work with IPU3, but I'm sure you know that already ;)
Thanks
j
> > > Best regards,
> > > Tomasz
> > >
> > > >
> > > > > [ 166.333022] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu] [
> > > > > 166.333032] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu] [
> > > > > 166.333043] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu] [
> > > > > 166.333056] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu] [ 166.333067]
> > > > > ? ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu] [ 166.333079]
> > > > > ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu] [ 166.333088]
> > > > > __vb2_queue_cancel+0xa8/0x705 [videobuf2_common] [ 166.333096] ?
> > > > > __mutex_lock_interruptible_slowpath+0xf/0xf
> > > > > [ 166.333104] vb2_core_streamoff+0x68/0xf8 [videobuf2_common] [
> > > > > 166.333123] __video_do_ioctl+0x625/0x887 [videodev] [
> > > > > 166.333142] ? copy_overflow+0x14/0x14 [videodev] [ 166.333147]
> > > > > ? slab_free_freelist_hook+0x46/0x94 [ 166.333151] ?
> > > > > kfree+0x107/0x1a0 [ 166.333169] video_usercopy+0x3a3/0x8ae
> > > > > [videodev] [ 166.333187] ? copy_overflow+0x14/0x14 [videodev] [
> > > > > 166.333203] ? v4l_enumstd+0x49/0x49 [videodev] [ 166.333207] ?
> > > > > __wake_up_common+0x342/0x342 [ 166.333215] ?
> > > > > atomic_long_add_return+0x15/0x24 [ 166.333219] ?
> > > > > ldsem_up_read+0x15/0x29 [ 166.333223] ? tty_write+0x4c6/0x4d8 [
> > > > > 166.333227] ? n_tty_receive_char_special+0x1152/0x1152
> > > > > [ 166.333244] ? video_usercopy+0x8ae/0x8ae [videodev] [
> > > > > 166.333260] v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.333266]
> > > > > vfs_ioctl+0x76/0x89 [ 166.333271] do_vfs_ioctl+0xb33/0xb7e [
> > > > > 166.333275] ? __switch_to_asm+0x40/0x70 [ 166.333279] ?
> > > > > __switch_to_asm+0x40/0x70 [ 166.333282] ?
> > > > > __switch_to_asm+0x34/0x70 [ 166.333286] ?
> > > > > __switch_to_asm+0x40/0x70 [ 166.333290] ?
> > > > > ioctl_preallocate+0x174/0x174 [ 166.333294] ?
> > > > > __switch_to+0x71c/0xb00 [ 166.333299] ?
> > > > > compat_start_thread+0x6b/0x6b [ 166.333302] ?
> > > > > __switch_to_asm+0x34/0x70 [ 166.333305] ?
> > > > > __switch_to_asm+0x40/0x70 [ 166.333309] ? mmdrop+0x12/0x23 [
> > > > > 166.333313] ? finish_task_switch+0x34d/0x3de [ 166.333319] ?
> > > > > __schedule+0x1004/0x1045 [ 166.333325] ?
> > > > > firmware_map_remove+0x119/0x119 [ 166.333330]
> > > > > ksys_ioctl+0x50/0x70 [ 166.333335] __x64_sys_ioctl+0x82/0x89 [
> > > > > 166.333340] do_syscall_64+0xa0/0xd2 [ 166.333345]
> > > > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > > [ 166.333349] RIP: 0033:0x7f2481541f47 [ 166.333354] Code: 00 00
> > > > > 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff
> > > > > c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01
> > > > > f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48 [
> > > > > 166.333357] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX:
> > > > > 0000000000000010 [ 166.333362] RAX: ffffffffffffffda RBX:
> > > > > 0000000000000000 RCX: 00007f2481541f47 [ 166.333364] RDX:
> > > > > 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003 [
> > > > > 166.333367] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09:
> > > > > 00007f2481c24700 [ 166.333369] R10: 0000000000000020 R11:
> > > > > 0000000000000246 R12: 0000555f1c494b06 [ 166.333372] R13:
> > > > > 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > > > >
> > > > > [ 166.333383] Allocated by task 1305:
> > > > > [ 166.333389] kasan_kmalloc+0x8a/0x98 [ 166.333392]
> > > > > slab_post_alloc_hook+0x31/0x51 [ 166.333396]
> > > > > kmem_cache_alloc+0xd7/0x174 [ 166.333399] alloc_iova+0x24/0x2ea
> > > > > [ 166.333407] ipu3_dmamap_alloc+0x193/0x83f [ipu3_imgu] [
> > > > > 166.333415] ipu3_css_pool_init+0x80/0xdf [ipu3_imgu] [
> > > > > 166.333424] ipu3_css_start_streaming+0x58df/0x5ddc [ipu3_imgu] [
> > > > > 166.333433] imgu_s_stream+0x2dd/0x6c0 [ipu3_imgu] [ 166.333442]
> > > > > ipu3_vb2_start_streaming+0x35f/0x3de [ipu3_imgu] [ 166.333449]
> > > > > vb2_start_streaming+0x164/0x33b [videobuf2_common] [ 166.333455]
> > > > > vb2_core_streamon+0x1a1/0x208 [videobuf2_common] [ 166.333471]
> > > > > __video_do_ioctl+0x625/0x887 [videodev] [ 166.333487]
> > > > > video_usercopy+0x3a3/0x8ae [videodev] [ 166.333501]
> > > > > v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.333505]
> > > > > vfs_ioctl+0x76/0x89 [ 166.333508] do_vfs_ioctl+0xb33/0xb7e [
> > > > > 166.333511] ksys_ioctl+0x50/0x70 [ 166.333514]
> > > > > __x64_sys_ioctl+0x82/0x89 [ 166.333518] do_syscall_64+0xa0/0xd2
> > > > > [ 166.333521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > >
> > > > > [ 166.333526] Freed by task 1301:
> > > > > [ 166.333532] __kasan_slab_free+0xfa/0x11c [ 166.333535]
> > > > > slab_free_freelist_hook+0x46/0x94 [ 166.333538]
> > > > > kmem_cache_free+0x7b/0x172 [ 166.333542] __free_iova+0x23/0x31 [
> > > > > 166.333550] ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu] [
> > > > > 166.333557] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu] [
> > > > > 166.333566] ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu] [
> > > > > 166.333574] ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu] [
> > > > > 166.333584] imgu_s_stream+0xc0/0x6c0 [ipu3_imgu] [ 166.333593]
> > > > > ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu] [ 166.333599]
> > > > > __vb2_queue_cancel+0xa8/0x705 [videobuf2_common] [ 166.333606]
> > > > > vb2_core_streamoff+0x68/0xf8 [videobuf2_common] [ 166.333621]
> > > > > __video_do_ioctl+0x625/0x887 [videodev] [ 166.333637]
> > > > > video_usercopy+0x3a3/0x8ae [videodev] [ 166.333652]
> > > > > v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.333655]
> > > > > vfs_ioctl+0x76/0x89 [ 166.333658] do_vfs_ioctl+0xb33/0xb7e [
> > > > > 166.333662] ksys_ioctl+0x50/0x70 [ 166.333665]
> > > > > __x64_sys_ioctl+0x82/0x89 [ 166.333668] do_syscall_64+0xa0/0xd2
> > > > > [ 166.333671] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > >
> > > > > [ 166.333678] The buggy address belongs to the object at
> > ffff888133823700
> > > > > which belongs to the cache iommu_iova of size 40 [
> > > > > 166.333685] The buggy address is located 24 bytes inside of
> > > > > 40-byte region [ffff888133823700,
> > > > > ffff888133823728) [ 166.333690] The buggy address belongs to the page:
> > > > > [ 166.333696] page:ffffea0004ce0880 count:1 mapcount:0
> > > > > mapping:ffff8881519e8640 index:0x0 compound_mapcount: 0 [
> > > > > 166.333703] flags: 0x200000000010200(slab|head) [ 166.333710]
> > > > > raw: 0200000000010200 ffffea0004dfc488 ffff88814bfbde70
> > > > > ffff8881519e8640 [ 166.333717] raw: 0000000000000000
> > > > > 0000000000120012 00000001ffffffff 0000000000000000 [ 166.333720]
> > > > > page dumped because: kasan: bad access detected
> > > > >
> > > > > [ 166.333726] Memory state around the buggy address:
> > > > > [ 166.333732] ffff888133823600: fc fc fc fc fc fc fc fc fc fc fc
> > > > > fc fc fc fc fc [ 166.333737] ffff888133823680: fc fc fc fc fc fc
> > > > > fc fc fc fc fc fc fc fc fc fc [ 166.333742] >ffff888133823700: fb fb fb fb fb fc fc
> > fc fc fc fc fc fc fc fc fc
> > > > > [ 166.333745] ^
> > > > > [ 166.333750] ffff888133823780: fc fc fc fc fc fc fc fc fc fc fc
> > > > > fc fc fc fc fc [ 166.333755] ffff888133823800: fc fc fc fc fc fc
> > > > > fc fc fc fc fc fc fc fc fc fc [ 166.333759]
> > > > >
> > ================================================================
> > ==
> > > > > [ 166.333762] Disabling lock debugging due to kernel taint [
> > > > > 166.333764]
> > > > >
> > ================================================================
> > ==
> > > > > [ 166.333770] BUG: KASAN: double-free or invalid-free in
> > > > > kmem_cache_free+0x7b/0x172
> > > > >
> > > > > [ 166.333780] CPU: 3 PID: 1305 Comm: yavta Tainted: G B C 4.20.0-
> > rc6+ #3
> > > > > [ 166.333782] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018 [
> > > > > 166.333783] Call Trace:
> > > > > [ 166.333789] dump_stack+0x5b/0x81 [ 166.333795]
> > > > > print_address_description+0x65/0x227
> > > > > [ 166.333799] ? kmem_cache_free+0x7b/0x172 [ 166.333803]
> > > > > kasan_report_invalid_free+0x67/0xa0
> > > > > [ 166.333807] ? kmem_cache_free+0x7b/0x172 [ 166.333812]
> > > > > __kasan_slab_free+0x86/0x11c [ 166.333817]
> > > > > slab_free_freelist_hook+0x46/0x94 [ 166.333822]
> > > > > kmem_cache_free+0x7b/0x172 [ 166.333826] ? __free_iova+0x23/0x31
> > > > > [ 166.333831] __free_iova+0x23/0x31 [ 166.333840]
> > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu] [ 166.333851]
> > > > > ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu] [ 166.333861]
> > > > > ipu3_css_pipeline_cleanup+0x79/0xcf [ipu3_imgu] [ 166.333872]
> > > > > ipu3_css_stop_streaming+0x2fe/0x4dc [ipu3_imgu] [ 166.333885]
> > > > > imgu_s_stream+0xc0/0x6c0 [ipu3_imgu] [ 166.333896] ?
> > > > > ipu3_all_nodes_streaming+0x1ee/0x20d [ipu3_imgu] [ 166.333908]
> > > > > ipu3_vb2_stop_streaming+0x27c/0x2d2 [ipu3_imgu] [ 166.333917]
> > > > > __vb2_queue_cancel+0xa8/0x705 [videobuf2_common] [ 166.333923] ?
> > > > > __mutex_lock_interruptible_slowpath+0xf/0xf
> > > > > [ 166.333932] vb2_core_streamoff+0x68/0xf8 [videobuf2_common] [
> > > > > 166.333950] __video_do_ioctl+0x625/0x887 [videodev] [
> > > > > 166.333970] ? copy_overflow+0x14/0x14 [videodev] [ 166.333974]
> > > > > ? slab_free_freelist_hook+0x46/0x94 [ 166.333979] ?
> > > > > kfree+0x107/0x1a0 [ 166.333997] video_usercopy+0x3a3/0x8ae
> > > > > [videodev] [ 166.334015] ? copy_overflow+0x14/0x14 [videodev] [
> > > > > 166.334031] ? v4l_enumstd+0x49/0x49 [videodev] [ 166.334035] ?
> > > > > __wake_up_common+0x342/0x342 [ 166.334042] ?
> > > > > atomic_long_add_return+0x15/0x24 [ 166.334046] ?
> > > > > ldsem_up_read+0x15/0x29 [ 166.334050] ? tty_write+0x4c6/0x4d8 [
> > > > > 166.334054] ? n_tty_receive_char_special+0x1152/0x1152
> > > > > [ 166.334071] ? video_usercopy+0x8ae/0x8ae [videodev] [
> > > > > 166.334087] v4l2_ioctl+0xb7/0xc5 [videodev] [ 166.334092]
> > > > > vfs_ioctl+0x76/0x89 [ 166.334097] do_vfs_ioctl+0xb33/0xb7e [
> > > > > 166.334101] ? __switch_to_asm+0x40/0x70 [ 166.334105] ?
> > > > > __switch_to_asm+0x40/0x70 [ 166.334108] ?
> > > > > __switch_to_asm+0x34/0x70 [ 166.334111] ?
> > > > > __switch_to_asm+0x40/0x70 [ 166.334116] ?
> > > > > ioctl_preallocate+0x174/0x174 [ 166.334120] ?
> > > > > __switch_to+0x71c/0xb00 [ 166.334124] ?
> > > > > compat_start_thread+0x6b/0x6b [ 166.334127] ?
> > > > > __switch_to_asm+0x34/0x70 [ 166.334130] ?
> > > > > __switch_to_asm+0x40/0x70 [ 166.334134] ? mmdrop+0x12/0x23 [
> > > > > 166.334137] ? finish_task_switch+0x34d/0x3de [ 166.334143] ?
> > > > > __schedule+0x1004/0x1045 [ 166.334148] ?
> > > > > firmware_map_remove+0x119/0x119 [ 166.334153]
> > > > > ksys_ioctl+0x50/0x70 [ 166.334158] __x64_sys_ioctl+0x82/0x89 [
> > > > > 166.334163] do_syscall_64+0xa0/0xd2 [ 166.334167]
> > > > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > > [ 166.334171] RIP: 0033:0x7f2481541f47 [ 166.334175] Code: 00 00
> > > > > 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff
> > > > > c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01
> > > > > f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48 [
> > > > > 166.334177] RSP: 002b:00007fffd6aff9b8 EFLAGS: 00000246 ORIG_RAX:
> > > > > 0000000000000010 [ 166.334181] RAX: ffffffffffffffda RBX:
> > > > > 0000000000000000 RCX: 00007f2481541f47 [ 166.334184] RDX:
> > > > > 00007fffd6aff9c4 RSI: 0000000040045613 RDI: 0000000000000003 [
> > > > > 166.334186] RBP: 0000555f1c494af8 R08: 00007f247f34c000 R09:
> > > > > 00007f2481c24700 [ 166.334189] R10: 0000000000000020 R11:
> > > > > 0000000000000246 R12: 0000555f1c494b06 [ 166.334191] R13:
> > > > > 0000000000000004 R14: 00007fffd6affb90 R15: 00007fffd6b00825
> > > > >
>
> [snip]
[-- Attachment #1.2: linux-v4.20-rc5-ipu3.config --]
[-- Type: text/plain, Size: 18022 bytes --]
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_LOG_BUF_SHIFT=18
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL_SYSCALL=y
# CONFIG_PCSPKR_PLATFORM is not set
CONFIG_KALLSYMS_ALL=y
CONFIG_BPF_SYSCALL=y
CONFIG_EMBEDDED=y
# CONFIG_COMPAT_BRK is not set
CONFIG_PROFILING=y
CONFIG_SMP=y
CONFIG_X86_X2APIC=y
CONFIG_X86_NUMACHIP=y
CONFIG_X86_INTEL_LPSS=y
CONFIG_IOSF_MBI_DEBUG=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_KVM_DEBUG_FS=y
CONFIG_PROCESSOR_SELECT=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_NR_CPUS=256
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_I8K=m
CONFIG_MICROCODE_AMD=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_NUMA=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_X86_PMEM_LEGACY=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_EFI=y
CONFIG_EFI_STUB=y
CONFIG_EFI_MIXED=y
CONFIG_HZ_1000=y
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_KEXEC_VERIFY_SIG=y
CONFIG_CRASH_DUMP=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_LIVEPATCH=y
CONFIG_HIBERNATION=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_PM_TRACE_RTC=y
CONFIG_WQ_POWER_EFFICIENT_DEFAULT=y
CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_PCI_SLOT=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_SBS=m
CONFIG_ACPI_BGRT=y
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
CONFIG_TPS68470_PMIC_OPREGION=y
CONFIG_SFI=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_INTEL_IDLE=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEASPM_DEBUG=y
CONFIG_PCI_MSI=y
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
CONFIG_PCI_STUB=m
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=y
CONFIG_IA32_EMULATION=y
CONFIG_X86_X32=y
CONFIG_EDD=y
CONFIG_EDD_OFF=y
CONFIG_DMI_SYSFS=m
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_EFI_VARS=y
CONFIG_EFI_VARS_PSTORE=m
# CONFIG_VIRTUALIZATION is not set
CONFIG_OPROFILE=m
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_BLK_DEV_THROTTLING=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_EESOX=y
CONFIG_ACORN_PARTITION_ICS=y
CONFIG_ACORN_PARTITION_ADFS=y
CONFIG_ACORN_PARTITION_POWERTEC=y
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_AIX_PARTITION=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_SYSV68_PARTITION=y
CONFIG_CMDLINE_PARTITION=y
CONFIG_IOSCHED_CFQ=m
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_IOSCHED_BFQ=y
CONFIG_BFQ_GROUP_IOSCHED=y
CONFIG_BINFMT_MISC=m
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_CMA=y
CONFIG_MEM_SOFT_DIRTY=y
CONFIG_ZSWAP=y
CONFIG_ZBUD=y
CONFIG_ZSMALLOC=y
CONFIG_PGTABLE_MAPPING=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_FIB_TRIE_STATS=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_SYN_COOKIES=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
# CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET6_XFRM_MODE_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_BEET is not set
# CONFIG_IPV6_SIT is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NETFILTER_NETLINK_OSF=m
CONFIG_NF_TABLES=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
CONFIG_NF_SOCKET_IPV4=m
CONFIG_NF_TPROXY_IPV4=m
CONFIG_NF_DUP_IPV4=m
CONFIG_NF_LOG_ARP=m
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_NF_SOCKET_IPV6=m
CONFIG_NF_TPROXY_IPV6=m
CONFIG_NF_DUP_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
CONFIG_DNS_RESOLVER=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_BPF_JIT=y
CONFIG_CFG80211=m
CONFIG_CFG80211_DEBUGFS=y
CONFIG_CFG80211_WEXT=y
CONFIG_MAC80211=m
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_MAC80211_MESSAGE_TRACING=y
CONFIG_RFKILL=y
CONFIG_RFKILL_INPUT=y
CONFIG_RFKILL_GPIO=m
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_STANDALONE is not set
CONFIG_DEBUG_DEVRES=y
CONFIG_CONNECTOR=y
CONFIG_OF=y
# CONFIG_PNP_DEBUG_MESSAGES is not set
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=m
CONFIG_ZRAM=m
CONFIG_BLK_DEV_UMEM=m
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_SKD=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=65536
CONFIG_BLK_DEV_NVME=m
CONFIG_SRAM=y
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_93CX6=m
CONFIG_CB710_CORE=m
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
# CONFIG_SCSI_MQ_DEFAULT is not set
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=m
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SRP_ATTRS=m
# CONFIG_SCSI_LOWLEVEL is not set
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_ATA=m
CONFIG_SATA_AHCI=m
CONFIG_SATA_AHCI_PLATFORM=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
CONFIG_LOOPBACK_TARGET=m
CONFIG_ISCSI_TARGET=m
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=m
CONFIG_NETDEVICES=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
CONFIG_EQUALIZER=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_RANDOM=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
CONFIG_IPVLAN=m
CONFIG_VXLAN=m
CONFIG_GENEVE=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_TUN=y
CONFIG_VETH=m
CONFIG_NLMON=m
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
# CONFIG_NET_VENDOR_AGERE is not set
# CONFIG_NET_VENDOR_ALACRITECH is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_NET_VENDOR_AMAZON is not set
# CONFIG_NET_VENDOR_AMD is not set
# CONFIG_NET_VENDOR_AQUANTIA is not set
# CONFIG_NET_VENDOR_ARC is not set
# CONFIG_NET_VENDOR_ATHEROS is not set
# CONFIG_NET_VENDOR_AURORA is not set
# CONFIG_NET_VENDOR_BROADCOM is not set
# CONFIG_NET_VENDOR_BROCADE is not set
# CONFIG_NET_VENDOR_CADENCE is not set
# CONFIG_NET_VENDOR_CAVIUM is not set
# CONFIG_NET_VENDOR_CHELSIO is not set
# CONFIG_NET_VENDOR_CISCO is not set
# CONFIG_NET_VENDOR_CORTINA is not set
# CONFIG_NET_VENDOR_DEC is not set
# CONFIG_NET_VENDOR_DLINK is not set
# CONFIG_NET_VENDOR_EMULEX is not set
# CONFIG_NET_VENDOR_EZCHIP is not set
# CONFIG_NET_VENDOR_HP is not set
# CONFIG_NET_VENDOR_HUAWEI is not set
# CONFIG_NET_VENDOR_I825XX is not set
CONFIG_E1000=m
CONFIG_IGBVF=m
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MELLANOX is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
# CONFIG_NET_VENDOR_MICROSEMI is not set
# CONFIG_NET_VENDOR_MYRI is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
# CONFIG_NET_VENDOR_NETERION is not set
# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_NI is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
# CONFIG_NET_VENDOR_OKI is not set
# CONFIG_NET_VENDOR_PACKET_ENGINES is not set
# CONFIG_NET_VENDOR_QLOGIC is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RDC is not set
# CONFIG_NET_VENDOR_REALTEK is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
# CONFIG_NET_VENDOR_SAMSUNG is not set
# CONFIG_NET_VENDOR_SEEQ is not set
# CONFIG_NET_VENDOR_SOLARFLARE is not set
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
# CONFIG_NET_VENDOR_SMSC is not set
# CONFIG_NET_VENDOR_SOCIONEXT is not set
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
# CONFIG_NET_VENDOR_SYNOPSYS is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
# CONFIG_NET_VENDOR_TI is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_PHYLIB=y
CONFIG_PPP=y
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPPOE=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_USB_NET_DRIVERS=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_CDC_EEM=m
CONFIG_USB_NET_CDC_MBIM=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_RNDIS_HOST=m
# CONFIG_USB_ARMLINUX is not set
# CONFIG_USB_NET_ZAURUS is not set
# CONFIG_WLAN_VENDOR_ADMTEK is not set
# CONFIG_WLAN_VENDOR_ATH is not set
# CONFIG_WLAN_VENDOR_ATMEL is not set
# CONFIG_WLAN_VENDOR_BROADCOM is not set
# CONFIG_WLAN_VENDOR_CISCO is not set
CONFIG_IWL4965=m
CONFIG_IWL3945=m
CONFIG_IWLWIFI=m
CONFIG_IWLDVM=m
CONFIG_IWLMVM=m
CONFIG_IWLWIFI_DEBUGFS=y
# CONFIG_WLAN_VENDOR_MARVELL is not set
# CONFIG_WLAN_VENDOR_MEDIATEK is not set
# CONFIG_WLAN_VENDOR_RALINK is not set
# CONFIG_WLAN_VENDOR_REALTEK is not set
# CONFIG_WLAN_VENDOR_RSI is not set
# CONFIG_WLAN_VENDOR_ST is not set
# CONFIG_WLAN_VENDOR_TI is not set
# CONFIG_WLAN_VENDOR_ZYDAS is not set
# CONFIG_WLAN_VENDOR_QUANTENNA is not set
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_EVBUG=m
CONFIG_KEYBOARD_CROS_EC=m
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
CONFIG_MOUSE_PS2_TOUCHKIT=y
CONFIG_MOUSE_PS2_VMMOUSE=y
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_ATMEL_MXT=y
CONFIG_TOUCHSCREEN_ELAN=y
CONFIG_TOUCHSCREEN_MELFAS_MIP4=y
CONFIG_TOUCHSCREEN_WDT87XX_I2C=y
CONFIG_TOUCHSCREEN_USB_COMPOSITE=m
CONFIG_TOUCHSCREEN_RM_TS=y
CONFIG_INPUT_MISC=y
CONFIG_INPUT_UINPUT=m
CONFIG_LEGACY_PTY_COUNT=0
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=48
CONFIG_SERIAL_8250_RUNTIME_UARTS=32
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_8250_DW=m
# CONFIG_SERIAL_8250_MID is not set
CONFIG_SERIAL_KGDB_NMI=y
CONFIG_TTY_PRINTK=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_SSIF=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_NVRAM=y
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_HPET=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y
CONFIG_TCG_TIS_I2C_ATMEL=m
CONFIG_TCG_TIS_I2C_INFINEON=m
CONFIG_TCG_TIS_I2C_NUVOTON=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
CONFIG_TELCLOCK=m
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=m
CONFIG_I2C_ISMT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_SCMI=m
CONFIG_I2C_DESIGNWARE_PCI=y
CONFIG_I2C_DESIGNWARE_BAYTRAIL=y
CONFIG_I2C_CROS_EC_TUNNEL=m
CONFIG_I2C_SLAVE=y
CONFIG_I2C_SLAVE_EEPROM=m
CONFIG_SPI=y
CONFIG_PINCTRL_BAYTRAIL=y
CONFIG_PINCTRL_CHERRYVIEW=y
CONFIG_PINCTRL_BROXTON=y
CONFIG_PINCTRL_SUNRISEPOINT=y
CONFIG_DEBUG_GPIO=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_GENERIC_PLATFORM=m
CONFIG_GPIO_ICH=m
CONFIG_GPIO_LYNXPOINT=m
CONFIG_GPIO_SCH=m
CONFIG_GPIO_TPS68470=y
CONFIG_POWER_RESET=y
CONFIG_PDA_POWER=m
CONFIG_GENERIC_ADC_BATTERY=m
CONFIG_BATTERY_SBS=m
CONFIG_CHARGER_GPIO=m
CONFIG_CHARGER_MANAGER=y
CONFIG_SENSORS_I5500=m
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_ACPI_POWER=m
CONFIG_THERMAL_GOV_FAIR_SHARE=y
CONFIG_THERMAL_GOV_BANG_BANG=y
CONFIG_THERMAL_EMULATION=y
CONFIG_INTEL_POWERCLAMP=m
CONFIG_INTEL_SOC_DTS_THERMAL=m
CONFIG_INT340X_THERMAL=m
CONFIG_MFD_CROS_EC=m
CONFIG_LPC_ICH=y
CONFIG_LPC_SCH=y
CONFIG_MFD_INTEL_LPSS_ACPI=y
CONFIG_MFD_INTEL_LPSS_PCI=y
CONFIG_MFD_TPS68470=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=m
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
CONFIG_REGULATOR_USERSPACE_CONSUMER=m
CONFIG_REGULATOR_GPIO=m
CONFIG_REGULATOR_TPS51632=m
CONFIG_REGULATOR_TPS62360=m
CONFIG_REGULATOR_TPS65023=m
CONFIG_REGULATOR_TPS6507X=m
CONFIG_MEDIA_SUPPORT=m
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_CONTROLLER=y
CONFIG_VIDEO_V4L2_SUBDEV_API=y
CONFIG_MEDIA_USB_SUPPORT=y
CONFIG_USB_VIDEO_CLASS=m
# CONFIG_USB_GSPCA is not set
CONFIG_MEDIA_PCI_SUPPORT=y
CONFIG_VIDEO_IPU3_CIO2=m
CONFIG_V4L_PLATFORM_DRIVERS=y
# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
CONFIG_VIDEO_DW9714=m
CONFIG_VIDEO_OV5670=m
CONFIG_VIDEO_OV7670=m
CONFIG_VIDEO_OV13858=m
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_I915=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_TILEBLITTING=y
CONFIG_FB_INTEL=m
CONFIG_FB_SIMPLE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=m
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
CONFIG_HID=m
CONFIG_HID_BATTERY_STRENGTH=y
CONFIG_HIDRAW=y
CONFIG_UHID=m
CONFIG_HID_MULTITOUCH=m
CONFIG_USB_HID=m
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_KBD=m
CONFIG_USB_MOUSE=m
CONFIG_I2C_HID=m
CONFIG_USB=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
CONFIG_USB_DYNAMIC_MINORS=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_HCD_PLATFORM=y
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PLATFORM=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=m
CONFIG_USB_UAS=m
CONFIG_NOP_USB_XCEIV=y
CONFIG_USB_GPIO_VBUS=m
CONFIG_MMC=y
CONFIG_MMC_BLOCK=m
CONFIG_SDIO_UART=m
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
# CONFIG_MMC_RICOH_MMC is not set
CONFIG_MMC_SDHCI_ACPI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_LEDS_CLASS=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_HID_SENSOR_TIME=m
CONFIG_DMADEVICES=y
CONFIG_INTEL_IOATDMA=m
CONFIG_DW_DMAC=y
CONFIG_ASYNC_TX_DMA=y
CONFIG_AUXDISPLAY=y
# CONFIG_VIRTIO_MENU is not set
CONFIG_STAGING=y
CONFIG_STAGING_MEDIA=y
CONFIG_VIDEO_IPU3_IMGU=m
CONFIG_DCDBAS=m
CONFIG_DELL_RBU=m
CONFIG_CHROMEOS_LAPTOP=m
CONFIG_CHROMEOS_PSTORE=m
CONFIG_CROS_EC_LPC=m
CONFIG_CROS_KBD_LED_BACKLIGHT=m
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_IRQ_REMAP=y
CONFIG_MEMORY=y
CONFIG_IIO_BUFFER=y
CONFIG_IIO_KFIFO_BUF=m
CONFIG_RESET_CONTROLLER=y
CONFIG_GENERIC_PHY=y
CONFIG_POWERCAP=y
CONFIG_INTEL_RAPL=m
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_EXT4_ENCRYPTION=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=y
CONFIG_CUSE=m
CONFIG_OVERLAY_FS=m
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
CONFIG_CACHEFILES=m
CONFIG_VFAT_FS=y
CONFIG_PROC_KCORE=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_EFIVAR_FS=y
CONFIG_PSTORE_RAM=m
CONFIG_NFS_FS=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
CONFIG_NFS_SWAP=y
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_NFS_V4_1_MIGRATION=y
CONFIG_NFS_FSCACHE=y
CONFIG_SUNRPC_DEBUG=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_UTF8=y
CONFIG_PERSISTENT_KEYRINGS=y
CONFIG_TRUSTED_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_INTEL_TXT=y
CONFIG_SECURITY_APPARMOR=y
# CONFIG_INTEGRITY is not set
CONFIG_CRYPTO_ECDH=m
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_CRC32C_INTEL=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_DRBG_HASH=y
CONFIG_CRYPTO_DRBG_CTR=y
CONFIG_CRYPTO_DEV_PADLOCK=y
CONFIG_CRYPTO_DEV_CCP=y
# CONFIG_CRYPTO_DEV_CCP_DD is not set
CONFIG_LIBCRC32C=y
# CONFIG_XZ_DEC_POWERPC is not set
# CONFIG_XZ_DEC_ARM is not set
# CONFIG_XZ_DEC_ARMTHUMB is not set
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_DDR=y
CONFIG_PRINTK_TIME=y
CONFIG_BOOT_PRINTK_DELAY=y
CONFIG_DYNAMIC_DEBUG=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_SCHEDSTATS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_NOTIFIER_ERROR_INJECTION=m
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_MMIOTRACE=y
CONFIG_MEMTEST=y
CONFIG_KGDB=y
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_EARLY_PRINTK_EFI=y
CONFIG_IO_DELAY_0XED=y
CONFIG_OPTIMIZE_INLINING=y
[-- Attachment #1.3: linux-v4.20-rc5-ipu3-kasan-kernel-log.config --]
[-- Type: text/plain, Size: 18148 bytes --]
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_LOG_CPU_MAX_BUF_SHIFT=17
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=17
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL_SYSCALL=y
# CONFIG_PCSPKR_PLATFORM is not set
CONFIG_KALLSYMS_ALL=y
CONFIG_BPF_SYSCALL=y
CONFIG_EMBEDDED=y
# CONFIG_COMPAT_BRK is not set
CONFIG_PROFILING=y
CONFIG_SMP=y
CONFIG_X86_X2APIC=y
CONFIG_X86_NUMACHIP=y
CONFIG_X86_INTEL_LPSS=y
CONFIG_IOSF_MBI_DEBUG=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_KVM_DEBUG_FS=y
CONFIG_PROCESSOR_SELECT=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_NR_CPUS=256
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_I8K=m
CONFIG_MICROCODE_AMD=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_NUMA=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_X86_PMEM_LEGACY=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_EFI=y
CONFIG_EFI_STUB=y
CONFIG_EFI_MIXED=y
CONFIG_HZ_1000=y
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_KEXEC_VERIFY_SIG=y
CONFIG_CRASH_DUMP=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_LIVEPATCH=y
CONFIG_HIBERNATION=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_PM_TRACE_RTC=y
CONFIG_WQ_POWER_EFFICIENT_DEFAULT=y
CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_PCI_SLOT=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_SBS=m
CONFIG_ACPI_BGRT=y
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
CONFIG_TPS68470_PMIC_OPREGION=y
CONFIG_SFI=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_INTEL_IDLE=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEASPM_DEBUG=y
CONFIG_PCI_MSI=y
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
CONFIG_PCI_STUB=m
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=y
CONFIG_IA32_EMULATION=y
CONFIG_X86_X32=y
CONFIG_EDD=y
CONFIG_EDD_OFF=y
CONFIG_DMI_SYSFS=m
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_EFI_VARS=y
CONFIG_EFI_VARS_PSTORE=m
# CONFIG_VIRTUALIZATION is not set
CONFIG_OPROFILE=m
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_BLK_DEV_THROTTLING=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_EESOX=y
CONFIG_ACORN_PARTITION_ICS=y
CONFIG_ACORN_PARTITION_ADFS=y
CONFIG_ACORN_PARTITION_POWERTEC=y
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_AIX_PARTITION=y
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_SYSV68_PARTITION=y
CONFIG_CMDLINE_PARTITION=y
CONFIG_IOSCHED_CFQ=m
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_IOSCHED_BFQ=y
CONFIG_BFQ_GROUP_IOSCHED=y
CONFIG_BINFMT_MISC=m
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_CMA=y
CONFIG_MEM_SOFT_DIRTY=y
CONFIG_ZSWAP=y
CONFIG_ZBUD=y
CONFIG_ZSMALLOC=y
CONFIG_PGTABLE_MAPPING=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_FIB_TRIE_STATS=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_SYN_COOKIES=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
# CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET6_XFRM_MODE_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_BEET is not set
# CONFIG_IPV6_SIT is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NETFILTER_NETLINK_OSF=m
CONFIG_NF_TABLES=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_QUEUE=m
CONFIG_NFT_REJECT=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
CONFIG_NF_SOCKET_IPV4=m
CONFIG_NF_TPROXY_IPV4=m
CONFIG_NF_DUP_IPV4=m
CONFIG_NF_LOG_ARP=m
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_NF_SOCKET_IPV6=m
CONFIG_NF_TPROXY_IPV6=m
CONFIG_NF_DUP_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
CONFIG_DNS_RESOLVER=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_BPF_JIT=y
CONFIG_CFG80211=m
CONFIG_CFG80211_DEBUGFS=y
CONFIG_CFG80211_WEXT=y
CONFIG_MAC80211=m
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_MAC80211_MESSAGE_TRACING=y
CONFIG_RFKILL=y
CONFIG_RFKILL_INPUT=y
CONFIG_RFKILL_GPIO=m
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_STANDALONE is not set
CONFIG_DEBUG_DEVRES=y
CONFIG_CONNECTOR=y
CONFIG_OF=y
# CONFIG_PNP_DEBUG_MESSAGES is not set
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=m
CONFIG_ZRAM=m
CONFIG_BLK_DEV_UMEM=m
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_SKD=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=65536
CONFIG_BLK_DEV_NVME=m
CONFIG_SRAM=y
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_93CX6=m
CONFIG_CB710_CORE=m
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
# CONFIG_SCSI_MQ_DEFAULT is not set
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=m
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SRP_ATTRS=m
# CONFIG_SCSI_LOWLEVEL is not set
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_ATA=m
CONFIG_SATA_AHCI=m
CONFIG_SATA_AHCI_PLATFORM=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
CONFIG_LOOPBACK_TARGET=m
CONFIG_ISCSI_TARGET=m
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=m
CONFIG_NETDEVICES=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
CONFIG_EQUALIZER=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_RANDOM=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
CONFIG_IPVLAN=m
CONFIG_VXLAN=m
CONFIG_GENEVE=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_TUN=y
CONFIG_VETH=m
CONFIG_NLMON=m
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
# CONFIG_NET_VENDOR_AGERE is not set
# CONFIG_NET_VENDOR_ALACRITECH is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_NET_VENDOR_AMAZON is not set
# CONFIG_NET_VENDOR_AMD is not set
# CONFIG_NET_VENDOR_AQUANTIA is not set
# CONFIG_NET_VENDOR_ARC is not set
# CONFIG_NET_VENDOR_ATHEROS is not set
# CONFIG_NET_VENDOR_AURORA is not set
# CONFIG_NET_VENDOR_BROADCOM is not set
# CONFIG_NET_VENDOR_BROCADE is not set
# CONFIG_NET_VENDOR_CADENCE is not set
# CONFIG_NET_VENDOR_CAVIUM is not set
# CONFIG_NET_VENDOR_CHELSIO is not set
# CONFIG_NET_VENDOR_CISCO is not set
# CONFIG_NET_VENDOR_CORTINA is not set
# CONFIG_NET_VENDOR_DEC is not set
# CONFIG_NET_VENDOR_DLINK is not set
# CONFIG_NET_VENDOR_EMULEX is not set
# CONFIG_NET_VENDOR_EZCHIP is not set
# CONFIG_NET_VENDOR_HP is not set
# CONFIG_NET_VENDOR_HUAWEI is not set
# CONFIG_NET_VENDOR_I825XX is not set
CONFIG_E1000=m
CONFIG_IGBVF=m
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MELLANOX is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
# CONFIG_NET_VENDOR_MICROSEMI is not set
# CONFIG_NET_VENDOR_MYRI is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
# CONFIG_NET_VENDOR_NETERION is not set
# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_NI is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
# CONFIG_NET_VENDOR_OKI is not set
# CONFIG_NET_VENDOR_PACKET_ENGINES is not set
# CONFIG_NET_VENDOR_QLOGIC is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RDC is not set
# CONFIG_NET_VENDOR_REALTEK is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
# CONFIG_NET_VENDOR_SAMSUNG is not set
# CONFIG_NET_VENDOR_SEEQ is not set
# CONFIG_NET_VENDOR_SOLARFLARE is not set
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
# CONFIG_NET_VENDOR_SMSC is not set
# CONFIG_NET_VENDOR_SOCIONEXT is not set
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
# CONFIG_NET_VENDOR_SYNOPSYS is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
# CONFIG_NET_VENDOR_TI is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_PHYLIB=y
CONFIG_PPP=y
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPPOE=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_USB_NET_DRIVERS=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_CDC_EEM=m
CONFIG_USB_NET_CDC_MBIM=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_RNDIS_HOST=m
# CONFIG_USB_ARMLINUX is not set
# CONFIG_USB_NET_ZAURUS is not set
# CONFIG_WLAN_VENDOR_ADMTEK is not set
# CONFIG_WLAN_VENDOR_ATH is not set
# CONFIG_WLAN_VENDOR_ATMEL is not set
# CONFIG_WLAN_VENDOR_BROADCOM is not set
# CONFIG_WLAN_VENDOR_CISCO is not set
CONFIG_IWL4965=m
CONFIG_IWL3945=m
CONFIG_IWLWIFI=m
CONFIG_IWLDVM=m
CONFIG_IWLMVM=m
CONFIG_IWLWIFI_DEBUGFS=y
# CONFIG_WLAN_VENDOR_MARVELL is not set
# CONFIG_WLAN_VENDOR_MEDIATEK is not set
# CONFIG_WLAN_VENDOR_RALINK is not set
# CONFIG_WLAN_VENDOR_REALTEK is not set
# CONFIG_WLAN_VENDOR_RSI is not set
# CONFIG_WLAN_VENDOR_ST is not set
# CONFIG_WLAN_VENDOR_TI is not set
# CONFIG_WLAN_VENDOR_ZYDAS is not set
# CONFIG_WLAN_VENDOR_QUANTENNA is not set
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_EVBUG=m
CONFIG_KEYBOARD_CROS_EC=m
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
CONFIG_MOUSE_PS2_TOUCHKIT=y
CONFIG_MOUSE_PS2_VMMOUSE=y
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_ATMEL_MXT=y
CONFIG_TOUCHSCREEN_ELAN=y
CONFIG_TOUCHSCREEN_MELFAS_MIP4=y
CONFIG_TOUCHSCREEN_WDT87XX_I2C=y
CONFIG_TOUCHSCREEN_USB_COMPOSITE=m
CONFIG_TOUCHSCREEN_RM_TS=y
CONFIG_INPUT_MISC=y
CONFIG_INPUT_UINPUT=m
CONFIG_LEGACY_PTY_COUNT=0
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=48
CONFIG_SERIAL_8250_RUNTIME_UARTS=32
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_8250_DW=m
# CONFIG_SERIAL_8250_MID is not set
CONFIG_SERIAL_KGDB_NMI=y
CONFIG_TTY_PRINTK=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_SSIF=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_NVRAM=y
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_HPET=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y
CONFIG_TCG_TIS_I2C_ATMEL=m
CONFIG_TCG_TIS_I2C_INFINEON=m
CONFIG_TCG_TIS_I2C_NUVOTON=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
CONFIG_TELCLOCK=m
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=m
CONFIG_I2C_ISMT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_SCMI=m
CONFIG_I2C_DESIGNWARE_PCI=y
CONFIG_I2C_DESIGNWARE_BAYTRAIL=y
CONFIG_I2C_CROS_EC_TUNNEL=m
CONFIG_I2C_SLAVE=y
CONFIG_I2C_SLAVE_EEPROM=m
CONFIG_SPI=y
CONFIG_PINCTRL_BAYTRAIL=y
CONFIG_PINCTRL_CHERRYVIEW=y
CONFIG_PINCTRL_BROXTON=y
CONFIG_PINCTRL_SUNRISEPOINT=y
CONFIG_DEBUG_GPIO=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_GENERIC_PLATFORM=m
CONFIG_GPIO_ICH=m
CONFIG_GPIO_LYNXPOINT=m
CONFIG_GPIO_SCH=m
CONFIG_GPIO_TPS68470=y
CONFIG_POWER_RESET=y
CONFIG_PDA_POWER=m
CONFIG_GENERIC_ADC_BATTERY=m
CONFIG_BATTERY_SBS=m
CONFIG_CHARGER_GPIO=m
CONFIG_CHARGER_MANAGER=y
CONFIG_SENSORS_I5500=m
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_ACPI_POWER=m
CONFIG_THERMAL_GOV_FAIR_SHARE=y
CONFIG_THERMAL_GOV_BANG_BANG=y
CONFIG_THERMAL_EMULATION=y
CONFIG_INTEL_POWERCLAMP=m
CONFIG_INTEL_SOC_DTS_THERMAL=m
CONFIG_INT340X_THERMAL=m
CONFIG_MFD_CROS_EC=m
CONFIG_LPC_ICH=y
CONFIG_LPC_SCH=y
CONFIG_MFD_INTEL_LPSS_ACPI=y
CONFIG_MFD_INTEL_LPSS_PCI=y
CONFIG_MFD_TPS68470=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=m
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
CONFIG_REGULATOR_USERSPACE_CONSUMER=m
CONFIG_REGULATOR_GPIO=m
CONFIG_REGULATOR_TPS51632=m
CONFIG_REGULATOR_TPS62360=m
CONFIG_REGULATOR_TPS65023=m
CONFIG_REGULATOR_TPS6507X=m
CONFIG_MEDIA_SUPPORT=m
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_CONTROLLER=y
CONFIG_VIDEO_V4L2_SUBDEV_API=y
CONFIG_MEDIA_USB_SUPPORT=y
CONFIG_USB_VIDEO_CLASS=m
# CONFIG_USB_GSPCA is not set
CONFIG_MEDIA_PCI_SUPPORT=y
CONFIG_VIDEO_IPU3_CIO2=m
CONFIG_V4L_PLATFORM_DRIVERS=y
# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
CONFIG_VIDEO_DW9714=m
CONFIG_VIDEO_OV5670=m
CONFIG_VIDEO_OV7670=m
CONFIG_VIDEO_OV13858=m
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_I915=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_TILEBLITTING=y
CONFIG_FB_INTEL=m
CONFIG_FB_SIMPLE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=m
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
CONFIG_HID=m
CONFIG_HID_BATTERY_STRENGTH=y
CONFIG_HIDRAW=y
CONFIG_UHID=m
CONFIG_HID_MULTITOUCH=m
CONFIG_USB_HID=m
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_KBD=m
CONFIG_USB_MOUSE=m
CONFIG_I2C_HID=m
CONFIG_USB=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
CONFIG_USB_DYNAMIC_MINORS=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_HCD_PLATFORM=y
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PLATFORM=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=m
CONFIG_USB_UAS=m
CONFIG_NOP_USB_XCEIV=y
CONFIG_USB_GPIO_VBUS=m
CONFIG_MMC=y
CONFIG_MMC_BLOCK=m
CONFIG_SDIO_UART=m
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
# CONFIG_MMC_RICOH_MMC is not set
CONFIG_MMC_SDHCI_ACPI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_LEDS_CLASS=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_HID_SENSOR_TIME=m
CONFIG_DMADEVICES=y
CONFIG_INTEL_IOATDMA=m
CONFIG_DW_DMAC=y
CONFIG_ASYNC_TX_DMA=y
CONFIG_AUXDISPLAY=y
# CONFIG_VIRTIO_MENU is not set
CONFIG_STAGING=y
CONFIG_STAGING_MEDIA=y
CONFIG_VIDEO_IPU3_IMGU=m
CONFIG_DCDBAS=m
CONFIG_DELL_RBU=m
CONFIG_CHROMEOS_LAPTOP=m
CONFIG_CHROMEOS_PSTORE=m
CONFIG_CROS_EC_LPC=m
CONFIG_CROS_KBD_LED_BACKLIGHT=m
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_IRQ_REMAP=y
CONFIG_MEMORY=y
CONFIG_IIO_BUFFER=y
CONFIG_IIO_KFIFO_BUF=m
CONFIG_RESET_CONTROLLER=y
CONFIG_GENERIC_PHY=y
CONFIG_POWERCAP=y
CONFIG_INTEL_RAPL=m
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_EXT4_ENCRYPTION=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=y
CONFIG_CUSE=m
CONFIG_OVERLAY_FS=m
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
CONFIG_CACHEFILES=m
CONFIG_VFAT_FS=y
CONFIG_PROC_KCORE=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_EFIVAR_FS=y
CONFIG_PSTORE_RAM=m
CONFIG_NFS_FS=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
CONFIG_NFS_SWAP=y
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_NFS_V4_1_MIGRATION=y
CONFIG_NFS_FSCACHE=y
CONFIG_SUNRPC_DEBUG=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_UTF8=y
CONFIG_PERSISTENT_KEYRINGS=y
CONFIG_TRUSTED_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_INTEL_TXT=y
CONFIG_SECURITY_APPARMOR=y
# CONFIG_INTEGRITY is not set
CONFIG_CRYPTO_ECDH=m
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_CRC32C_INTEL=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_DRBG_HASH=y
CONFIG_CRYPTO_DRBG_CTR=y
CONFIG_CRYPTO_DEV_PADLOCK=y
CONFIG_CRYPTO_DEV_CCP=y
# CONFIG_CRYPTO_DEV_CCP_DD is not set
CONFIG_LIBCRC32C=y
# CONFIG_XZ_DEC_POWERPC is not set
# CONFIG_XZ_DEC_ARM is not set
# CONFIG_XZ_DEC_ARMTHUMB is not set
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_DDR=y
CONFIG_PRINTK_TIME=y
CONFIG_BOOT_PRINTK_DELAY=y
CONFIG_DYNAMIC_DEBUG=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
CONFIG_KASAN=y
CONFIG_KASAN_EXTRA=y
CONFIG_KASAN_INLINE=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_SCHEDSTATS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_NOTIFIER_ERROR_INJECTION=m
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_MMIOTRACE=y
CONFIG_MEMTEST=y
CONFIG_KGDB=y
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_EARLY_PRINTK_EFI=y
CONFIG_IO_DELAY_0XED=y
CONFIG_OPTIMIZE_INLINING=y
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 17:25 ` Jacopo Mondi
@ 2019-01-09 18:01 ` Mani, Rajmohan
2019-01-09 18:20 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2019-01-09 18:01 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
Hi Jacopo,
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hello Raj,
>
> On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> > Hi Laurent, Tomasz, Jacopo,
> >
> > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > >
> > > Hello,
> > >
> > > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > > Hi Raj, Yong, Bingbu, Tianshu,
> > > >
> > > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa <tfiga@chromium.org>
> wrote:
> > > > >
> > > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > > >
> > > > > > Hellon
> > > > > >
> > > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > > > > > > Hello Yong,
> > > > > > >
> > > > > > > Could you please have a look at the crash reported below ?
> > > > > >
> > > > > > A bit more information to help you debugging this. I've
> > > > > > enabled KASAN in the kernel configuration, and get the
> > > > > > following use-after-free
> > > reports.
> > >
> > > I tested as well using the ipu-process.sh script shared by Laurent,
> > > with the following command line:
> > > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080
> > > frame-2592x1944.cio2
> > >
> > > and I got a very similar trace available at:
> > > https://paste.debian.net/hidden/5855e15a/
> > >
> > > Please note I have been able to process a set of images (with KASAN
> > > enabled the machine does not freeze) but the kernel log gets flooded
> > > and it is not possible to process any other frame after this.
> > >
> > > The issue is currently quite annoying and it's a blocker for
> > > libcamera development on IPU3. Please let me know if I can support with
> more testing.
> > >
> > > Thanks
> > > j
> > >
> > > > > >
> > > > > > [ 166.332920]
> > > > > >
> > >
> ================================================================
> > > ==
> > > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > [ 166.332944] Read of size 8 at addr ffff888133823718 by task
> > > > > > yavta/1305
> > > > > >
> > > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C
> 4.20.0-
> > > rc6+ #3
> > > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS
> > > > > > 08/30/2018 [ 166.332959] Call Trace:
> > > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > > print_address_description+0x65/0x227
> > > > > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > [ 166.332995] private_free_iova+0x57/0x6d [ 166.332999]
> > > > > > __free_iova+0x23/0x31 [ 166.333011]
> > > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > > >
> > > > > Thanks Laurent, I think this is a very good hint. It looks like
> > > > > we're basically freeing and already freed IOVA and corrupting
> > > > > some allocator state?
> > > >
> > > > Did you have any luck in reproducing and fixing this double free issue?
> > > >
> >
> > This issue is either hard to reproduce or comes with different
> > signatures with the updated yavta (that now supports meta output) with
> > the 4.4 kernel that I have been using.
> > I am switching to 4.20-rc6 for better reproducibility.
> > Enabling KASAN also results in storage space issues on my Chrome device.
> > Will enable this just for ImgU to get ahead and get back with more updates.
> >
>
> Thanks for testing this.
>
> For your informations I'm using the following branch, from Sakari's
> tree: git://linuxtv.org/sailus/media_tree.git ipu3
>
> Although it appears that the media tree master branch has everything that is
> there, with a few additional patches on top. I should move to use media tree
> master as well...
>
> I have here attached 2 configuration files for v4.20-rc5 I am using on Soraka, in
> case they might help you. One has KASAN enabled with an increased kernel
> log size, the other one is the one we use for daily development.
I think I am missing a trick here to override the default chrome os kernel
config with the one that you supplied.
In particular I am looking for steps to build the upstream kernel within chrome os
build environment using your config, so I can update my Soraka device.
>
> Also, please make sure to use (the most) recent media-ctl and yavta utilities, as
> the ones provided by most distros are usually not recent enough to work with
> IPU3, but I'm sure you know that already ;)
Ack
>
> Thanks
> j
>
[snip]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 18:01 ` Mani, Rajmohan
@ 2019-01-09 18:20 ` Jacopo Mondi
2019-01-09 18:36 ` Mani, Rajmohan
2019-01-12 2:30 ` Mani, Rajmohan
0 siblings, 2 replies; 123+ messages in thread
From: Jacopo Mondi @ 2019-01-09 18:20 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
[-- Attachment #1: Type: text/plain, Size: 5334 bytes --]
Hi Raj,
On Wed, Jan 09, 2019 at 06:01:39PM +0000, Mani, Rajmohan wrote:
> Hi Jacopo,
>
> > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> >
> > Hello Raj,
> >
> > On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> > > Hi Laurent, Tomasz, Jacopo,
> > >
> > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > >
> > > > Hello,
> > > >
> > > > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > > > Hi Raj, Yong, Bingbu, Tianshu,
> > > > >
> > > > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa <tfiga@chromium.org>
> > wrote:
> > > > > >
> > > > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > > > >
> > > > > > > Hellon
> > > > > > >
> > > > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart wrote:
> > > > > > > > Hello Yong,
> > > > > > > >
> > > > > > > > Could you please have a look at the crash reported below ?
> > > > > > >
> > > > > > > A bit more information to help you debugging this. I've
> > > > > > > enabled KASAN in the kernel configuration, and get the
> > > > > > > following use-after-free
> > > > reports.
> > > >
> > > > I tested as well using the ipu-process.sh script shared by Laurent,
> > > > with the following command line:
> > > > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080
> > > > frame-2592x1944.cio2
> > > >
> > > > and I got a very similar trace available at:
> > > > https://paste.debian.net/hidden/5855e15a/
> > > >
> > > > Please note I have been able to process a set of images (with KASAN
> > > > enabled the machine does not freeze) but the kernel log gets flooded
> > > > and it is not possible to process any other frame after this.
> > > >
> > > > The issue is currently quite annoying and it's a blocker for
> > > > libcamera development on IPU3. Please let me know if I can support with
> > more testing.
> > > >
> > > > Thanks
> > > > j
> > > >
> > > > > > >
> > > > > > > [ 166.332920]
> > > > > > >
> > > >
> > ================================================================
> > > > ==
> > > > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > [ 166.332944] Read of size 8 at addr ffff888133823718 by task
> > > > > > > yavta/1305
> > > > > > >
> > > > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C
> > 4.20.0-
> > > > rc6+ #3
> > > > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS
> > > > > > > 08/30/2018 [ 166.332959] Call Trace:
> > > > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > > > print_address_description+0x65/0x227
> > > > > > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > > > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > [ 166.332995] private_free_iova+0x57/0x6d [ 166.332999]
> > > > > > > __free_iova+0x23/0x31 [ 166.333011]
> > > > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > > > >
> > > > > > Thanks Laurent, I think this is a very good hint. It looks like
> > > > > > we're basically freeing and already freed IOVA and corrupting
> > > > > > some allocator state?
> > > > >
> > > > > Did you have any luck in reproducing and fixing this double free issue?
> > > > >
> > >
> > > This issue is either hard to reproduce or comes with different
> > > signatures with the updated yavta (that now supports meta output) with
> > > the 4.4 kernel that I have been using.
> > > I am switching to 4.20-rc6 for better reproducibility.
> > > Enabling KASAN also results in storage space issues on my Chrome device.
> > > Will enable this just for ImgU to get ahead and get back with more updates.
> > >
> >
> > Thanks for testing this.
> >
> > For your informations I'm using the following branch, from Sakari's
> > tree: git://linuxtv.org/sailus/media_tree.git ipu3
> >
> > Although it appears that the media tree master branch has everything that is
> > there, with a few additional patches on top. I should move to use media tree
> > master as well...
> >
> > I have here attached 2 configuration files for v4.20-rc5 I am using on Soraka, in
> > case they might help you. One has KASAN enabled with an increased kernel
> > log size, the other one is the one we use for daily development.
>
> I think I am missing a trick here to override the default chrome os kernel
> config with the one that you supplied.
>
> In particular I am looking for steps to build the upstream kernel within chrome os
> build environment using your config, so I can update my Soraka device.
I'm sorry I can not help much building 'withing chrome os build
environment'. Care to explain what you mean?
What I usually do, provided you're running a debian-based Linux distro
on your Soraka device, is compile the kernel on host with 'make bindeb-pkg'
and then upload and install the resulting .deb package on the
Soraka chromebook.
If that might work for you, we can share more details on how to do so
(tomorrow maybe :p )
Thanks
j
>
> >
> > Also, please make sure to use (the most) recent media-ctl and yavta utilities, as
> > the ones provided by most distros are usually not recent enough to work with
> > IPU3, but I'm sure you know that already ;)
>
> Ack
>
> >
> > Thanks
> > j
> >
>
> [snip]
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 18:20 ` Jacopo Mondi
@ 2019-01-09 18:36 ` Mani, Rajmohan
2019-01-10 8:19 ` Jacopo Mondi
2019-01-12 2:30 ` Mani, Rajmohan
1 sibling, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2019-01-09 18:36 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
Hi Jacopo,
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hi Raj,
>
> On Wed, Jan 09, 2019 at 06:01:39PM +0000, Mani, Rajmohan wrote:
> > Hi Jacopo,
> >
> > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > >
> > > Hello Raj,
> > >
> > > On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> > > > Hi Laurent, Tomasz, Jacopo,
> > > >
> > > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > > >
> > > > > Hello,
> > > > >
> > > > > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > > > > Hi Raj, Yong, Bingbu, Tianshu,
> > > > > >
> > > > > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa
> > > > > > <tfiga@chromium.org>
> > > wrote:
> > > > > > >
> > > > > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > > > > >
> > > > > > > > Hellon
> > > > > > > >
> > > > > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart
> wrote:
> > > > > > > > > Hello Yong,
> > > > > > > > >
> > > > > > > > > Could you please have a look at the crash reported below ?
> > > > > > > >
> > > > > > > > A bit more information to help you debugging this. I've
> > > > > > > > enabled KASAN in the kernel configuration, and get the
> > > > > > > > following use-after-free
> > > > > reports.
> > > > >
> > > > > I tested as well using the ipu-process.sh script shared by
> > > > > Laurent, with the following command line:
> > > > > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080
> > > > > frame-2592x1944.cio2
> > > > >
> > > > > and I got a very similar trace available at:
> > > > > https://paste.debian.net/hidden/5855e15a/
> > > > >
> > > > > Please note I have been able to process a set of images (with
> > > > > KASAN enabled the machine does not freeze) but the kernel log
> > > > > gets flooded and it is not possible to process any other frame after this.
> > > > >
> > > > > The issue is currently quite annoying and it's a blocker for
> > > > > libcamera development on IPU3. Please let me know if I can
> > > > > support with
> > > more testing.
> > > > >
> > > > > Thanks
> > > > > j
> > > > >
> > > > > > > >
> > > > > > > > [ 166.332920]
> > > > > > > >
> > > > >
> > >
> ================================================================
> > > > > ==
> > > > > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > [ 166.332944] Read of size 8 at addr ffff888133823718 by
> > > > > > > > task
> > > > > > > > yavta/1305
> > > > > > > >
> > > > > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C
> > > 4.20.0-
> > > > > rc6+ #3
> > > > > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS
> > > > > > > > 08/30/2018 [ 166.332959] Call Trace:
> > > > > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > > > > print_address_description+0x65/0x227
> > > > > > > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > [ 166.332995] private_free_iova+0x57/0x6d [ 166.332999]
> > > > > > > > __free_iova+0x23/0x31 [ 166.333011]
> > > > > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > > > > >
> > > > > > > Thanks Laurent, I think this is a very good hint. It looks
> > > > > > > like we're basically freeing and already freed IOVA and
> > > > > > > corrupting some allocator state?
> > > > > >
> > > > > > Did you have any luck in reproducing and fixing this double free
> issue?
> > > > > >
> > > >
> > > > This issue is either hard to reproduce or comes with different
> > > > signatures with the updated yavta (that now supports meta output)
> > > > with the 4.4 kernel that I have been using.
> > > > I am switching to 4.20-rc6 for better reproducibility.
> > > > Enabling KASAN also results in storage space issues on my Chrome
> device.
> > > > Will enable this just for ImgU to get ahead and get back with more
> updates.
> > > >
> > >
> > > Thanks for testing this.
> > >
> > > For your informations I'm using the following branch, from Sakari's
> > > tree: git://linuxtv.org/sailus/media_tree.git ipu3
> > >
> > > Although it appears that the media tree master branch has everything
> > > that is there, with a few additional patches on top. I should move
> > > to use media tree master as well...
> > >
> > > I have here attached 2 configuration files for v4.20-rc5 I am using
> > > on Soraka, in case they might help you. One has KASAN enabled with
> > > an increased kernel log size, the other one is the one we use for daily
> development.
> >
> > I think I am missing a trick here to override the default chrome os
> > kernel config with the one that you supplied.
> >
> > In particular I am looking for steps to build the upstream kernel
> > within chrome os build environment using your config, so I can update my
> Soraka device.
>
> I'm sorry I can not help much building 'withing chrome os build environment'.
> Care to explain what you mean?
>
This is part of the Chromium OS build environment and development workflow.
https://chromium.googlesource.com/chromiumos/docs/+/master/kernel_faq.md
No worries.
I will sync up with Tomasz, as he managed to get this working with 4.20 kernel.
> What I usually do, provided you're running a debian-based Linux distro on
> your Soraka device, is compile the kernel on host with 'make bindeb-pkg'
> and then upload and install the resulting .deb package on the Soraka
> chromebook.
>
> If that might work for you, we can share more details on how to do so
> (tomorrow maybe :p )
>
The above info would be very helpful for us to optimize use of upstream
kernel in Chrome OS devices.
Please share when your time permits.
> Thanks
> j
>
> >
> > >
> > > Also, please make sure to use (the most) recent media-ctl and yavta
> > > utilities, as the ones provided by most distros are usually not
> > > recent enough to work with IPU3, but I'm sure you know that already
> > > ;)
> >
> > Ack
> >
> > >
> > > Thanks
> > > j
> > >
> >
> > [snip]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 18:36 ` Mani, Rajmohan
@ 2019-01-10 8:19 ` Jacopo Mondi
2019-01-12 2:06 ` Mani, Rajmohan
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-01-10 8:19 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
[-- Attachment #1: Type: text/plain, Size: 6040 bytes --]
Hi Raj,
On Wed, Jan 09, 2019 at 06:36:02PM +0000, Mani, Rajmohan wrote:
> Hi Jacopo,
>
> > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> >
> > Hi Raj,
> >
> > On Wed, Jan 09, 2019 at 06:01:39PM +0000, Mani, Rajmohan wrote:
> > > Hi Jacopo,
> > >
> > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > >
> > > > Hello Raj,
> > > >
> > > > On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> > > > > Hi Laurent, Tomasz, Jacopo,
> > > > >
> > > > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > > > > > Hi Raj, Yong, Bingbu, Tianshu,
> > > > > > >
> > > > > > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa
> > > > > > > <tfiga@chromium.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > > > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > > > > > >
> > > > > > > > > Hellon
> > > > > > > > >
> > > > > > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent Pinchart
> > wrote:
> > > > > > > > > > Hello Yong,
> > > > > > > > > >
> > > > > > > > > > Could you please have a look at the crash reported below ?
> > > > > > > > >
> > > > > > > > > A bit more information to help you debugging this. I've
> > > > > > > > > enabled KASAN in the kernel configuration, and get the
> > > > > > > > > following use-after-free
> > > > > > reports.
> > > > > >
> > > > > > I tested as well using the ipu-process.sh script shared by
> > > > > > Laurent, with the following command line:
> > > > > > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080
> > > > > > frame-2592x1944.cio2
> > > > > >
> > > > > > and I got a very similar trace available at:
> > > > > > https://paste.debian.net/hidden/5855e15a/
> > > > > >
> > > > > > Please note I have been able to process a set of images (with
> > > > > > KASAN enabled the machine does not freeze) but the kernel log
> > > > > > gets flooded and it is not possible to process any other frame after this.
> > > > > >
> > > > > > The issue is currently quite annoying and it's a blocker for
> > > > > > libcamera development on IPU3. Please let me know if I can
> > > > > > support with
> > > > more testing.
> > > > > >
> > > > > > Thanks
> > > > > > j
> > > > > >
> > > > > > > > >
> > > > > > > > > [ 166.332920]
> > > > > > > > >
> > > > > >
> > > >
> > ================================================================
> > > > > > ==
> > > > > > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > [ 166.332944] Read of size 8 at addr ffff888133823718 by
> > > > > > > > > task
> > > > > > > > > yavta/1305
> > > > > > > > >
> > > > > > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C
> > > > 4.20.0-
> > > > > > rc6+ #3
> > > > > > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS
> > > > > > > > > 08/30/2018 [ 166.332959] Call Trace:
> > > > > > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > > > > > print_address_description+0x65/0x227
> > > > > > > > > [ 166.332979] ? __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > [ 166.332995] private_free_iova+0x57/0x6d [ 166.332999]
> > > > > > > > > __free_iova+0x23/0x31 [ 166.333011]
> > > > > > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > > > > > >
> > > > > > > > Thanks Laurent, I think this is a very good hint. It looks
> > > > > > > > like we're basically freeing and already freed IOVA and
> > > > > > > > corrupting some allocator state?
> > > > > > >
> > > > > > > Did you have any luck in reproducing and fixing this double free
> > issue?
> > > > > > >
> > > > >
> > > > > This issue is either hard to reproduce or comes with different
> > > > > signatures with the updated yavta (that now supports meta output)
> > > > > with the 4.4 kernel that I have been using.
> > > > > I am switching to 4.20-rc6 for better reproducibility.
> > > > > Enabling KASAN also results in storage space issues on my Chrome
> > device.
> > > > > Will enable this just for ImgU to get ahead and get back with more
> > updates.
> > > > >
> > > >
> > > > Thanks for testing this.
> > > >
> > > > For your informations I'm using the following branch, from Sakari's
> > > > tree: git://linuxtv.org/sailus/media_tree.git ipu3
> > > >
> > > > Although it appears that the media tree master branch has everything
> > > > that is there, with a few additional patches on top. I should move
> > > > to use media tree master as well...
> > > >
> > > > I have here attached 2 configuration files for v4.20-rc5 I am using
> > > > on Soraka, in case they might help you. One has KASAN enabled with
> > > > an increased kernel log size, the other one is the one we use for daily
> > development.
> > >
> > > I think I am missing a trick here to override the default chrome os
> > > kernel config with the one that you supplied.
> > >
> > > In particular I am looking for steps to build the upstream kernel
> > > within chrome os build environment using your config, so I can update my
> > Soraka device.
> >
> > I'm sorry I can not help much building 'withing chrome os build environment'.
> > Care to explain what you mean?
> >
>
> This is part of the Chromium OS build environment and development workflow.
> https://chromium.googlesource.com/chromiumos/docs/+/master/kernel_faq.md
>
> No worries.
> I will sync up with Tomasz, as he managed to get this working with 4.20 kernel.
>
I'm sorry I can't help much here. I suggest to work with mainline (or
better, media master) and install a GNU/Linux distro on the chromebook so
you can easily update your kernel.
I personally used https://chrx.org/ that makes installing gallium (or
else) very easy. Once you get that running, I find easy enough to
update the kernel installing a .deb package.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-10 8:19 ` Jacopo Mondi
@ 2019-01-12 2:06 ` Mani, Rajmohan
0 siblings, 0 replies; 123+ messages in thread
From: Mani, Rajmohan @ 2019-01-12 2:06 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
Hi Jacopo,
> Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hi Raj,
>
> On Wed, Jan 09, 2019 at 06:36:02PM +0000, Mani, Rajmohan wrote:
> > Hi Jacopo,
> >
> > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > >
> > > Hi Raj,
> > >
> > > On Wed, Jan 09, 2019 at 06:01:39PM +0000, Mani, Rajmohan wrote:
> > > > Hi Jacopo,
> > > >
> > > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > > >
> > > > > Hello Raj,
> > > > >
> > > > > On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> > > > > > Hi Laurent, Tomasz, Jacopo,
> > > > > >
> > > > > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > > > > > > Hi Raj, Yong, Bingbu, Tianshu,
> > > > > > > >
> > > > > > > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa
> > > > > > > > <tfiga@chromium.org>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > > > > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hellon
> > > > > > > > > >
> > > > > > > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent
> > > > > > > > > > Pinchart
> > > wrote:
> > > > > > > > > > > Hello Yong,
> > > > > > > > > > >
> > > > > > > > > > > Could you please have a look at the crash reported below ?
> > > > > > > > > >
> > > > > > > > > > A bit more information to help you debugging this.
> > > > > > > > > > I've enabled KASAN in the kernel configuration, and
> > > > > > > > > > get the following use-after-free
> > > > > > > reports.
> > > > > > >
> > > > > > > I tested as well using the ipu-process.sh script shared by
> > > > > > > Laurent, with the following command line:
> > > > > > > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080
> > > > > > > frame-2592x1944.cio2
> > > > > > >
> > > > > > > and I got a very similar trace available at:
> > > > > > > https://paste.debian.net/hidden/5855e15a/
> > > > > > >
> > > > > > > Please note I have been able to process a set of images
> > > > > > > (with KASAN enabled the machine does not freeze) but the
> > > > > > > kernel log gets flooded and it is not possible to process any other
> frame after this.
> > > > > > >
> > > > > > > The issue is currently quite annoying and it's a blocker for
> > > > > > > libcamera development on IPU3. Please let me know if I can
> > > > > > > support with
> > > > > more testing.
> > > > > > >
> > > > > > > Thanks
> > > > > > > j
> > > > > > >
> > > > > > > > > >
> > > > > > > > > > [ 166.332920]
> > > > > > > > > >
> > > > > > >
> > > > >
> > >
> ================================================================
> > > > > > > ==
> > > > > > > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > > [ 166.332944] Read of size 8 at addr ffff888133823718
> > > > > > > > > > by task
> > > > > > > > > > yavta/1305
> > > > > > > > > >
> > > > > > > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C
> > > > > 4.20.0-
> > > > > > > rc6+ #3
> > > > > > > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS
> > > > > > > > > > 08/30/2018 [ 166.332959] Call Trace:
> > > > > > > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > > > > > > print_address_description+0x65/0x227
> > > > > > > > > > [ 166.332979] ?
> > > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > > [ 166.332983] kasan_report+0x247/0x285 [
> > > > > > > > > > 166.332989]
> > > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > > [ 166.332995] private_free_iova+0x57/0x6d [
> > > > > > > > > > 166.332999]
> > > > > > > > > > __free_iova+0x23/0x31 [ 166.333011]
> > > > > > > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > > > > > > >
> > > > > > > > > Thanks Laurent, I think this is a very good hint. It
> > > > > > > > > looks like we're basically freeing and already freed
> > > > > > > > > IOVA and corrupting some allocator state?
> > > > > > > >
> > > > > > > > Did you have any luck in reproducing and fixing this
> > > > > > > > double free
> > > issue?
> > > > > > > >
> > > > > >
> > > > > > This issue is either hard to reproduce or comes with different
> > > > > > signatures with the updated yavta (that now supports meta
> > > > > > output) with the 4.4 kernel that I have been using.
> > > > > > I am switching to 4.20-rc6 for better reproducibility.
> > > > > > Enabling KASAN also results in storage space issues on my
> > > > > > Chrome
> > > device.
> > > > > > Will enable this just for ImgU to get ahead and get back with
> > > > > > more
> > > updates.
> > > > > >
> > > > >
> > > > > Thanks for testing this.
> > > > >
> > > > > For your informations I'm using the following branch, from
> > > > > Sakari's
> > > > > tree: git://linuxtv.org/sailus/media_tree.git ipu3
> > > > >
> > > > > Although it appears that the media tree master branch has
> > > > > everything that is there, with a few additional patches on top.
> > > > > I should move to use media tree master as well...
> > > > >
> > > > > I have here attached 2 configuration files for v4.20-rc5 I am
> > > > > using on Soraka, in case they might help you. One has KASAN
> > > > > enabled with an increased kernel log size, the other one is the
> > > > > one we use for daily
> > > development.
> > > >
> > > > I think I am missing a trick here to override the default chrome
> > > > os kernel config with the one that you supplied.
> > > >
> > > > In particular I am looking for steps to build the upstream kernel
> > > > within chrome os build environment using your config, so I can
> > > > update my
> > > Soraka device.
> > >
> > > I'm sorry I can not help much building 'withing chrome os build
> environment'.
> > > Care to explain what you mean?
> > >
> >
> > This is part of the Chromium OS build environment and development
> workflow.
> >
> https://chromium.googlesource.com/chromiumos/docs/+/master/kernel_faq.
> > md
> >
> > No worries.
> > I will sync up with Tomasz, as he managed to get this working with 4.20
> kernel.
> >
>
> I'm sorry I can't help much here. I suggest to work with mainline (or better,
> media master) and install a GNU/Linux distro on the chromebook so you can
> easily update your kernel.
>
> I personally used https://chrx.org/ that makes installing gallium (or
> else) very easy. Once you get that running, I find easy enough to update the
> kernel installing a .deb package.
Thanks for the pointers.
I will have a look at this workflow.
^ permalink raw reply [flat|nested] 123+ messages in thread
* RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-09 18:20 ` Jacopo Mondi
2019-01-09 18:36 ` Mani, Rajmohan
@ 2019-01-12 2:30 ` Mani, Rajmohan
2019-01-12 15:10 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Mani, Rajmohan @ 2019-01-12 2:30 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Laurent Pinchart, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, Hu, Jerry W, Toivonen,
Tuukka
Hi Laurent et al,
> Subject: RE: [PATCH v7 00/16] Intel IPU3 ImgU patchset
>
> Hi Jacopo,
>
> > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> >
> > Hi Raj,
> >
> > On Wed, Jan 09, 2019 at 06:01:39PM +0000, Mani, Rajmohan wrote:
> > > Hi Jacopo,
> > >
> > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > >
> > > > Hello Raj,
> > > >
> > > > On Wed, Jan 09, 2019 at 05:00:21PM +0000, Mani, Rajmohan wrote:
> > > > > Hi Laurent, Tomasz, Jacopo,
> > > > >
> > > > > > Subject: Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > On Tue, Jan 08, 2019 at 03:54:34PM +0900, Tomasz Figa wrote:
> > > > > > > Hi Raj, Yong, Bingbu, Tianshu,
> > > > > > >
> > > > > > > On Fri, Dec 21, 2018 at 12:04 PM Tomasz Figa
> > > > > > > <tfiga@chromium.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > On Fri, Dec 21, 2018 at 7:24 AM Laurent Pinchart
> > > > > > > > <laurent.pinchart@ideasonboard.com> wrote:
> > > > > > > > >
> > > > > > > > > Hellon
> > > > > > > > >
> > > > > > > > > On Sunday, 16 December 2018 09:26:18 EET Laurent
> > > > > > > > > Pinchart
> > wrote:
> > > > > > > > > > Hello Yong,
> > > > > > > > > >
> > > > > > > > > > Could you please have a look at the crash reported below ?
> > > > > > > > >
> > > > > > > > > A bit more information to help you debugging this. I've
> > > > > > > > > enabled KASAN in the kernel configuration, and get the
> > > > > > > > > following use-after-free
> > > > > > reports.
> > > > > >
> > > > > > I tested as well using the ipu-process.sh script shared by
> > > > > > Laurent, with the following command line:
> > > > > > ./ipu3-process.sh --out 2560x1920 --vf 1920x1080
> > > > > > frame-2592x1944.cio2
> > > > > >
> > > > > > and I got a very similar trace available at:
> > > > > > https://paste.debian.net/hidden/5855e15a/
> > > > > >
> > > > > > Please note I have been able to process a set of images (with
> > > > > > KASAN enabled the machine does not freeze) but the kernel log
> > > > > > gets flooded and it is not possible to process any other frame after
> this.
> > > > > >
> > > > > > The issue is currently quite annoying and it's a blocker for
> > > > > > libcamera development on IPU3. Please let me know if I can
> > > > > > support with
> > > > more testing.
> > > > > >
> > > > > > Thanks
> > > > > > j
> > > > > >
> > > > > > > > >
> > > > > > > > > [ 166.332920]
> > > > > > > > >
> > > > > >
> > > >
> >
> ================================================================
> > > > > > ==
> > > > > > > > > [ 166.332937] BUG: KASAN: use-after-free in
> > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > [ 166.332944] Read of size 8 at addr ffff888133823718
> > > > > > > > > by task
> > > > > > > > > yavta/1305
> > > > > > > > >
> > > > > > > > > [ 166.332955] CPU: 3 PID: 1305 Comm: yavta Tainted: G C
> > > > 4.20.0-
> > > > > > rc6+ #3
> > > > > > > > > [ 166.332958] Hardware name: HP Soraka/Soraka, BIOS
> > > > > > > > > 08/30/2018 [ 166.332959] Call Trace:
> > > > > > > > > [ 166.332967] dump_stack+0x5b/0x81 [ 166.332974]
> > > > > > > > > print_address_description+0x65/0x227
> > > > > > > > > [ 166.332979] ?
> > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > [ 166.332983] kasan_report+0x247/0x285 [ 166.332989]
> > > > > > > > > __cached_rbnode_delete_update+0x36/0x202
> > > > > > > > > [ 166.332995] private_free_iova+0x57/0x6d [
> > > > > > > > > 166.332999]
> > > > > > > > > __free_iova+0x23/0x31 [ 166.333011]
> > > > > > > > > ipu3_dmamap_free+0x118/0x1d6 [ipu3_imgu]
> > > > > > > >
> > > > > > > > Thanks Laurent, I think this is a very good hint. It looks
> > > > > > > > like we're basically freeing and already freed IOVA and
> > > > > > > > corrupting some allocator state?
> > > > > > >
> > > > > > > Did you have any luck in reproducing and fixing this double
> > > > > > > free
> > issue?
> > > > > > >
> > > > >
> > > > > This issue is either hard to reproduce or comes with different
> > > > > signatures with the updated yavta (that now supports meta
> > > > > output) with the 4.4 kernel that I have been using.
> > > > > I am switching to 4.20-rc6 for better reproducibility.
> > > > > Enabling KASAN also results in storage space issues on my Chrome
> > device.
> > > > > Will enable this just for ImgU to get ahead and get back with
> > > > > more
> > updates.
> > > > >
> > > >
> > > > Thanks for testing this.
> > > >
> > > > For your informations I'm using the following branch, from
> > > > Sakari's
> > > > tree: git://linuxtv.org/sailus/media_tree.git ipu3
> > > >
> > > > Although it appears that the media tree master branch has
> > > > everything that is there, with a few additional patches on top. I
> > > > should move to use media tree master as well...
> > > >
> > > > I have here attached 2 configuration files for v4.20-rc5 I am
> > > > using on Soraka, in case they might help you. One has KASAN
> > > > enabled with an increased kernel log size, the other one is the
> > > > one we use for daily
> > development.
> > >
> > > I think I am missing a trick here to override the default chrome os
> > > kernel config with the one that you supplied.
> > >
> > > In particular I am looking for steps to build the upstream kernel
> > > within chrome os build environment using your config, so I can
> > > update my
> > Soraka device.
> >
> > I'm sorry I can not help much building 'withing chrome os build
> environment'.
> > Care to explain what you mean?
> >
>
> This is part of the Chromium OS build environment and development
> workflow.
> https://chromium.googlesource.com/chromiumos/docs/+/master/kernel_faq.
> md
>
> No worries.
> I will sync up with Tomasz, as he managed to get this working with 4.20 kernel.
>
I finally managed to reproduce the issue with 4.20-rc6, with KASAN enabled and
with CONFIG_SLUB_DEBUG_ON with SLAB_STORE_USER.
The following line indicates the crash happens when yavta PID 10289 tries to free the memory.
[ 452.437844] BUG: KASAN: use-after-free in ipu3_dmamap_free+0x50/0x9c [ipu3_imgu]
[ 452.446123] Read of size 8 at addr ffff8881503481a0 by task yavta/10289
The above looks to be normal, since it's the same task that allocated this memory.
[ 452.685731] Allocated by task 10289:
Before the above happened, yavta/10187 came in and freed this memory per KASAN.
[ 452.787656] Freed by task 10187:
Is this (one instance of yavta freeing the memory allocated by another instance of yavta) expected?
Or does it indicate that mmap giving the same address across these 2 instances of yavta?
I need to debug / confirm the latter case.
With the help of local application that operates these pipes in a serial fashion, I do not see
this issue.
I have pasted the relevant parts of the dmesg.
[ 452.038082] WARNING: CPU: 1 PID: 10289 at /mnt/host/source/src/third_party/kernel/v4.4/drivers/staging/media/ipu3/ipu3-dmamap.c:172 ipu3_dmamap_unmap+0xf6/0x107 [ipu3_imgu]
[ 452.055293] Modules linked in: cmac rfcomm uinput snd_soc_kbl_rt5663_max98927 snd_soc_skl_ssp_clk snd_soc_hdac_hdmi snd_soc_dmic btusb btrtl btbcm asix usbnet btintel bluetooth snd_soc_skl snd_soc_skl_ipc ecdh_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_hda_core ipu3_imgu(C) ipu3_cio2 iova videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_rt5663 snd_soc_max98927 at24 snd_soc_rl6231 ov13858 ov5670 v4l2_fwnode dw9714 bridge stp llc acpi_als kfifo_buf industrialio ipt_MASQUERADE lzo lzo_compress zram xt_mark fuse snd_seq_dummy snd_seq snd_seq_device cfg80211 ip6table_filter r8152 mii joydev
[ 452.117513] CPU: 1 PID: 10289 Comm: yavta Tainted: G WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 452.128705] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 452.137476] RIP: 0010:ipu3_dmamap_unmap+0xf6/0x107 [ipu3_imgu]
[ 452.144007] Code: e1 48 d3 e2 48 8b 7d c8 48 89 de e8 7b f5 ff ff 48 8b 7d d0 4c 89 fe 48 83 c4 20 5b 41 5c 41 5d 41 5e 41 5f 5d e9 b2 f5 ee ff <0f> 0b 48 83 c4 20 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00
[ 452.165007] RSP: 0018:ffff88814ef67a20 EFLAGS: 00010246
[ 452.170857] RAX: 0000000000000000 RBX: 00000000000e527e RCX: 0000000000000001
[ 452.178842] RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8881179076b8
[ 452.186828] RBP: ffff88814ef67a68 R08: 0000000000000000 R09: ffffed1022f20ed8
[ 452.194812] R10: ffff8881179076bb R11: dffffc0000000000 R12: ffff8881179076e8
[ 452.202799] R13: ffff888117900028 R14: ffff8881179076b8 R15: 0000000000000000
[ 452.210784] FS: 00007a5d6524a700(0000) GS:ffff88815b680000(0000) knlGS:0000000000000000
[ 452.219837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 452.226271] CR2: 00005ba49b474078 CR3: 0000000129e1e002 CR4: 00000000003606e0
[ 452.234251] Call Trace:
[ 452.237003] ipu3_dmamap_free+0x41/0x9c [ipu3_imgu]
[ 452.242473] ipu3_css_pool_cleanup+0x24/0x37 [ipu3_imgu]
[ 452.248431] ipu3_css_pipeline_cleanup+0x61/0xb9 [ipu3_imgu]
[ 452.254772] ipu3_css_stop_streaming+0x1f2/0x321 [ipu3_imgu]
[ 452.261119] imgu_s_stream+0x94/0x443 [ipu3_imgu]
[ 452.266392] ? ipu3_vb2_buf_queue+0x280/0x280 [ipu3_imgu]
[ 452.272438] ? vb2_dma_sg_unmap_dmabuf+0x16/0x6f [videobuf2_dma_sg]
[ 452.279456] ? vb2_buffer_in_use+0x36/0x58 [videobuf2_common]
[ 452.285894] ipu3_vb2_stop_streaming+0xf9/0x135 [ipu3_imgu]
[ 452.292137] __vb2_queue_cancel+0x35/0x215 [videobuf2_common]
[ 452.298576] vb2_core_streamoff+0x19/0x73 [videobuf2_common]
[ 452.304920] __video_do_ioctl+0x34e/0x450
[ 452.309414] video_usercopy+0x25e/0x597
[ 452.313718] ? video_ioctl2+0x16/0x16
[ 452.317823] ? __switch_to_asm+0x34/0x70
[ 452.322215] v4l2_ioctl+0x45/0x49
[ 452.325932] vfs_ioctl+0x1b/0x30
[ 452.329551] do_vfs_ioctl+0x479/0x6d0
[ 452.333660] ksys_ioctl+0x53/0x79
[ 452.337375] __se_sys_ioctl+0xe/0x12
[ 452.341379] do_syscall_64+0x52/0x60
[ 452.345384] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 452.351035] RIP: 0033:0x7a5d64b73967
[ 452.355042] Code: 8a 66 90 48 8b 05 29 55 2b 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 54 2b 00 f7 d8 64 89 01 48
[ 452.376041] RSP: 002b:00007fff3483aca8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 452.384515] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007a5d64b73967
[ 452.392501] RDX: 00007fff3483acb4 RSI: 0000000040045613 RDI: 0000000000000003
[ 452.400484] RBP: 0000000000404c48 R08: fffffffffed7c030 R09: fffffffffed7c020
[ 452.408467] R10: fffffffffed7c010 R11: 0000000000000246 R12: 0000000000404c56
[ 452.416450] R13: 0000000000000001 R14: 00007fff3483c75c R15: 000000000062b800
[ 452.424432] ---[ end trace ed0895d0744ba932 ]---
[ 452.429752] ==================================================================
[ 452.437844] BUG: KASAN: use-after-free in ipu3_dmamap_free+0x50/0x9c [ipu3_imgu]
[ 452.446123] Read of size 8 at addr ffff8881503481a0 by task yavta/10289
[ 452.455191] CPU: 1 PID: 10289 Comm: yavta Tainted: G WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 452.466380] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 452.475133] Call Trace:
[ 452.477880] dump_stack+0x6a/0xb1
[ 452.481600] print_address_description+0x8e/0x279
[ 452.486873] ? ipu3_dmamap_free+0x50/0x9c [ipu3_imgu]
[ 452.492530] kasan_report+0x260/0x28a
[ 452.496637] ipu3_dmamap_free+0x50/0x9c [ipu3_imgu]
[ 452.502103] ipu3_css_pool_cleanup+0x24/0x37 [ipu3_imgu]
[ 452.508056] ipu3_css_pipeline_cleanup+0x61/0xb9 [ipu3_imgu]
[ 452.514395] ipu3_css_stop_streaming+0x1f2/0x321 [ipu3_imgu]
[ 452.520737] imgu_s_stream+0x94/0x443 [ipu3_imgu]
[ 452.526010] ? ipu3_vb2_buf_queue+0x280/0x280 [ipu3_imgu]
[ 452.532058] ? vb2_dma_sg_unmap_dmabuf+0x16/0x6f [videobuf2_dma_sg]
[ 452.539076] ? vb2_buffer_in_use+0x36/0x58 [videobuf2_common]
[ 452.545513] ipu3_vb2_stop_streaming+0xf9/0x135 [ipu3_imgu]
[ 452.551762] __vb2_queue_cancel+0x35/0x215 [videobuf2_common]
[ 452.558203] vb2_core_streamoff+0x19/0x73 [videobuf2_common]
[ 452.564542] __video_do_ioctl+0x34e/0x450
[ 452.569039] video_usercopy+0x25e/0x597
[ 452.573341] ? video_ioctl2+0x16/0x16
[ 452.577443] ? __switch_to_asm+0x34/0x70
[ 452.581838] v4l2_ioctl+0x45/0x49
[ 452.585559] vfs_ioctl+0x1b/0x30
[ 452.589178] do_vfs_ioctl+0x479/0x6d0
[ 452.593277] ksys_ioctl+0x53/0x79
[ 452.596991] __se_sys_ioctl+0xe/0x12
[ 452.601000] do_syscall_64+0x52/0x60
[ 452.605010] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 452.610668] RIP: 0033:0x7a5d64b73967
[ 452.614677] Code: 8a 66 90 48 8b 05 29 55 2b 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 54 2b 00 f7 d8 64 89 01 48
[ 452.635676] RSP: 002b:00007fff3483aca8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 452.644149] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007a5d64b73967
[ 452.652125] RDX: 00007fff3483acb4 RSI: 0000000040045613 RDI: 0000000000000003
[ 452.660109] RBP: 0000000000404c48 R08: fffffffffed7c030 R09: fffffffffed7c020
[ 452.668092] R10: fffffffffed7c010 R11: 0000000000000246 R12: 0000000000404c56
[ 452.676076] R13: 0000000000000001 R14: 00007fff3483c75c R15: 000000000062b800
[ 452.685731] Allocated by task 10289:
[ 452.689736] set_track+0x64/0xfb
[ 452.693354] __kmalloc+0x94/0x1af
[ 452.697066] __get_vm_area_node+0x9e/0x103
[ 452.701654] __get_vm_area+0x26/0x29
[ 452.705653] ipu3_dmamap_alloc+0x333/0x503 [ipu3_imgu]
[ 452.711408] ipu3_css_pool_init+0x43/0x99 [ipu3_imgu]
[ 452.717070] ipu3_css_start_streaming+0x25cf/0x29a7 [ipu3_imgu]
[ 452.723697] imgu_s_stream+0x133/0x443 [ipu3_imgu]
[ 452.729055] ipu3_vb2_start_streaming+0x1a3/0x1f1 [ipu3_imgu]
[ 452.735492] vb2_start_streaming+0x71/0x11c [videobuf2_common]
[ 452.742027] vb2_core_streamon+0xf8/0x118 [videobuf2_common]
[ 452.748371] __video_do_ioctl+0x34e/0x450
[ 452.752857] video_usercopy+0x25e/0x597
[ 452.757155] v4l2_ioctl+0x45/0x49
[ 452.760869] vfs_ioctl+0x1b/0x30
[ 452.764497] do_vfs_ioctl+0x479/0x6d0
[ 452.768602] ksys_ioctl+0x53/0x79
[ 452.772318] __se_sys_ioctl+0xe/0x12
[ 452.776322] do_syscall_64+0x52/0x60
[ 452.780330] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 452.787656] Freed by task 10187:
[ 452.791274] set_track+0x64/0xfb
[ 452.794897] __kasan_slab_free+0xde/0x101
[ 452.799393] slab_free_freelist_hook+0x4d/0x9e
[ 452.804373] kfree+0x8b/0x4d7
[ 452.807703] ipu3_dmamap_free+0x7e/0x9c [ipu3_imgu]
[ 452.813167] ipu3_css_pool_cleanup+0x24/0x37 [ipu3_imgu]
[ 452.819117] ipu3_css_pipeline_cleanup+0x61/0xb9 [ipu3_imgu]
[ 452.825454] ipu3_css_stop_streaming+0x1f2/0x321 [ipu3_imgu]
[ 452.831796] imgu_s_stream+0x94/0x443 [ipu3_imgu]
[ 452.837066] ipu3_vb2_stop_streaming+0xf9/0x135 [ipu3_imgu]
[ 452.843308] __vb2_queue_cancel+0x35/0x215 [videobuf2_common]
[ 452.849747] vb2_core_streamoff+0x19/0x73 [videobuf2_common]
[ 452.856086] __video_do_ioctl+0x34e/0x450
[ 452.860610] video_usercopy+0x25e/0x597
[ 452.864913] v4l2_ioctl+0x45/0x49
[ 452.868630] vfs_ioctl+0x1b/0x30
[ 452.872245] do_vfs_ioctl+0x479/0x6d0
[ 452.876349] ksys_ioctl+0x53/0x79
[ 452.880059] __se_sys_ioctl+0xe/0x12
[ 452.884068] do_syscall_64+0x52/0x60
[ 452.888075] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 452.895396] The buggy address belongs to the object at ffff888150348180
which belongs to the cache kmalloc-64 of size 64
[ 452.909203] The buggy address is located 32 bytes inside of
64-byte region [ffff888150348180, ffff8881503481c0)
[ 452.922142] The buggy address belongs to the page:
[ 452.927506] page:ffffea000540d200 count:1 mapcount:0 mapping:ffff88815ac0f840 index:0x0 compound_mapcount: 0
[ 452.938503] flags: 0x8000000000010200(slab|head)
[ 452.943675] raw: 8000000000010200 ffffea00055d1d08 ffffea0005345e08 ffff88815ac0f840
[ 452.952342] raw: 0000000000000000 0000000000150015 00000001ffffffff 0000000000000000
[ 452.961008] page dumped because: kasan: bad access detected
[ 452.968915] Memory state around the buggy address:
[ 452.974277] ffff888150348080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 452.982361] ffff888150348100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 452.990435] >ffff888150348180: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[ 452.998518] ^
[ 453.003291] ffff888150348200: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 453.011376] ffff888150348280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 453.019457] ==================================================================
[ 453.027537] Disabling lock debugging due to kernel taint
[ 453.034645] ==================================================================
[ 453.042736] BUG: KASAN: double-free or invalid-free in kfree+0x8b/0x4d7
[ 453.051817] CPU: 1 PID: 10289 Comm: yavta Tainted: G B WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 453.063006] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 453.071767] Call Trace:
[ 453.074513] dump_stack+0x6a/0xb1
[ 453.078233] ? kfree+0x8b/0x4d7
[ 453.081757] ? kfree+0x8b/0x4d7
[ 453.085284] print_address_description+0x8e/0x279
[ 453.090556] ? kfree+0x8b/0x4d7
[ 453.094082] ? kfree+0x8b/0x4d7
[ 453.097608] kasan_report_invalid_free+0x58/0x95
[ 453.102787] __kasan_slab_free+0x9f/0x101
[ 453.107286] slab_free_freelist_hook+0x4d/0x9e
[ 453.112266] ? ipu3_dmamap_free+0x6d/0x9c [ipu3_imgu]
[ 453.117930] kfree+0x8b/0x4d7
[ 453.121262] ? __free_pages+0x2f/0x71
[ 453.125368] ipu3_dmamap_free+0x6d/0x9c [ipu3_imgu]
[ 453.130833] ipu3_css_pool_cleanup+0x24/0x37 [ipu3_imgu]
[ 453.136786] ipu3_css_pipeline_cleanup+0x61/0xb9 [ipu3_imgu]
[ 453.143131] ipu3_css_stop_streaming+0x1f2/0x321 [ipu3_imgu]
[ 453.149476] imgu_s_stream+0x94/0x443 [ipu3_imgu]
[ 453.154750] ? ipu3_vb2_buf_queue+0x280/0x280 [ipu3_imgu]
[ 453.160798] ? vb2_dma_sg_unmap_dmabuf+0x16/0x6f [videobuf2_dma_sg]
[ 453.167821] ? vb2_buffer_in_use+0x36/0x58 [videobuf2_common]
[ 453.174263] ipu3_vb2_stop_streaming+0xf9/0x135 [ipu3_imgu]
[ 453.180500] __vb2_queue_cancel+0x35/0x215 [videobuf2_common]
[ 453.186938] vb2_core_streamoff+0x19/0x73 [videobuf2_common]
[ 453.193276] __video_do_ioctl+0x34e/0x450
[ 453.197789] video_usercopy+0x25e/0x597
[ 453.202088] ? video_ioctl2+0x16/0x16
[ 453.206193] ? __switch_to_asm+0x34/0x70
[ 453.210587] v4l2_ioctl+0x45/0x49
[ 453.214302] vfs_ioctl+0x1b/0x30
[ 453.217923] do_vfs_ioctl+0x479/0x6d0
[ 453.222033] ksys_ioctl+0x53/0x79
[ 453.225750] __se_sys_ioctl+0xe/0x12
[ 453.229757] do_syscall_64+0x52/0x60
[ 453.233765] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 453.239423] RIP: 0033:0x7a5d64b73967
[ 453.243429] Code: 8a 66 90 48 8b 05 29 55 2b 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 54 2b 00 f7 d8 64 89 01 48
[ 453.264428] RSP: 002b:00007fff3483aca8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 453.272904] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007a5d64b73967
[ 453.280892] RDX: 00007fff3483acb4 RSI: 0000000040045613 RDI: 0000000000000003
[ 453.288877] RBP: 0000000000404c48 R08: fffffffffed7c030 R09: fffffffffed7c020
[ 453.296861] R10: fffffffffed7c010 R11: 0000000000000246 R12: 0000000000404c56
[ 453.304842] R13: 0000000000000001 R14: 00007fff3483c75c R15: 000000000062b800
[ 453.314500] Allocated by task 10289:
[ 453.318507] set_track+0x64/0xfb
[ 453.322120] __kmalloc+0x94/0x1af
[ 453.325832] kvmalloc_node+0x4e/0x84
[ 453.329839] ipu3_dmamap_alloc+0xec/0x503 [ipu3_imgu]
[ 453.335488] ipu3_css_pool_init+0x43/0x99 [ipu3_imgu]
[ 453.341144] ipu3_css_start_streaming+0x25cf/0x29a7 [ipu3_imgu]
[ 453.347775] imgu_s_stream+0x133/0x443 [ipu3_imgu]
[ 453.353144] ipu3_vb2_start_streaming+0x1a3/0x1f1 [ipu3_imgu]
[ 453.359579] vb2_start_streaming+0x71/0x11c [videobuf2_common]
[ 453.366111] vb2_core_streamon+0xf8/0x118 [videobuf2_common]
[ 453.372449] __video_do_ioctl+0x34e/0x450
[ 453.376935] video_usercopy+0x25e/0x597
[ 453.381230] v4l2_ioctl+0x45/0x49
[ 453.384945] vfs_ioctl+0x1b/0x30
[ 453.388568] do_vfs_ioctl+0x479/0x6d0
[ 453.392672] ksys_ioctl+0x53/0x79
[ 453.396387] __se_sys_ioctl+0xe/0x12
[ 453.400393] do_syscall_64+0x52/0x60
[ 453.404395] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 453.411716] Freed by task 10187:
[ 453.415340] set_track+0x64/0xfb
[ 453.418960] __kasan_slab_free+0xde/0x101
[ 453.423452] slab_free_freelist_hook+0x4d/0x9e
[ 453.428430] kfree+0x8b/0x4d7
[ 453.431766] ipu3_dmamap_free+0x6d/0x9c [ipu3_imgu]
[ 453.437231] ipu3_css_pool_cleanup+0x24/0x37 [ipu3_imgu]
[ 453.443183] ipu3_css_pipeline_cleanup+0x61/0xb9 [ipu3_imgu]
[ 453.449522] ipu3_css_stop_streaming+0x1f2/0x321 [ipu3_imgu]
[ 453.455864] imgu_s_stream+0x94/0x443 [ipu3_imgu]
[ 453.461133] ipu3_vb2_stop_streaming+0xf9/0x135 [ipu3_imgu]
[ 453.467375] __vb2_queue_cancel+0x35/0x215 [videobuf2_common]
[ 453.473810] vb2_core_streamoff+0x19/0x73 [videobuf2_common]
[ 453.480154] __video_do_ioctl+0x34e/0x450
[ 453.484650] video_usercopy+0x25e/0x597
[ 453.488946] v4l2_ioctl+0x45/0x49
[ 453.492652] vfs_ioctl+0x1b/0x30
[ 453.496271] do_vfs_ioctl+0x479/0x6d0
[ 453.500376] ksys_ioctl+0x53/0x79
[ 453.504091] __se_sys_ioctl+0xe/0x12
[ 453.508094] do_syscall_64+0x52/0x60
[ 453.512099] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 453.519424] The buggy address belongs to the object at ffff888153eaf380
which belongs to the cache kmalloc-1k of size 1024
[ 453.533431] The buggy address is located 0 bytes inside of
1024-byte region [ffff888153eaf380, ffff888153eaf780)
[ 453.546466] The buggy address belongs to the page:
[ 453.551828] page:ffffea00054faa00 count:1 mapcount:0 mapping:ffff88815ac0f180 index:0x0 compound_mapcount: 0
[ 453.562829] flags: 0x8000000000010200(slab|head)
[ 453.568002] raw: 8000000000010200 ffffea0004b88a08 ffffea000565e808 ffff88815ac0f180
[ 453.576670] raw: 0000000000000000 0000000000180018 00000001ffffffff 0000000000000000
[ 453.585337] page dumped because: kasan: bad access detected
[ 453.593239] Memory state around the buggy address:
[ 453.598596] ffff888153eaf280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 453.606680] ffff888153eaf300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 453.614765] >ffff888153eaf380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 453.622846] ^
[ 453.626467] ffff888153eaf400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 453.634550] ffff888153eaf480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 453.642624] ==================================================================
[ 453.653315] ------------[ cut here ]------------
[ 453.658485] kernel BUG at /mnt/host/source/src/third_party/kernel/v4.4/mm/slub.c:3940!
[ 453.667369] invalid opcode: 0000 [#1] PREEMPT SMP KASAN PTI
[ 453.673604] CPU: 0 PID: 9 Comm: ksoftirqd/0 Tainted: G B WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 453.684990] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 453.693762] RIP: 0010:kfree+0x4d3/0x4d7
[ 453.698049] Code: 7d b0 48 8b 75 a0 e8 38 e6 6e 00 4c 89 ff 4c 89 f6 e8 3f a9 ff ff e9 22 fc ff ff 4c 89 ff 4c 89 f6 e8 1d b6 ff ff eb d6 0f 0b <0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 48 8b 07 48 8b 4f 08 48 89
[ 453.719051] RSP: 0018:ffff88815af17d20 EFLAGS: 00010246
[ 453.724904] RAX: ffffea0001d9a288 RBX: ffff88807644d860 RCX: ffffea0001d91300
[ 453.732891] RDX: ffffea0001d91340 RSI: 0000000000000004 RDI: 0000000001d91361
[ 453.740866] RBP: ffff88815af17da8 R08: 0000000000000000 R09: fffffbfff6521ca7
[ 453.748852] R10: ffffffffb290e533 R11: dffffc0000000000 R12: ffffffffb16dec02
[ 453.751044] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
[ 453.756835] R13: ffff88815b62ab70 R14: ffffea0001d91300 R15: ffff88815af08000
[ 453.756839] FS: 0000000000000000(0000) GS:ffff88815b600000(0000) knlGS:0000000000000000
[ 453.756841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 453.756843] CR2: 00007a8ac018f000 CR3: 00000000744ba006 CR4: 00000000003606f0
[ 453.756845] Call Trace:
[ 453.756855] rcu_process_callbacks+0x20a/0x437
[ 453.769422] BUG: Bad page state in process yavta pfn:74756
[ 453.771362] __do_softirq+0x16c/0x33e
[ 453.780406] page:ffffea0001d1d580 count:0 mapcount:0 mapping:ffff88815ae4c6c0 index:0x0
[ 453.786839] run_ksoftirqd+0x1d/0x34
[ 453.794809] flags: 0x4000000000000000()
[ 453.797551] smpboot_thread_fn+0x1bb/0x291
[ 453.802503] raw: 4000000000000000 dead000000000100 dead000000000200 ffff88815ae4c6c0
[ 453.808736] ? cpu_report_death+0x84/0x84
[ 453.812829] raw: 0000000000000000 0000000000100010 00000000ffffffff 0000000000000000
[ 453.821781] kthread+0xfd/0x10d
[ 453.825773] page dumped because: non-NULL mapping
[ 453.830072] ? cpu_report_death+0x84/0x84
[ 453.834649] Modules linked in: cmac rfcomm uinput snd_soc_kbl_rt5663_max98927 snd_soc_skl_ssp_clk snd_soc_hdac_hdmi snd_soc_dmic btusb btrtl btbcm asix usbnet btintel bluetooth snd_soc_skl snd_soc_skl_ipc ecdh_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_hda_core ipu3_imgu(C) ipu3_cio2 iova videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_rt5663 snd_soc_max98927 at24 snd_soc_rl6231 ov13858 ov5670 v4l2_fwnode dw9714 bridge stp llc acpi_als kfifo_buf industrialio ipt_MASQUERADE lzo lzo_compress zram xt_mark fuse snd_seq_dummy snd_seq snd_seq_device cfg80211 ip6table_filter r8152 mii joydev
[ 453.843306] ? kthread_destroy_worker+0x49/0x49
[ 453.847795] CPU: 1 PID: 10289 Comm: yavta Tainted: G B WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 453.856449] ret_from_fork+0x35/0x40
[ 453.859956] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 453.865208] Modules linked in: cmac rfcomm uinput snd_soc_kbl_rt5663_max98927 snd_soc_skl_ssp_clk snd_soc_hdac_hdmi snd_soc_dmic btusb btrtl btbcm asix usbnet btintel bluetooth snd_soc_skl snd_soc_skl_ipc ecdh_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_hda_core ipu3_imgu(C) ipu3_cio2 iova videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_rt5663 snd_soc_max98927 at24 snd_soc_rl6231 ov13858 ov5670 v4l2_fwnode dw9714 bridge stp llc acpi_als kfifo_buf industrialio ipt_MASQUERADE lzo lzo_compress zram xt_mark fuse snd_seq_dummy snd_seq snd_seq_device cfg80211 ip6table_filter r8152 mii joydev
[ 453.869692] Call Trace:
[ 453.933872] gsmi: Log Shutdown Reason 0x03
[ 453.938939] dump_stack+0x6a/0xb1
[ 453.950359] ---[ end trace ed0895d0744ba933 ]---
[ 453.954123] bad_page+0x140/0x14a
[ 453.954128] free_pages_check+0x87/0x95
[ 453.954132] free_pcppages_bulk+0xbd/0x218
[ 453.954137] free_unref_page+0x49/0x6e
[ 453.954142] __free_pages+0x4a/0x71
[ 453.962932] RIP: 0010:kfree+0x4d3/0x4d7
[ 454.025068] vb2_dma_sg_put+0x8f/0xec [videobuf2_dma_sg]
[ 454.025074] __vb2_buf_mem_free+0x39/0x75 [videobuf2_common]
[ 454.025079] __vb2_queue_free+0xb3/0x19f [videobuf2_common]
[ 454.025084] vb2_core_reqbufs+0x12a/0x312 [videobuf2_common]
[ 454.025090] vb2_ioctl_reqbufs+0x81/0xa8 [videobuf2_v4l2]
[ 454.025098] __video_do_ioctl+0x34e/0x450
[ 454.025105] video_usercopy+0x25e/0x597
[ 454.025109] ? video_ioctl2+0x16/0x16
[ 454.025116] v4l2_ioctl+0x45/0x49
[ 454.025121] vfs_ioctl+0x1b/0x30
[ 454.025125] do_vfs_ioctl+0x479/0x6d0
[ 454.025131] ksys_ioctl+0x53/0x79
[ 454.025136] __se_sys_ioctl+0xe/0x12
[ 454.027896] Code: 7d b0 48 8b 75 a0 e8 38 e6 6e 00 4c 89 ff 4c 89 f6 e8 3f a9 ff ff e9 22 fc ff ff 4c 89 ff 4c 89 f6 e8 1d b6 ff ff eb d6 0f 0b <0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 48 8b 07 48 8b 4f 08 48 89
[ 454.032459] do_syscall_64+0x52/0x60
[ 454.032465] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 454.032469] RIP: 0033:0x7a5d64b73967
[ 454.032473] Code: 8a 66 90 48 8b 05 29 55 2b 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 54 2b 00 f7 d8 64 89 01 48
[ 454.032475] RSP: 002b:00007fff3483acd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 454.032479] RAX: ffffffffffffffda RBX: 00000000023d97a0 RCX: 00007a5d64b73967
[ 454.036194] RSP: 0018:ffff88815af17d20 EFLAGS: 00010246
[ 454.041352] RDX: 00007fff3483ade0 RSI: 00000000c0145608 RDI: 0000000000000003
[ 454.041355] RBP: 0000000000000007 R08: 00007a5d64e287c0 R09: 0000000000000045
[ 454.041356] R10: fffffffffffff88f R11: 0000000000000246 R12: 0000000000000001
[ 454.041358] R13: 00000000023d9778 R14: 00000000023d9750 R15: 000000000062b800
[ 454.041364] BUG: Bad page state in process yavta pfn:7671e
[ 454.041368] page:ffffea0001d9c780 count:0 mapcount:0 mapping:ffff88815ae4c6c0 index:0x0
[ 454.041370] flags: 0x4000000000000000()
[ 454.041375] raw: 4000000000000000 dead000000000100 dead000000000200 ffff88815ae4c6c0
[ 454.041378] raw: 0000000000000000 0000000000100010 00000000ffffffff 0000000000000000
[ 454.041379] page dumped because: non-NULL mapping
[ 454.041380] Modules linked in: cmac rfcomm uinput snd_soc_kbl_rt5663_max98927 snd_soc_skl_ssp_clk snd_soc_hdac_hdmi snd_soc_dmic btusb btrtl btbcm asix usbnet btintel bluetooth snd_soc_skl snd_soc_skl_ipc ecdh_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_hda_core ipu3_imgu(C) ipu3_cio2 iova videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_rt5663 snd_soc_max98927 at24 snd_soc_rl6231 ov13858 ov5670 v4l2_fwnode dw9714 bridge stp llc acpi_als kfifo_buf industrialio ipt_MASQUERADE lzo lzo_compress zram xt_mark fuse snd_seq_dummy snd_seq snd_seq_device cfg80211 ip6table_filter r8152 mii joydev
[ 454.045094] RAX: ffffea0001d9a288 RBX: ffff88807644d860 RCX: ffffea0001d91300
[ 454.049378] CPU: 1 PID: 10289 Comm: yavta Tainted: G B D WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 454.049380] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 454.049381] Call Trace:
[ 454.049389] dump_stack+0x6a/0xb1
[ 454.049395] bad_page+0x140/0x14a
[ 454.049399] free_pages_check+0x87/0x95
[ 454.049403] free_pcppages_bulk+0xbd/0x218
[ 454.049408] free_unref_page+0x49/0x6e
[ 454.049412] __free_pages+0x4a/0x71
[ 454.049420] vb2_dma_sg_put+0x8f/0xec [videobuf2_dma_sg]
[ 454.049433] __vb2_buf_mem_free+0x39/0x75 [videobuf2_common]
[ 454.049438] __vb2_queue_free+0xb3/0x19f [videobuf2_common]
[ 454.049444] vb2_core_reqbufs+0x12a/0x312 [videobuf2_common]
[ 454.049450] vb2_ioctl_reqbufs+0x81/0xa8 [videobuf2_v4l2]
[ 454.049455] __video_do_ioctl+0x34e/0x450
[ 454.054060] RDX: ffffea0001d91340 RSI: 0000000000000004 RDI: 0000000001d91361
[ 454.058234] video_usercopy+0x25e/0x597
[ 454.058238] ? video_ioctl2+0x16/0x16
[ 454.058243] v4l2_ioctl+0x45/0x49
[ 454.062148] RBP: ffff88815af17da8 R08: 0000000000000000 R09: fffffbfff6521ca7
[ 454.066440] vfs_ioctl+0x1b/0x30
[ 454.066444] do_vfs_ioctl+0x479/0x6d0
[ 454.066448] ksys_ioctl+0x53/0x79
[ 454.066452] __se_sys_ioctl+0xe/0x12
[ 454.066456] do_syscall_64+0x52/0x60
[ 454.066461] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 454.066464] RIP: 0033:0x7a5d64b73967
[ 454.066468] Code: 8a 66 90 48 8b 05 29 55 2b 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 54 2b 00 f7 d8 64 89 01 48
[ 454.066469] RSP: 002b:00007fff3483acd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 454.066476] RAX: ffffffffffffffda RBX: 00000000023d97a0 RCX: 00007a5d64b73967
[ 454.072423] R10: ffffffffb290e533 R11: dffffc0000000000 R12: ffffffffb16dec02
[ 454.078745] RDX: 00007fff3483ade0 RSI: 00000000c0145608 RDI: 0000000000000003
[ 454.078747] RBP: 0000000000000007 R08: 00007a5d64e287c0 R09: 0000000000000045
[ 454.078749] R10: fffffffffffff88f R11: 0000000000000246 R12: 0000000000000001
[ 454.078750] R13: 00000000023d9778 R14: 00000000023d9750 R15: 000000000062b800
[ 454.078758] BUG: Bad page state in process yavta pfn:746a6
[ 454.085002] R13: ffff88815b62ab70 R14: ffffea0001d91300 R15: ffff88815af08000
[ 454.091322] page:ffffea0001d1a980 count:0 mapcount:0 mapping:ffff88815ae4c6c0 index:0x0
[ 454.091325] flags: 0x4000000000000000()
[ 454.091329] raw: 4000000000000000 dead000000000100 dead000000000200 ffff88815ae4c6c0
[ 454.091331] raw: 0000000000000000 0000000000100010 00000000ffffffff 0000000000000000
[ 454.097378] FS: 0000000000000000(0000) GS:ffff88815b600000(0000) knlGS:0000000000000000
[ 454.101841] page dumped because: non-NULL mapping
[ 454.101843] Modules linked in: cmac rfcomm uinput snd_soc_kbl_rt5663_max98927 snd_soc_skl_ssp_clk snd_soc_hdac_hdmi snd_soc_dmic btusb btrtl btbcm asix usbnet btintel bluetooth snd_soc_skl snd_soc_skl_ipc ecdh_generic snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_hda_core ipu3_imgu(C) ipu3_cio2 iova videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_rt5663 snd_soc_max98927 at24 snd_soc_rl6231 ov13858 ov5670 v4l2_fwnode dw9714 bridge stp llc acpi_als kfifo_buf industrialio ipt_MASQUERADE lzo lzo_compress zram xt_mark fuse snd_seq_dummy snd_seq snd_seq_device cfg80211 ip6table_filter r8152 mii joydev
[ 454.106153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 454.110240] CPU: 1 PID: 10289 Comm: yavta Tainted: G B D WC 4.20.0-rc6-00031-g3b32400169db-dirty #37
[ 454.110242] Hardware name: HP Soraka/Soraka, BIOS Google_Soraka.10431.17.0 03/22/2018
[ 454.110243] Call Trace:
[ 454.110250] dump_stack+0x6a/0xb1
[ 454.110265] bad_page+0x140/0x14a
[ 454.113968] CR2: 00007a8ac018f000 CR3: 00000000744ba006 CR4: 00000000003606f0
[ 454.117574] free_pages_check+0x87/0x95
[ 454.117579] free_pcppages_bulk+0xbd/0x218
[ 454.117583] free_unref_page+0x49/0x6e
[ 454.121691] Kernel panic - not syncing: Fatal exception in interrupt
[ 454.125400] __free_pages+0x4a/0x71
[snip]
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-12 2:30 ` Mani, Rajmohan
@ 2019-01-12 15:10 ` Laurent Pinchart
[not found] ` <6F87890CF0F5204F892DEA1EF0D77A599B323499@fmsmsx122.amr.corp.intel.com>
0 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2019-01-12 15:10 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Jacopo Mondi, Tomasz Figa, Zhi, Yong, Qiu, Tian Shu, Cao, Bingbu,
Linux Media Mailing List, Sakari Ailus, Mauro Carvalho Chehab,
Hans Verkuil, Hu, Jerry W, Toivonen, Tuukka
Hello Raj,
On Saturday, 12 January 2019 04:30:49 EET Mani, Rajmohan wrote:
[snip]
> I finally managed to reproduce the issue with 4.20-rc6, with KASAN enabled
> and with CONFIG_SLUB_DEBUG_ON with SLAB_STORE_USER.
Nice ! Thank you for your work.
> The following line indicates the crash happens when yavta PID 10289 tries to
> free the memory.
>
> [ 452.437844] BUG: KASAN: use-after-free in ipu3_dmamap_free+0x50/0x9c
> [ipu3_imgu]
> [ 452.446123] Read of size 8 at addr ffff8881503481a0 by task yavta/10289
>
> The above looks to be normal, since it's the same task that allocated this
> memory.
> [ 452.685731] Allocated by task 10289:
>
> Before the above happened, yavta/10187 came in and freed this memory per
> KASAN.
> [ 452.787656] Freed by task 10187:
>
> Is this (one instance of yavta freeing the memory allocated by another
> instance of yavta) expected? Or does it indicate that mmap giving the same
> address across these 2 instances of yavta? I need to debug / confirm the
> latter case.
KASAN prints the task name (and process ID) to help you debugging the problem,
but this doesn't mean that yavta is freeing the memory. yavta exercises the
V4L2 API exposed by the driver, and internally, down the call stack,
ipu3_dmamap_free() is called by the driver. According to the backtraces you
posted, this is in response to a VIDIOC_STREAMOFF call from yavta. I would
expect VIDIOC_STREAMOFF to free DMA mappings created for the buffers on the
corresponding video nodes, and thus allocated by the same task. The fact that
memory is allocated in one task and freed in another seems weird to me in this
case.
My guess is that when using multiple instances of yavta the calls to
VIDIOC_STREAMOFF on the different video nodes are asynchronous and happen in a
way that the driver does not expect. Regardless of how the API is exercised by
applications, in a good or bad way, the IPU3 driver must not crash. It needs
to be prepared for all V4L2 ioctls to be called at any time, and an
application could call VIDIOC_STREAMOFF on any video node while the IPU3 is
busy processing images.
> With the help of local application that operates these pipes in a serial
> fashion, I do not see this issue.
[snip]
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
[parent not found: <6F87890CF0F5204F892DEA1EF0D77A599B31FAF4@fmsmsx122.amr.corp.intel.com>]
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
[not found] ` <6F87890CF0F5204F892DEA1EF0D77A599B31FAF4@fmsmsx122.amr.corp.intel.com>
@ 2019-01-08 23:34 ` Laurent Pinchart
0 siblings, 0 replies; 123+ messages in thread
From: Laurent Pinchart @ 2019-01-08 23:34 UTC (permalink / raw)
To: Mani, Rajmohan
Cc: Tomasz Figa, Zhi, Yong, Linux Media Mailing List, Sakari Ailus,
Mauro Carvalho Chehab, Hans Verkuil, 'Zheng, Jian Xu',
Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu,
Jacopo Mondi
Hi Raj,
(CC'ing Jacopo Mondi)
On Saturday, 5 January 2019 04:26:16 EET Mani, Rajmohan wrote:
> >> On Tuesday, 11 December 2018 15:43:53 EET Laurent Pinchart wrote:
> >>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> >>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> >>>>
> >>>> [snip]
> >>>>
> >>>>> I can see a couple of steps missing in the script below.
> >>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-Nove
> >>>>> mber/000040.html)
> >>>>>
> >>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> >>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> >>>>> image processing"...
> >>>>>
> >>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> >>>>>
> >>>>> Also the pipe mode of the corresponding V4L2 subdev should be
> >>>>> set as desired (e.g 0 for video mode or 1 for still mode)
> >>>>> through the control id 0x009819a1 as below.
> >>>>>
> >>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> >>>>
> >>>> I assume the control takes a valid default value ? It's better to
> >>>> set it explicitly anyway, so I'll do so.
> >>>>
> >>>>> 2. ImgU pipeline needs to be configured for image processing as
> >>>>> below.
> >>>>>
> >>>>> RAW bayer frames go through the following ISP pipeline HW blocks
> >>>>> to have the processed image output to the DDR memory.
> >>>>>
> >>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >>>>> Geometric Distortion Correction (GDC) -> DDR
> >>>>>
> >>>>> The ImgU V4L2 subdev has to be configured with the supported
> >>>>> resolutions in all the above HW blocks, for a given input
> >>>>> resolution.
> >>>>>
> >>>>> For a given supported resolution for an input frame, the Input
> >>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured
> >>>>> with the supported resolutions. This information can be obtained
> >>>>> by looking at the following IPU3 ISP configuration table for
> >>>>> ov5670 sensor.
> >>>>>
> >>>>> https://chromium.googlesource.com/chromiumos/overlays/board-over
> >>>>> lays/+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-
> >>>>> poppy/files/gcss/graph_settings_ov5670.xml
> >>>>>
> >>>>> For the ov5670 example, for an input frame with a resolution of
> >>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> >>>>> corresponding resolutions for input feeder, BDS and GDC are
> >>>>> 2592x1944, 2592x1944 and
> >>>>> 2560x1920 respectively.
> >>>>
> >>>> How is the GDC output resolution computed from the input
> >>>> resolution ? Does the GDC always consume 32 columns and 22 lines ?
> >>>>
> >>>>> The following steps prepare the ImgU ISP pipeline for the image
> >>>>> processing.
> >>>>>
> >>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> >>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> >>>>> obtained above.
> >>>>
> >>>> If I understand things correctly, the GDC resolution is the
> >>>> pipeline output resolution. Why is it configured on pad 0 ?
> >>>>
> >>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
> >>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as
> >>>>> the target, using the input feeder height and width.
> >>>>>
> >>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with
> >>>>> V4L2_SEL_TGT_COMPOSE as the target, using the BDS height and width.
> >>>>>
> >>>>> Once these 2 steps are done, the raw bayer frames can be input
> >>>>> to the ImgU V4L2 subdev for processing.
> >>>>
> >>>> Do I need to capture from both the output and viewfinder nodes ?
> >>>> How are they related to the IF -> BDS -> GDC pipeline, are they
> >>>> both fed from the GDC output ? If so, how does the viewfinder
> >>>> scaler fit in that picture ?
> >>>>
> >>>> I have tried the above configuration with the IPU3 v8 driver, and
> >>>> while the kernel doesn't crash, no images get processed. The
> >>>> userspace processes wait forever for buffers to be ready. I then
> >>>> configured pad 2 to 2560x1920 and pad 3 to 1920x1080, and managed
> >>>> to capture images \o/
> >>>>
> >>>> There's one problem though: during capture, or very soon after it,
> >>>> the machine locks up completely. I suspect a memory corruption, as
> >>>> when it doesn't log immediately commands such as dmesg will not
> >>>> produce any output and just block, until the system freezes soon
> >>>> after (especially when moving the mouse).
> >>>>
> >>>> I would still call this an improvement to some extent, but there's
> >>>> definitely room for more improvements :-)
> >>>>
> >>>> To reproduce the issue, you can run the ipu3-process.sh script
> >>>> (attached to this e-mail) with the following arguments:
> >>>>
> >>>> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
> >>
> >> This should have read
> >>
> >> $ ipu3-process.sh --out 2560x1920 --vf 1920x1080 frame-2592x1944.cio2
> >>
> >> Without the --vf argument no images are processed.
> >>
> >> It seems that the Intel mail server blocked the mail that contained the
> >> script. You can find a copy at http://paste.debian.net/hidden/fd5bb8df/.
>
> Here's what I have so far.
>
> I made the following two changes to get image capture and processing
> working with the above script. I can see the output and vf ppm files to come
> out well.
>
> 1. Remove the "# Set formats" inside the configure_pipeline() from the
> above script and used a simple application (which I invoked from your
> script) to set pad 0 of the "ipu3-imgu 0" V4L2 subdev with these 3 sub
> steps.
>
> a. Set the data format using VIDIOC_SUBDEV_S_FMT on pad 0
> GDC width and height (2560x1920)
>
> b. Set cropping using VIDIOC_SUBDEV_S_SELECTION on pad 0
> with V4L2_SEL_TGT_CROP as target, using the input feeder width
> and height (2592x1944)
>
> c. Set composing using VIDIOC_SUBDEV_S_SELECTION on pad 0
> with V4L2_SEL_TGT_COMPOSE as the target, using
> BDS width and height (2592x1944)
This is exactly what the first media-ctl -V call just below the "Set formats"
comment does.
> 2. Modify yavta app to ignore the error (on "3A stat" node) from
> cap_get_buf_type(), so all 4 nodes can be running at the same time.
I don't get that error with the latest yavta version.
> Can you advise why the " # Set formats" part of configure_pipeline()
> in your script is not achieving the same results as the above 3 sub steps?
I tried removing the three media-ctl -V lines that configure format on pads 2,
3 and 4. The crash then didn't occur when running the script. However, after
rebooting the system and retrying the exact same sequence of operation, the
driver crashed again, and I haven't been able to process images without
crashing since then.
The driver is clearly very sensitive to how formats and selection rectangles
are configured, and crashes when userspace doesn't operate exactly as
expected. This should all be fixed, likely by rewriting the format and
selection rectangle configuration code, especially given that it doesn't
comply with how V4L2 configures scaling on subdevs.
We can discuss this issue with Sakari to decide on how the code should be
architectured. How long do you think it would then take to implement the
solution ?
> Per Bingbu, the driver might be lacking some implementation around this.
>
> >>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> >>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> >>>>> obtained above.
> >>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
> >>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as
> >>>>> the target, using the input feeder height and width.
> >>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE
> >>>>> as the target, using the BDS height and width.
> >
> > Sorry for the delay. I got interrupted with some other work.
> > I will have a look at this and get back soon.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-11 13:43 ` Laurent Pinchart
2018-12-11 14:20 ` Laurent Pinchart
@ 2018-12-12 4:55 ` Bingbu Cao
2018-12-13 22:24 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Bingbu Cao @ 2018-12-12 4:55 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> Hello Rajmohan,
>
> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
>>
>> [snip]
>>
>>> I can see a couple of steps missing in the script below.
>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/00004
>>> 0. html)
>>>
>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3 documentation",
>>> under section "Configuring ImgU V4L2 subdev for image processing"...
>>>
>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
>>>
>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
>>> desired (e.g 0 for video mode or 1 for still mode) through the control
>>> id 0x009819a1 as below.
>>>
>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
>> I assume the control takes a valid default value ? It's better to set it
>> explicitly anyway, so I'll do so.
The video mode is set by default. If you want to set to still mode or change
mode, you need set the subdev control.
>>
>>> 2. ImgU pipeline needs to be configured for image processing as below.
>>>
>>> RAW bayer frames go through the following ISP pipeline HW blocks to
>>> have the processed image output to the DDR memory.
>>>
>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
>>> Geometric Distortion Correction (GDC) -> DDR
>>>
>>> The ImgU V4L2 subdev has to be configured with the supported
>>> resolutions in all the above HW blocks, for a given input resolution.
>>>
>>> For a given supported resolution for an input frame, the Input Feeder,
>>> Bayer Down Scaling and GDC blocks should be configured with the
>>> supported resolutions. This information can be obtained by looking at
>>> the following IPU3 ISP configuration table for ov5670 sensor.
>>>
>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/mas
>>> te r /baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
>>> /graph_settings_ov5670.xml
>>>
>>> For the ov5670 example, for an input frame with a resolution of
>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
>>> 2560x1920 respectively.
>> How is the GDC output resolution computed from the input resolution ? Does
>> the GDC always consume 32 columns and 22 lines ?
All the intermediate resolutions in the pipeline are determined by the actual
use case, in other word determined by the IMGU input resolution(sensor output)
and the final output and viewfinder resolution.
BDS mainly do Bayer downscaling, it has limitation that the downscaling factor
must be a value a integer multiple of 1/32.
GDC output depends on the input and width should be x8 and height x4 alignment.
>>
>>> The following steps prepare the ImgU ISP pipeline for the image
>>> processing.
>>>
>>> 1. The ImgU V4L2 subdev data format should be set by using the
>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
>>> above.
>> If I understand things correctly, the GDC resolution is the pipeline output
>> resolution. Why is it configured on pad 0 ?
We see the GDC output resolution as the input of output system, the sink pad
format is used for output and viewfinder resolutions.
>>
>>> 2. The ImgU V4L2 subdev cropping should be set by using the
>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
>>> target, using the input feeder height and width.
>>>
>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>> target, using the BDS height and width.
>>>
>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>> ImgU V4L2 subdev for processing.
>> Do I need to capture from both the output and viewfinder nodes ? How are
>> they related to the IF -> BDS -> GDC pipeline, are they both fed from the
>> GDC output ? If so, how does the viewfinder scaler fit in that picture ?
The output capture should be set, the viewfinder can be disabled.
The IF and BDS are seen as crop and compose of the imgu input video
device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source pads.
>>
>> I have tried the above configuration with the IPU3 v8 driver, and while the
>> kernel doesn't crash, no images get processed. The userspace processes wait
>> forever for buffers to be ready. I then configured pad 2 to 2560x1920 and
>> pad 3 to 1920x1080, and managed to capture images \o/
>>
>> There's one problem though: during capture, or very soon after it, the
>> machine locks up completely. I suspect a memory corruption, as when it
>> doesn't log immediately commands such as dmesg will not produce any output
>> and just block, until the system freezes soon after (especially when moving
>> the mouse).
>>
>> I would still call this an improvement to some extent, but there's
>> definitely room for more improvements :-)
>>
>> To reproduce the issue, you can run the ipu3-process.sh script (attached to
>> this e-mail) with the following arguments:
>>
>> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
>>
>> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in the
>> IPU3-specific Bayer format (for a total of 6469632 bytes).
> I managed to get the dmesg output, and it doesn't look pretty.
>
> [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm mac80211
> iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
> 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
> intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev at24
> media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
> int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> drm_panel_orientation_quirks i2c_hid hid
> [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> 4.20.0-rc6+ #2
> [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3 48
> 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07 <0f> 0b
> 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> 000000000000000c
> [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> 00000000ffffffff
> [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> ffff8f5cfaba16f0
> [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> ffff8f5cf58f0028
> [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> ffff8f5cf58f04e8
> [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000) knlGS:
> 0000000000000000
> [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> 00000000003606e0
> [ 571.217301] Call Trace:
> [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> [ 571.217480] vfs_ioctl+0x1e/0x2b
> [ 571.217486] do_vfs_ioctl+0x531/0x559
> [ 571.217494] ? vfs_write+0xd1/0xdf
> [ 571.217500] ksys_ioctl+0x50/0x70
> [ 571.217506] __x64_sys_ioctl+0x16/0x19
> [ 571.217512] do_syscall_64+0x53/0x60
> [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48 c7
> c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d
> 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000010
> [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> 00007f85cf9b9f47
> [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> 0000000000000003
> [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> 00007f85d009c700
> [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> 000055f4c4dc0b06
> [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> 00007ffc59057825
> [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-12 4:55 ` Bingbu Cao
@ 2018-12-13 22:24 ` Laurent Pinchart
2018-12-14 2:53 ` Bingbu Cao
2018-12-17 3:14 ` Bingbu Cao
0 siblings, 2 replies; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-13 22:24 UTC (permalink / raw)
To: Bingbu Cao
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
Hello Bingbu,
On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> > On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> >> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> >>
> >> [snip]
> >>
> >>> I can see a couple of steps missing in the script below.
> >>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000
> >>> 040.html)
> >>>
> >>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> >>> documentation", under section "Configuring ImgU V4L2 subdev for image
> >>> processing"...
> >>>
> >>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> >>>
> >>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> >>> desired (e.g 0 for video mode or 1 for still mode) through the control
> >>> id 0x009819a1 as below.
> >>>
> >>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> >>
> >> I assume the control takes a valid default value ? It's better to set it
> >> explicitly anyway, so I'll do so.
>
> The video mode is set by default. If you want to set to still mode or change
> mode, you need set the subdev control.
>
> >>> 2. ImgU pipeline needs to be configured for image processing as below.
> >>>
> >>> RAW bayer frames go through the following ISP pipeline HW blocks to
> >>> have the processed image output to the DDR memory.
> >>>
> >>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >>> Geometric Distortion Correction (GDC) -> DDR
> >>>
> >>> The ImgU V4L2 subdev has to be configured with the supported
> >>> resolutions in all the above HW blocks, for a given input resolution.
> >>>
> >>> For a given supported resolution for an input frame, the Input Feeder,
> >>> Bayer Down Scaling and GDC blocks should be configured with the
> >>> supported resolutions. This information can be obtained by looking at
> >>> the following IPU3 ISP configuration table for ov5670 sensor.
> >>>
> >>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/m
> >>> aster/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/
> >>> gcss/graph_settings_ov5670.xml
> >>>
> >>> For the ov5670 example, for an input frame with a resolution of
> >>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> >>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> >>> 2560x1920 respectively.
> >>
> >> How is the GDC output resolution computed from the input resolution ?
> >> Does the GDC always consume 32 columns and 22 lines ?
>
> All the intermediate resolutions in the pipeline are determined by the
> actual use case, in other word determined by the IMGU input
> resolution(sensor output) and the final output and viewfinder resolution.
> BDS mainly do Bayer downscaling, it has limitation that the downscaling
> factor must be a value a integer multiple of 1/32.
> GDC output depends on the input and width should be x8 and height x4
> alignment.
Thank you for the information. This will need to be captured in the
documentation, along with information related to how each block in the
hardware pipeline interacts with the image size. It should be possible for a
developer to compute the output and viewfinder resolutions based on the
parameters of the image processing algorithms just with the information
contained in the driver documentation.
> >>> The following steps prepare the ImgU ISP pipeline for the image
> >>> processing.
> >>>
> >>> 1. The ImgU V4L2 subdev data format should be set by using the
> >>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> >>> above.
> >>
> >> If I understand things correctly, the GDC resolution is the pipeline
> >> output resolution. Why is it configured on pad 0 ?
>
> We see the GDC output resolution as the input of output system, the sink pad
> format is used for output and viewfinder resolutions.
The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be the
ImgU input, the format configured there should correspond to the format on the
connected video node, and should thus be the sensor format. You can then use
the crop and compose rectangles on pad 0, along with the format, crop and
compose rectangles on the output and viewfinder pads, to configure the device.
This should be fixed in the driver, and the documentation should then be
updated accordingly.
> >>> 2. The ImgU V4L2 subdev cropping should be set by using the
> >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> >>> target, using the input feeder height and width.
> >>>
> >>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >>> target, using the BDS height and width.
> >>>
> >>> Once these 2 steps are done, the raw bayer frames can be input to the
> >>> ImgU V4L2 subdev for processing.
> >>
> >> Do I need to capture from both the output and viewfinder nodes ? How are
> >> they related to the IF -> BDS -> GDC pipeline, are they both fed from the
> >> GDC output ? If so, how does the viewfinder scaler fit in that picture ?
>
> The output capture should be set, the viewfinder can be disabled.
> The IF and BDS are seen as crop and compose of the imgu input video
> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> pads.
The GDC is the last block in the pipeline according to the information
provided above. How can it be seen as the subdev sink pad ? That doesn't make
sense to me. I'm not asking for the MC graph to expose all internal blocks of
the ImgU, but if you want to retain a single subdev model, the format on the
sink pad needs to correspond to what is provided to the ImgU. Please see
figure 4.6 of https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for more information regarding how you can use the sink crop, sink
compose and source crop rectangles.
> >> I have tried the above configuration with the IPU3 v8 driver, and while
> >> the kernel doesn't crash, no images get processed. The userspace
> >> processes wait forever for buffers to be ready. I then configured pad 2
> >> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
> >>
> >> There's one problem though: during capture, or very soon after it, the
> >> machine locks up completely. I suspect a memory corruption, as when it
> >> doesn't log immediately commands such as dmesg will not produce any
> >> output and just block, until the system freezes soon after (especially
> >> when moving the mouse).
> >>
> >> I would still call this an improvement to some extent, but there's
> >> definitely room for more improvements :-)
> >>
> >> To reproduce the issue, you can run the ipu3-process.sh script (attached
> >> to this e-mail) with the following arguments:
> >>
> >> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
> >>
> >> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
> >> the IPU3-specific Bayer format (for a total of 6469632 bytes).
> >
> > I managed to get the dmesg output, and it doesn't look pretty.
> >
> > [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
> > libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
> > ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
> > mac80211
> > iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
> > 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
> > videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
> > intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev
> > at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
> > int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
> > mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
> > sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
> > drm_panel_orientation_quirks i2c_hid hid
> > [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
> > 4.20.0-rc6+ #2
> > [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
> > [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
> > [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3
> > 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
> > <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
> > [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
> > [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
> > 000000000000000c
> > [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
> > 00000000ffffffff
> > [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
> > ffff8f5cfaba16f0
> > [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
> > ffff8f5cf58f0028
> > [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
> > ffff8f5cf58f04e8
> > [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
> > knlGS:
> > 0000000000000000
> > [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
> > 00000000003606e0
> > [ 571.217301] Call Trace:
> > [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
> > [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
> > [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
> > [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
> > [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
> > [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
> > [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
> > [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
> > [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
> > [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
> > [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
> > [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
> > [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
> > [ 571.217480] vfs_ioctl+0x1e/0x2b
> > [ 571.217486] do_vfs_ioctl+0x531/0x559
> > [ 571.217494] ? vfs_write+0xd1/0xdf
> > [ 571.217500] ksys_ioctl+0x50/0x70
> > [ 571.217506] __x64_sys_ioctl+0x16/0x19
> > [ 571.217512] do_syscall_64+0x53/0x60
> > [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 571.217524] RIP: 0033:0x7f85cf9b9f47
> > [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
> > c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
> > <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
> > [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000010
> > [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> > 00007f85cf9b9f47
> > [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
> > 0000000000000003
> > [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
> > 00007f85d009c700
> > [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
> > 000055f4c4dc0b06
> > [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
> > 00007ffc59057825
> > [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
> > [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-13 22:24 ` Laurent Pinchart
@ 2018-12-14 2:53 ` Bingbu Cao
2018-12-17 3:14 ` Bingbu Cao
1 sibling, 0 replies; 123+ messages in thread
From: Bingbu Cao @ 2018-12-14 2:53 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> Hello Bingbu,
>
> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
>>>>
>>>> [snip]
>>>>
>>>>> I can see a couple of steps missing in the script below.
>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000
>>>>> 040.html)
>>>>>
>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
>>>>> documentation", under section "Configuring ImgU V4L2 subdev for image
>>>>> processing"...
>>>>>
>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
>>>>>
>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
>>>>> desired (e.g 0 for video mode or 1 for still mode) through the control
>>>>> id 0x009819a1 as below.
>>>>>
>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
>>>> I assume the control takes a valid default value ? It's better to set it
>>>> explicitly anyway, so I'll do so.
>> The video mode is set by default. If you want to set to still mode or change
>> mode, you need set the subdev control.
>>
>>>>> 2. ImgU pipeline needs to be configured for image processing as below.
>>>>>
>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
>>>>> have the processed image output to the DDR memory.
>>>>>
>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
>>>>> Geometric Distortion Correction (GDC) -> DDR
>>>>>
>>>>> The ImgU V4L2 subdev has to be configured with the supported
>>>>> resolutions in all the above HW blocks, for a given input resolution.
>>>>>
>>>>> For a given supported resolution for an input frame, the Input Feeder,
>>>>> Bayer Down Scaling and GDC blocks should be configured with the
>>>>> supported resolutions. This information can be obtained by looking at
>>>>> the following IPU3 ISP configuration table for ov5670 sensor.
>>>>>
>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/m
>>>>> aster/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/
>>>>> gcss/graph_settings_ov5670.xml
>>>>>
>>>>> For the ov5670 example, for an input frame with a resolution of
>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
>>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
>>>>> 2560x1920 respectively.
>>>> How is the GDC output resolution computed from the input resolution ?
>>>> Does the GDC always consume 32 columns and 22 lines ?
>> All the intermediate resolutions in the pipeline are determined by the
>> actual use case, in other word determined by the IMGU input
>> resolution(sensor output) and the final output and viewfinder resolution.
>> BDS mainly do Bayer downscaling, it has limitation that the downscaling
>> factor must be a value a integer multiple of 1/32.
>> GDC output depends on the input and width should be x8 and height x4
>> alignment.
> Thank you for the information. This will need to be captured in the
> documentation, along with information related to how each block in the
> hardware pipeline interacts with the image size. It should be possible for a
> developer to compute the output and viewfinder resolutions based on the
> parameters of the image processing algorithms just with the information
> contained in the driver documentation.
>
>>>>> The following steps prepare the ImgU ISP pipeline for the image
>>>>> processing.
>>>>>
>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
>>>>> above.
>>>> If I understand things correctly, the GDC resolution is the pipeline
>>>> output resolution. Why is it configured on pad 0 ?
>> We see the GDC output resolution as the input of output system, the sink pad
>> format is used for output and viewfinder resolutions.
> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be the
> ImgU input, the format configured there should correspond to the format on the
> connected video node, and should thus be the sensor format. You can then use
> the crop and compose rectangles on pad 0, along with the format, crop and
> compose rectangles on the output and viewfinder pads, to configure the device.
> This should be fixed in the driver, and the documentation should then be
> updated accordingly.
Ack.
>
>>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
>>>>> target, using the input feeder height and width.
>>>>>
>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>> target, using the BDS height and width.
>>>>>
>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>> ImgU V4L2 subdev for processing.
>>>> Do I need to capture from both the output and viewfinder nodes ? How are
>>>> they related to the IF -> BDS -> GDC pipeline, are they both fed from the
>>>> GDC output ? If so, how does the viewfinder scaler fit in that picture ?
>> The output capture should be set, the viewfinder can be disabled.
>> The IF and BDS are seen as crop and compose of the imgu input video
>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>> pads.
> The GDC is the last block in the pipeline according to the information
> provided above. How can it be seen as the subdev sink pad ? That doesn't make
> sense to me. I'm not asking for the MC graph to expose all internal blocks of
> the ImgU, but if you want to retain a single subdev model, the format on the
> sink pad needs to correspond to what is provided to the ImgU. Please see
> figure 4.6 of https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for more information regarding how you can use the sink crop, sink
> compose and source crop rectangles.
Ack, thanks!
>
>>>> I have tried the above configuration with the IPU3 v8 driver, and while
>>>> the kernel doesn't crash, no images get processed. The userspace
>>>> processes wait forever for buffers to be ready. I then configured pad 2
>>>> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
>>>>
>>>> There's one problem though: during capture, or very soon after it, the
>>>> machine locks up completely. I suspect a memory corruption, as when it
>>>> doesn't log immediately commands such as dmesg will not produce any
>>>> output and just block, until the system freezes soon after (especially
>>>> when moving the mouse).
>>>>
>>>> I would still call this an improvement to some extent, but there's
>>>> definitely room for more improvements :-)
>>>>
>>>> To reproduce the issue, you can run the ipu3-process.sh script (attached
>>>> to this e-mail) with the following arguments:
>>>>
>>>> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
>>>>
>>>> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
>>>> the IPU3-specific Bayer format (for a total of 6469632 bytes).
>>> I managed to get the dmesg output, and it doesn't look pretty.
>>>
>>> [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
>>> libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
>>> ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
>>> [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
>>> mac80211
>>> iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
>>> 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
>>> videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
>>> intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev
>>> at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
>>> int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
>>> mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
>>> sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
>>> drm_panel_orientation_quirks i2c_hid hid
>>> [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
>>> 4.20.0-rc6+ #2
>>> [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
>>> [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
>>> [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3
>>> 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
>>> <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
>>> [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
>>> [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
>>> 000000000000000c
>>> [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
>>> 00000000ffffffff
>>> [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
>>> ffff8f5cfaba16f0
>>> [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
>>> ffff8f5cf58f0028
>>> [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
>>> ffff8f5cf58f04e8
>>> [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
>>> knlGS:
>>> 0000000000000000
>>> [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
>>> 00000000003606e0
>>> [ 571.217301] Call Trace:
>>> [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
>>> [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
>>> [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
>>> [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
>>> [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
>>> [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
>>> [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
>>> [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
>>> [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
>>> [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
>>> [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
>>> [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
>>> [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
>>> [ 571.217480] vfs_ioctl+0x1e/0x2b
>>> [ 571.217486] do_vfs_ioctl+0x531/0x559
>>> [ 571.217494] ? vfs_write+0xd1/0xdf
>>> [ 571.217500] ksys_ioctl+0x50/0x70
>>> [ 571.217506] __x64_sys_ioctl+0x16/0x19
>>> [ 571.217512] do_syscall_64+0x53/0x60
>>> [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [ 571.217524] RIP: 0033:0x7f85cf9b9f47
>>> [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
>>> c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
>>> <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
>>> [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
>>> 0000000000000010
>>> [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
>>> 00007f85cf9b9f47
>>> [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
>>> 0000000000000003
>>> [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
>>> 00007f85d009c700
>>> [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
>>> 000055f4c4dc0b06
>>> [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
>>> 00007ffc59057825
>>> [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
>>> [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-13 22:24 ` Laurent Pinchart
2018-12-14 2:53 ` Bingbu Cao
@ 2018-12-17 3:14 ` Bingbu Cao
2018-12-26 11:03 ` Laurent Pinchart
1 sibling, 1 reply; 123+ messages in thread
From: Bingbu Cao @ 2018-12-17 3:14 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> Hello Bingbu,
>
> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
>>>>
>>>> [snip]
>>>>
>>>>> I can see a couple of steps missing in the script below.
>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/000
>>>>> 040.html)
>>>>>
>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
>>>>> documentation", under section "Configuring ImgU V4L2 subdev for image
>>>>> processing"...
>>>>>
>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
>>>>>
>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
>>>>> desired (e.g 0 for video mode or 1 for still mode) through the control
>>>>> id 0x009819a1 as below.
>>>>>
>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
>>>> I assume the control takes a valid default value ? It's better to set it
>>>> explicitly anyway, so I'll do so.
>> The video mode is set by default. If you want to set to still mode or change
>> mode, you need set the subdev control.
>>
>>>>> 2. ImgU pipeline needs to be configured for image processing as below.
>>>>>
>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
>>>>> have the processed image output to the DDR memory.
>>>>>
>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
>>>>> Geometric Distortion Correction (GDC) -> DDR
>>>>>
>>>>> The ImgU V4L2 subdev has to be configured with the supported
>>>>> resolutions in all the above HW blocks, for a given input resolution.
>>>>>
>>>>> For a given supported resolution for an input frame, the Input Feeder,
>>>>> Bayer Down Scaling and GDC blocks should be configured with the
>>>>> supported resolutions. This information can be obtained by looking at
>>>>> the following IPU3 ISP configuration table for ov5670 sensor.
>>>>>
>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/m
>>>>> aster/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/
>>>>> gcss/graph_settings_ov5670.xml
>>>>>
>>>>> For the ov5670 example, for an input frame with a resolution of
>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
>>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
>>>>> 2560x1920 respectively.
>>>> How is the GDC output resolution computed from the input resolution ?
>>>> Does the GDC always consume 32 columns and 22 lines ?
>> All the intermediate resolutions in the pipeline are determined by the
>> actual use case, in other word determined by the IMGU input
>> resolution(sensor output) and the final output and viewfinder resolution.
>> BDS mainly do Bayer downscaling, it has limitation that the downscaling
>> factor must be a value a integer multiple of 1/32.
>> GDC output depends on the input and width should be x8 and height x4
>> alignment.
> Thank you for the information. This will need to be captured in the
> documentation, along with information related to how each block in the
> hardware pipeline interacts with the image size. It should be possible for a
> developer to compute the output and viewfinder resolutions based on the
> parameters of the image processing algorithms just with the information
> contained in the driver documentation.
>
>>>>> The following steps prepare the ImgU ISP pipeline for the image
>>>>> processing.
>>>>>
>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
>>>>> above.
>>>> If I understand things correctly, the GDC resolution is the pipeline
>>>> output resolution. Why is it configured on pad 0 ?
>> We see the GDC output resolution as the input of output system, the sink pad
>> format is used for output and viewfinder resolutions.
> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be the
> ImgU input, the format configured there should correspond to the format on the
> connected video node, and should thus be the sensor format. You can then use
> the crop and compose rectangles on pad 0, along with the format, crop and
> compose rectangles on the output and viewfinder pads, to configure the device.
> This should be fixed in the driver, and the documentation should then be
> updated accordingly.
Hi, Laurent,
Thanks for your review.
I think it make sense for me that using Pad 0 as the ImgU input(IF).
However, I prefer using the 2 source pads for output and viewfinder.
It makes more sense because the output and viewfinder are independent
output.
The whole pipeline in ImgU looks like:
IF --> BDS --> GDC ---> OUTPUT
|
|-----> VF
The BDS is used to do Bayer downscaling and GDC can do cropping.
My understanding is that scaled size is configured on the CROP rectangle
by COMPOSE selection target, the order seems like not aligned with the
actual processing in ImgU if we set the crop/compose on sink pad.
Is there some rules for the order of the configuration in the subdev API?
Could I use crop selection based on the scaled size?
>>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
>>>>> target, using the input feeder height and width.
>>>>>
>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>> target, using the BDS height and width.
>>>>>
>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>> ImgU V4L2 subdev for processing.
>>>> Do I need to capture from both the output and viewfinder nodes ? How are
>>>> they related to the IF -> BDS -> GDC pipeline, are they both fed from the
>>>> GDC output ? If so, how does the viewfinder scaler fit in that picture ?
>> The output capture should be set, the viewfinder can be disabled.
>> The IF and BDS are seen as crop and compose of the imgu input video
>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>> pads.
> The GDC is the last block in the pipeline according to the information
> provided above. How can it be seen as the subdev sink pad ? That doesn't make
> sense to me. I'm not asking for the MC graph to expose all internal blocks of
> the ImgU, but if you want to retain a single subdev model, the format on the
> sink pad needs to correspond to what is provided to the ImgU. Please see
> figure 4.6 of https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for more information regarding how you can use the sink crop, sink
> compose and source crop rectangles.
>
>>>> I have tried the above configuration with the IPU3 v8 driver, and while
>>>> the kernel doesn't crash, no images get processed. The userspace
>>>> processes wait forever for buffers to be ready. I then configured pad 2
>>>> to 2560x1920 and pad 3 to 1920x1080, and managed to capture images \o/
>>>>
>>>> There's one problem though: during capture, or very soon after it, the
>>>> machine locks up completely. I suspect a memory corruption, as when it
>>>> doesn't log immediately commands such as dmesg will not produce any
>>>> output and just block, until the system freezes soon after (especially
>>>> when moving the mouse).
>>>>
>>>> I would still call this an improvement to some extent, but there's
>>>> definitely room for more improvements :-)
>>>>
>>>> To reproduce the issue, you can run the ipu3-process.sh script (attached
>>>> to this e-mail) with the following arguments:
>>>>
>>>> $ ipu3-process.sh --out 2560x1920 frame-2592x1944.cio2
>>>>
>>>> frame-2592x1944.cio2 is a binary file containing a 2592x1944 images in
>>>> the IPU3-specific Bayer format (for a total of 6469632 bytes).
>>> I managed to get the dmesg output, and it doesn't look pretty.
>>>
>>> [ 571.217192] WARNING: CPU: 3 PID: 1303 at /home/laurent/src/iob/oss/
>>> libcamera/linux/drivers/staging/media/ipu3/ipu3-dmamap.c:172
>>> ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
>>> [ 571.217196] Modules linked in: asix usbnet mii zram arc4 iwlmvm
>>> mac80211
>>> iwlwifi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp cfg80211
>>> 8250_dw hid_multitouch ipu3_cio2 ipu3_imgu(C) videobuf2_dma_sg
>>> videobuf2_memops videobuf2_v4l2 videobuf2_common processor_thermal_device
>>> intel_soc_dts_iosf ov13858 dw9714 ov5670 v4l2_fwnode v4l2_common videodev
>>> at24 media int3403_thermal int340x_thermal_zone cros_ec_lpcs cros_ec_core
>>> int3400_thermal chromeos_pstore mac_hid acpi_thermal_rel autofs4 usbhid
>>> mmc_block hid_generic i915 video i2c_algo_bit drm_kms_helper syscopyarea
>>> sysfillrect sdhci_pci sysimgblt fb_sys_fops cqhci sdhci drm
>>> drm_panel_orientation_quirks i2c_hid hid
>>> [ 571.217254] CPU: 3 PID: 1303 Comm: yavta Tainted: G C
>>> 4.20.0-rc6+ #2
>>> [ 571.217256] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018
>>> [ 571.217267] RIP: 0010:ipu3_dmamap_unmap+0x30/0x75 [ipu3_imgu]
>>> [ 571.217271] Code: 54 55 48 8d af d0 6e 00 00 53 48 8b 76 10 49 89 fc f3
>>> 48 0f bc 8f f0 6e 00 00 48 89 ef 48 d3 ee e8 e6 73 d9 e6 48 85 c0 75 07
>>> <0f> 0b 5b 5d 41 5c c3 48 8b 70 20 48 89 c3 48 8b 40 18 49 8b bc 24
>>> [ 571.217274] RSP: 0018:ffffb675021c7b38 EFLAGS: 00010246
>>> [ 571.217278] RAX: 0000000000000000 RBX: ffff8f5cf58f8448 RCX:
>>> 000000000000000c
>>> [ 571.217280] RDX: 0000000000000000 RSI: 0000000000000202 RDI:
>>> 00000000ffffffff
>>> [ 571.217283] RBP: ffff8f5cf58f6ef8 R08: 00000000000006c5 R09:
>>> ffff8f5cfaba16f0
>>> [ 571.217286] R10: ffff8f5cbf508f98 R11: 000000e03da27aba R12:
>>> ffff8f5cf58f0028
>>> [ 571.217289] R13: ffff8f5cf58f0028 R14: 0000000000000000 R15:
>>> ffff8f5cf58f04e8
>>> [ 571.217293] FS: 00007f85d009c700(0000) GS:ffff8f5cfab80000(0000)
>>> knlGS:
>>> 0000000000000000
>>> [ 571.217296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 571.217299] CR2: 00007f3440fce4b0 CR3: 000000014abf2001 CR4:
>>> 00000000003606e0
>>> [ 571.217301] Call Trace:
>>> [ 571.217316] ipu3_dmamap_free+0x5b/0x8f [ipu3_imgu]
>>> [ 571.217326] ipu3_css_pool_cleanup+0x25/0x2f [ipu3_imgu]
>>> [ 571.217338] ipu3_css_pipeline_cleanup+0x59/0x8f [ipu3_imgu]
>>> [ 571.217348] ipu3_css_stop_streaming+0x15b/0x20f [ipu3_imgu]
>>> [ 571.217360] imgu_s_stream+0x5a/0x30a [ipu3_imgu]
>>> [ 571.217371] ? ipu3_all_nodes_streaming+0x14f/0x16b [ipu3_imgu]
>>> [ 571.217382] ipu3_vb2_stop_streaming+0xe4/0x10f [ipu3_imgu]
>>> [ 571.217392] __vb2_queue_cancel+0x2b/0x1b8 [videobuf2_common]
>>> [ 571.217402] vb2_core_streamoff+0x30/0x71 [videobuf2_common]
>>> [ 571.217418] __video_do_ioctl+0x258/0x38e [videodev]
>>> [ 571.217438] video_usercopy+0x25f/0x4e5 [videodev]
>>> [ 571.217453] ? copy_overflow+0x14/0x14 [videodev]
>>> [ 571.217471] v4l2_ioctl+0x4d/0x58 [videodev]
>>> [ 571.217480] vfs_ioctl+0x1e/0x2b
>>> [ 571.217486] do_vfs_ioctl+0x531/0x559
>>> [ 571.217494] ? vfs_write+0xd1/0xdf
>>> [ 571.217500] ksys_ioctl+0x50/0x70
>>> [ 571.217506] __x64_sys_ioctl+0x16/0x19
>>> [ 571.217512] do_syscall_64+0x53/0x60
>>> [ 571.217519] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [ 571.217524] RIP: 0033:0x7f85cf9b9f47
>>> [ 571.217528] Code: 00 00 00 48 8b 05 51 6f 2c 00 64 c7 00 26 00 00 00 48
>>> c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05
>>> <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 21 6f 2c 00 f7 d8 64 89 01 48
>>> [ 571.217531] RSP: 002b:00007ffc59056b78 EFLAGS: 00000246 ORIG_RAX:
>>> 0000000000000010
>>> [ 571.217535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
>>> 00007f85cf9b9f47
>>> [ 571.217537] RDX: 00007ffc59056b84 RSI: 0000000040045613 RDI:
>>> 0000000000000003
>>> [ 571.217540] RBP: 000055f4c4dc0af8 R08: 00007f85cd7c4000 R09:
>>> 00007f85d009c700
>>> [ 571.217542] R10: 0000000000000020 R11: 0000000000000246 R12:
>>> 000055f4c4dc0b06
>>> [ 571.217545] R13: 0000000000000004 R14: 00007ffc59056d50 R15:
>>> 00007ffc59057825
>>> [ 571.217553] ---[ end trace 4b42bd84953eff53 ]---
>>> [ 571.318645] ipu3-imgu 0000:00:05.0: wait cio gate idle timeout
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-17 3:14 ` Bingbu Cao
@ 2018-12-26 11:03 ` Laurent Pinchart
2019-01-02 2:38 ` Bingbu Cao
0 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2018-12-26 11:03 UTC (permalink / raw)
To: Bingbu Cao
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
Hello Bingbu,
On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> > On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> >> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> >>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> >>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> >>>>
> >>>> [snip]
> >>>>
> >>>>> I can see a couple of steps missing in the script below.
> >>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
> >>>>> 00040.html)
> >>>>>
> >>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> >>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> >>>>> image processing"...
> >>>>>
> >>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> >>>>>
> >>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> >>>>> desired (e.g 0 for video mode or 1 for still mode) through the control
> >>>>> id 0x009819a1 as below.
> >>>>>
> >>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> >>>>
> >>>> I assume the control takes a valid default value ? It's better to set
> >>>> it explicitly anyway, so I'll do so.
> >>
> >> The video mode is set by default. If you want to set to still mode or
> >> change mode, you need set the subdev control.
> >>
> >>>>> 2. ImgU pipeline needs to be configured for image processing as below.
> >>>>>
> >>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> >>>>> have the processed image output to the DDR memory.
> >>>>>
> >>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >>>>> Geometric Distortion Correction (GDC) -> DDR
> >>>>>
> >>>>> The ImgU V4L2 subdev has to be configured with the supported
> >>>>> resolutions in all the above HW blocks, for a given input resolution.
> >>>>>
> >>>>> For a given supported resolution for an input frame, the Input Feeder,
> >>>>> Bayer Down Scaling and GDC blocks should be configured with the
> >>>>> supported resolutions. This information can be obtained by looking at
> >>>>> the following IPU3 ISP configuration table for ov5670 sensor.
> >>>>>
> >>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> >>>>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> >>>>> files/gcss/graph_settings_ov5670.xml
> >>>>>
> >>>>> For the ov5670 example, for an input frame with a resolution of
> >>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> >>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> >>>>> 2560x1920 respectively.
> >>>>
> >>>> How is the GDC output resolution computed from the input resolution ?
> >>>> Does the GDC always consume 32 columns and 22 lines ?
> >>
> >> All the intermediate resolutions in the pipeline are determined by the
> >> actual use case, in other word determined by the IMGU input
> >> resolution(sensor output) and the final output and viewfinder resolution.
> >> BDS mainly do Bayer downscaling, it has limitation that the downscaling
> >> factor must be a value a integer multiple of 1/32.
> >> GDC output depends on the input and width should be x8 and height x4
> >> alignment.
> >
> > Thank you for the information. This will need to be captured in the
> > documentation, along with information related to how each block in the
> > hardware pipeline interacts with the image size. It should be possible for
> > a developer to compute the output and viewfinder resolutions based on the
> > parameters of the image processing algorithms just with the information
> > contained in the driver documentation.
> >
> >>>>> The following steps prepare the ImgU ISP pipeline for the image
> >>>>> processing.
> >>>>>
> >>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> >>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
> >>>>> above.
> >>>>
> >>>> If I understand things correctly, the GDC resolution is the pipeline
> >>>> output resolution. Why is it configured on pad 0 ?
> >>
> >> We see the GDC output resolution as the input of output system, the sink
> >> pad format is used for output and viewfinder resolutions.
> >
> > The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> > the ImgU input, the format configured there should correspond to the
> > format on the connected video node, and should thus be the sensor format.
> > You can then use the crop and compose rectangles on pad 0, along with the
> > format, crop and compose rectangles on the output and viewfinder pads, to
> > configure the device. This should be fixed in the driver, and the
> > documentation should then be updated accordingly.
>
> Hi, Laurent,
>
> Thanks for your review.
>
> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> However, I prefer using the 2 source pads for output and viewfinder.
> It makes more sense because the output and viewfinder are independent
> output.
>
> The whole pipeline in ImgU looks like:
> IF --> BDS --> GDC ---> OUTPUT
> |
> |-----> VF
>
> The BDS is used to do Bayer downscaling and GDC can do cropping.
Does this mean that the main output and the viewfinder output share the same
scaler, and that the only difference in size between the two outputs is solely
due to cropping ?
> My understanding is that scaled size is configured on the CROP rectangle
> by COMPOSE selection target, the order seems like not aligned with the
> actual processing in ImgU if we set the crop/compose on sink pad.
>
> Is there some rules for the order of the configuration in the subdev API?
> Could I use crop selection based on the scaled size?
Please see figure 4.6 in https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/
dev-subdev.html. Scaling is configured on the sink pad through the crop and
compose rectangles, while the source crop rectangle is used to perform
cropping on the output. If you have a single scaler in the hardware pipeline
you can thus configure it on the sink pad, with output and viewfinder separate
cropping configure on the source pad.
> >>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
> >>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> >>>>> target, using the input feeder height and width.
> >>>>>
> >>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >>>>> target, using the BDS height and width.
> >>>>>
> >>>>> Once these 2 steps are done, the raw bayer frames can be input to the
> >>>>> ImgU V4L2 subdev for processing.
> >>>>
> >>>> Do I need to capture from both the output and viewfinder nodes ? How
> >>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> >>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
> >>>> picture ?
> >>
> >> The output capture should be set, the viewfinder can be disabled.
> >> The IF and BDS are seen as crop and compose of the imgu input video
> >> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> >> pads.
> >
> > The GDC is the last block in the pipeline according to the information
> > provided above. How can it be seen as the subdev sink pad ? That doesn't
> > make sense to me. I'm not asking for the MC graph to expose all internal
> > blocks of the ImgU, but if you want to retain a single subdev model, the
> > format on the sink pad needs to correspond to what is provided to the
> > ImgU. Please see figure 4.6 of
> > https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for
> > more information regarding how you can use the sink crop, sink compose
> > and source crop rectangles.
[snip]
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2018-12-26 11:03 ` Laurent Pinchart
@ 2019-01-02 2:38 ` Bingbu Cao
2019-01-02 8:20 ` Laurent Pinchart
2019-03-23 13:02 ` Jacopo Mondi
0 siblings, 2 replies; 123+ messages in thread
From: Bingbu Cao @ 2019-01-02 2:38 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> Hello Bingbu,
>
> On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
>> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
>>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
>>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
>>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
>>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>> I can see a couple of steps missing in the script below.
>>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November/0
>>>>>>> 00040.html)
>>>>>>>
>>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
>>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
>>>>>>> image processing"...
>>>>>>>
>>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
>>>>>>>
>>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
>>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the control
>>>>>>> id 0x009819a1 as below.
>>>>>>>
>>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
>>>>>> I assume the control takes a valid default value ? It's better to set
>>>>>> it explicitly anyway, so I'll do so.
>>>> The video mode is set by default. If you want to set to still mode or
>>>> change mode, you need set the subdev control.
>>>>
>>>>>>> 2. ImgU pipeline needs to be configured for image processing as below.
>>>>>>>
>>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
>>>>>>> have the processed image output to the DDR memory.
>>>>>>>
>>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
>>>>>>> Geometric Distortion Correction (GDC) -> DDR
>>>>>>>
>>>>>>> The ImgU V4L2 subdev has to be configured with the supported
>>>>>>> resolutions in all the above HW blocks, for a given input resolution.
>>>>>>>
>>>>>>> For a given supported resolution for an input frame, the Input Feeder,
>>>>>>> Bayer Down Scaling and GDC blocks should be configured with the
>>>>>>> supported resolutions. This information can be obtained by looking at
>>>>>>> the following IPU3 ISP configuration table for ov5670 sensor.
>>>>>>>
>>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
>>>>>>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
>>>>>>> files/gcss/graph_settings_ov5670.xml
>>>>>>>
>>>>>>> For the ov5670 example, for an input frame with a resolution of
>>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
>>>>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
>>>>>>> 2560x1920 respectively.
>>>>>> How is the GDC output resolution computed from the input resolution ?
>>>>>> Does the GDC always consume 32 columns and 22 lines ?
>>>> All the intermediate resolutions in the pipeline are determined by the
>>>> actual use case, in other word determined by the IMGU input
>>>> resolution(sensor output) and the final output and viewfinder resolution.
>>>> BDS mainly do Bayer downscaling, it has limitation that the downscaling
>>>> factor must be a value a integer multiple of 1/32.
>>>> GDC output depends on the input and width should be x8 and height x4
>>>> alignment.
>>> Thank you for the information. This will need to be captured in the
>>> documentation, along with information related to how each block in the
>>> hardware pipeline interacts with the image size. It should be possible for
>>> a developer to compute the output and viewfinder resolutions based on the
>>> parameters of the image processing algorithms just with the information
>>> contained in the driver documentation.
>>>
>>>>>>> The following steps prepare the ImgU ISP pipeline for the image
>>>>>>> processing.
>>>>>>>
>>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
>>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained
>>>>>>> above.
>>>>>> If I understand things correctly, the GDC resolution is the pipeline
>>>>>> output resolution. Why is it configured on pad 0 ?
>>>> We see the GDC output resolution as the input of output system, the sink
>>>> pad format is used for output and viewfinder resolutions.
>>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
>>> the ImgU input, the format configured there should correspond to the
>>> format on the connected video node, and should thus be the sensor format.
>>> You can then use the crop and compose rectangles on pad 0, along with the
>>> format, crop and compose rectangles on the output and viewfinder pads, to
>>> configure the device. This should be fixed in the driver, and the
>>> documentation should then be updated accordingly.
>> Hi, Laurent,
>>
>> Thanks for your review.
>>
>> I think it make sense for me that using Pad 0 as the ImgU input(IF).
>> However, I prefer using the 2 source pads for output and viewfinder.
>> It makes more sense because the output and viewfinder are independent
>> output.
>>
>> The whole pipeline in ImgU looks like:
>> IF --> BDS --> GDC ---> OUTPUT
>> |
>> |-----> VF
>>
>> The BDS is used to do Bayer downscaling and GDC can do cropping.
> Does this mean that the main output and the viewfinder output share the same
> scaler, and that the only difference in size between the two outputs is solely
> due to cropping ?
Laurent,
No, output only can do crop and viewfinder support crop and scaling, they share
same input.
>
>> My understanding is that scaled size is configured on the CROP rectangle
>> by COMPOSE selection target, the order seems like not aligned with the
>> actual processing in ImgU if we set the crop/compose on sink pad.
>>
>> Is there some rules for the order of the configuration in the subdev API?
>> Could I use crop selection based on the scaled size?
> Please see figure 4.6 in https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/
> dev-subdev.html. Scaling is configured on the sink pad through the crop and
> compose rectangles, while the source crop rectangle is used to perform
> cropping on the output. If you have a single scaler in the hardware pipeline
> you can thus configure it on the sink pad, with output and viewfinder separate
> cropping configure on the source pad.
>
>>>>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
>>>>>>> target, using the input feeder height and width.
>>>>>>>
>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>>>> target, using the BDS height and width.
>>>>>>>
>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>>>> ImgU V4L2 subdev for processing.
>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
>>>>>> picture ?
>>>> The output capture should be set, the viewfinder can be disabled.
>>>> The IF and BDS are seen as crop and compose of the imgu input video
>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>>>> pads.
>>> The GDC is the last block in the pipeline according to the information
>>> provided above. How can it be seen as the subdev sink pad ? That doesn't
>>> make sense to me. I'm not asking for the MC graph to expose all internal
>>> blocks of the ImgU, but if you want to retain a single subdev model, the
>>> format on the sink pad needs to correspond to what is provided to the
>>> ImgU. Please see figure 4.6 of
>>> https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for
>>> more information regarding how you can use the sink crop, sink compose
>>> and source crop rectangles.
> [snip]
>
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-02 2:38 ` Bingbu Cao
@ 2019-01-02 8:20 ` Laurent Pinchart
2019-01-02 20:26 ` Sakari Ailus
2019-03-23 13:02 ` Jacopo Mondi
1 sibling, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2019-01-02 8:20 UTC (permalink / raw)
To: Bingbu Cao
Cc: Mani, Rajmohan, Tomasz Figa, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu
Hello Bingbu,
On Wednesday, 2 January 2019 04:38:33 EET Bingbu Cao wrote:
> On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> >> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> >>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> >>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> >>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> >>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> >>>>>>
> >>>>>> [snip]
> >>>>>>
> >>>>>>> I can see a couple of steps missing in the script below.
> >>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November
> >>>>>>> /000040.html)
> >>>>>>>
> >>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> >>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> >>>>>>> image processing"...
> >>>>>>>
> >>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> >>>>>>>
> >>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> >>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the
> >>>>>>> control id 0x009819a1 as below.
> >>>>>>>
> >>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> >>>>>>
> >>>>>> I assume the control takes a valid default value ? It's better to set
> >>>>>> it explicitly anyway, so I'll do so.
> >>>>
> >>>> The video mode is set by default. If you want to set to still mode or
> >>>> change mode, you need set the subdev control.
> >>>>
> >>>>>>> 2. ImgU pipeline needs to be configured for image processing as
> >>>>>>> below.
> >>>>>>>
> >>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> >>>>>>> have the processed image output to the DDR memory.
> >>>>>>>
> >>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >>>>>>> Geometric Distortion Correction (GDC) -> DDR
> >>>>>>>
> >>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> >>>>>>> resolutions in all the above HW blocks, for a given input
> >>>>>>> resolution.
> >>>>>>>
> >>>>>>> For a given supported resolution for an input frame, the Input
> >>>>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured with
> >>>>>>> the supported resolutions. This information can be obtained by
> >>>>>>> looking at the following IPU3 ISP configuration table for ov5670
> >>>>>>> sensor.
> >>>>>>>
> >>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays
> >>>>>>> /+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> >>>>>>> files/gcss/graph_settings_ov5670.xml
> >>>>>>>
> >>>>>>> For the ov5670 example, for an input frame with a resolution of
> >>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> >>>>>>> corresponding resolutions for input feeder, BDS and GDC are
> >>>>>>> 2592x1944, 2592x1944 and 2560x1920 respectively.
> >>>>>>
> >>>>>> How is the GDC output resolution computed from the input resolution ?
> >>>>>> Does the GDC always consume 32 columns and 22 lines ?
> >>>>
> >>>> All the intermediate resolutions in the pipeline are determined by the
> >>>> actual use case, in other word determined by the IMGU input
> >>>> resolution(sensor output) and the final output and viewfinder
> >>>> resolution. BDS mainly do Bayer downscaling, it has limitation that the
> >>>> downscaling factor must be a value a integer multiple of 1/32.
> >>>> GDC output depends on the input and width should be x8 and height x4
> >>>> alignment.
> >>>
> >>> Thank you for the information. This will need to be captured in the
> >>> documentation, along with information related to how each block in the
> >>> hardware pipeline interacts with the image size. It should be possible
> >>> for a developer to compute the output and viewfinder resolutions based
> >>> on the parameters of the image processing algorithms just with the
> >>> information contained in the driver documentation.
> >>>
> >>>>>>> The following steps prepare the ImgU ISP pipeline for the image
> >>>>>>> processing.
> >>>>>>>
> >>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> >>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> >>>>>>> obtained
> >>>>>>> above.
> >>>>>>
> >>>>>> If I understand things correctly, the GDC resolution is the pipeline
> >>>>>> output resolution. Why is it configured on pad 0 ?
> >>>>
> >>>> We see the GDC output resolution as the input of output system, the
> >>>> sink pad format is used for output and viewfinder resolutions.
> >>>
> >>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> >>> the ImgU input, the format configured there should correspond to the
> >>> format on the connected video node, and should thus be the sensor
> >>> format. You can then use the crop and compose rectangles on pad 0, along
> >>> with the format, crop and compose rectangles on the output and
> >>> viewfinder pads, to configure the device. This should be fixed in the
> >>> driver, and the documentation should then be updated accordingly.
> >>
> >> Hi, Laurent,
> >>
> >> Thanks for your review.
> >>
> >> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> >> However, I prefer using the 2 source pads for output and viewfinder.
> >> It makes more sense because the output and viewfinder are independent
> >> output.
> >>
> >> The whole pipeline in ImgU looks like:
> >> IF --> BDS --> GDC ---> OUTPUT
> >> |-----> VF
> >>
> >> The BDS is used to do Bayer downscaling and GDC can do cropping.
> >
> > Does this mean that the main output and the viewfinder output share the
> > same scaler, and that the only difference in size between the two outputs
> > is solely due to cropping ?
>
> Laurent,
> No, output only can do crop and viewfinder support crop and scaling, they
> share same input.
Then you can't support this with a single subdev for the ImgU, you need at
least two subdevs. I can offer more guidance, but I'll need more information
about the GDC.
> >> My understanding is that scaled size is configured on the CROP rectangle
> >> by COMPOSE selection target, the order seems like not aligned with the
> >> actual processing in ImgU if we set the crop/compose on sink pad.
> >>
> >> Is there some rules for the order of the configuration in the subdev API?
> >> Could I use crop selection based on the scaled size?
> >
> > Please see figure 4.6 in
> > https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html.
> > Scaling is configured on the sink pad through the crop and compose
> > rectangles, while the source crop rectangle is used to perform cropping
> > on the output. If you have a single scaler in the hardware pipeline you
> > can thus configure it on the sink pad, with output and viewfinder
> > separate cropping configure on the source pad.
> >
> >>>>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
> >>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> >>>>>>> target, using the input feeder height and width.
> >>>>>>>
> >>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >>>>>>> target, using the BDS height and width.
> >>>>>>>
> >>>>>>> Once these 2 steps are done, the raw bayer frames can be input to
> >>>>>>> the ImgU V4L2 subdev for processing.
> >>>>>>
> >>>>>> Do I need to capture from both the output and viewfinder nodes ? How
> >>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> >>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in
> >>>>>> that picture ?
> >>>>
> >>>> The output capture should be set, the viewfinder can be disabled.
> >>>> The IF and BDS are seen as crop and compose of the imgu input video
> >>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> >>>> pads.
> >>>
> >>> The GDC is the last block in the pipeline according to the information
> >>> provided above. How can it be seen as the subdev sink pad ? That doesn't
> >>> make sense to me. I'm not asking for the MC graph to expose all internal
> >>> blocks of the ImgU, but if you want to retain a single subdev model, the
> >>> format on the sink pad needs to correspond to what is provided to the
> >>> ImgU. Please see figure 4.6 of
> >>> https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for
> >>> more information regarding how you can use the sink crop, sink compose
> >>> and source crop rectangles.
> >
> > [snip]
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-02 8:20 ` Laurent Pinchart
@ 2019-01-02 20:26 ` Sakari Ailus
2019-01-28 10:09 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Sakari Ailus @ 2019-01-02 20:26 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Bingbu Cao, Mani, Rajmohan, Tomasz Figa, Zhi, Yong,
Linux Media Mailing List, Mauro Carvalho Chehab, Hans Verkuil,
Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu,
Cao, Bingbu
Hi Laurent,
On Wed, Jan 02, 2019 at 10:20:13AM +0200, Laurent Pinchart wrote:
> Hello Bingbu,
>
> On Wednesday, 2 January 2019 04:38:33 EET Bingbu Cao wrote:
> > On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > > On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> > >> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> > >>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> > >>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> > >>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > >>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > >>>>>>
> > >>>>>> [snip]
> > >>>>>>
> > >>>>>>> I can see a couple of steps missing in the script below.
> > >>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November
> > >>>>>>> /000040.html)
> > >>>>>>>
> > >>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > >>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> > >>>>>>> image processing"...
> > >>>>>>>
> > >>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > >>>>>>>
> > >>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > >>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the
> > >>>>>>> control id 0x009819a1 as below.
> > >>>>>>>
> > >>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > >>>>>>
> > >>>>>> I assume the control takes a valid default value ? It's better to set
> > >>>>>> it explicitly anyway, so I'll do so.
> > >>>>
> > >>>> The video mode is set by default. If you want to set to still mode or
> > >>>> change mode, you need set the subdev control.
> > >>>>
> > >>>>>>> 2. ImgU pipeline needs to be configured for image processing as
> > >>>>>>> below.
> > >>>>>>>
> > >>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > >>>>>>> have the processed image output to the DDR memory.
> > >>>>>>>
> > >>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > >>>>>>> Geometric Distortion Correction (GDC) -> DDR
> > >>>>>>>
> > >>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> > >>>>>>> resolutions in all the above HW blocks, for a given input
> > >>>>>>> resolution.
> > >>>>>>>
> > >>>>>>> For a given supported resolution for an input frame, the Input
> > >>>>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured with
> > >>>>>>> the supported resolutions. This information can be obtained by
> > >>>>>>> looking at the following IPU3 ISP configuration table for ov5670
> > >>>>>>> sensor.
> > >>>>>>>
> > >>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays
> > >>>>>>> /+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> > >>>>>>> files/gcss/graph_settings_ov5670.xml
> > >>>>>>>
> > >>>>>>> For the ov5670 example, for an input frame with a resolution of
> > >>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> > >>>>>>> corresponding resolutions for input feeder, BDS and GDC are
> > >>>>>>> 2592x1944, 2592x1944 and 2560x1920 respectively.
> > >>>>>>
> > >>>>>> How is the GDC output resolution computed from the input resolution ?
> > >>>>>> Does the GDC always consume 32 columns and 22 lines ?
> > >>>>
> > >>>> All the intermediate resolutions in the pipeline are determined by the
> > >>>> actual use case, in other word determined by the IMGU input
> > >>>> resolution(sensor output) and the final output and viewfinder
> > >>>> resolution. BDS mainly do Bayer downscaling, it has limitation that the
> > >>>> downscaling factor must be a value a integer multiple of 1/32.
> > >>>> GDC output depends on the input and width should be x8 and height x4
> > >>>> alignment.
> > >>>
> > >>> Thank you for the information. This will need to be captured in the
> > >>> documentation, along with information related to how each block in the
> > >>> hardware pipeline interacts with the image size. It should be possible
> > >>> for a developer to compute the output and viewfinder resolutions based
> > >>> on the parameters of the image processing algorithms just with the
> > >>> information contained in the driver documentation.
> > >>>
> > >>>>>>> The following steps prepare the ImgU ISP pipeline for the image
> > >>>>>>> processing.
> > >>>>>>>
> > >>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> > >>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> > >>>>>>> obtained
> > >>>>>>> above.
> > >>>>>>
> > >>>>>> If I understand things correctly, the GDC resolution is the pipeline
> > >>>>>> output resolution. Why is it configured on pad 0 ?
> > >>>>
> > >>>> We see the GDC output resolution as the input of output system, the
> > >>>> sink pad format is used for output and viewfinder resolutions.
> > >>>
> > >>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> > >>> the ImgU input, the format configured there should correspond to the
> > >>> format on the connected video node, and should thus be the sensor
> > >>> format. You can then use the crop and compose rectangles on pad 0, along
> > >>> with the format, crop and compose rectangles on the output and
> > >>> viewfinder pads, to configure the device. This should be fixed in the
> > >>> driver, and the documentation should then be updated accordingly.
> > >>
> > >> Hi, Laurent,
> > >>
> > >> Thanks for your review.
> > >>
> > >> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> > >> However, I prefer using the 2 source pads for output and viewfinder.
> > >> It makes more sense because the output and viewfinder are independent
> > >> output.
> > >>
> > >> The whole pipeline in ImgU looks like:
> > >> IF --> BDS --> GDC ---> OUTPUT
> > >> |-----> VF
> > >>
> > >> The BDS is used to do Bayer downscaling and GDC can do cropping.
> > >
> > > Does this mean that the main output and the viewfinder output share the
> > > same scaler, and that the only difference in size between the two outputs
> > > is solely due to cropping ?
> >
> > Laurent,
> > No, output only can do crop and viewfinder support crop and scaling, they
> > share same input.
>
> Then you can't support this with a single subdev for the ImgU, you need at
> least two subdevs. I can offer more guidance, but I'll need more information
> about the GDC.
While the current documentation only defines the functionality of the
compose target for sink pads, there are a few sensor drivers supporting it
on source pads already. Some drivers such as the OMAP3 ISP also use the
format on source pads to configure scaling.
The current API certainly allows exposing the compose rectangle also on the
source pads, but to make that generic we'd need to amend the API to tell in
which order these steps take place. In the meantime the behaviour remains
device specific.
>
> > >> My understanding is that scaled size is configured on the CROP rectangle
> > >> by COMPOSE selection target, the order seems like not aligned with the
> > >> actual processing in ImgU if we set the crop/compose on sink pad.
> > >>
> > >> Is there some rules for the order of the configuration in the subdev API?
> > >> Could I use crop selection based on the scaled size?
> > >
> > > Please see figure 4.6 in
> > > https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html.
> > > Scaling is configured on the sink pad through the crop and compose
> > > rectangles, while the source crop rectangle is used to perform cropping
> > > on the output. If you have a single scaler in the hardware pipeline you
> > > can thus configure it on the sink pad, with output and viewfinder
> > > separate cropping configure on the source pad.
> > >
> > >>>>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
> > >>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > >>>>>>> target, using the input feeder height and width.
> > >>>>>>>
> > >>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> > >>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > >>>>>>> target, using the BDS height and width.
> > >>>>>>>
> > >>>>>>> Once these 2 steps are done, the raw bayer frames can be input to
> > >>>>>>> the ImgU V4L2 subdev for processing.
> > >>>>>>
> > >>>>>> Do I need to capture from both the output and viewfinder nodes ? How
> > >>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> > >>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in
> > >>>>>> that picture ?
> > >>>>
> > >>>> The output capture should be set, the viewfinder can be disabled.
> > >>>> The IF and BDS are seen as crop and compose of the imgu input video
> > >>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> > >>>> pads.
> > >>>
> > >>> The GDC is the last block in the pipeline according to the information
> > >>> provided above. How can it be seen as the subdev sink pad ? That doesn't
> > >>> make sense to me. I'm not asking for the MC graph to expose all internal
> > >>> blocks of the ImgU, but if you want to retain a single subdev model, the
> > >>> format on the sink pad needs to correspond to what is provided to the
> > >>> ImgU. Please see figure 4.6 of
> > >>> https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for
> > >>> more information regarding how you can use the sink crop, sink compose
> > >>> and source crop rectangles.
> > >
> > > [snip]
>
> --
> Regards,
>
> Laurent Pinchart
>
>
>
--
Regards,
Sakari Ailus
sakari.ailus@linux.intel.com
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-02 20:26 ` Sakari Ailus
@ 2019-01-28 10:09 ` Jacopo Mondi
2019-01-29 8:56 ` Tomasz Figa
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-01-28 10:09 UTC (permalink / raw)
To: Sakari Ailus
Cc: Laurent Pinchart, Bingbu Cao, Mani, Rajmohan, Tomasz Figa, Zhi,
Yong, Linux Media Mailing List, Mauro Carvalho Chehab,
Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu,
Tian Shu, Cao, Bingbu, libcamera-devel
[-- Attachment #1: Type: text/plain, Size: 11592 bytes --]
Hi Sakari, everyone..
On Wed, Jan 02, 2019 at 10:26:56PM +0200, Sakari Ailus wrote:
> Hi Laurent,
>
> On Wed, Jan 02, 2019 at 10:20:13AM +0200, Laurent Pinchart wrote:
> > Hello Bingbu,
> >
> > On Wednesday, 2 January 2019 04:38:33 EET Bingbu Cao wrote:
> > > On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > > > On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> > > >> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> > > >>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> > > >>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> > > >>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > >>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > >>>>>>
> > > >>>>>> [snip]
> > > >>>>>>
> > > >>>>>>> I can see a couple of steps missing in the script below.
> > > >>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November
> > > >>>>>>> /000040.html)
> > > >>>>>>>
> > > >>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > >>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> > > >>>>>>> image processing"...
> > > >>>>>>>
> > > >>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > >>>>>>>
> > > >>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > >>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the
> > > >>>>>>> control id 0x009819a1 as below.
> > > >>>>>>>
> > > >>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > >>>>>>
> > > >>>>>> I assume the control takes a valid default value ? It's better to set
> > > >>>>>> it explicitly anyway, so I'll do so.
> > > >>>>
> > > >>>> The video mode is set by default. If you want to set to still mode or
> > > >>>> change mode, you need set the subdev control.
> > > >>>>
> > > >>>>>>> 2. ImgU pipeline needs to be configured for image processing as
> > > >>>>>>> below.
> > > >>>>>>>
> > > >>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > >>>>>>> have the processed image output to the DDR memory.
> > > >>>>>>>
> > > >>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > >>>>>>> Geometric Distortion Correction (GDC) -> DDR
> > > >>>>>>>
> > > >>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> > > >>>>>>> resolutions in all the above HW blocks, for a given input
> > > >>>>>>> resolution.
> > > >>>>>>>
> > > >>>>>>> For a given supported resolution for an input frame, the Input
> > > >>>>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured with
> > > >>>>>>> the supported resolutions. This information can be obtained by
> > > >>>>>>> looking at the following IPU3 ISP configuration table for ov5670
> > > >>>>>>> sensor.
> > > >>>>>>>
> > > >>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays
> > > >>>>>>> /+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> > > >>>>>>> files/gcss/graph_settings_ov5670.xml
> > > >>>>>>>
> > > >>>>>>> For the ov5670 example, for an input frame with a resolution of
> > > >>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> > > >>>>>>> corresponding resolutions for input feeder, BDS and GDC are
> > > >>>>>>> 2592x1944, 2592x1944 and 2560x1920 respectively.
> > > >>>>>>
> > > >>>>>> How is the GDC output resolution computed from the input resolution ?
> > > >>>>>> Does the GDC always consume 32 columns and 22 lines ?
> > > >>>>
> > > >>>> All the intermediate resolutions in the pipeline are determined by the
> > > >>>> actual use case, in other word determined by the IMGU input
> > > >>>> resolution(sensor output) and the final output and viewfinder
> > > >>>> resolution. BDS mainly do Bayer downscaling, it has limitation that the
> > > >>>> downscaling factor must be a value a integer multiple of 1/32.
> > > >>>> GDC output depends on the input and width should be x8 and height x4
> > > >>>> alignment.
> > > >>>
> > > >>> Thank you for the information. This will need to be captured in the
> > > >>> documentation, along with information related to how each block in the
> > > >>> hardware pipeline interacts with the image size. It should be possible
> > > >>> for a developer to compute the output and viewfinder resolutions based
> > > >>> on the parameters of the image processing algorithms just with the
> > > >>> information contained in the driver documentation.
> > > >>>
> > > >>>>>>> The following steps prepare the ImgU ISP pipeline for the image
> > > >>>>>>> processing.
> > > >>>>>>>
> > > >>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > >>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> > > >>>>>>> obtained
> > > >>>>>>> above.
> > > >>>>>>
> > > >>>>>> If I understand things correctly, the GDC resolution is the pipeline
> > > >>>>>> output resolution. Why is it configured on pad 0 ?
> > > >>>>
> > > >>>> We see the GDC output resolution as the input of output system, the
> > > >>>> sink pad format is used for output and viewfinder resolutions.
> > > >>>
> > > >>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> > > >>> the ImgU input, the format configured there should correspond to the
> > > >>> format on the connected video node, and should thus be the sensor
> > > >>> format. You can then use the crop and compose rectangles on pad 0, along
> > > >>> with the format, crop and compose rectangles on the output and
> > > >>> viewfinder pads, to configure the device. This should be fixed in the
> > > >>> driver, and the documentation should then be updated accordingly.
> > > >>
> > > >> Hi, Laurent,
> > > >>
> > > >> Thanks for your review.
> > > >>
> > > >> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> > > >> However, I prefer using the 2 source pads for output and viewfinder.
> > > >> It makes more sense because the output and viewfinder are independent
> > > >> output.
> > > >>
> > > >> The whole pipeline in ImgU looks like:
> > > >> IF --> BDS --> GDC ---> OUTPUT
> > > >> |-----> VF
> > > >>
> > > >> The BDS is used to do Bayer downscaling and GDC can do cropping.
> > > >
> > > > Does this mean that the main output and the viewfinder output share the
> > > > same scaler, and that the only difference in size between the two outputs
> > > > is solely due to cropping ?
> > >
> > > Laurent,
> > > No, output only can do crop and viewfinder support crop and scaling, they
> > > share same input.
> >
> > Then you can't support this with a single subdev for the ImgU, you need at
> > least two subdevs. I can offer more guidance, but I'll need more information
> > about the GDC.
>
> While the current documentation only defines the functionality of the
> compose target for sink pads, there are a few sensor drivers supporting it
> on source pads already. Some drivers such as the OMAP3 ISP also use the
> format on source pads to configure scaling.
>
> The current API certainly allows exposing the compose rectangle also on the
> source pads, but to make that generic we'd need to amend the API to tell in
> which order these steps take place. In the meantime the behaviour remains
> device specific.
>
My understanding is that what is currently missing is the support
for viewfinder's ability to scale, as the scaler should get
programmed by configuring a composing rectangle on a source pad which
is not supported by the V4L2 APIs at the moment. Is my understanding correct?
As the composing rectangle is set for both 'output' and 'viewfinder'
through the image format sizes configured on the first sink pad (*),
the viewfinder output is obtained by cropping-only to the image format
sizes configured on source pad number 3 (though SUBDEV_S_FMT not through
SUBDEV_S_SELECTION, as SUBDEV_S_SELECTION is only allowed on sink pad
0 in the driver: see "ipu3_subdev_set_selection()").
As you mentioned "device specific behaviour", what is the intended one
for the ipu3? I assumed the viewfinder scaling/cropping was configured
on the 'viewfinder' video device node, through the VIDIOC_S_SELECTION
ioctl, but looking at the code, that doesn't seem to be listed as
supported in "ipu3_v4l2_ioctl_ops".
How am I supposed to configure scaling on the viewfinder output? Would
adding support for crop/compose to the 'output' and 'viewfinder' video
devices be supported by the V4L2 APIs? That would work with the single
subdevice model that is currently implemented in this patches...
Thanks
j
(*) and this should also be changed, as the image format sizes should
reflect what the imgu receives as input, and the crop() and compose()
rectangles should instead be used to configure BDS and GDC
respectively.
> >
> > > >> My understanding is that scaled size is configured on the CROP rectangle
> > > >> by COMPOSE selection target, the order seems like not aligned with the
> > > >> actual processing in ImgU if we set the crop/compose on sink pad.
> > > >>
> > > >> Is there some rules for the order of the configuration in the subdev API?
> > > >> Could I use crop selection based on the scaled size?
> > > >
> > > > Please see figure 4.6 in
> > > > https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html.
> > > > Scaling is configured on the sink pad through the crop and compose
> > > > rectangles, while the source crop rectangle is used to perform cropping
> > > > on the output. If you have a single scaler in the hardware pipeline you
> > > > can thus configure it on the sink pad, with output and viewfinder
> > > > separate cropping configure on the source pad.
> > > >
> > > >>>>>>> 2. The ImgU V4L2 subdev cropping should be set by using the
> > > >>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the
> > > >>>>>>> target, using the input feeder height and width.
> > > >>>>>>>
> > > >>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> > > >>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > > >>>>>>> target, using the BDS height and width.
> > > >>>>>>>
> > > >>>>>>> Once these 2 steps are done, the raw bayer frames can be input to
> > > >>>>>>> the ImgU V4L2 subdev for processing.
> > > >>>>>>
> > > >>>>>> Do I need to capture from both the output and viewfinder nodes ? How
> > > >>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> > > >>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in
> > > >>>>>> that picture ?
> > > >>>>
> > > >>>> The output capture should be set, the viewfinder can be disabled.
> > > >>>> The IF and BDS are seen as crop and compose of the imgu input video
> > > >>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> > > >>>> pads.
> > > >>>
> > > >>> The GDC is the last block in the pipeline according to the information
> > > >>> provided above. How can it be seen as the subdev sink pad ? That doesn't
> > > >>> make sense to me. I'm not asking for the MC graph to expose all internal
> > > >>> blocks of the ImgU, but if you want to retain a single subdev model, the
> > > >>> format on the sink pad needs to correspond to what is provided to the
> > > >>> ImgU. Please see figure 4.6 of
> > > >>> https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html for
> > > >>> more information regarding how you can use the sink crop, sink compose
> > > >>> and source crop rectangles.
> > > >
> > > > [snip]
> >
> > --
> > Regards,
> >
> > Laurent Pinchart
> >
> >
> >
>
> --
> Regards,
>
> Sakari Ailus
> sakari.ailus@linux.intel.com
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-28 10:09 ` Jacopo Mondi
@ 2019-01-29 8:56 ` Tomasz Figa
2019-02-01 10:04 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Tomasz Figa @ 2019-01-29 8:56 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Sakari Ailus, Laurent Pinchart, Bingbu Cao, Mani, Rajmohan, Zhi,
Yong, Linux Media Mailing List, Mauro Carvalho Chehab,
Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu,
Tian Shu, Cao, Bingbu, libcamera-devel
On Mon, Jan 28, 2019 at 7:08 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Hi Sakari, everyone..
>
> On Wed, Jan 02, 2019 at 10:26:56PM +0200, Sakari Ailus wrote:
> > Hi Laurent,
> >
> > On Wed, Jan 02, 2019 at 10:20:13AM +0200, Laurent Pinchart wrote:
> > > Hello Bingbu,
> > >
> > > On Wednesday, 2 January 2019 04:38:33 EET Bingbu Cao wrote:
> > > > On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > > > > On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> > > > >> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> > > > >>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> > > > >>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> > > > >>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > > >>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > > >>>>>>
> > > > >>>>>> [snip]
> > > > >>>>>>
> > > > >>>>>>> I can see a couple of steps missing in the script below.
> > > > >>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November
> > > > >>>>>>> /000040.html)
> > > > >>>>>>>
> > > > >>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > > >>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> > > > >>>>>>> image processing"...
> > > > >>>>>>>
> > > > >>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > > >>>>>>>
> > > > >>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > > >>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the
> > > > >>>>>>> control id 0x009819a1 as below.
> > > > >>>>>>>
> > > > >>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > > >>>>>>
> > > > >>>>>> I assume the control takes a valid default value ? It's better to set
> > > > >>>>>> it explicitly anyway, so I'll do so.
> > > > >>>>
> > > > >>>> The video mode is set by default. If you want to set to still mode or
> > > > >>>> change mode, you need set the subdev control.
> > > > >>>>
> > > > >>>>>>> 2. ImgU pipeline needs to be configured for image processing as
> > > > >>>>>>> below.
> > > > >>>>>>>
> > > > >>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > > >>>>>>> have the processed image output to the DDR memory.
> > > > >>>>>>>
> > > > >>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > > >>>>>>> Geometric Distortion Correction (GDC) -> DDR
> > > > >>>>>>>
> > > > >>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> > > > >>>>>>> resolutions in all the above HW blocks, for a given input
> > > > >>>>>>> resolution.
> > > > >>>>>>>
> > > > >>>>>>> For a given supported resolution for an input frame, the Input
> > > > >>>>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured with
> > > > >>>>>>> the supported resolutions. This information can be obtained by
> > > > >>>>>>> looking at the following IPU3 ISP configuration table for ov5670
> > > > >>>>>>> sensor.
> > > > >>>>>>>
> > > > >>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays
> > > > >>>>>>> /+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> > > > >>>>>>> files/gcss/graph_settings_ov5670.xml
> > > > >>>>>>>
> > > > >>>>>>> For the ov5670 example, for an input frame with a resolution of
> > > > >>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> > > > >>>>>>> corresponding resolutions for input feeder, BDS and GDC are
> > > > >>>>>>> 2592x1944, 2592x1944 and 2560x1920 respectively.
> > > > >>>>>>
> > > > >>>>>> How is the GDC output resolution computed from the input resolution ?
> > > > >>>>>> Does the GDC always consume 32 columns and 22 lines ?
> > > > >>>>
> > > > >>>> All the intermediate resolutions in the pipeline are determined by the
> > > > >>>> actual use case, in other word determined by the IMGU input
> > > > >>>> resolution(sensor output) and the final output and viewfinder
> > > > >>>> resolution. BDS mainly do Bayer downscaling, it has limitation that the
> > > > >>>> downscaling factor must be a value a integer multiple of 1/32.
> > > > >>>> GDC output depends on the input and width should be x8 and height x4
> > > > >>>> alignment.
> > > > >>>
> > > > >>> Thank you for the information. This will need to be captured in the
> > > > >>> documentation, along with information related to how each block in the
> > > > >>> hardware pipeline interacts with the image size. It should be possible
> > > > >>> for a developer to compute the output and viewfinder resolutions based
> > > > >>> on the parameters of the image processing algorithms just with the
> > > > >>> information contained in the driver documentation.
> > > > >>>
> > > > >>>>>>> The following steps prepare the ImgU ISP pipeline for the image
> > > > >>>>>>> processing.
> > > > >>>>>>>
> > > > >>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > > >>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> > > > >>>>>>> obtained
> > > > >>>>>>> above.
> > > > >>>>>>
> > > > >>>>>> If I understand things correctly, the GDC resolution is the pipeline
> > > > >>>>>> output resolution. Why is it configured on pad 0 ?
> > > > >>>>
> > > > >>>> We see the GDC output resolution as the input of output system, the
> > > > >>>> sink pad format is used for output and viewfinder resolutions.
> > > > >>>
> > > > >>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> > > > >>> the ImgU input, the format configured there should correspond to the
> > > > >>> format on the connected video node, and should thus be the sensor
> > > > >>> format. You can then use the crop and compose rectangles on pad 0, along
> > > > >>> with the format, crop and compose rectangles on the output and
> > > > >>> viewfinder pads, to configure the device. This should be fixed in the
> > > > >>> driver, and the documentation should then be updated accordingly.
> > > > >>
> > > > >> Hi, Laurent,
> > > > >>
> > > > >> Thanks for your review.
> > > > >>
> > > > >> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> > > > >> However, I prefer using the 2 source pads for output and viewfinder.
> > > > >> It makes more sense because the output and viewfinder are independent
> > > > >> output.
> > > > >>
> > > > >> The whole pipeline in ImgU looks like:
> > > > >> IF --> BDS --> GDC ---> OUTPUT
> > > > >> |-----> VF
> > > > >>
> > > > >> The BDS is used to do Bayer downscaling and GDC can do cropping.
> > > > >
> > > > > Does this mean that the main output and the viewfinder output share the
> > > > > same scaler, and that the only difference in size between the two outputs
> > > > > is solely due to cropping ?
> > > >
> > > > Laurent,
> > > > No, output only can do crop and viewfinder support crop and scaling, they
> > > > share same input.
> > >
> > > Then you can't support this with a single subdev for the ImgU, you need at
> > > least two subdevs. I can offer more guidance, but I'll need more information
> > > about the GDC.
> >
> > While the current documentation only defines the functionality of the
> > compose target for sink pads, there are a few sensor drivers supporting it
> > on source pads already. Some drivers such as the OMAP3 ISP also use the
> > format on source pads to configure scaling.
> >
> > The current API certainly allows exposing the compose rectangle also on the
> > source pads, but to make that generic we'd need to amend the API to tell in
> > which order these steps take place. In the meantime the behaviour remains
> > device specific.
> >
>
> My understanding is that what is currently missing is the support
> for viewfinder's ability to scale, as the scaler should get
> programmed by configuring a composing rectangle on a source pad which
> is not supported by the V4L2 APIs at the moment. Is my understanding correct?
>
> As the composing rectangle is set for both 'output' and 'viewfinder'
> through the image format sizes configured on the first sink pad (*),
> the viewfinder output is obtained by cropping-only to the image format
> sizes configured on source pad number 3 (though SUBDEV_S_FMT not through
> SUBDEV_S_SELECTION, as SUBDEV_S_SELECTION is only allowed on sink pad
> 0 in the driver: see "ipu3_subdev_set_selection()").
>
> As you mentioned "device specific behaviour", what is the intended one
> for the ipu3? I assumed the viewfinder scaling/cropping was configured
> on the 'viewfinder' video device node, through the VIDIOC_S_SELECTION
> ioctl, but looking at the code, that doesn't seem to be listed as
> supported in "ipu3_v4l2_ioctl_ops".
>
> How am I supposed to configure scaling on the viewfinder output? Would
> adding support for crop/compose to the 'output' and 'viewfinder' video
> devices be supported by the V4L2 APIs? That would work with the single
> subdevice model that is currently implemented in this patches...
Isn't this what the driver actually implements? My understanding was
that the format on the VF video node determined the scaling settings,
based on the source pad.
That was my understanding on how we should make things work, i.e. try
to put as much as possible into core V4L2 interfaces and only defer to
subdevices or media controller if that's impossible. Laurent didn't
seem to agree with that when we talked last time, though, and I can
see some reasons why actually moving simplifying the video devices as
much as possible and moving as much as possible into subdevices could
make sense (simpler interfaces of endpoints, finer granularity for
feature description, less redundance - only subdevices do processing,
video nodes are only DMAs, etc.).
Best regards,
Tomasz
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-29 8:56 ` Tomasz Figa
@ 2019-02-01 10:04 ` Jacopo Mondi
2019-02-05 6:01 ` Tomasz Figa
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-02-01 10:04 UTC (permalink / raw)
To: Tomasz Figa
Cc: Sakari Ailus, Laurent Pinchart, Bingbu Cao, Mani, Rajmohan, Zhi,
Yong, Linux Media Mailing List, Mauro Carvalho Chehab,
Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu,
Tian Shu, Cao, Bingbu, libcamera-devel
[-- Attachment #1: Type: text/plain, Size: 12318 bytes --]
Hi Tomasz,
On Tue, Jan 29, 2019 at 05:56:35PM +0900, Tomasz Figa wrote:
> On Mon, Jan 28, 2019 at 7:08 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> >
> > Hi Sakari, everyone..
> >
> > On Wed, Jan 02, 2019 at 10:26:56PM +0200, Sakari Ailus wrote:
> > > Hi Laurent,
> > >
> > > On Wed, Jan 02, 2019 at 10:20:13AM +0200, Laurent Pinchart wrote:
> > > > Hello Bingbu,
> > > >
> > > > On Wednesday, 2 January 2019 04:38:33 EET Bingbu Cao wrote:
> > > > > On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > > > > > On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> > > > > >> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> > > > > >>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> > > > > >>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> > > > > >>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > > > >>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > > > >>>>>>
> > > > > >>>>>> [snip]
> > > > > >>>>>>
> > > > > >>>>>>> I can see a couple of steps missing in the script below.
> > > > > >>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November
> > > > > >>>>>>> /000040.html)
> > > > > >>>>>>>
> > > > > >>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > > > >>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> > > > > >>>>>>> image processing"...
> > > > > >>>>>>>
> > > > > >>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > > > >>>>>>>
> > > > > >>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > > > >>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the
> > > > > >>>>>>> control id 0x009819a1 as below.
> > > > > >>>>>>>
> > > > > >>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > > > >>>>>>
> > > > > >>>>>> I assume the control takes a valid default value ? It's better to set
> > > > > >>>>>> it explicitly anyway, so I'll do so.
> > > > > >>>>
> > > > > >>>> The video mode is set by default. If you want to set to still mode or
> > > > > >>>> change mode, you need set the subdev control.
> > > > > >>>>
> > > > > >>>>>>> 2. ImgU pipeline needs to be configured for image processing as
> > > > > >>>>>>> below.
> > > > > >>>>>>>
> > > > > >>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > > > >>>>>>> have the processed image output to the DDR memory.
> > > > > >>>>>>>
> > > > > >>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > > > >>>>>>> Geometric Distortion Correction (GDC) -> DDR
> > > > > >>>>>>>
> > > > > >>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> > > > > >>>>>>> resolutions in all the above HW blocks, for a given input
> > > > > >>>>>>> resolution.
> > > > > >>>>>>>
> > > > > >>>>>>> For a given supported resolution for an input frame, the Input
> > > > > >>>>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured with
> > > > > >>>>>>> the supported resolutions. This information can be obtained by
> > > > > >>>>>>> looking at the following IPU3 ISP configuration table for ov5670
> > > > > >>>>>>> sensor.
> > > > > >>>>>>>
> > > > > >>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays
> > > > > >>>>>>> /+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> > > > > >>>>>>> files/gcss/graph_settings_ov5670.xml
> > > > > >>>>>>>
> > > > > >>>>>>> For the ov5670 example, for an input frame with a resolution of
> > > > > >>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> > > > > >>>>>>> corresponding resolutions for input feeder, BDS and GDC are
> > > > > >>>>>>> 2592x1944, 2592x1944 and 2560x1920 respectively.
> > > > > >>>>>>
> > > > > >>>>>> How is the GDC output resolution computed from the input resolution ?
> > > > > >>>>>> Does the GDC always consume 32 columns and 22 lines ?
> > > > > >>>>
> > > > > >>>> All the intermediate resolutions in the pipeline are determined by the
> > > > > >>>> actual use case, in other word determined by the IMGU input
> > > > > >>>> resolution(sensor output) and the final output and viewfinder
> > > > > >>>> resolution. BDS mainly do Bayer downscaling, it has limitation that the
> > > > > >>>> downscaling factor must be a value a integer multiple of 1/32.
> > > > > >>>> GDC output depends on the input and width should be x8 and height x4
> > > > > >>>> alignment.
> > > > > >>>
> > > > > >>> Thank you for the information. This will need to be captured in the
> > > > > >>> documentation, along with information related to how each block in the
> > > > > >>> hardware pipeline interacts with the image size. It should be possible
> > > > > >>> for a developer to compute the output and viewfinder resolutions based
> > > > > >>> on the parameters of the image processing algorithms just with the
> > > > > >>> information contained in the driver documentation.
> > > > > >>>
> > > > > >>>>>>> The following steps prepare the ImgU ISP pipeline for the image
> > > > > >>>>>>> processing.
> > > > > >>>>>>>
> > > > > >>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > > > >>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> > > > > >>>>>>> obtained
> > > > > >>>>>>> above.
> > > > > >>>>>>
> > > > > >>>>>> If I understand things correctly, the GDC resolution is the pipeline
> > > > > >>>>>> output resolution. Why is it configured on pad 0 ?
> > > > > >>>>
> > > > > >>>> We see the GDC output resolution as the input of output system, the
> > > > > >>>> sink pad format is used for output and viewfinder resolutions.
> > > > > >>>
> > > > > >>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> > > > > >>> the ImgU input, the format configured there should correspond to the
> > > > > >>> format on the connected video node, and should thus be the sensor
> > > > > >>> format. You can then use the crop and compose rectangles on pad 0, along
> > > > > >>> with the format, crop and compose rectangles on the output and
> > > > > >>> viewfinder pads, to configure the device. This should be fixed in the
> > > > > >>> driver, and the documentation should then be updated accordingly.
> > > > > >>
> > > > > >> Hi, Laurent,
> > > > > >>
> > > > > >> Thanks for your review.
> > > > > >>
> > > > > >> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> > > > > >> However, I prefer using the 2 source pads for output and viewfinder.
> > > > > >> It makes more sense because the output and viewfinder are independent
> > > > > >> output.
> > > > > >>
> > > > > >> The whole pipeline in ImgU looks like:
> > > > > >> IF --> BDS --> GDC ---> OUTPUT
> > > > > >> |-----> VF
> > > > > >>
> > > > > >> The BDS is used to do Bayer downscaling and GDC can do cropping.
> > > > > >
> > > > > > Does this mean that the main output and the viewfinder output share the
> > > > > > same scaler, and that the only difference in size between the two outputs
> > > > > > is solely due to cropping ?
> > > > >
> > > > > Laurent,
> > > > > No, output only can do crop and viewfinder support crop and scaling, they
> > > > > share same input.
> > > >
> > > > Then you can't support this with a single subdev for the ImgU, you need at
> > > > least two subdevs. I can offer more guidance, but I'll need more information
> > > > about the GDC.
> > >
> > > While the current documentation only defines the functionality of the
> > > compose target for sink pads, there are a few sensor drivers supporting it
> > > on source pads already. Some drivers such as the OMAP3 ISP also use the
> > > format on source pads to configure scaling.
> > >
> > > The current API certainly allows exposing the compose rectangle also on the
> > > source pads, but to make that generic we'd need to amend the API to tell in
> > > which order these steps take place. In the meantime the behaviour remains
> > > device specific.
> > >
> >
> > My understanding is that what is currently missing is the support
> > for viewfinder's ability to scale, as the scaler should get
> > programmed by configuring a composing rectangle on a source pad which
> > is not supported by the V4L2 APIs at the moment. Is my understanding correct?
> >
> > As the composing rectangle is set for both 'output' and 'viewfinder'
> > through the image format sizes configured on the first sink pad (*),
> > the viewfinder output is obtained by cropping-only to the image format
> > sizes configured on source pad number 3 (though SUBDEV_S_FMT not through
> > SUBDEV_S_SELECTION, as SUBDEV_S_SELECTION is only allowed on sink pad
> > 0 in the driver: see "ipu3_subdev_set_selection()").
> >
> > As you mentioned "device specific behaviour", what is the intended one
> > for the ipu3? I assumed the viewfinder scaling/cropping was configured
> > on the 'viewfinder' video device node, through the VIDIOC_S_SELECTION
> > ioctl, but looking at the code, that doesn't seem to be listed as
> > supported in "ipu3_v4l2_ioctl_ops".
> >
> > How am I supposed to configure scaling on the viewfinder output? Would
> > adding support for crop/compose to the 'output' and 'viewfinder' video
> > devices be supported by the V4L2 APIs? That would work with the single
> > subdevice model that is currently implemented in this patches...
>
> Isn't this what the driver actually implements? My understanding was
> that the format on the VF video node determined the scaling settings,
> based on the source pad.
>
I might surely be wrong, but VIDIOC_S_SELECTION seems not to be
supported in the "ipu3_v4l2_ioctl_ops" list.
So we can just confiure cropping sizes on the source pad number 3 of
the 'imgu' entity, and set the same sizes on the video device node
that represents the VF output.
Inspecting the images, and reading again the long mail thread, it
seems to me that the imgu perform composing internally, based on the
GDC output dimensions and the required VF ones, to maintain the field
of view.
I cannot find any link to post here to Bing Bu's reply to Sakari on
this, but seems like this discussion happened already, and my
understanding is that composing on VF cannot be set from user, but
calculated by the imgu internally. Sakari, Bing Bu, is that the
current state?
Thanks
j
------------------------------------------------------------------------------
Pasting down here the relevan bits from Bing Bu's and Sakari's
exchange from "Wed, 14 Nov 2018 15:02:37 +0800" which I cannot find a
suitable link to to paste here:
>>>>>>>> pad3: Source
>>>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
>>>>>>>> -> "ipu3-imgu 0 viewfinder":0 []
>>>>>>> Are there other differences between output and viewfinder?
>>>>>> output and viewfinder are the main and secondary output of output system.
>>>>>> 'main' output is not allowed to be scaled, only support crop. secondary
>>>>>> output 'viewfinder'
>>>>>> can support both cropping and scaling. User can select different nodes
>>>>>> to use
>>>>>> as preview and capture flexibly based on the actual use cases.
>>>>> If there's scaling to be configured, I'd expect to see the COMPOSE target
>>>>> supported.
>>>> Actually the viewfinder is the result of scaling, that means you can not
>>>> do more scaling.
>>> How do you configure the scaling of the viewfinder currently?
>> We consider that the viewfinder as a secondary output, and set the format by
>> subdev set_fmt() directly and all pads formats will be used to find
>> binary and
>> build pipeline.
> Ok.
>
> Could you instead use the compose target to configure the scaling? Setting
> the format on the source pad would have no effect.
Hi, Sakari,
For the secondary output (viewfinder), it support both cropping and
scaling, in order
to keep the aspect ratio, system will do [crop --> compose], sounds like
it should
have 2 selection targets, but firmware/driver did not provide clear
interfaces to allow
user to configure the cropping, driver calculate the scaler and cropping
parameters
based on the output-system input and output resolution to keep aspect ratio
and field of view. I think it is more succinct to set the actual output
by set format
instead of using selection targets.
------------------------------------------------------------------------------
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-02-01 10:04 ` Jacopo Mondi
@ 2019-02-05 6:01 ` Tomasz Figa
0 siblings, 0 replies; 123+ messages in thread
From: Tomasz Figa @ 2019-02-05 6:01 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Sakari Ailus, Laurent Pinchart, Bingbu Cao, Mani, Rajmohan, Zhi,
Yong, Linux Media Mailing List, Mauro Carvalho Chehab,
Hans Verkuil, Zheng, Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu,
Tian Shu, Cao, Bingbu, libcamera-devel
Hi Jacopo,
On Fri, Feb 1, 2019 at 7:04 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Hi Tomasz,
>
> On Tue, Jan 29, 2019 at 05:56:35PM +0900, Tomasz Figa wrote:
> > On Mon, Jan 28, 2019 at 7:08 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> > >
> > > Hi Sakari, everyone..
> > >
> > > On Wed, Jan 02, 2019 at 10:26:56PM +0200, Sakari Ailus wrote:
> > > > Hi Laurent,
> > > >
> > > > On Wed, Jan 02, 2019 at 10:20:13AM +0200, Laurent Pinchart wrote:
> > > > > Hello Bingbu,
> > > > >
> > > > > On Wednesday, 2 January 2019 04:38:33 EET Bingbu Cao wrote:
> > > > > > On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > > > > > > On Monday, 17 December 2018 05:14:44 EET Bingbu Cao wrote:
> > > > > > >> On 12/14/2018 06:24 AM, Laurent Pinchart wrote:
> > > > > > >>> On Wednesday, 12 December 2018 06:55:53 EET Bingbu Cao wrote:
> > > > > > >>>> On 12/11/2018 09:43 PM, Laurent Pinchart wrote:
> > > > > > >>>>> On Tuesday, 11 December 2018 15:34:49 EET Laurent Pinchart wrote:
> > > > > > >>>>>> On Wednesday, 5 December 2018 02:30:46 EET Mani, Rajmohan wrote:
> > > > > > >>>>>>
> > > > > > >>>>>> [snip]
> > > > > > >>>>>>
> > > > > > >>>>>>> I can see a couple of steps missing in the script below.
> > > > > > >>>>>>> (https://lists.libcamera.org/pipermail/libcamera-devel/2018-November
> > > > > > >>>>>>> /000040.html)
> > > > > > >>>>>>>
> > > > > > >>>>>>> From patch 02 of this v7 series "doc-rst: Add Intel IPU3
> > > > > > >>>>>>> documentation", under section "Configuring ImgU V4L2 subdev for
> > > > > > >>>>>>> image processing"...
> > > > > > >>>>>>>
> > > > > > >>>>>>> 1. The pipe mode needs to be configured for the V4L2 subdev.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Also the pipe mode of the corresponding V4L2 subdev should be set as
> > > > > > >>>>>>> desired (e.g 0 for video mode or 1 for still mode) through the
> > > > > > >>>>>>> control id 0x009819a1 as below.
> > > > > > >>>>>>>
> > > > > > >>>>>>> e.g v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
> > > > > > >>>>>>
> > > > > > >>>>>> I assume the control takes a valid default value ? It's better to set
> > > > > > >>>>>> it explicitly anyway, so I'll do so.
> > > > > > >>>>
> > > > > > >>>> The video mode is set by default. If you want to set to still mode or
> > > > > > >>>> change mode, you need set the subdev control.
> > > > > > >>>>
> > > > > > >>>>>>> 2. ImgU pipeline needs to be configured for image processing as
> > > > > > >>>>>>> below.
> > > > > > >>>>>>>
> > > > > > >>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> > > > > > >>>>>>> have the processed image output to the DDR memory.
> > > > > > >>>>>>>
> > > > > > >>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > > > > >>>>>>> Geometric Distortion Correction (GDC) -> DDR
> > > > > > >>>>>>>
> > > > > > >>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> > > > > > >>>>>>> resolutions in all the above HW blocks, for a given input
> > > > > > >>>>>>> resolution.
> > > > > > >>>>>>>
> > > > > > >>>>>>> For a given supported resolution for an input frame, the Input
> > > > > > >>>>>>> Feeder, Bayer Down Scaling and GDC blocks should be configured with
> > > > > > >>>>>>> the supported resolutions. This information can be obtained by
> > > > > > >>>>>>> looking at the following IPU3 ISP configuration table for ov5670
> > > > > > >>>>>>> sensor.
> > > > > > >>>>>>>
> > > > > > >>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays
> > > > > > >>>>>>> /+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> > > > > > >>>>>>> files/gcss/graph_settings_ov5670.xml
> > > > > > >>>>>>>
> > > > > > >>>>>>> For the ov5670 example, for an input frame with a resolution of
> > > > > > >>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the
> > > > > > >>>>>>> corresponding resolutions for input feeder, BDS and GDC are
> > > > > > >>>>>>> 2592x1944, 2592x1944 and 2560x1920 respectively.
> > > > > > >>>>>>
> > > > > > >>>>>> How is the GDC output resolution computed from the input resolution ?
> > > > > > >>>>>> Does the GDC always consume 32 columns and 22 lines ?
> > > > > > >>>>
> > > > > > >>>> All the intermediate resolutions in the pipeline are determined by the
> > > > > > >>>> actual use case, in other word determined by the IMGU input
> > > > > > >>>> resolution(sensor output) and the final output and viewfinder
> > > > > > >>>> resolution. BDS mainly do Bayer downscaling, it has limitation that the
> > > > > > >>>> downscaling factor must be a value a integer multiple of 1/32.
> > > > > > >>>> GDC output depends on the input and width should be x8 and height x4
> > > > > > >>>> alignment.
> > > > > > >>>
> > > > > > >>> Thank you for the information. This will need to be captured in the
> > > > > > >>> documentation, along with information related to how each block in the
> > > > > > >>> hardware pipeline interacts with the image size. It should be possible
> > > > > > >>> for a developer to compute the output and viewfinder resolutions based
> > > > > > >>> on the parameters of the image processing algorithms just with the
> > > > > > >>> information contained in the driver documentation.
> > > > > > >>>
> > > > > > >>>>>>> The following steps prepare the ImgU ISP pipeline for the image
> > > > > > >>>>>>> processing.
> > > > > > >>>>>>>
> > > > > > >>>>>>> 1. The ImgU V4L2 subdev data format should be set by using the
> > > > > > >>>>>>> VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height
> > > > > > >>>>>>> obtained
> > > > > > >>>>>>> above.
> > > > > > >>>>>>
> > > > > > >>>>>> If I understand things correctly, the GDC resolution is the pipeline
> > > > > > >>>>>> output resolution. Why is it configured on pad 0 ?
> > > > > > >>>>
> > > > > > >>>> We see the GDC output resolution as the input of output system, the
> > > > > > >>>> sink pad format is used for output and viewfinder resolutions.
> > > > > > >>>
> > > > > > >>> The ImgU subdev is supposed to represent the ImgU. Pad 0 should thus be
> > > > > > >>> the ImgU input, the format configured there should correspond to the
> > > > > > >>> format on the connected video node, and should thus be the sensor
> > > > > > >>> format. You can then use the crop and compose rectangles on pad 0, along
> > > > > > >>> with the format, crop and compose rectangles on the output and
> > > > > > >>> viewfinder pads, to configure the device. This should be fixed in the
> > > > > > >>> driver, and the documentation should then be updated accordingly.
> > > > > > >>
> > > > > > >> Hi, Laurent,
> > > > > > >>
> > > > > > >> Thanks for your review.
> > > > > > >>
> > > > > > >> I think it make sense for me that using Pad 0 as the ImgU input(IF).
> > > > > > >> However, I prefer using the 2 source pads for output and viewfinder.
> > > > > > >> It makes more sense because the output and viewfinder are independent
> > > > > > >> output.
> > > > > > >>
> > > > > > >> The whole pipeline in ImgU looks like:
> > > > > > >> IF --> BDS --> GDC ---> OUTPUT
> > > > > > >> |-----> VF
> > > > > > >>
> > > > > > >> The BDS is used to do Bayer downscaling and GDC can do cropping.
> > > > > > >
> > > > > > > Does this mean that the main output and the viewfinder output share the
> > > > > > > same scaler, and that the only difference in size between the two outputs
> > > > > > > is solely due to cropping ?
> > > > > >
> > > > > > Laurent,
> > > > > > No, output only can do crop and viewfinder support crop and scaling, they
> > > > > > share same input.
> > > > >
> > > > > Then you can't support this with a single subdev for the ImgU, you need at
> > > > > least two subdevs. I can offer more guidance, but I'll need more information
> > > > > about the GDC.
> > > >
> > > > While the current documentation only defines the functionality of the
> > > > compose target for sink pads, there are a few sensor drivers supporting it
> > > > on source pads already. Some drivers such as the OMAP3 ISP also use the
> > > > format on source pads to configure scaling.
> > > >
> > > > The current API certainly allows exposing the compose rectangle also on the
> > > > source pads, but to make that generic we'd need to amend the API to tell in
> > > > which order these steps take place. In the meantime the behaviour remains
> > > > device specific.
> > > >
> > >
> > > My understanding is that what is currently missing is the support
> > > for viewfinder's ability to scale, as the scaler should get
> > > programmed by configuring a composing rectangle on a source pad which
> > > is not supported by the V4L2 APIs at the moment. Is my understanding correct?
> > >
> > > As the composing rectangle is set for both 'output' and 'viewfinder'
> > > through the image format sizes configured on the first sink pad (*),
> > > the viewfinder output is obtained by cropping-only to the image format
> > > sizes configured on source pad number 3 (though SUBDEV_S_FMT not through
> > > SUBDEV_S_SELECTION, as SUBDEV_S_SELECTION is only allowed on sink pad
> > > 0 in the driver: see "ipu3_subdev_set_selection()").
> > >
> > > As you mentioned "device specific behaviour", what is the intended one
> > > for the ipu3? I assumed the viewfinder scaling/cropping was configured
> > > on the 'viewfinder' video device node, through the VIDIOC_S_SELECTION
> > > ioctl, but looking at the code, that doesn't seem to be listed as
> > > supported in "ipu3_v4l2_ioctl_ops".
> > >
> > > How am I supposed to configure scaling on the viewfinder output? Would
> > > adding support for crop/compose to the 'output' and 'viewfinder' video
> > > devices be supported by the V4L2 APIs? That would work with the single
> > > subdevice model that is currently implemented in this patches...
> >
> > Isn't this what the driver actually implements? My understanding was
> > that the format on the VF video node determined the scaling settings,
> > based on the source pad.
> >
>
> I might surely be wrong, but VIDIOC_S_SELECTION seems not to be
> supported in the "ipu3_v4l2_ioctl_ops" list.
>
VIDIOC_S_SELECTION is not implemented, but VIDIOC_S_FMT is. I might be
missing something, but it doesn't seem to consider pad formats and
anything that comes from the userspace and meets alignment and min/max
constraints seems to be accepted.
What happens if you set a resolution smaller than the pad on the VF
video node? (On OUT note it shouldn't be possible to do so, though...)
> So we can just confiure cropping sizes on the source pad number 3 of
> the 'imgu' entity, and set the same sizes on the video device node
> that represents the VF output.
>
> Inspecting the images, and reading again the long mail thread, it
> seems to me that the imgu perform composing internally, based on the
> GDC output dimensions and the required VF ones, to maintain the field
> of view.
>
Yeah, I think we may be missing the separate cropping control between
VF and OUT, but I'm not sure if there is any real use case which would
need it, since one would normally want to have the preview match the
full frame, but possibly scaled down.
> I cannot find any link to post here to Bing Bu's reply to Sakari on
> this, but seems like this discussion happened already, and my
> understanding is that composing on VF cannot be set from user, but
> calculated by the imgu internally. Sakari, Bing Bu, is that the
> current state?
>
> Thanks
> j
>
> ------------------------------------------------------------------------------
> Pasting down here the relevan bits from Bing Bu's and Sakari's
> exchange from "Wed, 14 Nov 2018 15:02:37 +0800" which I cannot find a
> suitable link to to paste here:
>
> >>>>>>>> pad3: Source
> >>>>>>>> [fmt:UYVY8_2X8/1920x1080 field:none colorspace:unknown]
> >>>>>>>> -> "ipu3-imgu 0 viewfinder":0 []
> >>>>>>> Are there other differences between output and viewfinder?
> >>>>>> output and viewfinder are the main and secondary output of output system.
> >>>>>> 'main' output is not allowed to be scaled, only support crop. secondary
> >>>>>> output 'viewfinder'
> >>>>>> can support both cropping and scaling. User can select different nodes
> >>>>>> to use
> >>>>>> as preview and capture flexibly based on the actual use cases.
> >>>>> If there's scaling to be configured, I'd expect to see the COMPOSE target
> >>>>> supported.
> >>>> Actually the viewfinder is the result of scaling, that means you can not
> >>>> do more scaling.
> >>> How do you configure the scaling of the viewfinder currently?
> >> We consider that the viewfinder as a secondary output, and set the format by
> >> subdev set_fmt() directly and all pads formats will be used to find
> >> binary and
> >> build pipeline.
> > Ok.
> >
> > Could you instead use the compose target to configure the scaling? Setting
> > the format on the source pad would have no effect.
> Hi, Sakari,
>
> For the secondary output (viewfinder), it support both cropping and
> scaling, in order
> to keep the aspect ratio, system will do [crop --> compose], sounds like
> it should
> have 2 selection targets, but firmware/driver did not provide clear
> interfaces to allow
> user to configure the cropping, driver calculate the scaler and cropping
> parameters
> based on the output-system input and output resolution to keep aspect ratio
> and field of view. I think it is more succinct to set the actual output
> by set format
> instead of using selection targets.
> ------------------------------------------------------------------------------
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-01-02 2:38 ` Bingbu Cao
2019-01-02 8:20 ` Laurent Pinchart
@ 2019-03-23 13:02 ` Jacopo Mondi
2019-03-25 3:45 ` Bingbu Cao
1 sibling, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-03-23 13:02 UTC (permalink / raw)
To: Bingbu Cao, Laurent Pinchart, Tomasz Figa
Cc: Mani, Rajmohan, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Zheng,
Jian Xu, Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao,
Bingbu, libcamera-devel
[-- Attachment #1: Type: text/plain, Size: 5730 bytes --]
Hello,
sorry for resurrecting the thread.
The development of libcamera has IPU3 devices as first target, and
we're now at the point where some clarifications are required.
I'll re-use this email, as some of points were already stated here
and, as they've become pressing for libcamera development, I would like
to have them clarified here on the list.
On Wed, Jan 02, 2019 at 10:38:33AM +0800, Bingbu Cao wrote:
>
> On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> > Hello Bingbu,
> >
[snip]
> > > > > > > > 2. ImgU pipeline needs to be configured for image processing as below.
> > > > > > > >
> > > > > > > > RAW bayer frames go through the following ISP pipeline HW blocks to
> > > > > > > > have the processed image output to the DDR memory.
> > > > > > > >
> > > > > > > > RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> > > > > > > > Geometric Distortion Correction (GDC) -> DDR
> > > > > > > >
> > > > > > > > The ImgU V4L2 subdev has to be configured with the supported
> > > > > > > > resolutions in all the above HW blocks, for a given input resolution.
> > > > > > > >
> > > > > > > > For a given supported resolution for an input frame, the Input Feeder,
> > > > > > > > Bayer Down Scaling and GDC blocks should be configured with the
> > > > > > > > supported resolutions. This information can be obtained by looking at
> > > > > > > > the following IPU3 ISP configuration table for ov5670 sensor.
> > > > > > > >
> > > > > > > > https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> > > > > > > > /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> > > > > > > > files/gcss/graph_settings_ov5670.xml
> > > > > > > >
> > > > > > > > For the ov5670 example, for an input frame with a resolution of
> > > > > > > > 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> > > > > > > > resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> > > > > > > > 2560x1920 respectively.
> > > > > > > How is the GDC output resolution computed from the input resolution ?
> > > > > > > Does the GDC always consume 32 columns and 22 lines ?
> > > > > All the intermediate resolutions in the pipeline are determined by the
> > > > > actual use case, in other word determined by the IMGU input
> > > > > resolution(sensor output) and the final output and viewfinder resolution.
> > > > > BDS mainly do Bayer downscaling, it has limitation that the downscaling
> > > > > factor must be a value a integer multiple of 1/32.
> > > > > GDC output depends on the input and width should be x8 and height x4
> > > > > alignment.
> > > > Thank you for the information. This will need to be captured in the
> > > > documentation, along with information related to how each block in the
> > > > hardware pipeline interacts with the image size. It should be possible for
> > > > a developer to compute the output and viewfinder resolutions based on the
> > > > parameters of the image processing algorithms just with the information
> > > > contained in the driver documentation.
In libcamera development we're now at the point of having to calculate
the sizes to apply to all intermediate pipeline stages based on the
following informations:
1) Main output resolution
2) Secondary output resolution (optional)
3) Image sensor's available resolutions
Right now that informations are captured in the xml file you linked
here above, but we need a programmatic way to do the calculation,
without going through an XML file, that refers to two specific sensors
only.
As Laurent said here, this should come as part of the documentation
for driver users and would unblock libcamera IPU3 support
development.
Could you provide documentation on how to calculate each
intermediate step resolutions?
> > > >
[snip]
> > > > > > > > 3. The ImgU V4L2 subdev composing should be set by using the
> > > > > > > > VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > > > > > > > target, using the BDS height and width.
> > > > > > > >
> > > > > > > > Once these 2 steps are done, the raw bayer frames can be input to the
> > > > > > > > ImgU V4L2 subdev for processing.
> > > > > > > Do I need to capture from both the output and viewfinder nodes ? How
> > > > > > > are they related to the IF -> BDS -> GDC pipeline, are they both fed
> > > > > > > from the GDC output ? If so, how does the viewfinder scaler fit in that
> > > > > > > picture ?
> > > > > The output capture should be set, the viewfinder can be disabled.
> > > > > The IF and BDS are seen as crop and compose of the imgu input video
> > > > > device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> > > > > pads.
This is another point that we would like to have clarified:
1) which outputs are mandatory and which one are not
2) which operations are mandatory on un-used outputs
3) does the 'ipu_pipe_mode' control impact this
As you mentioned here, "output" seems to be mandatory, while
"viewfinder" and "stat" are optional. We have tried using the "output"
video node only but the system hangs to an un-recoverable state.
What I have noticed is instead that the viewfinder and stat nodes
needs to be:
1) Linked to the respective "ImgU" subdevice pads
2) Format configured
3) Memory reserved
4) video device nodes started
It it not required to queue/dequeue buffers from viewfinder and stat,
but steps 1-4 have to be performed.
Can you confirm this is intended?
Could you please list all the steps that have to be applied to the
ImgU's capture video nodes, and which ones are mandatory and which ones
are optional, for the following use cases:
1) Main output capture only
2) Main + secondary output capture
3) Secondary capture only.
Thanks
j
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-03-23 13:02 ` Jacopo Mondi
@ 2019-03-25 3:45 ` Bingbu Cao
2019-03-25 4:06 ` Laurent Pinchart
0 siblings, 1 reply; 123+ messages in thread
From: Bingbu Cao @ 2019-03-25 3:45 UTC (permalink / raw)
To: Jacopo Mondi, Laurent Pinchart, Tomasz Figa
Cc: Mani, Rajmohan, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Qiu, Tian Shu,
Hu, Jerry W, Toivonen, Tuukka, Qiu, Tian Shu, Cao, Bingbu,
libcamera-devel
On 3/23/19 9:02 PM, Jacopo Mondi wrote:
> Hello,
> sorry for resurrecting the thread.
>
> The development of libcamera has IPU3 devices as first target, and
> we're now at the point where some clarifications are required.
> I'll re-use this email, as some of points were already stated here
> and, as they've become pressing for libcamera development, I would like
> to have them clarified here on the list.
>
> On Wed, Jan 02, 2019 at 10:38:33AM +0800, Bingbu Cao wrote:
>>
>> On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
>>> Hello Bingbu,
>>>
>
> [snip]
>
>>>>>>>>> 2. ImgU pipeline needs to be configured for image processing as below.
>>>>>>>>>
>>>>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
>>>>>>>>> have the processed image output to the DDR memory.
>>>>>>>>>
>>>>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
>>>>>>>>> Geometric Distortion Correction (GDC) -> DDR
>>>>>>>>>
>>>>>>>>> The ImgU V4L2 subdev has to be configured with the supported
>>>>>>>>> resolutions in all the above HW blocks, for a given input resolution.
>>>>>>>>>
>>>>>>>>> For a given supported resolution for an input frame, the Input Feeder,
>>>>>>>>> Bayer Down Scaling and GDC blocks should be configured with the
>>>>>>>>> supported resolutions. This information can be obtained by looking at
>>>>>>>>> the following IPU3 ISP configuration table for ov5670 sensor.
>>>>>>>>>
>>>>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
>>>>>>>>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
>>>>>>>>> files/gcss/graph_settings_ov5670.xml
>>>>>>>>>
>>>>>>>>> For the ov5670 example, for an input frame with a resolution of
>>>>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
>>>>>>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
>>>>>>>>> 2560x1920 respectively.
>>>>>>>> How is the GDC output resolution computed from the input resolution ?
>>>>>>>> Does the GDC always consume 32 columns and 22 lines ?
>>>>>> All the intermediate resolutions in the pipeline are determined by the
>>>>>> actual use case, in other word determined by the IMGU input
>>>>>> resolution(sensor output) and the final output and viewfinder resolution.
>>>>>> BDS mainly do Bayer downscaling, it has limitation that the downscaling
>>>>>> factor must be a value a integer multiple of 1/32.
>>>>>> GDC output depends on the input and width should be x8 and height x4
>>>>>> alignment.
>>>>> Thank you for the information. This will need to be captured in the
>>>>> documentation, along with information related to how each block in the
>>>>> hardware pipeline interacts with the image size. It should be possible for
>>>>> a developer to compute the output and viewfinder resolutions based on the
>>>>> parameters of the image processing algorithms just with the information
>>>>> contained in the driver documentation.
>
> In libcamera development we're now at the point of having to calculate
> the sizes to apply to all intermediate pipeline stages based on the
> following informations:
>
> 1) Main output resolution
> 2) Secondary output resolution (optional)
> 3) Image sensor's available resolutions
>
> Right now that informations are captured in the xml file you linked
> here above, but we need a programmatic way to do the calculation,
> without going through an XML file, that refers to two specific sensors
> only.
>
> As Laurent said here, this should come as part of the documentation
> for driver users and would unblock libcamera IPU3 support
> development.
>
> Could you provide documentation on how to calculate each
> intermediate step resolutions?
All the intermediate step resolutions are generated by the specific tool
with sensor input and outputs resolutions.
The tool try to keep maximum fov and has the knowledge of all the
limitations of each intermediate hardware components(mainly BDS and GDC).
Currently, there is not a very simple calculation to get the
intermediate resolutions.
Let's take some effort to try find a programmatic way to do calculation
instead of the tool.
>
>>>>>
>
> [snip]
>
>>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>>>>>> target, using the BDS height and width.
>>>>>>>>>
>>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>>>>>> ImgU V4L2 subdev for processing.
>>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
>>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
>>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
>>>>>>>> picture ?
>>>>>> The output capture should be set, the viewfinder can be disabled.
>>>>>> The IF and BDS are seen as crop and compose of the imgu input video
>>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>>>>>> pads.
>
> This is another point that we would like to have clarified:
> 1) which outputs are mandatory and which one are not
> 2) which operations are mandatory on un-used outputs
> 3) does the 'ipu_pipe_mode' control impact this
>
> As you mentioned here, "output" seems to be mandatory, while
> "viewfinder" and "stat" are optional. We have tried using the "output"
> video node only but the system hangs to an un-recoverable state.
Yes, main output is mandatory, 'vf' and 'stat' are optional.
>
> What I have noticed is instead that the viewfinder and stat nodes
> needs to be:
> 1) Linked to the respective "ImgU" subdevice pads
> 2) Format configured
> 3) Memory reserved
> 4) video device nodes started
>
> It it not required to queue/dequeue buffers from viewfinder and stat,
> but steps 1-4 have to be performed.
>
> Can you confirm this is intended?
viewfinder and stats are enabled when the link for respective subdev
pads enabled, and then driver can use these input conditions to find the
binary to run.
> Could you please list all the steps that have to be applied to the
> ImgU's capture video nodes, and which ones are mandatory and which ones
> are optional, for the following use cases:
> 1) Main output capture only
> 2) Main + secondary output capture
> 3) Secondary capture only.
I think the 3) is not supported.
The steps are:
1). link necessary the respective subdevices
input --> imgu -->output
| -->vf
| -->3a stats
2). set all the formats for input, output and intermediate resolutions.
3). start stream
The ipu pipe_mode will not impact the whole pipe behavior. It just ask
firmware to run different processing to generate same format outputs.
>
> Thanks
> j
>
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-03-25 3:45 ` Bingbu Cao
@ 2019-03-25 4:06 ` Laurent Pinchart
2019-03-25 8:11 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Laurent Pinchart @ 2019-03-25 4:06 UTC (permalink / raw)
To: Bingbu Cao
Cc: Jacopo Mondi, Tomasz Figa, Mani, Rajmohan, Zhi, Yong,
Linux Media Mailing List, Sakari Ailus, Mauro Carvalho Chehab,
Hans Verkuil, Qiu, Tian Shu, Hu, Jerry W, Toivonen, Tuukka, Cao,
Bingbu, libcamera-devel
Hi Bingbu,
On Mon, Mar 25, 2019 at 11:45:58AM +0800, Bingbu Cao wrote:
> On 3/23/19 9:02 PM, Jacopo Mondi wrote:
> > Hello,
> > sorry for resurrecting the thread.
> >
> > The development of libcamera has IPU3 devices as first target, and
> > we're now at the point where some clarifications are required.
> > I'll re-use this email, as some of points were already stated here
> > and, as they've become pressing for libcamera development, I would like
> > to have them clarified here on the list.
> >
> > On Wed, Jan 02, 2019 at 10:38:33AM +0800, Bingbu Cao wrote:
> >> On 12/26/2018 07:03 PM, Laurent Pinchart wrote:
> >>> Hello Bingbu,
> >>>
> >
> > [snip]
> >
> >>>>>>>>> 2. ImgU pipeline needs to be configured for image processing as below.
> >>>>>>>>>
> >>>>>>>>> RAW bayer frames go through the following ISP pipeline HW blocks to
> >>>>>>>>> have the processed image output to the DDR memory.
> >>>>>>>>>
> >>>>>>>>> RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) ->
> >>>>>>>>> Geometric Distortion Correction (GDC) -> DDR
> >>>>>>>>>
> >>>>>>>>> The ImgU V4L2 subdev has to be configured with the supported
> >>>>>>>>> resolutions in all the above HW blocks, for a given input resolution.
> >>>>>>>>>
> >>>>>>>>> For a given supported resolution for an input frame, the Input Feeder,
> >>>>>>>>> Bayer Down Scaling and GDC blocks should be configured with the
> >>>>>>>>> supported resolutions. This information can be obtained by looking at
> >>>>>>>>> the following IPU3 ISP configuration table for ov5670 sensor.
> >>>>>>>>>
> >>>>>>>>> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+
> >>>>>>>>> /master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/
> >>>>>>>>> files/gcss/graph_settings_ov5670.xml
> >>>>>>>>>
> >>>>>>>>> For the ov5670 example, for an input frame with a resolution of
> >>>>>>>>> 2592x1944 (which is input to the ImgU subdev pad 0), the corresponding
> >>>>>>>>> resolutions for input feeder, BDS and GDC are 2592x1944, 2592x1944 and
> >>>>>>>>> 2560x1920 respectively.
> >>>>>>>>
> >>>>>>>> How is the GDC output resolution computed from the input resolution ?
> >>>>>>>> Does the GDC always consume 32 columns and 22 lines ?
> >>>>>>
> >>>>>> All the intermediate resolutions in the pipeline are determined by the
> >>>>>> actual use case, in other word determined by the IMGU input
> >>>>>> resolution(sensor output) and the final output and viewfinder resolution.
> >>>>>> BDS mainly do Bayer downscaling, it has limitation that the downscaling
> >>>>>> factor must be a value a integer multiple of 1/32.
> >>>>>> GDC output depends on the input and width should be x8 and height x4
> >>>>>> alignment.
> >>>>>
> >>>>> Thank you for the information. This will need to be captured in the
> >>>>> documentation, along with information related to how each block in the
> >>>>> hardware pipeline interacts with the image size. It should be possible for
> >>>>> a developer to compute the output and viewfinder resolutions based on the
> >>>>> parameters of the image processing algorithms just with the information
> >>>>> contained in the driver documentation.
> >
> > In libcamera development we're now at the point of having to calculate
> > the sizes to apply to all intermediate pipeline stages based on the
> > following informations:
> >
> > 1) Main output resolution
> > 2) Secondary output resolution (optional)
> > 3) Image sensor's available resolutions
> >
> > Right now that informations are captured in the xml file you linked
> > here above, but we need a programmatic way to do the calculation,
> > without going through an XML file, that refers to two specific sensors
> > only.
> >
> > As Laurent said here, this should come as part of the documentation
> > for driver users and would unblock libcamera IPU3 support
> > development.
> >
> > Could you provide documentation on how to calculate each
> > intermediate step resolutions?
>
> All the intermediate step resolutions are generated by the specific tool
> with sensor input and outputs resolutions.
>
> The tool try to keep maximum fov and has the knowledge of all the
> limitations of each intermediate hardware components(mainly BDS and GDC).
That's exactly what we want to do in software in libcamera :-) And
that's why we need more infirmation about the limitations of each
intermediate hardware component. Eventually those limitations should be
documented in the IPU3 driver documentation in the kernel sources, but
for now we can move forward if they're just communicated by e-mail (if
time permits we may be able to submit a kernel patch to integrate that
in the documentation).
> Currently, there is not a very simple calculation to get the
> intermediate resolutions.
> Let's take some effort to try find a programmatic way to do calculation
> instead of the tool.
>
> > [snip]
> >
> >>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >>>>>>>>> target, using the BDS height and width.
> >>>>>>>>>
> >>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
> >>>>>>>>> ImgU V4L2 subdev for processing.
> >>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
> >>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> >>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
> >>>>>>>> picture ?
> >>>>>> The output capture should be set, the viewfinder can be disabled.
> >>>>>> The IF and BDS are seen as crop and compose of the imgu input video
> >>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> >>>>>> pads.
> >
> > This is another point that we would like to have clarified:
> > 1) which outputs are mandatory and which one are not
> > 2) which operations are mandatory on un-used outputs
> > 3) does the 'ipu_pipe_mode' control impact this
> >
> > As you mentioned here, "output" seems to be mandatory, while
> > "viewfinder" and "stat" are optional. We have tried using the "output"
> > video node only but the system hangs to an un-recoverable state.
>
> Yes, main output is mandatory, 'vf' and 'stat' are optional.
I will let Jacopo confirm this, but unless I'm mistaken, when he tried
to use the main output only (with the links between the ImgU subdev and
the vf and stat video nodes disabled), the driver would hang without
processing any frame. I believe this was a complete system hang,
requiring a hard reboot to recover.
> > What I have noticed is instead that the viewfinder and stat nodes
> > needs to be:
> > 1) Linked to the respective "ImgU" subdevice pads
> > 2) Format configured
> > 3) Memory reserved
> > 4) video device nodes started
> >
> > It it not required to queue/dequeue buffers from viewfinder and stat,
> > but steps 1-4 have to be performed.
> >
> > Can you confirm this is intended?
>
> viewfinder and stats are enabled when the link for respective subdev
> pads enabled, and then driver can use these input conditions to find the
> binary to run.
>
> > Could you please list all the steps that have to be applied to the
> > ImgU's capture video nodes, and which ones are mandatory and which ones
> > are optional, for the following use cases:
> > 1) Main output capture only
> > 2) Main + secondary output capture
> > 3) Secondary capture only.
>
> I think the 3) is not supported.
>
> The steps are:
> 1). link necessary the respective subdevices
> input --> imgu -->output
> | -->vf
> | -->3a stats
>
> 2). set all the formats for input, output and intermediate resolutions.
> 3). start stream
>
> The ipu pipe_mode will not impact the whole pipe behavior. It just ask
> firmware to run different processing to generate same format outputs.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-03-25 4:06 ` Laurent Pinchart
@ 2019-03-25 8:11 ` Jacopo Mondi
2019-03-25 10:07 ` Bingbu Cao
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-03-25 8:11 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Bingbu Cao, Tomasz Figa, Mani, Rajmohan, Zhi, Yong,
Linux Media Mailing List, Sakari Ailus, Mauro Carvalho Chehab,
Hans Verkuil, Qiu, Tian Shu, Hu, Jerry W, Toivonen, Tuukka, Cao,
Bingbu, libcamera-devel
[-- Attachment #1: Type: text/plain, Size: 6460 bytes --]
Hi Laurent, Bingbu
On Mon, Mar 25, 2019 at 06:06:30AM +0200, Laurent Pinchart wrote:
> Hi Bingbu,
>
[snip]
> > >>>>>
> > >>>>> Thank you for the information. This will need to be captured in the
> > >>>>> documentation, along with information related to how each block in the
> > >>>>> hardware pipeline interacts with the image size. It should be possible for
> > >>>>> a developer to compute the output and viewfinder resolutions based on the
> > >>>>> parameters of the image processing algorithms just with the information
> > >>>>> contained in the driver documentation.
> > >
> > > In libcamera development we're now at the point of having to calculate
> > > the sizes to apply to all intermediate pipeline stages based on the
> > > following informations:
> > >
> > > 1) Main output resolution
> > > 2) Secondary output resolution (optional)
> > > 3) Image sensor's available resolutions
> > >
> > > Right now that informations are captured in the xml file you linked
> > > here above, but we need a programmatic way to do the calculation,
> > > without going through an XML file, that refers to two specific sensors
> > > only.
> > >
> > > As Laurent said here, this should come as part of the documentation
> > > for driver users and would unblock libcamera IPU3 support
> > > development.
> > >
> > > Could you provide documentation on how to calculate each
> > > intermediate step resolutions?
> >
> > All the intermediate step resolutions are generated by the specific tool
> > with sensor input and outputs resolutions.
> >
> > The tool try to keep maximum fov and has the knowledge of all the
> > limitations of each intermediate hardware components(mainly BDS and GDC).
>
> That's exactly what we want to do in software in libcamera :-) And
> that's why we need more infirmation about the limitations of each
> intermediate hardware component. Eventually those limitations should be
> documented in the IPU3 driver documentation in the kernel sources, but
> for now we can move forward if they're just communicated by e-mail (if
> time permits we may be able to submit a kernel patch to integrate that
> in the documentation).
>
> > Currently, there is not a very simple calculation to get the
> > intermediate resolutions.
> > Let's take some effort to try find a programmatic way to do calculation
> > instead of the tool.
Thank you for your effort.
> >
> > > [snip]
> > >
> > >>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> > >>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> > >>>>>>>>> target, using the BDS height and width.
> > >>>>>>>>>
> > >>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
> > >>>>>>>>> ImgU V4L2 subdev for processing.
> > >>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
> > >>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> > >>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
> > >>>>>>>> picture ?
> > >>>>>> The output capture should be set, the viewfinder can be disabled.
> > >>>>>> The IF and BDS are seen as crop and compose of the imgu input video
> > >>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> > >>>>>> pads.
> > >
> > > This is another point that we would like to have clarified:
> > > 1) which outputs are mandatory and which one are not
> > > 2) which operations are mandatory on un-used outputs
> > > 3) does the 'ipu_pipe_mode' control impact this
> > >
> > > As you mentioned here, "output" seems to be mandatory, while
> > > "viewfinder" and "stat" are optional. We have tried using the "output"
> > > video node only but the system hangs to an un-recoverable state.
> >
> > Yes, main output is mandatory, 'vf' and 'stat' are optional.
>
> I will let Jacopo confirm this, but unless I'm mistaken, when he tried
> to use the main output only (with the links between the ImgU subdev and
> the vf and stat video nodes disabled), the driver would hang without
> processing any frame. I believe this was a complete system hang,
> requiring a hard reboot to recover.
>
Yes, that's what I have noticed.
On the other hand, if I link, configure, prepare buffers and start the
'vf' and 'stat' nodes, but never queue buffers there, I can capture
from output only.
> > > What I have noticed is instead that the viewfinder and stat nodes
> > > needs to be:
> > > 1) Linked to the respective "ImgU" subdevice pads
> > > 2) Format configured
> > > 3) Memory reserved
> > > 4) video device nodes started
> > >
> > > It it not required to queue/dequeue buffers from viewfinder and stat,
> > > but steps 1-4 have to be performed.
> > >
> > > Can you confirm this is intended?
> >
> > viewfinder and stats are enabled when the link for respective subdev
> > pads enabled, and then driver can use these input conditions to find the
> > binary to run.
> >
As Laurent reported above, if I leave the the 'vf' and 'stat' links
disabled, the system hangs.
> > > Could you please list all the steps that have to be applied to the
> > > ImgU's capture video nodes, and which ones are mandatory and which ones
> > > are optional, for the following use cases:
> > > 1) Main output capture only
> > > 2) Main + secondary output capture
> > > 3) Secondary capture only.
> >
> > I think the 3) is not supported.
> >
> > The steps are:
> > 1). link necessary the respective subdevices
> > input --> imgu -->output
> > | -->vf
> > | -->3a stats
For which use case, in the above reported list?
1) Main output capture only
Does 'vf' and 'stat' links needs to be enabled?
2) Main + secondary output capture
Does 'stat' link need to be enabled?
3) Secondary capture only.
not supported
> >
> > 2). set all the formats for input, output and intermediate resolutions.
> > 3). start stream
> >
> > The ipu pipe_mode will not impact the whole pipe behavior. It just ask
> > firmware to run different processing to generate same format outputs.
>
I would apreciate to have a better description of the pipe_mode
control, in order to better understand when and if the library has to
modify its value and which mode to use (0=video, 1=still_capture).
Thanks
j
> --
> Regards,
>
> Laurent Pinchart
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-03-25 8:11 ` Jacopo Mondi
@ 2019-03-25 10:07 ` Bingbu Cao
2019-03-26 11:16 ` Jacopo Mondi
0 siblings, 1 reply; 123+ messages in thread
From: Bingbu Cao @ 2019-03-25 10:07 UTC (permalink / raw)
To: Jacopo Mondi, Laurent Pinchart
Cc: Tomasz Figa, Mani, Rajmohan, Zhi, Yong, Linux Media Mailing List,
Sakari Ailus, Mauro Carvalho Chehab, Hans Verkuil, Qiu, Tian Shu,
Hu, Jerry W, Toivonen, Tuukka, Cao, Bingbu, libcamera-devel
On 3/25/19 4:11 PM, Jacopo Mondi wrote:
> Hi Laurent, Bingbu
>
> On Mon, Mar 25, 2019 at 06:06:30AM +0200, Laurent Pinchart wrote:
>> Hi Bingbu,
>>
> [snip]
>
>>>>>>>>
>>>>>>>> Thank you for the information. This will need to be captured in the
>>>>>>>> documentation, along with information related to how each block in the
>>>>>>>> hardware pipeline interacts with the image size. It should be possible for
>>>>>>>> a developer to compute the output and viewfinder resolutions based on the
>>>>>>>> parameters of the image processing algorithms just with the information
>>>>>>>> contained in the driver documentation.
>>>>
>>>> In libcamera development we're now at the point of having to calculate
>>>> the sizes to apply to all intermediate pipeline stages based on the
>>>> following informations:
>>>>
>>>> 1) Main output resolution
>>>> 2) Secondary output resolution (optional)
>>>> 3) Image sensor's available resolutions
>>>>
>>>> Right now that informations are captured in the xml file you linked
>>>> here above, but we need a programmatic way to do the calculation,
>>>> without going through an XML file, that refers to two specific sensors
>>>> only.
>>>>
>>>> As Laurent said here, this should come as part of the documentation
>>>> for driver users and would unblock libcamera IPU3 support
>>>> development.
>>>>
>>>> Could you provide documentation on how to calculate each
>>>> intermediate step resolutions?
>>>
>>> All the intermediate step resolutions are generated by the specific tool
>>> with sensor input and outputs resolutions.
>>>
>>> The tool try to keep maximum fov and has the knowledge of all the
>>> limitations of each intermediate hardware components(mainly BDS and GDC).
>>
>> That's exactly what we want to do in software in libcamera :-) And
>> that's why we need more infirmation about the limitations of each
>> intermediate hardware component. Eventually those limitations should be
>> documented in the IPU3 driver documentation in the kernel sources, but
>> for now we can move forward if they're just communicated by e-mail (if
>> time permits we may be able to submit a kernel patch to integrate that
>> in the documentation).
>>
>>> Currently, there is not a very simple calculation to get the
>>> intermediate resolutions.
>>> Let's take some effort to try find a programmatic way to do calculation
>>> instead of the tool.
>
> Thank you for your effort.
>
>>>
>>>> [snip]
>>>>
>>>>>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>>>>>>>>> target, using the BDS height and width.
>>>>>>>>>>>>
>>>>>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>>>>>>>>> ImgU V4L2 subdev for processing.
>>>>>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
>>>>>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
>>>>>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
>>>>>>>>>>> picture ?
>>>>>>>>> The output capture should be set, the viewfinder can be disabled.
>>>>>>>>> The IF and BDS are seen as crop and compose of the imgu input video
>>>>>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>>>>>>>>> pads.
>>>>
>>>> This is another point that we would like to have clarified:
>>>> 1) which outputs are mandatory and which one are not
>>>> 2) which operations are mandatory on un-used outputs
>>>> 3) does the 'ipu_pipe_mode' control impact this
>>>>
>>>> As you mentioned here, "output" seems to be mandatory, while
>>>> "viewfinder" and "stat" are optional. We have tried using the "output"
>>>> video node only but the system hangs to an un-recoverable state.
>>>
>>> Yes, main output is mandatory, 'vf' and 'stat' are optional.
>>
>> I will let Jacopo confirm this, but unless I'm mistaken, when he tried
>> to use the main output only (with the links between the ImgU subdev and
>> the vf and stat video nodes disabled), the driver would hang without
>> processing any frame. I believe this was a complete system hang,
>> requiring a hard reboot to recover.
>>
>
> Yes, that's what I have noticed.
>
> On the other hand, if I link, configure, prepare buffers and start the
> 'vf' and 'stat' nodes, but never queue buffers there, I can capture
> from output only.
>
>>>> What I have noticed is instead that the viewfinder and stat nodes
>>>> needs to be:
>>>> 1) Linked to the respective "ImgU" subdevice pads
>>>> 2) Format configured
>>>> 3) Memory reserved
>>>> 4) video device nodes started
>>>>
>>>> It it not required to queue/dequeue buffers from viewfinder and stat,
>>>> but steps 1-4 have to be performed.
>>>>
>>>> Can you confirm this is intended?
>>>
>>> viewfinder and stats are enabled when the link for respective subdev
>>> pads enabled, and then driver can use these input conditions to find the
>>> binary to run.
>>>
>
> As Laurent reported above, if I leave the the 'vf' and 'stat' links
> disabled, the system hangs.
>
>>>> Could you please list all the steps that have to be applied to the
>>>> ImgU's capture video nodes, and which ones are mandatory and which ones
>>>> are optional, for the following use cases:
>>>> 1) Main output capture only
>>>> 2) Main + secondary output capture
>>>> 3) Secondary capture only.
>>>
>>> I think the 3) is not supported.
>>>
>>> The steps are:
>>> 1). link necessary the respective subdevices
>>> input --> imgu -->output
>>> | -->vf
>>> | -->3a stats
>
> For which use case, in the above reported list?
>
> 1) Main output capture only
> Does 'vf' and 'stat' links needs to be enabled?
>
> 2) Main + secondary output capture
> Does 'stat' link need to be enabled?
>
> 3) Secondary capture only.
> not supported
The list above is a typical use, all outputs enabled, you can setup link
for main output only.
>
>>>
>>> 2). set all the formats for input, output and intermediate resolutions.
>>> 3). start stream
>>>
>>> The ipu pipe_mode will not impact the whole pipe behavior. It just ask
>>> firmware to run different processing to generate same format outputs.
>>
>
> I would apreciate to have a better description of the pipe_mode
> control, in order to better understand when and if the library has to
> modify its value and which mode to use (0=video, 1=still_capture).
In some application, it will request continuous viewfinder, that means
you must keep preview continuous when take capture. That means you can
not switch out pipeline (preview and still mode, back and force), so the
driver need create 2 pipelines to satisfy this usage, 1 video mode pipe
and another is still mode. Both 2 pipes are created at first, run the
pipe as you command. It also can support still during video usage.
>
> Thanks
> j
>
>> --
>> Regards,
>>
>> Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-03-25 10:07 ` Bingbu Cao
@ 2019-03-26 11:16 ` Jacopo Mondi
2019-04-08 6:35 ` Bingbu Cao
0 siblings, 1 reply; 123+ messages in thread
From: Jacopo Mondi @ 2019-03-26 11:16 UTC (permalink / raw)
To: Bingbu Cao
Cc: Laurent Pinchart, Tomasz Figa, Mani, Rajmohan, Zhi, Yong,
Linux Media Mailing List, Sakari Ailus, Mauro Carvalho Chehab,
Hans Verkuil, Qiu, Tian Shu, Hu, Jerry W, Toivonen, Tuukka, Cao,
Bingbu, libcamera-devel
[-- Attachment #1: Type: text/plain, Size: 9373 bytes --]
Hi Bingbu,
On Mon, Mar 25, 2019 at 06:07:58PM +0800, Bingbu Cao wrote:
>
>
> On 3/25/19 4:11 PM, Jacopo Mondi wrote:
> > Hi Laurent, Bingbu
> >
> > On Mon, Mar 25, 2019 at 06:06:30AM +0200, Laurent Pinchart wrote:
> >> Hi Bingbu,
> >>
> > [snip]
> >
> >>>>>>>>
> >>>>>>>> Thank you for the information. This will need to be captured in the
> >>>>>>>> documentation, along with information related to how each block in the
> >>>>>>>> hardware pipeline interacts with the image size. It should be possible for
> >>>>>>>> a developer to compute the output and viewfinder resolutions based on the
> >>>>>>>> parameters of the image processing algorithms just with the information
> >>>>>>>> contained in the driver documentation.
> >>>>
> >>>> In libcamera development we're now at the point of having to calculate
> >>>> the sizes to apply to all intermediate pipeline stages based on the
> >>>> following informations:
> >>>>
> >>>> 1) Main output resolution
> >>>> 2) Secondary output resolution (optional)
> >>>> 3) Image sensor's available resolutions
> >>>>
> >>>> Right now that informations are captured in the xml file you linked
> >>>> here above, but we need a programmatic way to do the calculation,
> >>>> without going through an XML file, that refers to two specific sensors
> >>>> only.
> >>>>
> >>>> As Laurent said here, this should come as part of the documentation
> >>>> for driver users and would unblock libcamera IPU3 support
> >>>> development.
> >>>>
> >>>> Could you provide documentation on how to calculate each
> >>>> intermediate step resolutions?
> >>>
> >>> All the intermediate step resolutions are generated by the specific tool
> >>> with sensor input and outputs resolutions.
> >>>
> >>> The tool try to keep maximum fov and has the knowledge of all the
> >>> limitations of each intermediate hardware components(mainly BDS and GDC).
> >>
> >> That's exactly what we want to do in software in libcamera :-) And
> >> that's why we need more infirmation about the limitations of each
> >> intermediate hardware component. Eventually those limitations should be
> >> documented in the IPU3 driver documentation in the kernel sources, but
> >> for now we can move forward if they're just communicated by e-mail (if
> >> time permits we may be able to submit a kernel patch to integrate that
> >> in the documentation).
> >>
> >>> Currently, there is not a very simple calculation to get the
> >>> intermediate resolutions.
> >>> Let's take some effort to try find a programmatic way to do calculation
> >>> instead of the tool.
> >
> > Thank you for your effort.
> >
> >>>
> >>>> [snip]
> >>>>
> >>>>>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
> >>>>>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
> >>>>>>>>>>>> target, using the BDS height and width.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
> >>>>>>>>>>>> ImgU V4L2 subdev for processing.
> >>>>>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
> >>>>>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
> >>>>>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
> >>>>>>>>>>> picture ?
> >>>>>>>>> The output capture should be set, the viewfinder can be disabled.
> >>>>>>>>> The IF and BDS are seen as crop and compose of the imgu input video
> >>>>>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
> >>>>>>>>> pads.
> >>>>
> >>>> This is another point that we would like to have clarified:
> >>>> 1) which outputs are mandatory and which one are not
> >>>> 2) which operations are mandatory on un-used outputs
> >>>> 3) does the 'ipu_pipe_mode' control impact this
> >>>>
> >>>> As you mentioned here, "output" seems to be mandatory, while
> >>>> "viewfinder" and "stat" are optional. We have tried using the "output"
> >>>> video node only but the system hangs to an un-recoverable state.
> >>>
> >>> Yes, main output is mandatory, 'vf' and 'stat' are optional.
> >>
> >> I will let Jacopo confirm this, but unless I'm mistaken, when he tried
> >> to use the main output only (with the links between the ImgU subdev and
> >> the vf and stat video nodes disabled), the driver would hang without
> >> processing any frame. I believe this was a complete system hang,
> >> requiring a hard reboot to recover.
> >>
> >
> > Yes, that's what I have noticed.
> >
> > On the other hand, if I link, configure, prepare buffers and start the
> > 'vf' and 'stat' nodes, but never queue buffers there, I can capture
> > from output only.
> >
> >>>> What I have noticed is instead that the viewfinder and stat nodes
> >>>> needs to be:
> >>>> 1) Linked to the respective "ImgU" subdevice pads
> >>>> 2) Format configured
> >>>> 3) Memory reserved
> >>>> 4) video device nodes started
> >>>>
> >>>> It it not required to queue/dequeue buffers from viewfinder and stat,
> >>>> but steps 1-4 have to be performed.
> >>>>
> >>>> Can you confirm this is intended?
> >>>
> >>> viewfinder and stats are enabled when the link for respective subdev
> >>> pads enabled, and then driver can use these input conditions to find the
> >>> binary to run.
> >>>
> >
> > As Laurent reported above, if I leave the the 'vf' and 'stat' links
> > disabled, the system hangs.
> >
> >>>> Could you please list all the steps that have to be applied to the
> >>>> ImgU's capture video nodes, and which ones are mandatory and which ones
> >>>> are optional, for the following use cases:
> >>>> 1) Main output capture only
> >>>> 2) Main + secondary output capture
> >>>> 3) Secondary capture only.
> >>>
> >>> I think the 3) is not supported.
> >>>
> >>> The steps are:
> >>> 1). link necessary the respective subdevices
> >>> input --> imgu -->output
> >>> | -->vf
> >>> | -->3a stats
> >
> > For which use case, in the above reported list?
> >
> > 1) Main output capture only
> > Does 'vf' and 'stat' links needs to be enabled?
> >
> > 2) Main + secondary output capture
> > Does 'stat' link need to be enabled?
> >
> > 3) Secondary capture only.
> > not supported
>
> The list above is a typical use, all outputs enabled, you can setup link
> for main output only.
I think we should clarify better what you mean by 'enabled'.
From my testing what I see is that in order to operate the main
output I have to:
- link the stat and vf nodes (as well as input and output of course)
- reserve memory buffers on stat and vf video nodes, even if not used
- set format on all device ndoes
- start the all video devices
If one of these steps is not performed, the ImgU processing stalls and
I need to hard reboot the device to have it operational again.
On the other hand, I see that there is no need to queue any buffer to
any of the output capture devices to have frames processed by the
ImgU.
IF links are setup as explained above, format configured and all the video
device node started, I can queue all the frames I want to the ImgU input
and never queue anything on its outputs, and they will get processed and
returned to userspace nicely from the ImgU input device node. As soon
as I queue a buffer to the ImgU main output video device, I see it
returned filled with the processed data. This is good, as it doesen't
stall the pipeline if there are no capture buffers to queue to the
ImgU output, but now I wonder what you meant with "main output is
mandatory"
> >
> >>>
> >>> 2). set all the formats for input, output and intermediate resolutions.
> >>> 3). start stream
> >>>
> >>> The ipu pipe_mode will not impact the whole pipe behavior. It just ask
> >>> firmware to run different processing to generate same format outputs.
> >>
> >
> > I would apreciate to have a better description of the pipe_mode
> > control, in order to better understand when and if the library has to
> > modify its value and which mode to use (0=video, 1=still_capture).
>
> In some application, it will request continuous viewfinder, that means
> you must keep preview continuous when take capture. That means you can
> not switch out pipeline (preview and still mode, back and force), so the
> driver need create 2 pipelines to satisfy this usage, 1 video mode pipe
> and another is still mode. Both 2 pipes are created at first, run the
> pipe as you command. It also can support still during video usage.
>
Thanks for the explanation, but it is still vague to me and it worries
me a bit as in this typical usage scenario (viewfinder + sporadic
capture) -both- ImgU pipes have to be used, preventing usage of two
cameras at the same time (one camera assigned to one ImgU pipe
instance).
Why would you use both the ImgU pipes in this case? Shouldn't you
always capture from viewfinder (discarding main output frames
if not required), and when requested by the application capture from
both the main and secondary output from the same ImgU pipe? Why would I
need to change the pipe_mode for doing this?
Thank for your patience to answer all this questions :)
> >
> > Thanks
> > j
> >
> >> --
> >> Regards,
> >>
> >> Laurent Pinchart
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 123+ messages in thread
* Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
2019-03-26 11:16 ` Jacopo Mondi
@ 2019-04-08 6:35 ` Bingbu Cao
0 siblings, 0 replies; 123+ messages in thread
From: Bingbu Cao @ 2019-04-08 6:35 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Laurent Pinchart, Tomasz Figa, Mani, Rajmohan, Zhi, Yong,
Linux Media Mailing List, Sakari Ailus, Mauro Carvalho Chehab,
Hans Verkuil, Qiu, Tian Shu, Hu, Jerry W, Toivonen, Tuukka, Cao,
Bingbu, libcamera-devel
On 3/26/19 7:16 PM, Jacopo Mondi wrote:
> Hi Bingbu,
>
> On Mon, Mar 25, 2019 at 06:07:58PM +0800, Bingbu Cao wrote:
>>
>>
>> On 3/25/19 4:11 PM, Jacopo Mondi wrote:
>>> Hi Laurent, Bingbu
>>>
>>> On Mon, Mar 25, 2019 at 06:06:30AM +0200, Laurent Pinchart wrote:
>>>> Hi Bingbu,
>>>>
>>> [snip]
>>>
>>>>>>>>>>
>>>>>>>>>> Thank you for the information. This will need to be captured in the
>>>>>>>>>> documentation, along with information related to how each block in the
>>>>>>>>>> hardware pipeline interacts with the image size. It should be possible for
>>>>>>>>>> a developer to compute the output and viewfinder resolutions based on the
>>>>>>>>>> parameters of the image processing algorithms just with the information
>>>>>>>>>> contained in the driver documentation.
>>>>>>
>>>>>> In libcamera development we're now at the point of having to calculate
>>>>>> the sizes to apply to all intermediate pipeline stages based on the
>>>>>> following informations:
>>>>>>
>>>>>> 1) Main output resolution
>>>>>> 2) Secondary output resolution (optional)
>>>>>> 3) Image sensor's available resolutions
>>>>>>
>>>>>> Right now that informations are captured in the xml file you linked
>>>>>> here above, but we need a programmatic way to do the calculation,
>>>>>> without going through an XML file, that refers to two specific sensors
>>>>>> only.
>>>>>>
>>>>>> As Laurent said here, this should come as part of the documentation
>>>>>> for driver users and would unblock libcamera IPU3 support
>>>>>> development.
>>>>>>
>>>>>> Could you provide documentation on how to calculate each
>>>>>> intermediate step resolutions?
>>>>>
>>>>> All the intermediate step resolutions are generated by the specific tool
>>>>> with sensor input and outputs resolutions.
>>>>>
>>>>> The tool try to keep maximum fov and has the knowledge of all the
>>>>> limitations of each intermediate hardware components(mainly BDS and GDC).
>>>>
>>>> That's exactly what we want to do in software in libcamera :-) And
>>>> that's why we need more infirmation about the limitations of each
>>>> intermediate hardware component. Eventually those limitations should be
>>>> documented in the IPU3 driver documentation in the kernel sources, but
>>>> for now we can move forward if they're just communicated by e-mail (if
>>>> time permits we may be able to submit a kernel patch to integrate that
>>>> in the documentation).
>>>>
>>>>> Currently, there is not a very simple calculation to get the
>>>>> intermediate resolutions.
>>>>> Let's take some effort to try find a programmatic way to do calculation
>>>>> instead of the tool.
>>>
>>> Thank you for your effort.
>>>
>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>>>>>>>>> 3. The ImgU V4L2 subdev composing should be set by using the
>>>>>>>>>>>>>> VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the
>>>>>>>>>>>>>> target, using the BDS height and width.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Once these 2 steps are done, the raw bayer frames can be input to the
>>>>>>>>>>>>>> ImgU V4L2 subdev for processing.
>>>>>>>>>>>>> Do I need to capture from both the output and viewfinder nodes ? How
>>>>>>>>>>>>> are they related to the IF -> BDS -> GDC pipeline, are they both fed
>>>>>>>>>>>>> from the GDC output ? If so, how does the viewfinder scaler fit in that
>>>>>>>>>>>>> picture ?
>>>>>>>>>>> The output capture should be set, the viewfinder can be disabled.
>>>>>>>>>>> The IF and BDS are seen as crop and compose of the imgu input video
>>>>>>>>>>> device. The GDC is seen as the subdev sink pad and OUTPUT/VF are source
>>>>>>>>>>> pads.
>>>>>>
>>>>>> This is another point that we would like to have clarified:
>>>>>> 1) which outputs are mandatory and which one are not
>>>>>> 2) which operations are mandatory on un-used outputs
>>>>>> 3) does the 'ipu_pipe_mode' control impact this
>>>>>>
>>>>>> As you mentioned here, "output" seems to be mandatory, while
>>>>>> "viewfinder" and "stat" are optional. We have tried using the "output"
>>>>>> video node only but the system hangs to an un-recoverable state.
>>>>>
>>>>> Yes, main output is mandatory, 'vf' and 'stat' are optional.
>>>>
>>>> I will let Jacopo confirm this, but unless I'm mistaken, when he tried
>>>> to use the main output only (with the links between the ImgU subdev and
>>>> the vf and stat video nodes disabled), the driver would hang without
>>>> processing any frame. I believe this was a complete system hang,
>>>> requiring a hard reboot to recover.
>>>>
>>>
>>> Yes, that's what I have noticed.
>>>
>>> On the other hand, if I link, configure, prepare buffers and start the
>>> 'vf' and 'stat' nodes, but never queue buffers there, I can capture
>>> from output only.
>>>
>>>>>> What I have noticed is instead that the viewfinder and stat nodes
>>>>>> needs to be:
>>>>>> 1) Linked to the respective "ImgU" subdevice pads
>>>>>> 2) Format configured
>>>>>> 3) Memory reserved
>>>>>> 4) video device nodes started
>>>>>>
>>>>>> It it not required to queue/dequeue buffers from viewfinder and stat,
>>>>>> but steps 1-4 have to be performed.
>>>>>>
>>>>>> Can you confirm this is intended?
>>>>>
>>>>> viewfinder and stats are enabled when the link for respective subdev
>>>>> pads enabled, and then driver can use these input conditions to find the
>>>>> binary to run.
>>>>>
>>>
>>> As Laurent reported above, if I leave the the 'vf' and 'stat' links
>>> disabled, the system hangs.
>>>
>>>>>> Could you please list all the steps that have to be applied to the
>>>>>> ImgU's capture video nodes, and which ones are mandatory and which ones
>>>>>> are optional, for the following use cases:
>>>>>> 1) Main output capture only
>>>>>> 2) Main + secondary output capture
>>>>>> 3) Secondary capture only.
>>>>>
>>>>> I think the 3) is not supported.
>>>>>
>>>>> The steps are:
>>>>> 1). link necessary the respective subdevices
>>>>> input --> imgu -->output
>>>>> | -->vf
>>>>> | -->3a stats
>>>
>>> For which use case, in the above reported list?
>>>
>>> 1) Main output capture only
>>> Does 'vf' and 'stat' links needs to be enabled?
>>>
>>> 2) Main + secondary output capture
>>> Does 'stat' link need to be enabled?
>>>
>>> 3) Secondary capture only.
>>> not supported
>>
>> The list above is a typical use, all outputs enabled, you can setup link
>> for main output only.
>
> I think we should clarify better what you mean by 'enabled'.
>
> From my testing what I see is that in order to operate the main
> output I have to:
> - link the stat and vf nodes (as well as input and output of course)
> - reserve memory buffers on stat and vf video nodes, even if not used
> - set format on all device ndoes
> - start the all video devices
>
> If one of these steps is not performed, the ImgU processing stalls and
> I need to hard reboot the device to have it operational again.
>
> On the other hand, I see that there is no need to queue any buffer to
> any of the output capture devices to have frames processed by the
> ImgU.
>
> IF links are setup as explained above, format configured and all the video
> device node started, I can queue all the frames I want to the ImgU input
> and never queue anything on its outputs, and they will get processed and
> returned to userspace nicely from the ImgU input device node. As soon
> as I queue a buffer to the ImgU main output video device, I see it
> returned filled with the processed data. This is good, as it doesen't
> stall the pipeline if there are no capture buffers to queue to the
> ImgU output, but now I wonder what you meant with "main output is
> mandatory"
Hi, Mondi
Sorry for late response.
I think you do not need setup link for vf and stat if you did not need
them. If you try setup link (enable) some queues (not input) and not
queue the buffers, driver will queue some dummy buffers to make the pipe
run.
It needs time for me to check all the cases and try to answer you. I am
a little busy on other work items recently, so forgive me late response
for your questions.
>
>>>
>>>>>
>>>>> 2). set all the formats for input, output and intermediate resolutions.
>>>>> 3). start stream
>>>>>
>>>>> The ipu pipe_mode will not impact the whole pipe behavior. It just ask
>>>>> firmware to run different processing to generate same format outputs.
>>>>
>>>
>>> I would apreciate to have a better description of the pipe_mode
>>> control, in order to better understand when and if the library has to
>>> modify its value and which mode to use (0=video, 1=still_capture).
>>
>> In some application, it will request continuous viewfinder, that means
>> you must keep preview continuous when take capture. That means you can
>> not switch out pipeline (preview and still mode, back and force), so the
>> driver need create 2 pipelines to satisfy this usage, 1 video mode pipe
>> and another is still mode. Both 2 pipes are created at first, run the
>> pipe as you command. It also can support still during video usage.
>>
>
> Thanks for the explanation, but it is still vague to me and it worries
> me a bit as in this typical usage scenario (viewfinder + sporadic
> capture) -both- ImgU pipes have to be used, preventing usage of two
> cameras at the same time (one camera assigned to one ImgU pipe
> instance).
>
> Why would you use both the ImgU pipes in this case? Shouldn't you
> always capture from viewfinder (discarding main output frames
> if not required), and when requested by the application capture from
> both the main and secondary output from the same ImgU pipe? Why would I
> need to change the pipe_mode for doing this?
Each pipe has its own pipe_mode and outputs.
So you can run 2 parallel pipes with different pipe mode, different pipe
modes mean to run different pipe in firmware, typically, 1 is still pipe
which used to capture and 1 is video pipe which used for video or
preview. You can only enable the main output for still pipe, enable
viewfinder and main for video pipe, viewfinder for video preview and
main for video recording. It depends on the real use case and they are
not fixed.
>
> Thank for your patience to answer all this questions :)
>
>
>>>
>>> Thanks
>>> j
>>>
>>>> --
>>>> Regards,
>>>>
>>>> Laurent Pinchart
^ permalink raw reply [flat|nested] 123+ messages in thread