linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v8 00/14] Rockchip ISP1 Driver
@ 2019-07-30 18:42 Helen Koike
  2019-07-30 18:42 ` [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format Helen Koike
                   ` (15 more replies)
  0 siblings, 16 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

Hello,

I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
rk3399 SoC.

I didn't change much from the last version, just applying the
suggestions made in the previous one.

This patchset is also available at:
https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8

Libcamera patched to work with this version:
https://gitlab.collabora.com/koike/libcamera
(also sent to the mailing list)

I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
Scarlet Chromebook.

Known issues (same as in v7):
-------------
- Reloading the module doesn't work (there is some missing cleanup when
unloading)
- When capturing in bayer format, changing the size doesn't seem to
affect the image.
- crop needs more tests
- v4l2-compliance error:
        fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
It seems that if controls are supported, v4l2-compliance says that
controls of type 'Image Processing Controls' are mandatory, is this
correct?
- It seems there are still some issues with interrupts, but I couldn't
isolate them yet.

Previous changelog:
-------------------

changes in V6:
  - add mipi txrx phy support
  - remove bool and enum from uapi header
  - add buf_prepare op
  - correct some spelling problems
  - return all queued buffers when starting stream failed

changes in V5: Sync with local changes,
  - fix the SP height limit
  - speed up the second stream capture
  - the second stream can't force sync for rsz when start/stop streaming
  - add frame id to param vb2 buf
  - enable luminance maximum threshold

changes in V4:
  - fix some bugs during development
  - move quantization settings to rkisp1 subdev
  - correct some spelling problems
  - describe ports in dt-binding documents

changes in V3:
  - add some comments
  - fix wrong use of v4l2_async_subdev_notifier_register
  - optimize two paths capture at a time
  - remove compose
  - re-struct headers
  - add a tmp wiki page: http://opensource.rock-chips.com/wiki_Rockchip-isp1

changes in V2:
  mipi-phy:
    - use async probing
    - make it be a child device of the GRF
  isp:
    - add dummy buffer
    - change the way to get bus configuration, which make it possible to
            add parallel sensor support in the future(without mipi-phy driver).

------------------

Changes in v8:
- Add SPDX in the header
- Remove emacs configs
- Fix doc style
- Remove boiler plate license text

Changes in v7:
- s/IPU3/RK_ISP1
- s/correspond/corresponding
- s/use/uses
- s/docuemnt/document
- Fix checkpatch errors (lines over 80 and SPDX)
- Add TODO to improve docs
- Migrate dphy specific code from
drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
to drivers/phy/rockchip/phy-rockchip-dphy.c
- Drop support for rk3288
- Drop support for dphy txrx
- code styling and checkpatch fixes
- fixed warning because of unknown entity type
- fixed v4l2-compliance errors regarding rkisp1 formats, try formats
and default values
- fix typo riksp1/rkisp1
- redesign: remove mipi/csi subdevice, sensors connect directly to the
isp subdevice in the media topology now. As a consequence, remove the
hack in mipidphy_g_mbus_config() where information from the sensor was
being propagated through the topology.
- From the old dphy:
        * cache get_remote_sensor() in s_stream
        * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
- Replace stream state with a boolean
- code styling and checkpatch fixes
- fix stop_stream (return after calling stop, do not reenable the stream)
- fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
- fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
- s/intput/input
- remove #define sd_to_isp_sd(_sd), add a static inline as it will be
reused by the capture
- s/strlcpy/strscpy
- sort out the locks in isp stats
- code styling and checkpatch fixes
- s/strlcpy/strscpy
- s/strcpy/strscpy
- fix config lsc error
LSC data table size is 17x17, but when configuring data to ISP,
should be aligned to 18x17. That means every last data of last
line should be filled with 0, and not filled with the data of
next line.
- Update new ISP parameters immediately
For those sub modules that have shadow registers in core isp, the
new programing parameters would not be active if both
CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT and CFG_UPD are not set. Now
we configure CFG_UPD to force update the shadow registers when new
ISP parameters are configured.
- fix some ISP parameters config error
Some ISP parameter config functions may override the old enable
bit value, because the enable bits of these modules are in the
same registers with parameters. So we should save the old enable
bits firstly.
- code styling and checkpatch fixes
- s/strlcpy/strscpy
- Fix v4l2-compliance issues:
        * remove input ioctls
media api can be used to define the topology, this input api is not
required. Besides it, if an input is enumerated, v4l2-compliance is not
happy with G_FMT returning the default colorspace instead of something
more specific.
        * return the pixelformat to the userspace
G_/S_/TRY_ FORMAT should return a valid pixelformat to the user, even if
the user gave an invalid one
        * add missing default colorspace and ycbcr
        * fix wrong pixformat in mp_fmts[] table
        * add buf type check in s_/g_selection
        * queue_setup - check sizes
        * normalize bus_info name
        * fix field any v4l2-compliance -s complain - set field none
        when streaming
- Fix compiling error: s/vidioc_enum_fmt_vid_cap_mplane/vidioc_enum_fmt_vid_cap
- Replace stream state with a boolean
The rkisp1_state enum consists only of 3 entries, where 1 is completely
unused and the other two respectively mean not streaming or streaming.
Replace it with a boolean called "streaming".
- Simplify MI interrupt handling
Rather than adding unnecessary indirection, just use stream index to
handle MI interrupt enable/disable/clear, since the stream index matches
the order of bits now, thanks to previous patch. While at it, remove
some dead code.
- code styling and checkpatch fixes
- add link_validate: don't allow a link with bayer/non-bayer mismatch
- VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
- add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
- Fix compilation and runtime errors due to bitrotting
The code has bit-rotten since March 2018, fix compilation errors.
The new V4L2 async notifier API requires notifiers to be initialized by
a call to v4l2_async_notifier_init() before being used, do so.
- Add missing module device table
- use clk_bulk framework
- add missing notifiers cleanups
- s/strlcpy/strscpy
- normalize bus_info name
- fix s_stream error path, stream_cnt wans't being decremented properly
- use devm_platform_ioremap_resource() helper
- s/deice/device
- redesign: remove mipi/csi subdevice, sensors connect directly to the
isp subdevice in the media topology now.
- remove "saved_state" member from rkisp1_stream struct
- Reverse the order of MIs
- Simplify MI interrupt handling
Rather than adding unnecessary indirection, just use stream index to
handle MI interrupt enable/disable/clear, since the stream index matches
the order of bits now, thanks to previous patch. While at it, remove
some dead code.
- code styling and checkpatch fixes
- update document with new design and tested example
- updated doc with new design and tested example
- add phy properties
- add ports
- add phy-cells

Helen Koike (1):
  MAINTAINERS: add entry for Rockchip ISP1 driver

Jacob Chen (9):
  media: doc: add document for rkisp1 meta buffer format
  media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  media: rkisp1: add Rockchip ISP1 subdev driver
  media: rkisp1: add ISP1 statistics driver
  media: rkisp1: add ISP1 params driver
  media: rkisp1: add capture device driver
  media: rkisp1: add rockchip isp1 core driver
  dt-bindings: Document the Rockchip ISP1 bindings
  dt-bindings: Document the Rockchip MIPI RX D-PHY bindings

Jeffy Chen (1):
  media: rkisp1: Add user space ABI definitions

Shunqian Zheng (3):
  media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
  arm64: dts: rockchip: add isp0 node for rk3399
  arm64: dts: rockchip: add rx0 mipi-phy for rk3399

 .../bindings/media/rockchip-isp1.txt          |   71 +
 .../bindings/media/rockchip-mipi-dphy.txt     |   38 +
 Documentation/media/uapi/v4l/meta-formats.rst |    2 +
 .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    |   23 +
 .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      |   22 +
 MAINTAINERS                                   |    8 +
 arch/arm64/boot/dts/rockchip/rk3399.dtsi      |   36 +
 drivers/media/platform/Kconfig                |   12 +
 drivers/media/platform/Makefile               |    1 +
 drivers/media/platform/rockchip/isp1/Makefile |    7 +
 .../media/platform/rockchip/isp1/capture.c    | 1754 +++++++++++++++++
 .../media/platform/rockchip/isp1/capture.h    |  164 ++
 drivers/media/platform/rockchip/isp1/common.h |  101 +
 drivers/media/platform/rockchip/isp1/dev.c    |  675 +++++++
 drivers/media/platform/rockchip/isp1/dev.h    |   97 +
 .../media/platform/rockchip/isp1/isp_params.c | 1604 +++++++++++++++
 .../media/platform/rockchip/isp1/isp_params.h |   50 +
 .../media/platform/rockchip/isp1/isp_stats.c  |  508 +++++
 .../media/platform/rockchip/isp1/isp_stats.h  |   60 +
 drivers/media/platform/rockchip/isp1/regs.c   |  223 +++
 drivers/media/platform/rockchip/isp1/regs.h   | 1525 ++++++++++++++
 drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 ++++++++++++
 drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
 drivers/media/v4l2-core/v4l2-ioctl.c          |    2 +
 drivers/phy/rockchip/Kconfig                  |    8 +
 drivers/phy/rockchip/Makefile                 |    1 +
 drivers/phy/rockchip/phy-rockchip-dphy.c      |  408 ++++
 include/uapi/linux/rkisp1-config.h            |  816 ++++++++
 include/uapi/linux/videodev2.h                |    4 +
 29 files changed, 9617 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
 create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
 create mode 100644 drivers/media/platform/rockchip/isp1/capture.c
 create mode 100644 drivers/media/platform/rockchip/isp1/capture.h
 create mode 100644 drivers/media/platform/rockchip/isp1/common.h
 create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
 create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.c
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.h
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h
 create mode 100644 drivers/media/platform/rockchip/isp1/regs.c
 create mode 100644 drivers/media/platform/rockchip/isp1/regs.h
 create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
 create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
 create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
 create mode 100644 include/uapi/linux/rkisp1-config.h

-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-15 13:30   ` Laurent Pinchart
  2019-07-30 18:42 ` [PATCH v8 02/14] media: doc: add document for " Helen Koike
                   ` (14 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

From: Shunqian Zheng <zhengsq@rock-chips.com>

Add the Rockchip ISP1 specific processing parameter format
V4L2_META_FMT_RK_ISP1_PARAMS and metadata format
V4L2_META_FMT_RK_ISP1_STAT_3A for 3A.

Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- s/IPU3/RK_ISP1

 drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
 include/uapi/linux/videodev2.h       | 4 ++++
 2 files changed, 6 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index bb5b4926538a..2df20801a299 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1330,6 +1330,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_META_FMT_VSP1_HGT:	descr = "R-Car VSP1 2-D Histogram"; break;
 	case V4L2_META_FMT_UVC:		descr = "UVC Payload Header Metadata"; break;
 	case V4L2_META_FMT_D4XX:	descr = "Intel D4xx UVC Metadata"; break;
+	case V4L2_META_FMT_RK_ISP1_PARAMS:	descr = "Rockchip ISP1 3A params"; break;
+	case V4L2_META_FMT_RK_ISP1_STAT_3A:	descr = "Rockchip ISP1 3A statistics"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 2427bc4d8eba..ae1e375a10d7 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -756,6 +756,10 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_UVC         v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
 #define V4L2_META_FMT_D4XX        v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
 
+/* Vendor specific - used for RK_ISP1 camera sub-system */
+#define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 params */
+#define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A statistics */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 02/14] media: doc: add document for rkisp1 meta buffer format
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
  2019-07-30 18:42 ` [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-07 13:09   ` Sakari Ailus
  2019-08-15 13:51   ` Laurent Pinchart
  2019-07-30 18:42 ` [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions Helen Koike
                   ` (13 subsequent siblings)
  15 siblings, 2 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Jacob Chen,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

This commit add document for rkisp1 meta buffer format

Signed-off-by: Jacob Chen <jacob-chen@rock-chips.com>
Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8:
- Add SPDX in the header
- Remove emacs configs
- Fix doc style

Changes in v7:
- s/correspond/corresponding
- s/use/uses
- s/docuemnt/document

 Documentation/media/uapi/v4l/meta-formats.rst |  2 ++
 .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    | 23 +++++++++++++++++++
 .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      | 22 ++++++++++++++++++
 3 files changed, 47 insertions(+)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst

diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
index b10ca9ee3968..5de621fea3cc 100644
--- a/Documentation/media/uapi/v4l/meta-formats.rst
+++ b/Documentation/media/uapi/v4l/meta-formats.rst
@@ -24,3 +24,5 @@ These formats are used for the :ref:`metadata` interface only.
     pixfmt-meta-uvc
     pixfmt-meta-vsp1-hgo
     pixfmt-meta-vsp1-hgt
+    pixfmt-meta-rkisp1-params
+    pixfmt-meta-rkisp1-stat
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
new file mode 100644
index 000000000000..103b5cb79b7c
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
@@ -0,0 +1,23 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+
+.. _v4l2-meta-fmt-rkisp1-params:
+
+============================
+V4L2_META_FMT_RK_ISP1_PARAMS
+============================
+
+Rockchip ISP1 Parameters Data
+
+Description
+===========
+
+This format describes input parameters for the Rockchip ISP1.
+
+It uses c-struct :c:type:`rkisp1_isp_params_cfg`, which is defined in
+the ``linux/rkisp1-config.h`` header file.
+
+The parameters consist of multiple modules.
+The module won't be updated if the corresponding bit was not set in module_*_update.
+
+.. kernel-doc:: include/uapi/linux/rkisp1-config.h
+   :functions: rkisp1_isp_params_cfg
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
new file mode 100644
index 000000000000..4ad303f96421
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
@@ -0,0 +1,22 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+
+.. _v4l2-meta-fmt-rkisp1-stat:
+
+=============================
+V4L2_META_FMT_RK_ISP1_STAT_3A
+=============================
+
+
+Rockchip ISP1 Statistics Data
+
+Description
+===========
+
+This format describes image color statistics information generated by the Rockchip
+ISP1.
+
+It uses c-struct :c:type:`rkisp1_stat_buffer`, which is defined in
+the ``linux/rkisp1-config.h`` header file.
+
+.. kernel-doc:: include/uapi/linux/rkisp1-config.h
+   :functions: rkisp1_stat_buffer
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
  2019-07-30 18:42 ` [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format Helen Koike
  2019-07-30 18:42 ` [PATCH v8 02/14] media: doc: add document for " Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-15 18:46   ` Laurent Pinchart
  2019-07-30 18:42 ` [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver Helen Koike
                   ` (12 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

From: Jeffy Chen <jeffy.chen@rock-chips.com>

Add the header for userspace

Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---
Hi,

I don't have the "REF_01 - ISP_user_manual, Rev 2.57" document that was
mentioned in previous version, so I'm adding a TODO for now to improve
the docs once we collect the information.
If Rockchip people could help here it would be great.

Thanks
Helen

Changes in v8: None
Changes in v7:
- Fix checkpatch errors (lines over 80 and SPDX)
- Add TODO to improve docs

 include/uapi/linux/rkisp1-config.h | 816 +++++++++++++++++++++++++++++
 1 file changed, 816 insertions(+)
 create mode 100644 include/uapi/linux/rkisp1-config.h

diff --git a/include/uapi/linux/rkisp1-config.h b/include/uapi/linux/rkisp1-config.h
new file mode 100644
index 000000000000..9ab979bb4adb
--- /dev/null
+++ b/include/uapi/linux/rkisp1-config.h
@@ -0,0 +1,816 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
+/*
+ * Rockchip isp1 driver
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+/*
+ * TODO: Improve documentation, mostly regarding abbreviation and hardware
+ * specificities.
+ */
+
+#ifndef _UAPI_RKISP1_CONFIG_H
+#define _UAPI_RKISP1_CONFIG_H
+
+#include <linux/types.h>
+#include <linux/v4l2-controls.h>
+
+#define CIFISP_MODULE_DPCC              (1 << 0)
+#define CIFISP_MODULE_BLS               (1 << 1)
+#define CIFISP_MODULE_SDG               (1 << 2)
+#define CIFISP_MODULE_HST               (1 << 3)
+#define CIFISP_MODULE_LSC               (1 << 4)
+#define CIFISP_MODULE_AWB_GAIN          (1 << 5)
+#define CIFISP_MODULE_FLT               (1 << 6)
+#define CIFISP_MODULE_BDM               (1 << 7)
+#define CIFISP_MODULE_CTK               (1 << 8)
+#define CIFISP_MODULE_GOC               (1 << 9)
+#define CIFISP_MODULE_CPROC             (1 << 10)
+#define CIFISP_MODULE_AFC               (1 << 11)
+#define CIFISP_MODULE_AWB               (1 << 12)
+#define CIFISP_MODULE_IE                (1 << 13)
+#define CIFISP_MODULE_AEC               (1 << 14)
+#define CIFISP_MODULE_WDR               (1 << 15)
+#define CIFISP_MODULE_DPF               (1 << 16)
+#define CIFISP_MODULE_DPF_STRENGTH      (1 << 17)
+
+#define CIFISP_CTK_COEFF_MAX            0x100
+#define CIFISP_CTK_OFFSET_MAX           0x800
+
+#define CIFISP_AE_MEAN_MAX              25
+#define CIFISP_HIST_BIN_N_MAX           16
+#define CIFISP_AFM_MAX_WINDOWS          3
+#define CIFISP_DEGAMMA_CURVE_SIZE       17
+
+#define CIFISP_BDM_MAX_TH               0xFF
+
+/*
+ * Black level compensation
+ */
+/* maximum value for horizontal start address */
+#define CIFISP_BLS_START_H_MAX             0x00000FFF
+/* maximum value for horizontal stop address */
+#define CIFISP_BLS_STOP_H_MAX              0x00000FFF
+/* maximum value for vertical start address */
+#define CIFISP_BLS_START_V_MAX             0x00000FFF
+/* maximum value for vertical stop address */
+#define CIFISP_BLS_STOP_V_MAX              0x00000FFF
+/* maximum is 2^18 = 262144*/
+#define CIFISP_BLS_SAMPLES_MAX             0x00000012
+/* maximum value for fixed black level */
+#define CIFISP_BLS_FIX_SUB_MAX             0x00000FFF
+/* minimum value for fixed black level */
+#define CIFISP_BLS_FIX_SUB_MIN             0xFFFFF000
+/* 13 bit range (signed)*/
+#define CIFISP_BLS_FIX_MASK                0x00001FFF
+
+/*
+ * Automatic white balance measurments
+ */
+#define CIFISP_AWB_MAX_GRID                1
+#define CIFISP_AWB_MAX_FRAMES              7
+
+/*
+ * Gamma out
+ */
+/* Maximum number of color samples supported */
+#define CIFISP_GAMMA_OUT_MAX_SAMPLES       17
+
+/*
+ * Lens shade correction
+ */
+#define CIFISP_LSC_GRAD_TBL_SIZE           8
+#define CIFISP_LSC_SIZE_TBL_SIZE           8
+/*
+ * The following matches the tuning process,
+ * not the max capabilities of the chip.
+ * Last value unused.
+ */
+#define	CIFISP_LSC_DATA_TBL_SIZE           290
+
+/*
+ * Histogram calculation
+ */
+/* Last 3 values unused. */
+#define CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE 28
+
+/*
+ * Defect Pixel Cluster Correction
+ */
+#define CIFISP_DPCC_METHODS_MAX       3
+
+/*
+ * Denoising pre filter
+ */
+#define CIFISP_DPF_MAX_NLF_COEFFS      17
+#define CIFISP_DPF_MAX_SPATIAL_COEFFS  6
+
+/*
+ * Measurement types
+ */
+#define CIFISP_STAT_AWB           (1 << 0)
+#define CIFISP_STAT_AUTOEXP       (1 << 1)
+#define CIFISP_STAT_AFM_FIN       (1 << 2)
+#define CIFISP_STAT_HIST          (1 << 3)
+
+enum cifisp_histogram_mode {
+	CIFISP_HISTOGRAM_MODE_DISABLE,
+	CIFISP_HISTOGRAM_MODE_RGB_COMBINED,
+	CIFISP_HISTOGRAM_MODE_R_HISTOGRAM,
+	CIFISP_HISTOGRAM_MODE_G_HISTOGRAM,
+	CIFISP_HISTOGRAM_MODE_B_HISTOGRAM,
+	CIFISP_HISTOGRAM_MODE_Y_HISTOGRAM
+};
+
+enum cifisp_awb_mode_type {
+	CIFISP_AWB_MODE_MANUAL,
+	CIFISP_AWB_MODE_RGB,
+	CIFISP_AWB_MODE_YCBCR
+};
+
+enum cifisp_flt_mode {
+	CIFISP_FLT_STATIC_MODE,
+	CIFISP_FLT_DYNAMIC_MODE
+};
+
+/**
+ * enum cifisp_exp_ctrl_autostop - stop modes
+ * @CIFISP_EXP_CTRL_AUTOSTOP_0: continuous measurement
+ * @CIFISP_EXP_CTRL_AUTOSTOP_1: stop measuring after a complete frame
+ */
+enum cifisp_exp_ctrl_autostop {
+	CIFISP_EXP_CTRL_AUTOSTOP_0 = 0,
+	CIFISP_EXP_CTRL_AUTOSTOP_1 = 1,
+};
+
+/**
+ * enum cifisp_exp_meas_mode - Exposure measure mode
+ * @CIFISP_EXP_MEASURING_MODE_0: Y = 16 + 0.25R + 0.5G + 0.1094B
+ * @CIFISP_EXP_MEASURING_MODE_1: Y = (R + G + B) x (85/256)
+ */
+enum cifisp_exp_meas_mode {
+	CIFISP_EXP_MEASURING_MODE_0,
+	CIFISP_EXP_MEASURING_MODE_1,
+};
+
+/*---------- PART1: Input Parameters ------------*/
+
+struct cifisp_window {
+	__u16 h_offs;
+	__u16 v_offs;
+	__u16 h_size;
+	__u16 v_size;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_bls_fixed_val - BLS fixed subtraction values
+ *
+ * The values will be subtracted from the sensor
+ * values. Therefore a negative value means addition instead of subtraction!
+ *
+ * @r: Fixed (signed!) subtraction value for Bayer pattern R
+ * @gr: Fixed (signed!) subtraction value for Bayer pattern Gr
+ * @gb: Fixed (signed!) subtraction value for Bayer pattern Gb
+ * @b: Fixed (signed!) subtraction value for Bayer pattern B
+ */
+struct cifisp_bls_fixed_val {
+	__s16 r;
+	__s16 gr;
+	__s16 gb;
+	__s16 b;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_bls_config - Configuration used by black level subtraction
+ *
+ * @enable_auto: Automatic mode activated means that the measured values
+ *		 are subtracted. Otherwise the fixed subtraction
+ *		 values will be subtracted.
+ * @en_windows: enabled window
+ * @bls_window1: Measurement window 1 size
+ * @bls_window2: Measurement window 2 size
+ * @bls_samples: Set amount of measured pixels for each Bayer position
+ *		 (A, B,C and D) to 2^bls_samples.
+ * @cifisp_bls_fixed_val: Fixed subtraction values
+ */
+struct cifisp_bls_config {
+	__u8 enable_auto;
+	__u8 en_windows;
+	struct cifisp_window bls_window1;
+	struct cifisp_window bls_window2;
+	__u8 bls_samples;
+	struct cifisp_bls_fixed_val fixed_val;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpcc_methods_config - Methods Configuration used by DPCC
+ *
+ * Methods Configuration used by Defect Pixel Cluster Correction
+ *
+ * @method: Method enable bits
+ * @line_thresh: Line threshold
+ * @line_mad_fac: Line MAD factor
+ * @pg_fac: Peak gradient factor
+ * @rnd_thresh: Rank Neighbor Difference threshold
+ * @rg_fac: Rank gradient factor
+ */
+struct cifisp_dpcc_methods_config {
+	__u32 method;
+	__u32 line_thresh;
+	__u32 line_mad_fac;
+	__u32 pg_fac;
+	__u32 rnd_thresh;
+	__u32 rg_fac;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpcc_methods_config - Configuration used by DPCC
+ *
+ * Configuration used by Defect Pixel Cluster Correction
+ *
+ * @mode: dpcc output mode
+ * @output_mode: whether use hard coded methods
+ * @set_use: stage1 methods set
+ * @methods: methods config
+ * @ro_limits: rank order limits
+ * @rnd_offs: differential rank offsets for rank neighbor difference
+ */
+struct cifisp_dpcc_config {
+	__u32 mode;
+	__u32 output_mode;
+	__u32 set_use;
+	struct cifisp_dpcc_methods_config methods[CIFISP_DPCC_METHODS_MAX];
+	__u32 ro_limits;
+	__u32 rnd_offs;
+} __attribute__ ((packed));
+
+struct cifisp_gamma_corr_curve {
+	__u16 gamma_y[CIFISP_DEGAMMA_CURVE_SIZE];
+} __attribute__ ((packed));
+
+struct cifisp_gamma_curve_x_axis_pnts {
+	__u32 gamma_dx0;
+	__u32 gamma_dx1;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_gamma_corr_curve - Configuration used by sensor degamma
+ *
+ * @curve_x: gamma curve point definition axis for x
+ * @xa_pnts: x increments
+ */
+struct cifisp_sdg_config {
+	struct cifisp_gamma_corr_curve curve_r;
+	struct cifisp_gamma_corr_curve curve_g;
+	struct cifisp_gamma_corr_curve curve_b;
+	struct cifisp_gamma_curve_x_axis_pnts xa_pnts;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_lsc_config - Configuration used by Lens shading correction
+ *
+ * refer to REF_01 for details
+ */
+struct cifisp_lsc_config {
+	__u32 r_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
+	__u32 gr_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
+	__u32 gb_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
+	__u32 b_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
+
+	__u32 x_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
+	__u32 y_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
+
+	__u32 x_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
+	__u32 y_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
+	__u16 config_width;
+	__u16 config_height;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_ie_config - Configuration used by image effects
+ *
+ * @eff_mat_1: 3x3 Matrix Coefficients for Emboss Effect 1
+ * @eff_mat_2: 3x3 Matrix Coefficients for Emboss Effect 2
+ * @eff_mat_3: 3x3 Matrix Coefficients for Emboss 3/Sketch 1
+ * @eff_mat_4: 3x3 Matrix Coefficients for Sketch Effect 2
+ * @eff_mat_5: 3x3 Matrix Coefficients for Sketch Effect 3
+ * @eff_tint: Chrominance increment values of tint (used for sepia effect)
+ */
+struct cifisp_ie_config {
+	__u16 effect;
+	__u16 color_sel;
+	__u16 eff_mat_1;
+	__u16 eff_mat_2;
+	__u16 eff_mat_3;
+	__u16 eff_mat_4;
+	__u16 eff_mat_5;
+	__u16 eff_tint;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_cproc_config - Configuration used by Color Processing
+ *
+ * @c_out_range: Chrominance pixel clipping range at output.
+ *		 (0 for limit, 1 for full)
+ * @y_in_range: Luminance pixel clipping range at output.
+ * @y_out_range: Luminance pixel clipping range at output.
+ * @contrast: 00~ff, 0.0~1.992
+ * @brightness: 80~7F, -128~+127
+ * @sat: saturation, 00~FF, 0.0~1.992
+ * @hue: 80~7F, -90~+87.188
+ */
+struct cifisp_cproc_config {
+	__u8 c_out_range;
+	__u8 y_in_range;
+	__u8 y_out_range;
+	__u8 contrast;
+	__u8 brightness;
+	__u8 sat;
+	__u8 hue;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_awb_meas_config - Configuration used by auto white balance
+ *
+ * @awb_wnd: white balance measurement window (in pixels)
+ *	     (from enum cifisp_awb_mode_type)
+ * @max_y: only pixels values < max_y contribute to awb measurement, set to 0
+ *	   to disable this feature
+ * @min_y: only pixels values > min_y contribute to awb measurement
+ * @max_csum: Chrominance sum maximum value, only consider pixels with Cb+Cr,
+ *	      smaller than threshold for awb measurements
+ * @min_c: Chrominance minimum value, only consider pixels with Cb/Cr
+ *	   each greater than threshold value for awb measurements
+ * @frames: number of frames - 1 used for mean value calculation
+ *	    (ucFrames=0 means 1 Frame)
+ * @awb_ref_cr: reference Cr value for AWB regulation, target for AWB
+ * @awb_ref_cb: reference Cb value for AWB regulation, target for AWB
+ */
+struct cifisp_awb_meas_config {
+	/*
+	 * Note: currently the h and v offsets are mapped to grid offsets
+	 */
+	struct cifisp_window awb_wnd;
+	__u32 awb_mode;
+	__u8 max_y;
+	__u8 min_y;
+	__u8 max_csum;
+	__u8 min_c;
+	__u8 frames;
+	__u8 awb_ref_cr;
+	__u8 awb_ref_cb;
+	__u8 enable_ymax_cmp;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_awb_gain_config - Configuration used by auto white balance gain
+ *
+ * out_data_x = ( AWB_GEAIN_X * in_data + 128) >> 8
+ */
+struct cifisp_awb_gain_config {
+	__u16 gain_red;
+	__u16 gain_green_r;
+	__u16 gain_blue;
+	__u16 gain_green_b;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_flt_config - Configuration used by ISP filtering
+ *
+ * @mode: ISP_FILT_MODE register fields (from enum cifisp_flt_mode)
+ * @grn_stage1: ISP_FILT_MODE register fields
+ * @chr_h_mode: ISP_FILT_MODE register fields
+ * @chr_v_mode: ISP_FILT_MODE register fields
+ *
+ * refer to REF_01 for details.
+ */
+
+struct cifisp_flt_config {
+	__u32 mode;
+	__u8 grn_stage1;
+	__u8 chr_h_mode;
+	__u8 chr_v_mode;
+	__u32 thresh_bl0;
+	__u32 thresh_bl1;
+	__u32 thresh_sh0;
+	__u32 thresh_sh1;
+	__u32 lum_weight;
+	__u32 fac_sh1;
+	__u32 fac_sh0;
+	__u32 fac_mid;
+	__u32 fac_bl0;
+	__u32 fac_bl1;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_bdm_config - Configuration used by Bayer DeMosaic
+ *
+ * @demosaic_th: threshod for bayer demosaicing texture detection
+ */
+struct cifisp_bdm_config {
+	__u8 demosaic_th;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_ctk_config - Configuration used by Cross Talk correction
+ *
+ * @coeff: color correction matrix
+ * @ct_offset_b: offset for the crosstalk correction matrix
+ */
+struct cifisp_ctk_config {
+	__u16 coeff0;
+	__u16 coeff1;
+	__u16 coeff2;
+	__u16 coeff3;
+	__u16 coeff4;
+	__u16 coeff5;
+	__u16 coeff6;
+	__u16 coeff7;
+	__u16 coeff8;
+	__u16 ct_offset_r;
+	__u16 ct_offset_g;
+	__u16 ct_offset_b;
+} __attribute__ ((packed));
+
+enum cifisp_goc_mode {
+	CIFISP_GOC_MODE_LOGARITHMIC,
+	CIFISP_GOC_MODE_EQUIDISTANT
+};
+
+/**
+ * struct cifisp_goc_config - Configuration used by Gamma Out correction
+ *
+ * @mode: goc mode (from enum cifisp_goc_mode)
+ * @gamma_y: gamma out curve y-axis for all color components
+ */
+struct cifisp_goc_config {
+	__u32 mode;
+	__u16 gamma_y[CIFISP_GAMMA_OUT_MAX_SAMPLES];
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_hst_config - Configuration used by Histogram
+ *
+ * @mode: histogram mode (from enum cifisp_histogram_mode)
+ * @histogram_predivider: process every stepsize pixel, all other pixels are
+ *			  skipped
+ * @meas_window: coordinates of the measure window
+ * @hist_weight: weighting factor for sub-windows
+ */
+struct cifisp_hst_config {
+	__u32 mode;
+	__u8 histogram_predivider;
+	struct cifisp_window meas_window;
+	__u8 hist_weight[CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE];
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_aec_config - Configuration used by Auto Exposure Control
+ *
+ * @mode: Exposure measure mode (from enum cifisp_exp_meas_mode)
+ * @autostop: stop mode (from enum cifisp_exp_ctrl_autostop)
+ * @meas_window: coordinates of the measure window
+ */
+struct cifisp_aec_config {
+	__u32 mode;
+	__u32 autostop;
+	struct cifisp_window meas_window;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_afc_config - Configuration used by Auto Focus Control
+ *
+ * @num_afm_win: max CIFISP_AFM_MAX_WINDOWS
+ * @afm_win: coordinates of the meas window
+ * @thres: threshold used for minimizing the influence of noise
+ * @var_shift: the number of bits for the shift operation at the end of the
+ *	       calculation chain.
+ */
+struct cifisp_afc_config {
+	__u8 num_afm_win;
+	struct cifisp_window afm_win[CIFISP_AFM_MAX_WINDOWS];
+	__u32 thres;
+	__u32 var_shift;
+} __attribute__ ((packed));
+
+/**
+ * enum cifisp_dpf_gain_usage - dpf gain usage
+ * @CIFISP_DPF_GAIN_USAGE_DISABLED: don't use any gains in preprocessing stage
+ * @CIFISP_DPF_GAIN_USAGE_NF_GAINS: use only the noise function gains from
+ *				    registers DPF_NF_GAIN_R, ...
+ * @CIFISP_DPF_GAIN_USAGE_LSC_GAINS:  use only the gains from LSC module
+ * @CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS: use the noise function gains and the
+ *					gains from LSC module
+ * @CIFISP_DPF_GAIN_USAGE_AWB_GAINS: use only the gains from AWB module
+ * @CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS: use the gains from AWB and LSC module
+ * @CIFISP_DPF_GAIN_USAGE_MAX: upper border (only for an internal evaluation)
+ */
+enum cifisp_dpf_gain_usage {
+	CIFISP_DPF_GAIN_USAGE_DISABLED,
+	CIFISP_DPF_GAIN_USAGE_NF_GAINS,
+	CIFISP_DPF_GAIN_USAGE_LSC_GAINS,
+	CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS,
+	CIFISP_DPF_GAIN_USAGE_AWB_GAINS,
+	CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS,
+	CIFISP_DPF_GAIN_USAGE_MAX
+};
+
+/**
+ * enum cifisp_dpf_gain_usage - dpf gain usage
+ * @CIFISP_DPF_RB_FILTERSIZE_13x9: red and blue filter kernel size 13x9
+ *				   (means 7x5 active pixel)
+ * @CIFISP_DPF_RB_FILTERSIZE_9x9: red and blue filter kernel size 9x9
+ *				   (means 5x5 active pixel)
+ */
+enum cifisp_dpf_rb_filtersize {
+	CIFISP_DPF_RB_FILTERSIZE_13x9,
+	CIFISP_DPF_RB_FILTERSIZE_9x9,
+};
+
+/**
+ * enum cifisp_dpf_nll_scale_mode - dpf noise level scale mode
+ * @CIFISP_NLL_SCALE_LINEAR: use a linear scaling
+ * @CIFISP_NLL_SCALE_LOGARITHMIC: use a logarithmic scaling
+ */
+enum cifisp_dpf_nll_scale_mode {
+	CIFISP_NLL_SCALE_LINEAR,
+	CIFISP_NLL_SCALE_LOGARITHMIC,
+};
+
+/**
+ * struct cifisp_dpf_nll - Noise level lookup
+ *
+ * @coeff: Noise level Lookup coefficient
+ * @scale_mode: dpf noise level scale mode (from enum cifisp_dpf_nll_scale_mode)
+ */
+struct cifisp_dpf_nll {
+	__u16 coeff[CIFISP_DPF_MAX_NLF_COEFFS];
+	__u32 scale_mode;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpf_rb_flt - Red blue filter config
+ *
+ * @fltsize: The filter size for the red and blue pixels
+ *	     (from enum cifisp_dpf_rb_filtersize)
+ * @spatial_coeff: Spatial weights
+ * @r_enable: enable filter processing for red pixels
+ * @b_enable: enable filter processing for blue pixels
+ */
+struct cifisp_dpf_rb_flt {
+	__u32 fltsize;
+	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
+	__u8 r_enable;
+	__u8 b_enable;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpf_g_flt - Green filter Configuration
+ *
+ * @spatial_coeff: Spatial weights
+ * @gr_enable: enable filter processing for green pixels in green/red lines
+ * @gb_enable: enable filter processing for green pixels in green/blue lines
+ */
+struct cifisp_dpf_g_flt {
+	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
+	__u8 gr_enable;
+	__u8 gb_enable;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpf_gain - Noise function Configuration
+ *
+ * @mode: dpf gain usage  (from enum cifisp_dpf_gain_usage)
+ * @nf_r_gain: Noise function Gain that replaces the AWB gain for red pixels
+ * @nf_b_gain: Noise function Gain that replaces the AWB gain for blue pixels
+ * @nf_gr_gain: Noise function Gain that replaces the AWB gain
+ *		for green pixels in a red line
+ * @nf_gb_gain: Noise function Gain that replaces the AWB gain
+ *		for green pixels in a blue line
+ */
+struct cifisp_dpf_gain {
+	__u32 mode;
+	__u16 nf_r_gain;
+	__u16 nf_b_gain;
+	__u16 nf_gr_gain;
+	__u16 nf_gb_gain;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpf_config - Configuration used by De-noising pre-filter
+ *
+ * @gain: noise function gain
+ * @g_flt: green filter config
+ * @rb_flt: red blue filter config
+ * @nll: noise level lookup
+ */
+struct cifisp_dpf_config {
+	struct cifisp_dpf_gain gain;
+	struct cifisp_dpf_g_flt g_flt;
+	struct cifisp_dpf_rb_flt rb_flt;
+	struct cifisp_dpf_nll nll;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_dpf_strength_config - strength of the filter
+ *
+ * @r: filter strength of the RED filter
+ * @g: filter strength of the GREEN filter
+ * @b: filter strength of the BLUE filter
+ */
+struct cifisp_dpf_strength_config {
+	__u8 r;
+	__u8 g;
+	__u8 b;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_isp_other_cfg - Parameters for some blocks in rockchip isp1
+ *
+ * @dpcc_config: Defect Pixel Cluster Correction config
+ * @bls_config: Black Level Subtraction config
+ * @sdg_config: sensor degamma config
+ * @lsc_config: Lens Shade config
+ * @awb_gain_config: Auto White balance gain config
+ * @flt_config: filter config
+ * @bdm_config: demosaic config
+ * @ctk_config: cross talk config
+ * @goc_config: gamma out config
+ * @bls_config: black level subtraction config
+ * @dpf_config: De-noising pre-filter config
+ * @dpf_strength_config: dpf strength config
+ * @cproc_config: color process config
+ * @ie_config: image effects config
+ */
+struct cifisp_isp_other_cfg {
+	struct cifisp_dpcc_config dpcc_config;
+	struct cifisp_bls_config bls_config;
+	struct cifisp_sdg_config sdg_config;
+	struct cifisp_lsc_config lsc_config;
+	struct cifisp_awb_gain_config awb_gain_config;
+	struct cifisp_flt_config flt_config;
+	struct cifisp_bdm_config bdm_config;
+	struct cifisp_ctk_config ctk_config;
+	struct cifisp_goc_config goc_config;
+	struct cifisp_dpf_config dpf_config;
+	struct cifisp_dpf_strength_config dpf_strength_config;
+	struct cifisp_cproc_config cproc_config;
+	struct cifisp_ie_config ie_config;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_isp_meas_cfg - Rockchip ISP1 Measure Parameters
+ *
+ * @awb_meas_config: auto white balance config
+ * @hst_config: histogram config
+ * @aec_config: auto exposure config
+ * @afc_config: auto focus config
+ */
+struct cifisp_isp_meas_cfg {
+	struct cifisp_awb_meas_config awb_meas_config;
+	struct cifisp_hst_config hst_config;
+	struct cifisp_aec_config aec_config;
+	struct cifisp_afc_config afc_config;
+} __attribute__ ((packed));
+
+/**
+ * struct rkisp1_isp_params_cfg - Rockchip ISP1 Input Parameters Meta Data
+ *
+ * @module_en_update: mask the enable bits of which module should be updated
+ * @module_ens: mask the enable value of each module, only update the module
+ *		which correspond bit was set in module_en_update
+ * @module_cfg_update: mask the config bits of which module should be updated
+ * @meas: measurement config
+ * @others: other config
+ */
+struct rkisp1_isp_params_cfg {
+	__u32 module_en_update;
+	__u32 module_ens;
+	__u32 module_cfg_update;
+
+	struct cifisp_isp_meas_cfg meas;
+	struct cifisp_isp_other_cfg others;
+} __attribute__ ((packed));
+
+/*---------- PART2: Measurement Statistics ------------*/
+
+/**
+ * struct cifisp_bls_meas_val - AWB measured values
+ *
+ * @cnt: White pixel count, number of "white pixels" found during laster
+ *	 measurement
+ * @mean_y_or_g: Mean value of Y within window and frames,
+ *		 Green if RGB is selected.
+ * @mean_cb_or_b: Mean value of Cb within window and frames,
+ *		  Blue if RGB is selected.
+ * @mean_cr_or_r: Mean value of Cr within window and frames,
+ *		  Red if RGB is selected.
+ */
+struct cifisp_awb_meas {
+	__u32 cnt;
+	__u8 mean_y_or_g;
+	__u8 mean_cb_or_b;
+	__u8 mean_cr_or_r;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_awb_stat - statistics automatic white balance data
+ *
+ * @awb_mean: Mean measured data
+ */
+struct cifisp_awb_stat {
+	struct cifisp_awb_meas awb_mean[CIFISP_AWB_MAX_GRID];
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_bls_meas_val - BLS measured values
+ *
+ * @meas_r: Mean measured value for Bayer pattern R
+ * @meas_gr: Mean measured value for Bayer pattern Gr
+ * @meas_gb: Mean measured value for Bayer pattern Gb
+ * @meas_b: Mean measured value for Bayer pattern B
+ */
+struct cifisp_bls_meas_val {
+	__u16 meas_r;
+	__u16 meas_gr;
+	__u16 meas_gb;
+	__u16 meas_b;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_ae_stat - statistics auto exposure data
+ *
+ * @exp_mean: Mean luminance value of block xx
+ * @bls_val:  BLS measured values
+ *
+ * Image is divided into 5x5 blocks.
+ */
+struct cifisp_ae_stat {
+	__u8 exp_mean[CIFISP_AE_MEAN_MAX];
+	struct cifisp_bls_meas_val bls_val;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_af_meas_val - AF measured values
+ *
+ * @sum: sharpness, refer to REF_01 for definition
+ * @lum: luminance, refer to REF_01 for definition
+ */
+struct cifisp_af_meas_val {
+	__u32 sum;
+	__u32 lum;
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_af_stat - statistics auto focus data
+ *
+ * @window: AF measured value of window x
+ *
+ * The module measures the sharpness in 3 windows of selectable size via
+ * register settings(ISP_AFM_*_A/B/C)
+ */
+struct cifisp_af_stat {
+	struct cifisp_af_meas_val window[CIFISP_AFM_MAX_WINDOWS];
+} __attribute__ ((packed));
+
+/**
+ * struct cifisp_hist_stat - statistics histogram data
+ *
+ * @hist_bins: measured bin counters
+ *
+ * Measurement window divided into 25 sub-windows, set
+ * with ISP_HIST_XXX
+ */
+struct cifisp_hist_stat {
+	__u16 hist_bins[CIFISP_HIST_BIN_N_MAX];
+} __attribute__ ((packed));
+
+/**
+ * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Data
+ *
+ * @cifisp_awb_stat: statistics data for automatic white balance
+ * @cifisp_ae_stat: statistics data for auto exposure
+ * @cifisp_af_stat: statistics data for auto focus
+ * @cifisp_hist_stat: statistics histogram data
+ */
+struct cifisp_stat {
+	struct cifisp_awb_stat awb;
+	struct cifisp_ae_stat ae;
+	struct cifisp_af_stat af;
+	struct cifisp_hist_stat hist;
+} __attribute__ ((packed));
+
+/**
+ * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Meta Data
+ *
+ * @meas_type: measurement types (CIFISP_STAT_ definitions)
+ * @frame_id: frame ID for sync
+ * @params: statistics data
+ */
+struct rkisp1_stat_buffer {
+	__u32 meas_type;
+	__u32 frame_id;
+	struct cifisp_stat params;
+} __attribute__ ((packed));
+
+#endif /* _UAPI_RKISP1_CONFIG_H */
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (2 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-07 13:05   ` Sakari Ailus
  2019-07-30 18:42 ` [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver Helen Koike
                   ` (11 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
[migrate to phy framework]
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8:
- Remove boiler plate license text

Changes in v7:
- Migrate dphy specific code from
drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
to drivers/phy/rockchip/phy-rockchip-dphy.c
- Drop support for rk3288
- Drop support for dphy txrx
- code styling and checkpatch fixes

 drivers/phy/rockchip/Kconfig             |   8 +
 drivers/phy/rockchip/Makefile            |   1 +
 drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
 3 files changed, 417 insertions(+)
 create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c

diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
index c454c90cd99e..afd072f135e6 100644
--- a/drivers/phy/rockchip/Kconfig
+++ b/drivers/phy/rockchip/Kconfig
@@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
 	help
 	  Enable this to support the Rockchip Display Port PHY.
 
+config PHY_ROCKCHIP_DPHY
+	tristate "Rockchip MIPI Synopsys DPHY driver"
+	depends on ARCH_ROCKCHIP && OF
+	select GENERIC_PHY_MIPI_DPHY
+	select GENERIC_PHY
+	help
+	  Enable this to support the Rockchip MIPI Synopsys DPHY.
+
 config PHY_ROCKCHIP_EMMC
 	tristate "Rockchip EMMC PHY Driver"
 	depends on ARCH_ROCKCHIP && OF
diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
index fd21cbaf40dd..f62e9010bcaf 100644
--- a/drivers/phy/rockchip/Makefile
+++ b/drivers/phy/rockchip/Makefile
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
+obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
 obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
 obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
 obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
new file mode 100644
index 000000000000..3a29976c2dff
--- /dev/null
+++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
@@ -0,0 +1,408 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Rockchip MIPI Synopsys DPHY driver
+ *
+ * Based on:
+ *
+ * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
+ * Author: Yakir Yang <ykk@@rock-chips.com>
+ */
+
+#include <linux/clk.h>
+#include <linux/io.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/phy/phy.h>
+#include <linux/phy/phy-mipi-dphy.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+
+#define RK3399_GRF_SOC_CON9	0x6224
+#define RK3399_GRF_SOC_CON21	0x6254
+#define RK3399_GRF_SOC_CON22	0x6258
+#define RK3399_GRF_SOC_CON23	0x625c
+#define RK3399_GRF_SOC_CON24	0x6260
+#define RK3399_GRF_SOC_CON25	0x6264
+#define RK3399_GRF_SOC_STATUS1	0xe2a4
+
+#define CLOCK_LANE_HS_RX_CONTROL		0x34
+#define LANE0_HS_RX_CONTROL			0x44
+#define LANE1_HS_RX_CONTROL			0x54
+#define LANE2_HS_RX_CONTROL			0x84
+#define LANE3_HS_RX_CONTROL			0x94
+#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
+
+#define MAX_DPHY_CLK 8
+
+#define PHY_TESTEN_ADDR			(0x1 << 16)
+#define PHY_TESTEN_DATA			(0x0 << 16)
+#define PHY_TESTCLK			(0x1 << 1)
+#define PHY_TESTCLR			(0x1 << 0)
+#define THS_SETTLE_COUNTER_THRESHOLD	0x04
+
+#define HIWORD_UPDATE(val, mask, shift) \
+	((val) << (shift) | (mask) << ((shift) + 16))
+
+#define GRF_SOC_CON12                           0x0274
+
+#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
+#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
+
+#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
+#define GRF_EDP_PHY_SIDDQ_ON                    0
+#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)
+
+struct hsfreq_range {
+	u32 range_h;
+	u8 cfg_bit;
+};
+
+static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
+	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
+	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
+	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
+	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
+	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
+	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
+	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
+	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
+	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
+	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}
+};
+
+static const char * const rk3399_mipidphy_clks[] = {
+	"dphy-ref",
+	"dphy-cfg",
+	"grf",
+};
+
+enum dphy_reg_id {
+	GRF_DPHY_RX0_TURNDISABLE = 0,
+	GRF_DPHY_RX0_FORCERXMODE,
+	GRF_DPHY_RX0_FORCETXSTOPMODE,
+	GRF_DPHY_RX0_ENABLE,
+	GRF_DPHY_RX0_TESTCLR,
+	GRF_DPHY_RX0_TESTCLK,
+	GRF_DPHY_RX0_TESTEN,
+	GRF_DPHY_RX0_TESTDIN,
+	GRF_DPHY_RX0_TURNREQUEST,
+	GRF_DPHY_RX0_TESTDOUT,
+	GRF_DPHY_TX0_TURNDISABLE,
+	GRF_DPHY_TX0_FORCERXMODE,
+	GRF_DPHY_TX0_FORCETXSTOPMODE,
+	GRF_DPHY_TX0_TURNREQUEST,
+	GRF_DPHY_TX1RX1_TURNDISABLE,
+	GRF_DPHY_TX1RX1_FORCERXMODE,
+	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
+	GRF_DPHY_TX1RX1_ENABLE,
+	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
+	GRF_DPHY_TX1RX1_BASEDIR,
+	GRF_DPHY_TX1RX1_ENABLECLK,
+	GRF_DPHY_TX1RX1_TURNREQUEST,
+	GRF_DPHY_RX1_SRC_SEL,
+	/* rk3288 only */
+	GRF_CON_DISABLE_ISP,
+	GRF_CON_ISP_DPHY_SEL,
+	GRF_DSI_CSI_TESTBUS_SEL,
+	GRF_DVP_V18SEL,
+	/* below is for rk3399 only */
+	GRF_DPHY_RX0_CLK_INV_SEL,
+	GRF_DPHY_RX1_CLK_INV_SEL,
+};
+
+struct dphy_reg {
+	u32 offset;
+	u32 mask;
+	u32 shift;
+};
+
+#define PHY_REG(_offset, _width, _shift) \
+	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
+
+static const struct dphy_reg rk3399_grf_dphy_regs[] = {
+	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
+	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
+	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
+	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
+	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
+	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
+	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
+	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
+	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
+	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
+	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
+	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
+	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
+	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
+	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
+	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
+	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
+	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
+	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
+	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
+	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
+	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
+	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
+	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
+	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),
+};
+
+struct dphy_drv_data {
+	const char * const *clks;
+	int num_clks;
+	const struct hsfreq_range *hsfreq_ranges;
+	int num_hsfreq_ranges;
+	const struct dphy_reg *regs;
+};
+
+struct rockchip_dphy {
+	struct device *dev;
+	struct regmap *grf;
+	const struct dphy_reg *grf_regs;
+	struct clk_bulk_data clks[MAX_DPHY_CLK];
+
+	const struct dphy_drv_data *drv_data;
+	struct phy_configure_opts_mipi_dphy config;
+};
+
+static inline void write_grf_reg(struct rockchip_dphy *priv,
+				 int index, u8 value)
+{
+	const struct dphy_reg *reg = &priv->grf_regs[index];
+	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
+
+	WARN_ON(!reg->offset);
+	regmap_write(priv->grf, reg->offset, val);
+}
+
+static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
+			     u8 test_code, u8 test_data)
+{
+	/*
+	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
+	 * is latched internally as the current test code. Test data is
+	 * programmed internally by rising edge on TESTCLK.
+	 */
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
+}
+
+/* should be move to power_on */
+static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
+{
+	const struct dphy_drv_data *drv_data = priv->drv_data;
+	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
+	struct phy_configure_opts_mipi_dphy *config = &priv->config;
+	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
+	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
+
+	do_div(data_rate_mbps, 1000 * 1000);
+
+	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
+		__func__, config->lanes, data_rate_mbps);
+
+	for (i = 0; i < num_hsfreq_ranges; i++) {
+		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
+			hsfreq = hsfreq_ranges[i].cfg_bit;
+			break;
+		}
+	}
+
+	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
+	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
+
+	/* Disable lan turn around, which is ignored in receive mode */
+	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
+	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
+
+	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
+
+	/* dphy start */
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
+	usleep_range(100, 150);
+	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
+	usleep_range(100, 150);
+
+	/* set clock lane */
+	/* HS hsfreq_range & lane 0  settle bypass */
+	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
+	/* HS RX Control of lane0 */
+	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
+	/* HS RX Control of lane1 */
+	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
+	/* HS RX Control of lane2 */
+	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
+	/* HS RX Control of lane3 */
+	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);
+	/* HS RX Data Lanes Settle State Time Control */
+	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
+			 THS_SETTLE_COUNTER_THRESHOLD);
+
+	/* Normal operation */
+	mipidphy0_wr_reg(priv, 0x0, 0);
+
+	return 0;
+}
+
+static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
+{
+	struct rockchip_dphy *priv = phy_get_drvdata(phy);
+	int ret;
+
+	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */
+	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
+	if (ret)
+		return ret;
+
+	memcpy(&priv->config, opts, sizeof(priv->config));
+
+	return 0;
+}
+
+static int rockchip_dphy_power_on(struct phy *phy)
+{
+	struct rockchip_dphy *priv = phy_get_drvdata(phy);
+	int ret;
+
+	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
+	if (ret)
+		return ret;
+
+	return mipidphy_rx_stream_on(priv);
+}
+
+static int rockchip_dphy_power_off(struct phy *phy)
+{
+	struct rockchip_dphy *priv = phy_get_drvdata(phy);
+
+	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
+	return 0;
+}
+
+static int rockchip_dphy_init(struct phy *phy)
+{
+	struct rockchip_dphy *priv = phy_get_drvdata(phy);
+	int ret;
+
+	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);
+	if (ret)
+		return ret;
+	return 0;
+}
+
+static int rockchip_dphy_exit(struct phy *phy)
+{
+	struct rockchip_dphy *priv = phy_get_drvdata(phy);
+
+	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
+	return 0;
+}
+
+static const struct phy_ops rockchip_dphy_ops = {
+	.power_on	= rockchip_dphy_power_on,
+	.power_off	= rockchip_dphy_power_off,
+	.init		= rockchip_dphy_init,
+	.exit		= rockchip_dphy_exit,
+	.configure	= rockchip_dphy_configure,
+	.owner		= THIS_MODULE,
+};
+
+static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
+	.clks = rk3399_mipidphy_clks,
+	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
+	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
+	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
+	.regs = rk3399_grf_dphy_regs,
+};
+
+static const struct of_device_id rockchip_dphy_dt_ids[] = {
+	{
+		.compatible = "rockchip,rk3399-mipi-dphy",
+		.data = &rk3399_mipidphy_drv_data,
+	},
+	{}
+};
+MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
+
+static int rockchip_dphy_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct device_node *np = dev->of_node;
+	const struct dphy_drv_data *drv_data;
+	struct phy_provider *phy_provider;
+	const struct of_device_id *of_id;
+	struct rockchip_dphy *priv;
+	struct regmap *grf;
+	struct phy *phy;
+	unsigned int i;
+	int ret;
+
+	if (!dev->parent || !dev->parent->of_node)
+		return -ENODEV;
+
+	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
+		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");
+		return -EINVAL;
+	}
+
+	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+	priv->dev = dev;
+
+	grf = syscon_node_to_regmap(dev->parent->of_node);
+	if (IS_ERR(grf)) {
+		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
+						      "rockchip,grf");
+		if (IS_ERR(grf)) {
+			dev_err(dev, "Can't find GRF syscon\n");
+			return -ENODEV;
+		}
+	}
+	priv->grf = grf;
+
+	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
+	if (!of_id)
+		return -EINVAL;
+
+	drv_data = of_id->data;
+	priv->grf_regs = drv_data->regs;
+	priv->drv_data = drv_data;
+	for (i = 0; i < drv_data->num_clks; i++)
+		priv->clks[i].id = drv_data->clks[i];
+	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
+	if (ret)
+		return ret;
+
+	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
+	if (IS_ERR(phy)) {
+		dev_err(dev, "failed to create phy\n");
+		return PTR_ERR(phy);
+	}
+	phy_set_drvdata(phy, priv);
+
+	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+	return PTR_ERR_OR_ZERO(phy_provider);
+}
+
+static struct platform_driver rockchip_dphy_driver = {
+	.probe = rockchip_dphy_probe,
+	.driver = {
+		.name	= "rockchip-mipi-dphy",
+		.of_match_table = rockchip_dphy_dt_ids,
+	},
+};
+module_platform_driver(rockchip_dphy_driver);
+
+MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
+MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
+MODULE_LICENSE("Dual MIT/GPL");
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (3 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-06 18:51   ` Helen Koike
                     ` (2 more replies)
  2019-07-30 18:42 ` [PATCH v8 06/14] media: rkisp1: add ISP1 statistics driver Helen Koike
                   ` (10 subsequent siblings)
  15 siblings, 3 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Allon Huang, Jacob Chen,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, Helen Koike, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

Add the subdev driver for rockchip isp1.

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
Signed-off-by: Jacob Chen <cc@rock-chips.com>
Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
[fixed unknown entity type / switched to PIXEL_RATE]
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- fixed warning because of unknown entity type
- fixed v4l2-compliance errors regarding rkisp1 formats, try formats
and default values
- fix typo riksp1/rkisp1
- redesign: remove mipi/csi subdevice, sensors connect directly to the
isp subdevice in the media topology now. As a consequence, remove the
hack in mipidphy_g_mbus_config() where information from the sensor was
being propagated through the topology.
- From the old dphy:
        * cache get_remote_sensor() in s_stream
        * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
- Replace stream state with a boolean
- code styling and checkpatch fixes
- fix stop_stream (return after calling stop, do not reenable the stream)
- fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
- fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
- s/intput/input
- remove #define sd_to_isp_sd(_sd), add a static inline as it will be
reused by the capture

 drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
 drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
 2 files changed, 1397 insertions(+)
 create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
 create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h

diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
new file mode 100644
index 000000000000..6d0c0ffb5e03
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
@@ -0,0 +1,1286 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#include <linux/iopoll.h>
+#include <linux/phy/phy.h>
+#include <linux/phy/phy-mipi-dphy.h>
+#include <linux/pm_runtime.h>
+#include <linux/videodev2.h>
+#include <linux/vmalloc.h>
+#include <media/v4l2-event.h>
+
+#include "common.h"
+#include "regs.h"
+
+#define CIF_ISP_INPUT_W_MAX		4032
+#define CIF_ISP_INPUT_H_MAX		3024
+#define CIF_ISP_INPUT_W_MIN		32
+#define CIF_ISP_INPUT_H_MIN		32
+#define CIF_ISP_OUTPUT_W_MAX		CIF_ISP_INPUT_W_MAX
+#define CIF_ISP_OUTPUT_H_MAX		CIF_ISP_INPUT_H_MAX
+#define CIF_ISP_OUTPUT_W_MIN		CIF_ISP_INPUT_W_MIN
+#define CIF_ISP_OUTPUT_H_MIN		CIF_ISP_INPUT_H_MIN
+
+/*
+ * NOTE: MIPI controller and input MUX are also configured in this file,
+ * because ISP Subdev is not only describe ISP submodule(input size,format,
+ * output size, format), but also a virtual route device.
+ */
+
+/*
+ * There are many variables named with format/frame in below code,
+ * please see here for their meaning.
+ *
+ * Cropping regions of ISP
+ *
+ * +---------------------------------------------------------+
+ * | Sensor image                                            |
+ * | +---------------------------------------------------+   |
+ * | | ISP_ACQ (for black level)                         |   |
+ * | | in_frm                                            |   |
+ * | | +--------------------------------------------+    |   |
+ * | | |    ISP_OUT                                 |    |   |
+ * | | |    in_crop                                 |    |   |
+ * | | |    +---------------------------------+     |    |   |
+ * | | |    |   ISP_IS                        |     |    |   |
+ * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
+ * | | |    +---------------------------------+     |    |   |
+ * | | +--------------------------------------------+    |   |
+ * | +---------------------------------------------------+   |
+ * +---------------------------------------------------------+
+ */
+
+static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
+{
+	return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
+}
+
+/* Get sensor by enabled media link */
+static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
+{
+	struct media_pad *local, *remote;
+	struct media_entity *sensor_me;
+
+	local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
+	remote = media_entity_remote_pad(local);
+	if (!remote) {
+		v4l2_warn(sd, "No link between isp and sensor\n");
+		return NULL;
+	}
+
+	sensor_me = media_entity_remote_pad(local)->entity;
+	return media_entity_to_v4l2_subdev(sensor_me);
+}
+
+static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
+					       struct v4l2_subdev *sd)
+{
+	struct rkisp1_sensor *sensor;
+
+	list_for_each_entry(sensor, &dev->sensors, list)
+		if (sensor->sd == sd)
+			return sensor;
+
+	return NULL;
+}
+
+/****************  register operations ****************/
+
+/*
+ * Image Stabilization.
+ * This should only be called when configuring CIF
+ * or at the frame end interrupt
+ */
+static void rkisp1_config_ism(struct rkisp1_device *dev)
+{
+	void __iomem *base = dev->base_addr;
+	struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
+	u32 val;
+
+	writel(0, base + CIF_ISP_IS_RECENTER);
+	writel(0, base + CIF_ISP_IS_MAX_DX);
+	writel(0, base + CIF_ISP_IS_MAX_DY);
+	writel(0, base + CIF_ISP_IS_DISPLACE);
+	writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
+	writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
+	writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
+	writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
+
+	/* IS(Image Stabilization) is always on, working as output crop */
+	writel(1, base + CIF_ISP_IS_CTRL);
+	val = readl(base + CIF_ISP_CTRL);
+	val |= CIF_ISP_CTRL_ISP_CFG_UPD;
+	writel(val, base + CIF_ISP_CTRL);
+}
+
+/*
+ * configure isp blocks with input format, size......
+ */
+static int rkisp1_config_isp(struct rkisp1_device *dev)
+{
+	u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
+	struct v4l2_rect *out_crop, *in_crop;
+	void __iomem *base = dev->base_addr;
+	struct v4l2_mbus_framefmt *in_frm;
+	struct ispsd_out_fmt *out_fmt;
+	struct rkisp1_sensor *sensor;
+	struct ispsd_in_fmt *in_fmt;
+
+	sensor = dev->active_sensor;
+	in_frm = &dev->isp_sdev.in_frm;
+	in_fmt = &dev->isp_sdev.in_fmt;
+	out_fmt = &dev->isp_sdev.out_fmt;
+	out_crop = &dev->isp_sdev.out_crop;
+	in_crop = &dev->isp_sdev.in_crop;
+
+	if (in_fmt->fmt_type == FMT_BAYER) {
+		acq_mult = 1;
+		if (out_fmt->fmt_type == FMT_BAYER) {
+			if (sensor->mbus.type == V4L2_MBUS_BT656)
+				isp_ctrl =
+					CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
+			else
+				isp_ctrl =
+					CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
+		} else {
+			writel(CIF_ISP_DEMOSAIC_TH(0xc),
+			       base + CIF_ISP_DEMOSAIC);
+
+			if (sensor->mbus.type == V4L2_MBUS_BT656)
+				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
+			else
+				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
+		}
+	} else if (in_fmt->fmt_type == FMT_YUV) {
+		acq_mult = 2;
+		if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
+			isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
+		} else {
+			if (sensor->mbus.type == V4L2_MBUS_BT656)
+				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
+			else
+				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
+
+		}
+
+		irq_mask |= CIF_ISP_DATA_LOSS;
+	}
+
+	/* Set up input acquisition properties */
+	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
+	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
+		if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
+			signal = CIF_ISP_ACQ_PROP_POS_EDGE;
+	}
+
+	if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
+		if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
+			signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
+
+		if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
+			signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
+	}
+
+	writel(isp_ctrl, base + CIF_ISP_CTRL);
+	writel(signal | in_fmt->yuv_seq |
+	       CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
+	       CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
+	writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
+
+	/* Acquisition Size */
+	writel(0, base + CIF_ISP_ACQ_H_OFFS);
+	writel(0, base + CIF_ISP_ACQ_V_OFFS);
+	writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
+	writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
+
+	/* ISP Out Area */
+	writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
+	writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
+	writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
+	writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
+
+	/* interrupt mask */
+	irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
+		    CIF_ISP_FRAME_IN;
+	writel(irq_mask, base + CIF_ISP_IMSC);
+
+	if (out_fmt->fmt_type == FMT_BAYER)
+		rkisp1_params_disable_isp(&dev->params_vdev);
+	else
+		rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
+					    dev->isp_sdev.quantization);
+
+	return 0;
+}
+
+static int rkisp1_config_dvp(struct rkisp1_device *dev)
+{
+	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
+	void __iomem *base = dev->base_addr;
+	u32 val, input_sel;
+
+	switch (in_fmt->bus_width) {
+	case 8:
+		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
+		break;
+	case 10:
+		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
+		break;
+	case 12:
+		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
+		break;
+	default:
+		v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
+		return -EINVAL;
+	}
+
+	val = readl(base + CIF_ISP_ACQ_PROP);
+	writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
+
+	return 0;
+}
+
+static int rkisp1_config_mipi(struct rkisp1_device *dev)
+{
+	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
+	struct rkisp1_sensor *sensor = dev->active_sensor;
+	void __iomem *base = dev->base_addr;
+	unsigned int lanes;
+	u32 mipi_ctrl;
+
+	/*
+	 * sensor->mbus is set in isp or d-phy notifier_bound function
+	 */
+	switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
+	case V4L2_MBUS_CSI2_4_LANE:
+		lanes = 4;
+		break;
+	case V4L2_MBUS_CSI2_3_LANE:
+		lanes = 3;
+		break;
+	case V4L2_MBUS_CSI2_2_LANE:
+		lanes = 2;
+		break;
+	case V4L2_MBUS_CSI2_1_LANE:
+		lanes = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
+		    CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
+		    CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
+		    CIF_MIPI_CTRL_CLOCKLANE_ENA;
+
+	writel(mipi_ctrl, base + CIF_MIPI_CTRL);
+
+	/* Configure Data Type and Virtual Channel */
+	writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
+	       base + CIF_MIPI_IMG_DATA_SEL);
+
+	/* Clear MIPI interrupts */
+	writel(~0, base + CIF_MIPI_ICR);
+	/*
+	 * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
+	 * isp bus may be dead when switch isp.
+	 */
+	writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
+	       CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
+	       base + CIF_MIPI_IMSC);
+
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
+		 "  MIPI_IMG_DATA_SEL 0x%08x\n"
+		 "  MIPI_STATUS 0x%08x\n"
+		 "  MIPI_IMSC 0x%08x\n",
+		 readl(base + CIF_MIPI_CTRL),
+		 readl(base + CIF_MIPI_IMG_DATA_SEL),
+		 readl(base + CIF_MIPI_STATUS),
+		 readl(base + CIF_MIPI_IMSC));
+
+	return 0;
+}
+
+/* Configure MUX */
+static int rkisp1_config_path(struct rkisp1_device *dev)
+{
+	struct rkisp1_sensor *sensor = dev->active_sensor;
+	u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
+	int ret = 0;
+
+	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
+	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
+		ret = rkisp1_config_dvp(dev);
+		dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
+	} else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
+		ret = rkisp1_config_mipi(dev);
+		dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
+	}
+
+	writel(dpcl, dev->base_addr + CIF_VI_DPCL);
+
+	return ret;
+}
+
+/* Hareware configure Entry */
+static int rkisp1_config_cif(struct rkisp1_device *dev)
+{
+	u32 cif_id;
+	int ret;
+
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
+		 "SP streaming = %d, MP streaming = %d\n",
+		 dev->stream[RKISP1_STREAM_SP].streaming,
+		 dev->stream[RKISP1_STREAM_MP].streaming);
+
+	cif_id = readl(dev->base_addr + CIF_VI_ID);
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
+
+	ret = rkisp1_config_isp(dev);
+	if (ret < 0)
+		return ret;
+	ret = rkisp1_config_path(dev);
+	if (ret < 0)
+		return ret;
+	rkisp1_config_ism(dev);
+
+	return 0;
+}
+
+/* Mess register operations to stop isp */
+static int rkisp1_isp_stop(struct rkisp1_device *dev)
+{
+	void __iomem *base = dev->base_addr;
+	u32 val;
+
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
+		 "SP streaming = %d, MP streaming = %d\n",
+		 dev->stream[RKISP1_STREAM_SP].streaming,
+		 dev->stream[RKISP1_STREAM_MP].streaming);
+
+	/*
+	 * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
+	 * Stop ISP(isp) ->wait for ISP isp off
+	 */
+	/* stop and clear MI, MIPI, and ISP interrupts */
+	writel(0, base + CIF_MIPI_IMSC);
+	writel(~0, base + CIF_MIPI_ICR);
+
+	writel(0, base + CIF_ISP_IMSC);
+	writel(~0, base + CIF_ISP_ICR);
+
+	writel(0, base + CIF_MI_IMSC);
+	writel(~0, base + CIF_MI_ICR);
+	val = readl(base + CIF_MIPI_CTRL);
+	writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
+	/* stop ISP */
+	val = readl(base + CIF_ISP_CTRL);
+	val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
+	writel(val, base + CIF_ISP_CTRL);
+
+	val = readl(base + CIF_ISP_CTRL);
+	writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
+
+	readx_poll_timeout(readl, base + CIF_ISP_RIS,
+			   val, val & CIF_ISP_OFF, 20, 100);
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
+		"streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
+		 dev->stream[RKISP1_STREAM_SP].streaming,
+		 dev->stream[RKISP1_STREAM_MP].streaming,
+		 readl(base + CIF_MI_CTRL),
+		 readl(base + CIF_ISP_CTRL),
+		 readl(base + CIF_MIPI_CTRL));
+
+	writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
+	writel(0x0, base + CIF_IRCL);
+
+	return 0;
+}
+
+/* Mess register operations to start isp */
+static int rkisp1_isp_start(struct rkisp1_device *dev)
+{
+	struct rkisp1_sensor *sensor = dev->active_sensor;
+	void __iomem *base = dev->base_addr;
+	u32 val;
+
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
+		 "SP streaming = %d, MP streaming = %d\n",
+		 dev->stream[RKISP1_STREAM_SP].streaming,
+		 dev->stream[RKISP1_STREAM_MP].streaming);
+
+	/* Activate MIPI */
+	if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
+		val = readl(base + CIF_MIPI_CTRL);
+		writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
+	}
+	/* Activate ISP */
+	val = readl(base + CIF_ISP_CTRL);
+	val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
+	       CIF_ISP_CTRL_ISP_INFORM_ENABLE;
+	writel(val, base + CIF_ISP_CTRL);
+
+	/* XXX: Is the 1000us too long?
+	 * CIF spec says to wait for sufficient time after enabling
+	 * the MIPI interface and before starting the sensor output.
+	 */
+	usleep_range(1000, 1200);
+
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
+		 "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
+		 "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
+		 dev->stream[RKISP1_STREAM_SP].streaming,
+		 dev->stream[RKISP1_STREAM_MP].streaming,
+		 readl(base + CIF_MI_CTRL),
+		 readl(base + CIF_ISP_CTRL),
+		 readl(base + CIF_MIPI_CTRL));
+
+	return 0;
+}
+
+static void rkisp1_config_clk(struct rkisp1_device *dev)
+{
+	u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
+		  CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
+		  CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
+
+	writel(val, dev->base_addr + CIF_ICCL);
+}
+
+/***************************** isp sub-devs *******************************/
+
+static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
+	{
+		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW10,
+		.bayer_pat	= RAW_BGGR,
+		.bus_width	= 10,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW10,
+		.bayer_pat	= RAW_RGGB,
+		.bus_width	= 10,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW10,
+		.bayer_pat	= RAW_GBRG,
+		.bus_width	= 10,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW10,
+		.bayer_pat	= RAW_GRBG,
+		.bus_width	= 10,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW12,
+		.bayer_pat	= RAW_RGGB,
+		.bus_width	= 12,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW12,
+		.bayer_pat	= RAW_BGGR,
+		.bus_width	= 12,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW12,
+		.bayer_pat	= RAW_GBRG,
+		.bus_width	= 12,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW12,
+		.bayer_pat	= RAW_GRBG,
+		.bus_width	= 12,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW8,
+		.bayer_pat	= RAW_RGGB,
+		.bus_width	= 8,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW8,
+		.bayer_pat	= RAW_BGGR,
+		.bus_width	= 8,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW8,
+		.bayer_pat	= RAW_GBRG,
+		.bus_width	= 8,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
+		.fmt_type	= FMT_BAYER,
+		.mipi_dt	= CIF_CSI2_DT_RAW8,
+		.bayer_pat	= RAW_GRBG,
+		.bus_width	= 8,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
+		.fmt_type	= FMT_YUV,
+		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
+		.yuv_seq	= CIF_ISP_ACQ_PROP_YCBYCR,
+		.bus_width	= 16,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_YVYU8_1X16,
+		.fmt_type	= FMT_YUV,
+		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
+		.yuv_seq	= CIF_ISP_ACQ_PROP_YCRYCB,
+		.bus_width	= 16,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_UYVY8_1X16,
+		.fmt_type	= FMT_YUV,
+		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
+		.yuv_seq	= CIF_ISP_ACQ_PROP_CBYCRY,
+		.bus_width	= 16,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_VYUY8_1X16,
+		.fmt_type	= FMT_YUV,
+		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
+		.yuv_seq	= CIF_ISP_ACQ_PROP_CRYCBY,
+		.bus_width	= 16,
+	},
+};
+
+static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
+	{
+		.mbus_code	= MEDIA_BUS_FMT_YUYV8_2X8,
+		.fmt_type	= FMT_YUV,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
+		.fmt_type	= FMT_BAYER,
+	}, {
+		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
+		.fmt_type	= FMT_BAYER,
+	},
+};
+
+static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
+{
+	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
+	const struct ispsd_in_fmt *fmt;
+
+	for (i = 0; i < array_size; i++) {
+		fmt = &rkisp1_isp_input_formats[i];
+		if (fmt->mbus_code == mbus_code)
+			return fmt;
+	}
+
+	return NULL;
+}
+
+static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
+{
+	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
+	const struct ispsd_out_fmt *fmt;
+
+	for (i = 0; i < array_size; i++) {
+		fmt = &rkisp1_isp_output_formats[i];
+		if (fmt->mbus_code == mbus_code)
+			return fmt;
+	}
+
+	return NULL;
+}
+
+static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
+					struct v4l2_subdev_pad_config *cfg,
+					struct v4l2_subdev_mbus_code_enum *code)
+{
+	unsigned int i = code->index;
+
+	if ((code->pad != RKISP1_ISP_PAD_SINK) &&
+	    (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
+		if (i > 0)
+			return -EINVAL;
+		code->code = MEDIA_BUS_FMT_FIXED;
+		return 0;
+	}
+
+	if (code->pad == RKISP1_ISP_PAD_SINK) {
+		if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
+			return -EINVAL;
+		code->code = rkisp1_isp_input_formats[i].mbus_code;
+	} else {
+		if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
+			return -EINVAL;
+		code->code = rkisp1_isp_output_formats[i].mbus_code;
+	}
+
+	return 0;
+}
+
+static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_pad_config *cfg)
+{
+	struct v4l2_rect *mf_in_crop, *mf_out_crop;
+	struct v4l2_mbus_framefmt *mf_in, *mf_out;
+
+	mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
+	mf_in->width = RKISP1_DEFAULT_WIDTH;
+	mf_in->height = RKISP1_DEFAULT_HEIGHT;
+	mf_in->field = V4L2_FIELD_NONE;
+	mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
+
+	mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
+	mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
+	mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
+	mf_in_crop->left = 0;
+	mf_in_crop->top = 0;
+
+	mf_out = v4l2_subdev_get_try_format(sd, cfg,
+					    RKISP1_ISP_PAD_SOURCE_PATH);
+	*mf_out = *mf_in;
+	mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
+	mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+
+	mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
+					       RKISP1_ISP_PAD_SOURCE_PATH);
+	*mf_out_crop = *mf_in_crop;
+
+	return 0;
+}
+
+static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_pad_config *cfg,
+				 struct v4l2_subdev_format *fmt)
+{
+	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
+	struct v4l2_mbus_framefmt *mf = &fmt->format;
+
+	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
+	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
+		fmt->format.code = MEDIA_BUS_FMT_FIXED;
+		/*
+		 * NOTE: setting a format here doesn't make much sense
+		 * but v4l2-compliance complains
+		 */
+		fmt->format.width = RKISP1_DEFAULT_WIDTH;
+		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
+		fmt->format.field = V4L2_FIELD_NONE;
+		return 0;
+	}
+
+	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
+		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
+		fmt->format = *mf;
+		return 0;
+	}
+
+	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
+		*mf = isp_sd->in_frm;
+	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
+		/* format of source pad */
+		*mf = isp_sd->in_frm;
+		mf->code = isp_sd->out_fmt.mbus_code;
+		/* window size of source pad */
+		mf->width = isp_sd->out_crop.width;
+		mf->height = isp_sd->out_crop.height;
+		mf->quantization = isp_sd->quantization;
+	}
+	mf->field = V4L2_FIELD_NONE;
+
+	return 0;
+}
+
+static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
+				  unsigned int pad,
+				  struct v4l2_mbus_framefmt *fmt)
+{
+	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
+	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
+	const struct ispsd_out_fmt *out_fmt;
+	const struct ispsd_in_fmt *in_fmt;
+
+	switch (pad) {
+	case RKISP1_ISP_PAD_SINK:
+		in_fmt = find_in_fmt(fmt->code);
+		if (in_fmt)
+			fmt->code = in_fmt->mbus_code;
+		else
+			fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
+		fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
+				      CIF_ISP_INPUT_W_MAX);
+		fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
+				      CIF_ISP_INPUT_H_MAX);
+		break;
+	case RKISP1_ISP_PAD_SOURCE_PATH:
+		out_fmt = find_out_fmt(fmt->code);
+		if (out_fmt)
+			fmt->code = out_fmt->mbus_code;
+		else
+			fmt->code = rkisp1_isp_output_formats[0].mbus_code;
+		/* window size is set in s_selection */
+		fmt->width  = isp_sd->out_crop.width;
+		fmt->height = isp_sd->out_crop.height;
+		/* full range by default */
+		if (!fmt->quantization)
+			fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+		break;
+	}
+
+	fmt->field = V4L2_FIELD_NONE;
+}
+
+static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_pad_config *cfg,
+				 struct v4l2_subdev_format *fmt)
+{
+	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
+	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
+	struct v4l2_mbus_framefmt *mf = &fmt->format;
+
+	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
+	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
+		return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
+
+	rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
+
+	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
+		struct v4l2_mbus_framefmt *try_mf;
+
+		try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
+		*try_mf = *mf;
+		return 0;
+	}
+
+	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
+		const struct ispsd_in_fmt *in_fmt;
+
+		in_fmt = find_in_fmt(mf->code);
+		isp_sd->in_fmt = *in_fmt;
+		isp_sd->in_frm = *mf;
+	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
+		const struct ispsd_out_fmt *out_fmt;
+
+		/* Ignore width/height */
+		out_fmt = find_out_fmt(mf->code);
+		isp_sd->out_fmt = *out_fmt;
+		/*
+		 * It is quantization for output,
+		 * isp use bt601 limit-range in internal
+		 */
+		isp_sd->quantization = mf->quantization;
+	}
+
+	return 0;
+}
+
+static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_pad_config *cfg,
+				  struct v4l2_subdev_selection *sel)
+{
+	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
+	struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
+	struct v4l2_rect in_crop = isp_sd->in_crop;
+	struct v4l2_rect *input = &sel->r;
+
+	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
+		in_frm = *v4l2_subdev_get_try_format(sd, cfg,
+						     RKISP1_ISP_PAD_SINK);
+		in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
+						    RKISP1_ISP_PAD_SINK);
+	}
+
+	input->left = ALIGN(input->left, 2);
+	input->width = ALIGN(input->width, 2);
+
+	if (sel->pad == RKISP1_ISP_PAD_SINK) {
+		input->left = clamp_t(u32, input->left, 0, in_frm.width);
+		input->top = clamp_t(u32, input->top, 0, in_frm.height);
+		input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
+				       in_frm.width - input->left);
+		input->height = clamp_t(u32, input->height,
+					CIF_ISP_INPUT_H_MIN,
+					in_frm.height - input->top);
+	} else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
+		input->left = clamp_t(u32, input->left, 0, in_crop.width);
+		input->top = clamp_t(u32, input->top, 0, in_crop.height);
+		input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
+				       in_crop.width - input->left);
+		input->height = clamp_t(u32, input->height,
+					CIF_ISP_OUTPUT_H_MIN,
+					in_crop.height - input->top);
+	}
+}
+
+static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
+				       struct v4l2_subdev_pad_config *cfg,
+				       struct v4l2_subdev_selection *sel)
+{
+	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
+	struct v4l2_mbus_framefmt *frm;
+	struct v4l2_rect *rect;
+
+	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
+	    sel->pad != RKISP1_ISP_PAD_SINK)
+		return -EINVAL;
+
+	switch (sel->target) {
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		if (sel->pad == RKISP1_ISP_PAD_SINK) {
+			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+				frm = v4l2_subdev_get_try_format(sd, cfg,
+								 sel->pad);
+			else
+				frm = &isp_sd->in_frm;
+
+			sel->r.height = frm->height;
+			sel->r.width = frm->width;
+			sel->r.left = 0;
+			sel->r.top = 0;
+		} else {
+			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+				rect = v4l2_subdev_get_try_crop(sd, cfg,
+							RKISP1_ISP_PAD_SINK);
+			else
+				rect = &isp_sd->in_crop;
+			sel->r = *rect;
+		}
+		break;
+	case V4L2_SEL_TGT_CROP:
+		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+			rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
+		else if (sel->pad == RKISP1_ISP_PAD_SINK)
+			rect = &isp_sd->in_crop;
+		else
+			rect = &isp_sd->out_crop;
+		sel->r = *rect;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
+				       struct v4l2_subdev_pad_config *cfg,
+				       struct v4l2_subdev_selection *sel)
+{
+	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
+	struct rkisp1_device *dev = sd_to_isp_dev(sd);
+
+	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
+	    sel->pad != RKISP1_ISP_PAD_SINK)
+		return -EINVAL;
+	if (sel->target != V4L2_SEL_TGT_CROP)
+		return -EINVAL;
+
+	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
+		 "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
+		 sel->r.left, sel->r.top, sel->r.width, sel->r.height);
+	rkisp1_isp_sd_try_crop(sd, cfg, sel);
+
+	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
+		struct v4l2_rect *try_sel =
+			v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
+
+		*try_sel = sel->r;
+		return 0;
+	}
+
+	if (sel->pad == RKISP1_ISP_PAD_SINK)
+		isp_sd->in_crop = sel->r;
+	else
+		isp_sd->out_crop = sel->r;
+
+	return 0;
+}
+
+static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
+				    struct rkisp1_sensor *sensor)
+{
+	union phy_configure_opts opts = { 0 };
+	struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
+	struct v4l2_ctrl *pixel_rate;
+	s64 pixel_clock;
+
+	pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
+				    V4L2_CID_PIXEL_RATE);
+	if (!pixel_rate) {
+		v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
+		return -EPIPE;
+	}
+
+	pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
+	if (!pixel_clock) {
+		v4l2_err(sensor->sd, "Invalid pixel rate value\n");
+		return -EINVAL;
+	}
+
+	phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
+					 sensor->lanes, cfg);
+	phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
+	phy_configure(sensor->dphy, &opts);
+	phy_power_on(sensor->dphy);
+
+	return 0;
+}
+
+static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
+{
+	phy_power_off(sensor->dphy);
+}
+
+static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
+{
+	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
+	struct v4l2_subdev *sensor_sd;
+	int ret = 0;
+
+	if (!on) {
+		ret = rkisp1_isp_stop(isp_dev);
+		if (ret < 0)
+			return ret;
+		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
+		return 0;
+	}
+
+	sensor_sd = get_remote_sensor(sd);
+	if (!sensor_sd)
+		return -ENODEV;
+
+	isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
+	if (!isp_dev->active_sensor)
+		return -ENODEV;
+
+	atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
+	ret = rkisp1_config_cif(isp_dev);
+	if (ret < 0)
+		return ret;
+
+	/* TODO: support other interfaces */
+	if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
+		return -EINVAL;
+
+	ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
+				       isp_dev->active_sensor);
+	if (ret < 0)
+		return ret;
+
+	ret = rkisp1_isp_start(isp_dev);
+	if (ret)
+		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
+
+	return ret;
+}
+
+static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
+	int ret;
+
+	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
+
+	if (on) {
+		ret = pm_runtime_get_sync(isp_dev->dev);
+		if (ret < 0)
+			return ret;
+
+		rkisp1_config_clk(isp_dev);
+	} else {
+		ret = pm_runtime_put(isp_dev->dev);
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int rkisp1_subdev_link_validate(struct media_link *link)
+{
+	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
+		return 0;
+
+	return v4l2_subdev_link_validate(link);
+}
+
+static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
+					struct media_link *link,
+					struct v4l2_subdev_format *source_fmt,
+					struct v4l2_subdev_format *sink_fmt)
+{
+	if (source_fmt->format.code != sink_fmt->format.code)
+		return -EINVAL;
+
+	/* Crop is available */
+	if (source_fmt->format.width < sink_fmt->format.width ||
+	    source_fmt->format.height < sink_fmt->format.height)
+		return -EINVAL;
+
+	return 0;
+}
+
+static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
+{
+	struct v4l2_event event = {
+		.type = V4L2_EVENT_FRAME_SYNC,
+		.u.frame_sync.frame_sequence =
+			atomic_inc_return(&isp->frm_sync_seq) - 1,
+	};
+	v4l2_event_queue(isp->sd.devnode, &event);
+}
+
+static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
+				  struct v4l2_event_subscription *sub)
+{
+	if (sub->type != V4L2_EVENT_FRAME_SYNC)
+		return -EINVAL;
+
+	/* Line number. For now only zero accepted. */
+	if (sub->id != 0)
+		return -EINVAL;
+
+	return v4l2_event_subscribe(fh, sub, 0, NULL);
+}
+
+static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
+	.enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
+	.get_selection = rkisp1_isp_sd_get_selection,
+	.set_selection = rkisp1_isp_sd_set_selection,
+	.init_cfg = rkisp1_isp_sd_init_config,
+	.get_fmt = rkisp1_isp_sd_get_fmt,
+	.set_fmt = rkisp1_isp_sd_set_fmt,
+	.link_validate = rkisp1_subdev_fmt_link_validate,
+};
+
+static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
+	.link_validate = rkisp1_subdev_link_validate,
+};
+
+static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
+	.s_stream = rkisp1_isp_sd_s_stream,
+};
+
+static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
+	.subscribe_event = rkisp1_isp_sd_subs_evt,
+	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
+	.s_power = rkisp1_isp_sd_s_power,
+};
+
+static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {
+	.core = &rkisp1_isp_core_ops,
+	.video = &rkisp1_isp_sd_video_ops,
+	.pad = &rkisp1_isp_sd_pad_ops,
+};
+
+static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
+{
+	struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
+	struct v4l2_rect *in_crop = &isp_sd->in_crop;
+	struct v4l2_rect *out_crop = &isp_sd->out_crop;
+	struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
+	struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
+
+	*in_fmt = rkisp1_isp_input_formats[0];
+	in_frm->width = RKISP1_DEFAULT_WIDTH;
+	in_frm->height = RKISP1_DEFAULT_HEIGHT;
+	in_frm->code = in_fmt->mbus_code;
+
+	in_crop->width = in_frm->width;
+	in_crop->height = in_frm->height;
+	in_crop->left = 0;
+	in_crop->top = 0;
+
+	/* propagate to source */
+	*out_crop = *in_crop;
+	*out_fmt = rkisp1_isp_output_formats[0];
+	isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+}
+
+int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
+			       struct v4l2_device *v4l2_dev)
+{
+	struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
+	struct v4l2_subdev *sd = &isp_sdev->sd;
+	int ret;
+
+	v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
+	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	sd->entity.ops = &rkisp1_isp_sd_media_ops;
+	snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
+
+	isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
+	isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
+	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
+	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
+	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
+				     isp_sdev->pads);
+	if (ret < 0)
+		return ret;
+
+	sd->owner = THIS_MODULE;
+	v4l2_set_subdevdata(sd, isp_dev);
+
+	sd->grp_id = GRP_ID_ISP;
+	ret = v4l2_device_register_subdev(v4l2_dev, sd);
+	if (ret < 0) {
+		v4l2_err(sd, "Failed to register isp subdev\n");
+		goto err_cleanup_media_entity;
+	}
+
+	rkisp1_isp_sd_init_default_fmt(isp_sdev);
+
+	return 0;
+err_cleanup_media_entity:
+	media_entity_cleanup(&sd->entity);
+	return ret;
+}
+
+void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
+{
+	struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
+
+	v4l2_device_unregister_subdev(sd);
+	media_entity_cleanup(&sd->entity);
+}
+
+/****************  Interrupter Handler ****************/
+
+void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
+{
+	struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
+	void __iomem *base = dev->base_addr;
+	u32 val;
+
+	writel(~0, base + CIF_MIPI_ICR);
+
+	/*
+	 * Disable DPHY errctrl interrupt, because this dphy
+	 * erctrl signal is asserted until the next changes
+	 * of line state. This time is may be too long and cpu
+	 * is hold in this interrupt.
+	 */
+	if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
+		val = readl(base + CIF_MIPI_IMSC);
+		writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
+		dev->isp_sdev.dphy_errctrl_disabled = true;
+	}
+
+	/*
+	 * Enable DPHY errctrl interrupt again, if mipi have receive
+	 * the whole frame without any error.
+	 */
+	if (mis == CIF_MIPI_FRAME_END) {
+		/*
+		 * Enable DPHY errctrl interrupt again, if mipi have receive
+		 * the whole frame without any error.
+		 */
+		if (dev->isp_sdev.dphy_errctrl_disabled) {
+			val = readl(base + CIF_MIPI_IMSC);
+			val |= CIF_MIPI_ERR_CTRL(0x0f);
+			writel(val, base + CIF_MIPI_IMSC);
+			dev->isp_sdev.dphy_errctrl_disabled = false;
+		}
+	} else {
+		v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
+	}
+}
+
+void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
+{
+	void __iomem *base = dev->base_addr;
+	unsigned int isp_mis_tmp = 0;
+	unsigned int isp_err = 0;
+
+	/* start edge of v_sync */
+	if (isp_mis & CIF_ISP_V_START) {
+		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
+
+		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
+		isp_mis_tmp = readl(base + CIF_ISP_MIS);
+		if (isp_mis_tmp & CIF_ISP_V_START)
+			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
+				 isp_mis_tmp);
+	}
+
+	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
+		/* Clear pic_size_error */
+		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
+		isp_err = readl(base + CIF_ISP_ERR);
+		v4l2_err(&dev->v4l2_dev,
+			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
+		writel(isp_err, base + CIF_ISP_ERR_CLR);
+	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
+		/* Clear data_loss */
+		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
+		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
+		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
+	}
+
+	/* sampled input frame is complete */
+	if (isp_mis & CIF_ISP_FRAME_IN) {
+		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
+		isp_mis_tmp = readl(base + CIF_ISP_MIS);
+		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
+			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
+				 isp_mis_tmp);
+	}
+
+	/* frame was completely put out */
+	if (isp_mis & CIF_ISP_FRAME) {
+		u32 isp_ris = 0;
+		/* Clear Frame In (ISP) */
+		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
+		isp_mis_tmp = readl(base + CIF_ISP_MIS);
+		if (isp_mis_tmp & CIF_ISP_FRAME)
+			v4l2_err(&dev->v4l2_dev,
+				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
+
+		isp_ris = readl(base + CIF_ISP_RIS);
+		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
+			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
+			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
+	}
+
+	/*
+	 * Then update changed configs. Some of them involve
+	 * lot of register writes. Do those only one per frame.
+	 * Do the updates in the order of the processing flow.
+	 */
+	rkisp1_params_isr(&dev->params_vdev, isp_mis);
+}
diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
new file mode 100644
index 000000000000..b0366e354ec2
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
@@ -0,0 +1,111 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#ifndef _RKISP1_H
+#define _RKISP1_H
+
+#include <linux/platform_device.h>
+#include <media/v4l2-fwnode.h>
+
+#include "common.h"
+
+struct rkisp1_stream;
+
+/*
+ * struct ispsd_in_fmt - ISP intput-pad format
+ *
+ * Translate mbus_code to hardware format values
+ *
+ * @bus_width: used for parallel
+ */
+struct ispsd_in_fmt {
+	u32 mbus_code;
+	u8 fmt_type;
+	u32 mipi_dt;
+	u32 yuv_seq;
+	enum rkisp1_fmt_raw_pat_type bayer_pat;
+	u8 bus_width;
+};
+
+struct ispsd_out_fmt {
+	u32 mbus_code;
+	u8 fmt_type;
+};
+
+struct rkisp1_ie_config {
+	unsigned int effect;
+};
+
+enum rkisp1_isp_pad {
+	RKISP1_ISP_PAD_SINK,
+	RKISP1_ISP_PAD_SINK_PARAMS,
+	RKISP1_ISP_PAD_SOURCE_PATH,
+	RKISP1_ISP_PAD_SOURCE_STATS,
+	RKISP1_ISP_PAD_MAX
+};
+
+/*
+ * struct rkisp1_isp_subdev - ISP sub-device
+ *
+ * See Cropping regions of ISP in rkisp1.c for details
+ * @in_frm: input size, don't have to be equal to sensor size
+ * @in_fmt: input format
+ * @in_crop: crop for sink pad
+ * @out_fmt: output format
+ * @out_crop: output size
+ *
+ * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)
+ * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
+ * @quantization: output quantization
+ */
+struct rkisp1_isp_subdev {
+	struct v4l2_subdev sd;
+	struct media_pad pads[RKISP1_ISP_PAD_MAX];
+	struct v4l2_ctrl_handler ctrl_handler;
+	struct v4l2_mbus_framefmt in_frm;
+	struct ispsd_in_fmt in_fmt;
+	struct v4l2_rect in_crop;
+	struct ispsd_out_fmt out_fmt;
+	struct v4l2_rect out_crop;
+	bool dphy_errctrl_disabled;
+	atomic_t frm_sync_seq;
+	enum v4l2_quantization quantization;
+};
+
+int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
+			       struct v4l2_device *v4l2_dev);
+
+void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
+
+void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
+
+void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
+
+static inline
+struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
+{
+	return &isp_sdev->out_fmt;
+}
+
+static inline
+struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
+{
+	return &isp_sdev->in_fmt;
+}
+
+static inline
+struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
+{
+	return &isp_sdev->out_crop;
+}
+
+static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct rkisp1_isp_subdev, sd);
+}
+
+#endif /* _RKISP1_H */
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 06/14] media: rkisp1: add ISP1 statistics driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (4 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-08  9:37   ` Sakari Ailus
  2019-07-30 18:42 ` [PATCH v8 07/14] media: rkisp1: add ISP1 params driver Helen Koike
                   ` (9 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Allon Huang, Jacob Chen,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, Helen Koike, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

Add the capture video driver for rockchip isp1 statistics block.

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
Signed-off-by: Jacob Chen <cc@rock-chips.com>
Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Acked-by: Hans Verkuil <hans.verkuil@cisco.com>

---

Changes in v8: None
Changes in v7:
- s/strlcpy/strscpy
- sort out the locks in isp stats
- code styling and checkpatch fixes

 .../media/platform/rockchip/isp1/isp_stats.c  | 508 ++++++++++++++++++
 .../media/platform/rockchip/isp1/isp_stats.h  |  60 +++
 2 files changed, 568 insertions(+)
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h

diff --git a/drivers/media/platform/rockchip/isp1/isp_stats.c b/drivers/media/platform/rockchip/isp1/isp_stats.c
new file mode 100644
index 000000000000..01d947867c70
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/isp_stats.c
@@ -0,0 +1,508 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#include <media/v4l2-common.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-vmalloc.h>	/* for ISP statistics */
+
+#include "dev.h"
+#include "regs.h"
+
+#define RKISP1_ISP_STATS_REQ_BUFS_MIN 2
+#define RKISP1_ISP_STATS_REQ_BUFS_MAX 8
+
+static int rkisp1_stats_enum_fmt_meta_cap(struct file *file, void *priv,
+					  struct v4l2_fmtdesc *f)
+{
+	struct video_device *video = video_devdata(file);
+	struct rkisp1_isp_stats_vdev *stats_vdev = video_get_drvdata(video);
+
+	if (f->index > 0 || f->type != video->queue->type)
+		return -EINVAL;
+
+	f->pixelformat = stats_vdev->vdev_fmt.fmt.meta.dataformat;
+	return 0;
+}
+
+static int rkisp1_stats_g_fmt_meta_cap(struct file *file, void *priv,
+				       struct v4l2_format *f)
+{
+	struct video_device *video = video_devdata(file);
+	struct rkisp1_isp_stats_vdev *stats_vdev = video_get_drvdata(video);
+	struct v4l2_meta_format *meta = &f->fmt.meta;
+
+	if (f->type != video->queue->type)
+		return -EINVAL;
+
+	memset(meta, 0, sizeof(*meta));
+	meta->dataformat = stats_vdev->vdev_fmt.fmt.meta.dataformat;
+	meta->buffersize = stats_vdev->vdev_fmt.fmt.meta.buffersize;
+
+	return 0;
+}
+
+static int rkisp1_stats_querycap(struct file *file,
+				 void *priv, struct v4l2_capability *cap)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	strscpy(cap->driver, DRIVER_NAME, sizeof(cap->driver));
+	strscpy(cap->card, vdev->name, sizeof(cap->card));
+	strscpy(cap->bus_info, "platform: " DRIVER_NAME, sizeof(cap->bus_info));
+
+	return 0;
+}
+
+/* ISP video device IOCTLs */
+static const struct v4l2_ioctl_ops rkisp1_stats_ioctl = {
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_enum_fmt_meta_cap = rkisp1_stats_enum_fmt_meta_cap,
+	.vidioc_g_fmt_meta_cap = rkisp1_stats_g_fmt_meta_cap,
+	.vidioc_s_fmt_meta_cap = rkisp1_stats_g_fmt_meta_cap,
+	.vidioc_try_fmt_meta_cap = rkisp1_stats_g_fmt_meta_cap,
+	.vidioc_querycap = rkisp1_stats_querycap,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+struct v4l2_file_operations rkisp1_stats_fops = {
+	.mmap = vb2_fop_mmap,
+	.unlocked_ioctl = video_ioctl2,
+	.poll = vb2_fop_poll,
+	.open = v4l2_fh_open,
+	.release = vb2_fop_release
+};
+
+static int rkisp1_stats_vb2_queue_setup(struct vb2_queue *vq,
+					unsigned int *num_buffers,
+					unsigned int *num_planes,
+					unsigned int sizes[],
+					struct device *alloc_devs[])
+{
+	struct rkisp1_isp_stats_vdev *stats_vdev = vq->drv_priv;
+
+	*num_planes = 1;
+
+	*num_buffers = clamp_t(u32, *num_buffers, RKISP1_ISP_STATS_REQ_BUFS_MIN,
+			       RKISP1_ISP_STATS_REQ_BUFS_MAX);
+
+	sizes[0] = sizeof(struct rkisp1_stat_buffer);
+
+	INIT_LIST_HEAD(&stats_vdev->stat);
+
+	return 0;
+}
+
+static void rkisp1_stats_vb2_buf_queue(struct vb2_buffer *vb)
+{
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct rkisp1_buffer *stats_buf = to_rkisp1_buffer(vbuf);
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct rkisp1_isp_stats_vdev *stats_dev = vq->drv_priv;
+
+	stats_buf->vaddr[0] = vb2_plane_vaddr(vb, 0);
+
+	mutex_lock(&stats_dev->wq_lock);
+	list_add_tail(&stats_buf->queue, &stats_dev->stat);
+	mutex_unlock(&stats_dev->wq_lock);
+}
+
+static int rkisp1_stats_vb2_buf_prepare(struct vb2_buffer *vb)
+{
+	if (vb2_plane_size(vb, 0) < sizeof(struct rkisp1_stat_buffer))
+		return -EINVAL;
+
+	vb2_set_plane_payload(vb, 0, sizeof(struct rkisp1_stat_buffer));
+
+	return 0;
+}
+
+static void rkisp1_stats_vb2_stop_streaming(struct vb2_queue *vq)
+{
+	struct rkisp1_isp_stats_vdev *stats_vdev = vq->drv_priv;
+	struct rkisp1_buffer *buf;
+	unsigned long flags;
+	unsigned int i;
+
+	/* Make sure no new work queued in isr before draining wq */
+	spin_lock_irqsave(&stats_vdev->irq_lock, flags);
+	stats_vdev->streamon = false;
+	spin_unlock_irqrestore(&stats_vdev->irq_lock, flags);
+
+	drain_workqueue(stats_vdev->readout_wq);
+
+	mutex_lock(&stats_vdev->wq_lock);
+	for (i = 0; i < RKISP1_ISP_STATS_REQ_BUFS_MAX; i++) {
+		if (list_empty(&stats_vdev->stat))
+			break;
+		buf = list_first_entry(&stats_vdev->stat,
+				       struct rkisp1_buffer, queue);
+		list_del(&buf->queue);
+		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+	mutex_unlock(&stats_vdev->wq_lock);
+}
+
+static int
+rkisp1_stats_vb2_start_streaming(struct vb2_queue *queue,
+				 unsigned int count)
+{
+	struct rkisp1_isp_stats_vdev *stats_vdev = queue->drv_priv;
+
+	stats_vdev->streamon = true;
+
+	return 0;
+}
+
+static struct vb2_ops rkisp1_stats_vb2_ops = {
+	.queue_setup = rkisp1_stats_vb2_queue_setup,
+	.buf_queue = rkisp1_stats_vb2_buf_queue,
+	.buf_prepare = rkisp1_stats_vb2_buf_prepare,
+	.wait_prepare = vb2_ops_wait_prepare,
+	.wait_finish = vb2_ops_wait_finish,
+	.stop_streaming = rkisp1_stats_vb2_stop_streaming,
+	.start_streaming = rkisp1_stats_vb2_start_streaming,
+};
+
+static int rkisp1_stats_init_vb2_queue(struct vb2_queue *q,
+				       struct rkisp1_isp_stats_vdev *stats_vdev)
+{
+	struct rkisp1_vdev_node *node;
+
+	node = queue_to_node(q);
+
+	q->type = V4L2_BUF_TYPE_META_CAPTURE;
+	q->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
+	q->drv_priv = stats_vdev;
+	q->ops = &rkisp1_stats_vb2_ops;
+	q->mem_ops = &vb2_vmalloc_memops;
+	q->buf_struct_size = sizeof(struct rkisp1_buffer);
+	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	q->lock = &node->vlock;
+
+	return vb2_queue_init(q);
+}
+
+static void rkisp1_stats_get_awb_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
+				      struct rkisp1_stat_buffer *pbuf)
+{
+	/* Protect against concurrent access from ISR? */
+	u32 reg_val;
+
+	pbuf->meas_type |= CIFISP_STAT_AWB;
+	reg_val = readl(stats_vdev->dev->base_addr + CIF_ISP_AWB_WHITE_CNT);
+	pbuf->params.awb.awb_mean[0].cnt = CIF_ISP_AWB_GET_PIXEL_CNT(reg_val);
+	reg_val = readl(stats_vdev->dev->base_addr + CIF_ISP_AWB_MEAN);
+
+	pbuf->params.awb.awb_mean[0].mean_cr_or_r =
+		CIF_ISP_AWB_GET_MEAN_CR_R(reg_val);
+	pbuf->params.awb.awb_mean[0].mean_cb_or_b =
+		CIF_ISP_AWB_GET_MEAN_CB_B(reg_val);
+	pbuf->params.awb.awb_mean[0].mean_y_or_g =
+		CIF_ISP_AWB_GET_MEAN_Y_G(reg_val);
+}
+
+static void rkisp1_stats_get_aec_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
+				      struct rkisp1_stat_buffer *pbuf)
+{
+	void __iomem *addr = stats_vdev->dev->base_addr + CIF_ISP_EXP_MEAN_00;
+	unsigned int i;
+
+	pbuf->meas_type |= CIFISP_STAT_AUTOEXP;
+	for (i = 0; i < CIFISP_AE_MEAN_MAX; i++)
+		pbuf->params.ae.exp_mean[i] = (u8)readl(addr + i * 4);
+}
+
+static void rkisp1_stats_get_afc_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
+				      struct rkisp1_stat_buffer *pbuf)
+{
+	void __iomem *base_addr;
+	struct cifisp_af_stat *af;
+
+	pbuf->meas_type = CIFISP_STAT_AFM_FIN;
+
+	af = &pbuf->params.af;
+	base_addr = stats_vdev->dev->base_addr;
+	af->window[0].sum = readl(base_addr + CIF_ISP_AFM_SUM_A);
+	af->window[0].lum = readl(base_addr + CIF_ISP_AFM_LUM_A);
+	af->window[1].sum = readl(base_addr + CIF_ISP_AFM_SUM_B);
+	af->window[1].lum = readl(base_addr + CIF_ISP_AFM_LUM_B);
+	af->window[2].sum = readl(base_addr + CIF_ISP_AFM_SUM_C);
+	af->window[2].lum = readl(base_addr + CIF_ISP_AFM_LUM_C);
+}
+
+static void rkisp1_stats_get_hst_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
+				      struct rkisp1_stat_buffer *pbuf)
+{
+	void __iomem *addr = stats_vdev->dev->base_addr + CIF_ISP_HIST_BIN_0;
+	unsigned int i;
+
+	pbuf->meas_type |= CIFISP_STAT_HIST;
+	for (i = 0; i < CIFISP_HIST_BIN_N_MAX; i++)
+		pbuf->params.hist.hist_bins[i] = readl(addr + (i * 4));
+}
+
+static void rkisp1_stats_get_bls_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
+				      struct rkisp1_stat_buffer *pbuf)
+{
+	struct rkisp1_device *dev = stats_vdev->dev;
+	const struct ispsd_in_fmt *in_fmt =
+			rkisp1_get_ispsd_in_fmt(&dev->isp_sdev);
+	void __iomem *base = stats_vdev->dev->base_addr;
+	struct cifisp_bls_meas_val *bls_val;
+
+	bls_val = &pbuf->params.ae.bls_val;
+	if (in_fmt->bayer_pat == RAW_BGGR) {
+		bls_val->meas_b = readl(base + CIF_ISP_BLS_A_MEASURED);
+		bls_val->meas_gb = readl(base + CIF_ISP_BLS_B_MEASURED);
+		bls_val->meas_gr = readl(base + CIF_ISP_BLS_C_MEASURED);
+		bls_val->meas_r = readl(base + CIF_ISP_BLS_D_MEASURED);
+	} else if (in_fmt->bayer_pat == RAW_GBRG) {
+		bls_val->meas_gb = readl(base + CIF_ISP_BLS_A_MEASURED);
+		bls_val->meas_b = readl(base + CIF_ISP_BLS_B_MEASURED);
+		bls_val->meas_r = readl(base + CIF_ISP_BLS_C_MEASURED);
+		bls_val->meas_gr = readl(base + CIF_ISP_BLS_D_MEASURED);
+	} else if (in_fmt->bayer_pat == RAW_GRBG) {
+		bls_val->meas_gr = readl(base + CIF_ISP_BLS_A_MEASURED);
+		bls_val->meas_r = readl(base + CIF_ISP_BLS_B_MEASURED);
+		bls_val->meas_b = readl(base + CIF_ISP_BLS_C_MEASURED);
+		bls_val->meas_gb = readl(base + CIF_ISP_BLS_D_MEASURED);
+	} else if (in_fmt->bayer_pat == RAW_RGGB) {
+		bls_val->meas_r = readl(base + CIF_ISP_BLS_A_MEASURED);
+		bls_val->meas_gr = readl(base + CIF_ISP_BLS_B_MEASURED);
+		bls_val->meas_gb = readl(base + CIF_ISP_BLS_C_MEASURED);
+		bls_val->meas_b = readl(base + CIF_ISP_BLS_D_MEASURED);
+	}
+}
+
+static void
+rkisp1_stats_send_measurement(struct rkisp1_isp_stats_vdev *stats_vdev,
+			      struct rkisp1_isp_readout_work *meas_work)
+{
+	struct rkisp1_stat_buffer *cur_stat_buf;
+	struct rkisp1_buffer *cur_buf = NULL;
+	unsigned int cur_frame_id = -1;
+
+	cur_frame_id = atomic_read(&stats_vdev->dev->isp_sdev.frm_sync_seq) - 1;
+	if (cur_frame_id != meas_work->frame_id) {
+		v4l2_warn(stats_vdev->vnode.vdev.v4l2_dev,
+			  "Measurement late(%d, %d)\n",
+			  cur_frame_id, meas_work->frame_id);
+		cur_frame_id = meas_work->frame_id;
+	}
+
+	mutex_lock(&stats_vdev->wq_lock);
+	/* get one empty buffer */
+	if (!list_empty(&stats_vdev->stat)) {
+		cur_buf = list_first_entry(&stats_vdev->stat,
+					   struct rkisp1_buffer, queue);
+		list_del(&cur_buf->queue);
+	}
+	mutex_unlock(&stats_vdev->wq_lock);
+
+	if (!cur_buf)
+		return;
+
+	cur_stat_buf =
+		(struct rkisp1_stat_buffer *)(cur_buf->vaddr[0]);
+
+	if (meas_work->isp_ris & CIF_ISP_AWB_DONE) {
+		rkisp1_stats_get_awb_meas(stats_vdev, cur_stat_buf);
+		cur_stat_buf->meas_type |= CIFISP_STAT_AWB;
+	}
+
+	if (meas_work->isp_ris & CIF_ISP_AFM_FIN) {
+		rkisp1_stats_get_afc_meas(stats_vdev, cur_stat_buf);
+		cur_stat_buf->meas_type |= CIFISP_STAT_AFM_FIN;
+	}
+
+	if (meas_work->isp_ris & CIF_ISP_EXP_END) {
+		rkisp1_stats_get_aec_meas(stats_vdev, cur_stat_buf);
+		rkisp1_stats_get_bls_meas(stats_vdev, cur_stat_buf);
+		cur_stat_buf->meas_type |= CIFISP_STAT_AUTOEXP;
+	}
+
+	if (meas_work->isp_ris & CIF_ISP_HIST_MEASURE_RDY) {
+		rkisp1_stats_get_hst_meas(stats_vdev, cur_stat_buf);
+		cur_stat_buf->meas_type |= CIFISP_STAT_HIST;
+	}
+
+	vb2_set_plane_payload(&cur_buf->vb.vb2_buf, 0,
+			      sizeof(struct rkisp1_stat_buffer));
+	cur_buf->vb.sequence = cur_frame_id;
+	cur_buf->vb.vb2_buf.timestamp = ktime_get_ns();
+	vb2_buffer_done(&cur_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
+}
+
+static void rkisp1_stats_readout_work(struct work_struct *work)
+{
+	struct rkisp1_isp_readout_work *readout_work = container_of(work,
+						struct rkisp1_isp_readout_work,
+						work);
+	struct rkisp1_isp_stats_vdev *stats_vdev = readout_work->stats_vdev;
+
+	if (readout_work->readout == RKISP1_ISP_READOUT_MEAS)
+		rkisp1_stats_send_measurement(stats_vdev, readout_work);
+
+	kfree(readout_work);
+}
+
+int rkisp1_stats_isr(struct rkisp1_isp_stats_vdev *stats_vdev, u32 isp_ris)
+{
+	unsigned int cur_frame_id =
+		atomic_read(&stats_vdev->dev->isp_sdev.frm_sync_seq) - 1;
+	struct rkisp1_isp_readout_work *work;
+	unsigned int isp_mis_tmp = 0;
+#ifdef LOG_ISR_EXE_TIME
+	ktime_t in_t = ktime_get();
+#endif
+
+	spin_lock(&stats_vdev->irq_lock);
+
+	writel((CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN | CIF_ISP_EXP_END |
+		CIF_ISP_HIST_MEASURE_RDY),
+		stats_vdev->dev->base_addr + CIF_ISP_ICR);
+
+	isp_mis_tmp = readl(stats_vdev->dev->base_addr + CIF_ISP_MIS);
+	if (isp_mis_tmp &
+		(CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
+		 CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
+		v4l2_err(stats_vdev->vnode.vdev.v4l2_dev,
+			 "isp icr 3A info err: 0x%x\n",
+			 isp_mis_tmp);
+
+	if (!stats_vdev->streamon)
+		goto unlock;
+	if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN | CIF_ISP_EXP_END |
+		CIF_ISP_HIST_MEASURE_RDY)) {
+		work = (struct rkisp1_isp_readout_work *)
+			kzalloc(sizeof(struct rkisp1_isp_readout_work),
+				GFP_ATOMIC);
+		if (work) {
+			INIT_WORK(&work->work,
+				  rkisp1_stats_readout_work);
+			work->readout = RKISP1_ISP_READOUT_MEAS;
+			work->stats_vdev = stats_vdev;
+			work->frame_id = cur_frame_id;
+			work->isp_ris = isp_ris;
+			if (!queue_work(stats_vdev->readout_wq,
+					&work->work))
+				kfree(work);
+		} else {
+			v4l2_err(stats_vdev->vnode.vdev.v4l2_dev,
+				 "Could not allocate work\n");
+		}
+	}
+
+#ifdef LOG_ISR_EXE_TIME
+	if (isp_ris & (CIF_ISP_EXP_END | CIF_ISP_AWB_DONE |
+		       CIF_ISP_FRAME | CIF_ISP_HIST_MEASURE_RDY)) {
+		unsigned int diff_us =
+				ktime_to_us(ktime_sub(ktime_get(), in_t));
+
+		if (diff_us > g_longest_isr_time)
+			g_longest_isr_time = diff_us;
+
+		v4l2_info(stats_vdev->vnode.vdev.v4l2_dev,
+			  "isp_isr time %d %d\n", diff_us, g_longest_isr_time);
+	}
+#endif
+
+unlock:
+	spin_unlock(&stats_vdev->irq_lock);
+
+	return 0;
+}
+
+static void rkisp1_init_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev)
+{
+	stats_vdev->vdev_fmt.fmt.meta.dataformat =
+		V4L2_META_FMT_RK_ISP1_STAT_3A;
+	stats_vdev->vdev_fmt.fmt.meta.buffersize =
+		sizeof(struct rkisp1_stat_buffer);
+}
+
+int rkisp1_register_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev,
+			       struct v4l2_device *v4l2_dev,
+			       struct rkisp1_device *dev)
+{
+	struct rkisp1_vdev_node *node = &stats_vdev->vnode;
+	struct video_device *vdev = &node->vdev;
+	unsigned int ret;
+
+	stats_vdev->dev = dev;
+	mutex_init(&stats_vdev->wq_lock);
+	mutex_init(&node->vlock);
+	INIT_LIST_HEAD(&stats_vdev->stat);
+	spin_lock_init(&stats_vdev->irq_lock);
+
+	strlcpy(vdev->name, "rkisp1-statistics", sizeof(vdev->name));
+
+	video_set_drvdata(vdev, stats_vdev);
+	vdev->ioctl_ops = &rkisp1_stats_ioctl;
+	vdev->fops = &rkisp1_stats_fops;
+	vdev->release = video_device_release_empty;
+	vdev->lock = &node->vlock;
+	vdev->v4l2_dev = v4l2_dev;
+	vdev->queue = &node->buf_queue;
+	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
+	vdev->vfl_dir =  VFL_DIR_RX;
+	rkisp1_stats_init_vb2_queue(vdev->queue, stats_vdev);
+	rkisp1_init_stats_vdev(stats_vdev);
+	video_set_drvdata(vdev, stats_vdev);
+
+	node->pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_pads_init(&vdev->entity, 1, &node->pad);
+	if (ret < 0)
+		goto err_release_queue;
+
+	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
+	if (ret < 0) {
+		dev_err(&vdev->dev,
+			"could not register Video for Linux device\n");
+		goto err_cleanup_media_entity;
+	}
+
+	stats_vdev->readout_wq =
+	    alloc_workqueue("measurement_queue",
+			    WQ_UNBOUND | WQ_MEM_RECLAIM, 1);
+
+	if (!stats_vdev->readout_wq) {
+		ret = -ENOMEM;
+			goto err_unreg_vdev;
+	}
+
+	return 0;
+err_unreg_vdev:
+	video_unregister_device(vdev);
+err_cleanup_media_entity:
+	media_entity_cleanup(&vdev->entity);
+err_release_queue:
+	vb2_queue_release(vdev->queue);
+	return ret;
+}
+
+void rkisp1_unregister_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev)
+{
+	struct rkisp1_vdev_node *node = &stats_vdev->vnode;
+	struct video_device *vdev = &node->vdev;
+
+	destroy_workqueue(stats_vdev->readout_wq);
+	video_unregister_device(vdev);
+	media_entity_cleanup(&vdev->entity);
+	vb2_queue_release(vdev->queue);
+}
diff --git a/drivers/media/platform/rockchip/isp1/isp_stats.h b/drivers/media/platform/rockchip/isp1/isp_stats.h
new file mode 100644
index 000000000000..b46c8537e1c7
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/isp_stats.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#ifndef _RKISP1_ISP_STATS_H
+#define _RKISP1_ISP_STATS_H
+
+#include <linux/rkisp1-config.h>
+
+#include "common.h"
+
+struct rkisp1_isp_stats_vdev;
+
+enum rkisp1_isp_readout_cmd {
+	RKISP1_ISP_READOUT_MEAS,
+	RKISP1_ISP_READOUT_META,
+};
+
+struct rkisp1_isp_readout_work {
+	struct work_struct work;
+	struct rkisp1_isp_stats_vdev *stats_vdev;
+
+	unsigned int frame_id;
+	unsigned int isp_ris;
+	enum rkisp1_isp_readout_cmd readout;
+	struct vb2_buffer *vb;
+};
+
+/*
+ * struct rkisp1_isp_stats_vdev - ISP Statistics device
+ *
+ * @irq_lock: buffer queue lock
+ * @stat: stats buffer list
+ * @readout_wq: workqueue for statistics information read
+ */
+struct rkisp1_isp_stats_vdev {
+	struct rkisp1_vdev_node vnode;
+	struct rkisp1_device *dev;
+
+	spinlock_t irq_lock;
+	struct list_head stat;
+	struct v4l2_format vdev_fmt;
+	bool streamon;
+
+	struct workqueue_struct *readout_wq;
+	struct mutex wq_lock;
+};
+
+int rkisp1_stats_isr(struct rkisp1_isp_stats_vdev *stats_vdev, u32 isp_ris);
+
+int rkisp1_register_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev,
+			       struct v4l2_device *v4l2_dev,
+			       struct rkisp1_device *dev);
+
+void rkisp1_unregister_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev);
+
+#endif /* _RKISP1_ISP_STATS_H */
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 07/14] media: rkisp1: add ISP1 params driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (5 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 06/14] media: rkisp1: add ISP1 statistics driver Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-07-30 18:42 ` [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver Helen Koike
                   ` (8 subsequent siblings)
  15 siblings, 0 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Allon Huang, Jacob Chen,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, Helen Koike, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

Add the output video driver that accept params from userspace.

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
Signed-off-by: Jacob Chen <cc@rock-chips.com>
Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- s/strlcpy/strscpy
- s/strcpy/strscpy
- fix config lsc error
LSC data table size is 17x17, but when configuring data to ISP,
should be aligned to 18x17. That means every last data of last
line should be filled with 0, and not filled with the data of
next line.
- Update new ISP parameters immediately
For those sub modules that have shadow registers in core isp, the
new programing parameters would not be active if both
CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT and CFG_UPD are not set. Now
we configure CFG_UPD to force update the shadow registers when new
ISP parameters are configured.
- fix some ISP parameters config error
Some ISP parameter config functions may override the old enable
bit value, because the enable bits of these modules are in the
same registers with parameters. So we should save the old enable
bits firstly.
- code styling and checkpatch fixes

 .../media/platform/rockchip/isp1/isp_params.c | 1604 +++++++++++++++++
 .../media/platform/rockchip/isp1/isp_params.h |   50 +
 2 files changed, 1654 insertions(+)
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.c
 create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.h

diff --git a/drivers/media/platform/rockchip/isp1/isp_params.c b/drivers/media/platform/rockchip/isp1/isp_params.c
new file mode 100644
index 000000000000..7b470b6bf90f
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/isp_params.c
@@ -0,0 +1,1604 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#include <media/v4l2-common.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-core.h>
+#include <media/videobuf2-vmalloc.h>	/* for ISP params */
+
+#include "dev.h"
+#include "regs.h"
+
+#define RKISP1_ISP_PARAMS_REQ_BUFS_MIN	2
+#define RKISP1_ISP_PARAMS_REQ_BUFS_MAX	8
+
+#define BLS_START_H_MAX_IS_VALID(val)	((val) < CIFISP_BLS_START_H_MAX)
+#define BLS_STOP_H_MAX_IS_VALID(val)	((val) < CIFISP_BLS_STOP_H_MAX)
+
+#define BLS_START_V_MAX_IS_VALID(val)	((val) < CIFISP_BLS_START_V_MAX)
+#define BLS_STOP_V_MAX_IS_VALID(val)	((val) < CIFISP_BLS_STOP_V_MAX)
+
+#define BLS_SAMPLE_MAX_IS_VALID(val)	((val) < CIFISP_BLS_SAMPLES_MAX)
+
+#define BLS_FIX_SUB_IS_VALID(val)	\
+	((val) > (s16) CIFISP_BLS_FIX_SUB_MIN && (val) < CIFISP_BLS_FIX_SUB_MAX)
+
+#define RKISP1_ISP_DPCC_LINE_THRESH(n)	(CIF_ISP_DPCC_LINE_THRESH_1 + 0x14 * (n))
+#define RKISP1_ISP_DPCC_LINE_MAD_FAC(n) (CIF_ISP_DPCC_LINE_MAD_FAC_1 + 0x14 * (n))
+#define RKISP1_ISP_DPCC_PG_FAC(n)	(CIF_ISP_DPCC_PG_FAC_1 + 0x14 * (n))
+#define RKISP1_ISP_DPCC_RND_THRESH(n)	(CIF_ISP_DPCC_RND_THRESH_1 + 0x14 * (n))
+#define RKISP1_ISP_DPCC_RG_FAC(n)	(CIF_ISP_DPCC_RG_FAC_1 + 0x14 * (n))
+#define RKISP1_ISP_CC_COEFF(n)		(CIF_ISP_CC_COEFF_0 + (n) * 4)
+
+static inline void rkisp1_iowrite32(struct rkisp1_isp_params_vdev *params_vdev,
+				    u32 value, u32 addr)
+{
+	iowrite32(value, params_vdev->dev->base_addr + addr);
+}
+
+static inline u32 rkisp1_ioread32(struct rkisp1_isp_params_vdev *params_vdev,
+				  u32 addr)
+{
+	return ioread32(params_vdev->dev->base_addr + addr);
+}
+
+static inline void isp_param_set_bits(struct rkisp1_isp_params_vdev
+					     *params_vdev,
+				      u32 reg, u32 bit_mask)
+{
+	u32 val;
+
+	val = rkisp1_ioread32(params_vdev, reg);
+	rkisp1_iowrite32(params_vdev, val | bit_mask, reg);
+}
+
+static inline void isp_param_clear_bits(struct rkisp1_isp_params_vdev
+					       *params_vdev,
+					u32 reg, u32 bit_mask)
+{
+	u32 val;
+
+	val = rkisp1_ioread32(params_vdev, reg);
+	rkisp1_iowrite32(params_vdev, val & ~bit_mask, reg);
+}
+
+/* ISP BP interface function */
+static void dpcc_config(struct rkisp1_isp_params_vdev *params_vdev,
+			const struct cifisp_dpcc_config *arg)
+{
+	unsigned int i;
+	u32 mode;
+
+	/* avoid to override the old enable value */
+	mode = rkisp1_ioread32(params_vdev, CIF_ISP_DPCC_MODE);
+	mode &= CIF_ISP_DPCC_ENA;
+	mode |= arg->mode & ~CIF_ISP_DPCC_ENA;
+	rkisp1_iowrite32(params_vdev, mode, CIF_ISP_DPCC_MODE);
+	rkisp1_iowrite32(params_vdev, arg->output_mode,
+			 CIF_ISP_DPCC_OUTPUT_MODE);
+	rkisp1_iowrite32(params_vdev, arg->set_use, CIF_ISP_DPCC_SET_USE);
+
+	rkisp1_iowrite32(params_vdev, arg->methods[0].method,
+			 CIF_ISP_DPCC_METHODS_SET_1);
+	rkisp1_iowrite32(params_vdev, arg->methods[1].method,
+			 CIF_ISP_DPCC_METHODS_SET_2);
+	rkisp1_iowrite32(params_vdev, arg->methods[2].method,
+			 CIF_ISP_DPCC_METHODS_SET_3);
+	for (i = 0; i < CIFISP_DPCC_METHODS_MAX; i++) {
+		rkisp1_iowrite32(params_vdev, arg->methods[i].line_thresh,
+				 RKISP1_ISP_DPCC_LINE_THRESH(i));
+		rkisp1_iowrite32(params_vdev, arg->methods[i].line_mad_fac,
+				 RKISP1_ISP_DPCC_LINE_MAD_FAC(i));
+		rkisp1_iowrite32(params_vdev, arg->methods[i].pg_fac,
+				 RKISP1_ISP_DPCC_PG_FAC(i));
+		rkisp1_iowrite32(params_vdev, arg->methods[i].rnd_thresh,
+				 RKISP1_ISP_DPCC_RND_THRESH(i));
+		rkisp1_iowrite32(params_vdev, arg->methods[i].rg_fac,
+				 RKISP1_ISP_DPCC_RG_FAC(i));
+	}
+
+	rkisp1_iowrite32(params_vdev, arg->rnd_offs, CIF_ISP_DPCC_RND_OFFS);
+	rkisp1_iowrite32(params_vdev, arg->ro_limits, CIF_ISP_DPCC_RO_LIMITS);
+}
+
+/* ISP black level subtraction interface function */
+static void bls_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_bls_config *arg)
+{
+	/* avoid to override the old enable value */
+	u32 new_control;
+
+	new_control = rkisp1_ioread32(params_vdev, CIF_ISP_BLS_CTRL);
+	new_control &= CIF_ISP_BLS_ENA;
+	/* fixed subtraction values */
+	if (!arg->enable_auto) {
+		const struct cifisp_bls_fixed_val *pval = &arg->fixed_val;
+
+		switch (params_vdev->raw_type) {
+		case RAW_BGGR:
+			rkisp1_iowrite32(params_vdev,
+					 pval->r, CIF_ISP_BLS_D_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gr, CIF_ISP_BLS_C_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gb, CIF_ISP_BLS_B_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->b, CIF_ISP_BLS_A_FIXED);
+			break;
+		case RAW_GBRG:
+			rkisp1_iowrite32(params_vdev,
+					 pval->r, CIF_ISP_BLS_C_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gr, CIF_ISP_BLS_D_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gb, CIF_ISP_BLS_A_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->b, CIF_ISP_BLS_B_FIXED);
+			break;
+		case RAW_GRBG:
+			rkisp1_iowrite32(params_vdev,
+					 pval->r, CIF_ISP_BLS_B_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gr, CIF_ISP_BLS_A_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gb, CIF_ISP_BLS_D_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->b, CIF_ISP_BLS_C_FIXED);
+			break;
+		case RAW_RGGB:
+			rkisp1_iowrite32(params_vdev,
+					 pval->r, CIF_ISP_BLS_A_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gr, CIF_ISP_BLS_B_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->gb, CIF_ISP_BLS_C_FIXED);
+			rkisp1_iowrite32(params_vdev,
+					 pval->b, CIF_ISP_BLS_D_FIXED);
+			break;
+		default:
+			break;
+		}
+
+	} else {
+		if (arg->en_windows & BIT(1)) {
+			rkisp1_iowrite32(params_vdev, arg->bls_window2.h_offs,
+					 CIF_ISP_BLS_H2_START);
+			rkisp1_iowrite32(params_vdev, arg->bls_window2.h_size,
+					 CIF_ISP_BLS_H2_STOP);
+			rkisp1_iowrite32(params_vdev, arg->bls_window2.v_offs,
+					 CIF_ISP_BLS_V2_START);
+			rkisp1_iowrite32(params_vdev, arg->bls_window2.v_size,
+					 CIF_ISP_BLS_V2_STOP);
+			new_control |= CIF_ISP_BLS_WINDOW_2;
+		}
+
+		if (arg->en_windows & BIT(0)) {
+			rkisp1_iowrite32(params_vdev, arg->bls_window1.h_offs,
+					 CIF_ISP_BLS_H1_START);
+			rkisp1_iowrite32(params_vdev, arg->bls_window1.h_size,
+					 CIF_ISP_BLS_H1_STOP);
+			rkisp1_iowrite32(params_vdev, arg->bls_window1.v_offs,
+					 CIF_ISP_BLS_V1_START);
+			rkisp1_iowrite32(params_vdev, arg->bls_window1.v_size,
+					 CIF_ISP_BLS_V1_STOP);
+			new_control |= CIF_ISP_BLS_WINDOW_1;
+		}
+
+		rkisp1_iowrite32(params_vdev, arg->bls_samples,
+				 CIF_ISP_BLS_SAMPLES);
+
+		new_control |= CIF_ISP_BLS_MODE_MEASURED;
+	}
+	rkisp1_iowrite32(params_vdev, new_control, CIF_ISP_BLS_CTRL);
+}
+
+/* ISP LS correction interface function */
+static void
+__lsc_correct_matrix_config(struct rkisp1_isp_params_vdev *params_vdev,
+			    const struct cifisp_lsc_config *pconfig)
+{
+	unsigned int isp_lsc_status, sram_addr, isp_lsc_table_sel;
+	unsigned int i, j, data;
+
+	isp_lsc_status = rkisp1_ioread32(params_vdev, CIF_ISP_LSC_STATUS);
+
+	/* CIF_ISP_LSC_TABLE_ADDRESS_153 = ( 17 * 18 ) >> 1 */
+	sram_addr = (isp_lsc_status & CIF_ISP_LSC_ACTIVE_TABLE) ?
+		     CIF_ISP_LSC_TABLE_ADDRESS_0 :
+		     CIF_ISP_LSC_TABLE_ADDRESS_153;
+	rkisp1_iowrite32(params_vdev, sram_addr, CIF_ISP_LSC_R_TABLE_ADDR);
+	rkisp1_iowrite32(params_vdev, sram_addr, CIF_ISP_LSC_GR_TABLE_ADDR);
+	rkisp1_iowrite32(params_vdev, sram_addr, CIF_ISP_LSC_GB_TABLE_ADDR);
+	rkisp1_iowrite32(params_vdev, sram_addr, CIF_ISP_LSC_B_TABLE_ADDR);
+
+	/* program data tables (table size is 9 * 17 = 153) */
+	for (i = 0; i < CIF_ISP_LSC_SECTORS_MAX * CIF_ISP_LSC_SECTORS_MAX;
+	     i += CIF_ISP_LSC_SECTORS_MAX) {
+		/*
+		 * 17 sectors with 2 values in one DWORD = 9
+		 * DWORDs (2nd value of last DWORD unused)
+		 */
+		for (j = 0; j < CIF_ISP_LSC_SECTORS_MAX - 1; j += 2) {
+			data = CIF_ISP_LSC_TABLE_DATA(
+					pconfig->r_data_tbl[i + j],
+					pconfig->r_data_tbl[i + j + 1]);
+			rkisp1_iowrite32(params_vdev, data,
+					 CIF_ISP_LSC_R_TABLE_DATA);
+
+			data = CIF_ISP_LSC_TABLE_DATA(
+					pconfig->gr_data_tbl[i + j],
+					pconfig->gr_data_tbl[i + j + 1]);
+			rkisp1_iowrite32(params_vdev, data,
+					 CIF_ISP_LSC_GR_TABLE_DATA);
+
+			data = CIF_ISP_LSC_TABLE_DATA(
+					pconfig->gb_data_tbl[i + j],
+					pconfig->gb_data_tbl[i + j + 1]);
+			rkisp1_iowrite32(params_vdev, data,
+					 CIF_ISP_LSC_GB_TABLE_DATA);
+
+			data = CIF_ISP_LSC_TABLE_DATA(
+					pconfig->b_data_tbl[i + j],
+					pconfig->b_data_tbl[i + j + 1]);
+			rkisp1_iowrite32(params_vdev, data,
+					 CIF_ISP_LSC_B_TABLE_DATA);
+		}
+		data = CIF_ISP_LSC_TABLE_DATA(
+				pconfig->r_data_tbl[i + j],
+				0);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_R_TABLE_DATA);
+
+		data = CIF_ISP_LSC_TABLE_DATA(
+				pconfig->gr_data_tbl[i + j],
+				0);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_GR_TABLE_DATA);
+
+		data = CIF_ISP_LSC_TABLE_DATA(
+				pconfig->gb_data_tbl[i + j],
+				0);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_GB_TABLE_DATA);
+
+		data = CIF_ISP_LSC_TABLE_DATA(
+				pconfig->b_data_tbl[i + j],
+				0);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_B_TABLE_DATA);
+	}
+	isp_lsc_table_sel = (isp_lsc_status & CIF_ISP_LSC_ACTIVE_TABLE) ?
+				CIF_ISP_LSC_TABLE_0 : CIF_ISP_LSC_TABLE_1;
+	rkisp1_iowrite32(params_vdev, isp_lsc_table_sel, CIF_ISP_LSC_TABLE_SEL);
+}
+
+static void lsc_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_lsc_config *arg)
+{
+	unsigned int i, data;
+	u32 lsc_ctrl;
+
+	/* To config must be off , store the current status firstly */
+	lsc_ctrl = rkisp1_ioread32(params_vdev, CIF_ISP_LSC_CTRL);
+	isp_param_clear_bits(params_vdev, CIF_ISP_LSC_CTRL,
+			     CIF_ISP_LSC_CTRL_ENA);
+	__lsc_correct_matrix_config(params_vdev, arg);
+
+	for (i = 0; i < 4; i++) {
+		/* program x size tables */
+		data = CIF_ISP_LSC_SECT_SIZE(arg->x_size_tbl[i * 2],
+					arg->x_size_tbl[i * 2 + 1]);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_XSIZE_01 + i * 4);
+
+		/* program x grad tables */
+		data = CIF_ISP_LSC_SECT_SIZE(arg->x_grad_tbl[i * 2],
+					arg->x_grad_tbl[i * 2 + 1]);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_XGRAD_01 + i * 4);
+
+		/* program y size tables */
+		data = CIF_ISP_LSC_SECT_SIZE(arg->y_size_tbl[i * 2],
+					arg->y_size_tbl[i * 2 + 1]);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_YSIZE_01 + i * 4);
+
+		/* program y grad tables */
+		data = CIF_ISP_LSC_SECT_SIZE(arg->y_grad_tbl[i * 2],
+					arg->y_grad_tbl[i * 2 + 1]);
+		rkisp1_iowrite32(params_vdev, data,
+				 CIF_ISP_LSC_YGRAD_01 + i * 4);
+	}
+
+	/* restore the lsc ctrl status */
+	if (lsc_ctrl & CIF_ISP_LSC_CTRL_ENA) {
+		isp_param_set_bits(params_vdev,
+				   CIF_ISP_LSC_CTRL,
+				   CIF_ISP_LSC_CTRL_ENA);
+	} else {
+		isp_param_clear_bits(params_vdev,
+				     CIF_ISP_LSC_CTRL,
+				     CIF_ISP_LSC_CTRL_ENA);
+	}
+}
+
+/* ISP Filtering function */
+static void flt_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_flt_config *arg)
+{
+	u32 filt_mode;
+
+	rkisp1_iowrite32(params_vdev, arg->thresh_bl0, CIF_ISP_FILT_THRESH_BL0);
+	rkisp1_iowrite32(params_vdev, arg->thresh_bl1, CIF_ISP_FILT_THRESH_BL1);
+	rkisp1_iowrite32(params_vdev, arg->thresh_sh0, CIF_ISP_FILT_THRESH_SH0);
+	rkisp1_iowrite32(params_vdev, arg->thresh_sh1, CIF_ISP_FILT_THRESH_SH1);
+	rkisp1_iowrite32(params_vdev, arg->fac_bl0, CIF_ISP_FILT_FAC_BL0);
+	rkisp1_iowrite32(params_vdev, arg->fac_bl1, CIF_ISP_FILT_FAC_BL1);
+	rkisp1_iowrite32(params_vdev, arg->fac_mid, CIF_ISP_FILT_FAC_MID);
+	rkisp1_iowrite32(params_vdev, arg->fac_sh0, CIF_ISP_FILT_FAC_SH0);
+	rkisp1_iowrite32(params_vdev, arg->fac_sh1, CIF_ISP_FILT_FAC_SH1);
+	rkisp1_iowrite32(params_vdev, arg->lum_weight, CIF_ISP_FILT_LUM_WEIGHT);
+
+	rkisp1_iowrite32(params_vdev, (arg->mode ? CIF_ISP_FLT_MODE_DNR : 0) |
+			 CIF_ISP_FLT_CHROMA_V_MODE(arg->chr_v_mode) |
+			 CIF_ISP_FLT_CHROMA_H_MODE(arg->chr_h_mode) |
+			 CIF_ISP_FLT_GREEN_STAGE1(arg->grn_stage1),
+			 CIF_ISP_FILT_MODE);
+
+	/* avoid to override the old enable value */
+	filt_mode = rkisp1_ioread32(params_vdev, CIF_ISP_FILT_MODE);
+	filt_mode &= CIF_ISP_FLT_ENA;
+	if (arg->mode)
+		filt_mode |= CIF_ISP_FLT_MODE_DNR;
+	filt_mode |= CIF_ISP_FLT_CHROMA_V_MODE(arg->chr_v_mode) |
+				 CIF_ISP_FLT_CHROMA_H_MODE(arg->chr_h_mode) |
+				 CIF_ISP_FLT_GREEN_STAGE1(arg->grn_stage1);
+	rkisp1_iowrite32(params_vdev, filt_mode, CIF_ISP_FILT_MODE);
+}
+
+/* ISP demosaic interface function */
+static int bdm_config(struct rkisp1_isp_params_vdev *params_vdev,
+		      const struct cifisp_bdm_config *arg)
+{
+	u32 bdm_th;
+
+	/* avoid to override the old enable value */
+	bdm_th = rkisp1_ioread32(params_vdev, CIF_ISP_DEMOSAIC);
+	bdm_th &= CIF_ISP_DEMOSAIC_BYPASS;
+	bdm_th |= arg->demosaic_th & ~CIF_ISP_DEMOSAIC_BYPASS;
+	/* set demosaic threshold */
+	rkisp1_iowrite32(params_vdev, bdm_th, CIF_ISP_DEMOSAIC);
+	return 0;
+}
+
+/* ISP GAMMA correction interface function */
+static void sdg_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_sdg_config *arg)
+{
+	unsigned int i;
+
+	rkisp1_iowrite32(params_vdev,
+			 arg->xa_pnts.gamma_dx0, CIF_ISP_GAMMA_DX_LO);
+	rkisp1_iowrite32(params_vdev,
+			 arg->xa_pnts.gamma_dx1, CIF_ISP_GAMMA_DX_HI);
+
+	for (i = 0; i < CIFISP_DEGAMMA_CURVE_SIZE; i++) {
+		rkisp1_iowrite32(params_vdev, arg->curve_r.gamma_y[i],
+				 CIF_ISP_GAMMA_R_Y0 + i * 4);
+		rkisp1_iowrite32(params_vdev, arg->curve_g.gamma_y[i],
+				 CIF_ISP_GAMMA_G_Y0 + i * 4);
+		rkisp1_iowrite32(params_vdev, arg->curve_b.gamma_y[i],
+				 CIF_ISP_GAMMA_B_Y0 + i * 4);
+	}
+}
+
+/* ISP GAMMA correction interface function */
+static void goc_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_goc_config *arg)
+{
+	unsigned int i;
+
+	isp_param_clear_bits(params_vdev, CIF_ISP_CTRL,
+			     CIF_ISP_CTRL_ISP_GAMMA_OUT_ENA);
+	rkisp1_iowrite32(params_vdev, arg->mode, CIF_ISP_GAMMA_OUT_MODE);
+
+	for (i = 0; i < CIFISP_GAMMA_OUT_MAX_SAMPLES; i++)
+		rkisp1_iowrite32(params_vdev, arg->gamma_y[i],
+				 CIF_ISP_GAMMA_OUT_Y_0 + i * 4);
+}
+
+/* ISP Cross Talk */
+static void ctk_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_ctk_config *arg)
+{
+	rkisp1_iowrite32(params_vdev, arg->coeff0, CIF_ISP_CT_COEFF_0);
+	rkisp1_iowrite32(params_vdev, arg->coeff1, CIF_ISP_CT_COEFF_1);
+	rkisp1_iowrite32(params_vdev, arg->coeff2, CIF_ISP_CT_COEFF_2);
+	rkisp1_iowrite32(params_vdev, arg->coeff3, CIF_ISP_CT_COEFF_3);
+	rkisp1_iowrite32(params_vdev, arg->coeff4, CIF_ISP_CT_COEFF_4);
+	rkisp1_iowrite32(params_vdev, arg->coeff5, CIF_ISP_CT_COEFF_5);
+	rkisp1_iowrite32(params_vdev, arg->coeff6, CIF_ISP_CT_COEFF_6);
+	rkisp1_iowrite32(params_vdev, arg->coeff7, CIF_ISP_CT_COEFF_7);
+	rkisp1_iowrite32(params_vdev, arg->coeff8, CIF_ISP_CT_COEFF_8);
+	rkisp1_iowrite32(params_vdev, arg->ct_offset_r, CIF_ISP_CT_OFFSET_R);
+	rkisp1_iowrite32(params_vdev, arg->ct_offset_g, CIF_ISP_CT_OFFSET_G);
+	rkisp1_iowrite32(params_vdev, arg->ct_offset_b, CIF_ISP_CT_OFFSET_B);
+}
+
+static void ctk_enable(struct rkisp1_isp_params_vdev *params_vdev, bool en)
+{
+	if (en)
+		return;
+
+	/* Write back the default values. */
+	rkisp1_iowrite32(params_vdev, 0x80, CIF_ISP_CT_COEFF_0);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_COEFF_1);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_COEFF_2);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_COEFF_3);
+	rkisp1_iowrite32(params_vdev, 0x80, CIF_ISP_CT_COEFF_4);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_COEFF_5);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_COEFF_6);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_COEFF_7);
+	rkisp1_iowrite32(params_vdev, 0x80, CIF_ISP_CT_COEFF_8);
+
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_OFFSET_R);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_OFFSET_G);
+	rkisp1_iowrite32(params_vdev, 0, CIF_ISP_CT_OFFSET_B);
+}
+
+/* ISP White Balance Mode */
+static void awb_meas_config(struct rkisp1_isp_params_vdev *params_vdev,
+			    const struct cifisp_awb_meas_config *arg)
+{
+	u32 reg_val = 0;
+	/* based on the mode,configure the awb module */
+	if (arg->awb_mode == CIFISP_AWB_MODE_YCBCR) {
+		/* Reference Cb and Cr */
+		rkisp1_iowrite32(params_vdev,
+				 CIF_ISP_AWB_REF_CR_SET(arg->awb_ref_cr) |
+				 arg->awb_ref_cb, CIF_ISP_AWB_REF);
+		/* Yc Threshold */
+		rkisp1_iowrite32(params_vdev,
+				 CIF_ISP_AWB_MAX_Y_SET(arg->max_y) |
+				 CIF_ISP_AWB_MIN_Y_SET(arg->min_y) |
+				 CIF_ISP_AWB_MAX_CS_SET(arg->max_csum) |
+				 arg->min_c, CIF_ISP_AWB_THRESH);
+	}
+
+	reg_val = rkisp1_ioread32(params_vdev, CIF_ISP_AWB_PROP);
+	if (arg->enable_ymax_cmp)
+		reg_val |= CIF_ISP_AWB_YMAX_CMP_EN;
+	else
+		reg_val &= ~CIF_ISP_AWB_YMAX_CMP_EN;
+	rkisp1_iowrite32(params_vdev, reg_val, CIF_ISP_AWB_PROP);
+
+	/* window offset */
+	rkisp1_iowrite32(params_vdev,
+			 arg->awb_wnd.v_offs, CIF_ISP_AWB_WND_V_OFFS);
+	rkisp1_iowrite32(params_vdev,
+			 arg->awb_wnd.h_offs, CIF_ISP_AWB_WND_H_OFFS);
+	/* AWB window size */
+	rkisp1_iowrite32(params_vdev,
+			 arg->awb_wnd.v_size, CIF_ISP_AWB_WND_V_SIZE);
+	rkisp1_iowrite32(params_vdev,
+			 arg->awb_wnd.h_size, CIF_ISP_AWB_WND_H_SIZE);
+	/* Number of frames */
+	rkisp1_iowrite32(params_vdev,
+			 arg->frames, CIF_ISP_AWB_FRAMES);
+}
+
+static void awb_meas_enable(struct rkisp1_isp_params_vdev *params_vdev,
+			    const struct cifisp_awb_meas_config *arg, bool en)
+{
+	u32 reg_val = rkisp1_ioread32(params_vdev, CIF_ISP_AWB_PROP);
+
+	/* switch off */
+	reg_val &= CIF_ISP_AWB_MODE_MASK_NONE;
+
+	if (en) {
+		if (arg->awb_mode == CIFISP_AWB_MODE_RGB)
+			reg_val |= CIF_ISP_AWB_MODE_RGB_EN;
+		else
+			reg_val |= CIF_ISP_AWB_MODE_YCBCR_EN;
+
+		rkisp1_iowrite32(params_vdev, reg_val, CIF_ISP_AWB_PROP);
+
+		/* Measurements require AWB block be active. */
+		/* TODO: need to enable here ? awb_gain_enable has done this */
+		isp_param_set_bits(params_vdev, CIF_ISP_CTRL,
+				   CIF_ISP_CTRL_ISP_AWB_ENA);
+	} else {
+		rkisp1_iowrite32(params_vdev,
+				 reg_val, CIF_ISP_AWB_PROP);
+		isp_param_clear_bits(params_vdev, CIF_ISP_CTRL,
+				     CIF_ISP_CTRL_ISP_AWB_ENA);
+	}
+}
+
+static void awb_gain_config(struct rkisp1_isp_params_vdev *params_vdev,
+			    const struct cifisp_awb_gain_config *arg)
+{
+	rkisp1_iowrite32(params_vdev,
+			 CIF_ISP_AWB_GAIN_R_SET(arg->gain_green_r) |
+			 arg->gain_green_b, CIF_ISP_AWB_GAIN_G);
+
+	rkisp1_iowrite32(params_vdev, CIF_ISP_AWB_GAIN_R_SET(arg->gain_red) |
+			 arg->gain_blue, CIF_ISP_AWB_GAIN_RB);
+}
+
+static void aec_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_aec_config *arg)
+{
+	unsigned int block_hsize, block_vsize;
+	u32 exp_ctrl;
+
+	/* avoid to override the old enable value */
+	exp_ctrl = rkisp1_ioread32(params_vdev, CIF_ISP_EXP_CTRL);
+	exp_ctrl &= CIF_ISP_EXP_ENA;
+	if (arg->autostop)
+		exp_ctrl |= CIF_ISP_EXP_CTRL_AUTOSTOP;
+	if (arg->mode == CIFISP_EXP_MEASURING_MODE_1)
+		exp_ctrl |= CIF_ISP_EXP_CTRL_MEASMODE_1;
+	rkisp1_iowrite32(params_vdev, exp_ctrl, CIF_ISP_EXP_CTRL);
+
+	rkisp1_iowrite32(params_vdev,
+			 arg->meas_window.h_offs, CIF_ISP_EXP_H_OFFSET);
+	rkisp1_iowrite32(params_vdev,
+			 arg->meas_window.v_offs, CIF_ISP_EXP_V_OFFSET);
+
+	block_hsize = arg->meas_window.h_size / CIF_ISP_EXP_COLUMN_NUM - 1;
+	block_vsize = arg->meas_window.v_size / CIF_ISP_EXP_ROW_NUM - 1;
+
+	rkisp1_iowrite32(params_vdev, CIF_ISP_EXP_H_SIZE_SET(block_hsize),
+			 CIF_ISP_EXP_H_SIZE);
+	rkisp1_iowrite32(params_vdev, CIF_ISP_EXP_V_SIZE_SET(block_vsize),
+			 CIF_ISP_EXP_V_SIZE);
+}
+
+static void cproc_config(struct rkisp1_isp_params_vdev *params_vdev,
+			 const struct cifisp_cproc_config *arg)
+{
+	struct cifisp_isp_other_cfg *cur_other_cfg =
+						&params_vdev->cur_params.others;
+	struct cifisp_ie_config *cur_ie_config = &cur_other_cfg->ie_config;
+	u32 effect = cur_ie_config->effect;
+	u32 quantization = params_vdev->quantization;
+
+	rkisp1_iowrite32(params_vdev, arg->contrast, CIF_C_PROC_CONTRAST);
+	rkisp1_iowrite32(params_vdev, arg->hue, CIF_C_PROC_HUE);
+	rkisp1_iowrite32(params_vdev, arg->sat, CIF_C_PROC_SATURATION);
+	rkisp1_iowrite32(params_vdev, arg->brightness, CIF_C_PROC_BRIGHTNESS);
+
+	if (quantization != V4L2_QUANTIZATION_FULL_RANGE ||
+	    effect != V4L2_COLORFX_NONE) {
+		isp_param_clear_bits(params_vdev, CIF_C_PROC_CTRL,
+				     CIF_C_PROC_YOUT_FULL |
+				     CIF_C_PROC_YIN_FULL |
+				     CIF_C_PROC_COUT_FULL);
+	} else {
+		isp_param_set_bits(params_vdev, CIF_C_PROC_CTRL,
+				   CIF_C_PROC_YOUT_FULL |
+				   CIF_C_PROC_YIN_FULL |
+				   CIF_C_PROC_COUT_FULL);
+	}
+}
+
+static void hst_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_hst_config *arg)
+{
+	unsigned int block_hsize, block_vsize;
+	static const u32 hist_weight_regs[] = {
+		CIF_ISP_HIST_WEIGHT_00TO30, CIF_ISP_HIST_WEIGHT_40TO21,
+		CIF_ISP_HIST_WEIGHT_31TO12, CIF_ISP_HIST_WEIGHT_22TO03,
+		CIF_ISP_HIST_WEIGHT_13TO43, CIF_ISP_HIST_WEIGHT_04TO34,
+		CIF_ISP_HIST_WEIGHT_44,
+	};
+	const u8 *weight;
+	unsigned int i;
+	u32 hist_prop;
+
+	/* avoid to override the old enable value */
+	hist_prop = rkisp1_ioread32(params_vdev, CIF_ISP_HIST_PROP);
+	hist_prop &= CIF_ISP_HIST_PROP_MODE_MASK;
+	hist_prop |= CIF_ISP_HIST_PREDIV_SET(arg->histogram_predivider);
+	rkisp1_iowrite32(params_vdev, hist_prop, CIF_ISP_HIST_PROP);
+	rkisp1_iowrite32(params_vdev,
+			arg->meas_window.h_offs,
+			CIF_ISP_HIST_H_OFFS);
+	rkisp1_iowrite32(params_vdev,
+			arg->meas_window.v_offs,
+			CIF_ISP_HIST_V_OFFS);
+
+	block_hsize = arg->meas_window.h_size / CIF_ISP_HIST_COLUMN_NUM - 1;
+	block_vsize = arg->meas_window.v_size / CIF_ISP_HIST_ROW_NUM - 1;
+
+	rkisp1_iowrite32(params_vdev, block_hsize, CIF_ISP_HIST_H_SIZE);
+	rkisp1_iowrite32(params_vdev, block_vsize, CIF_ISP_HIST_V_SIZE);
+
+	weight = arg->hist_weight;
+	for (i = 0; i < ARRAY_SIZE(hist_weight_regs); ++i, weight += 4)
+		rkisp1_iowrite32(params_vdev, CIF_ISP_HIST_WEIGHT_SET(
+				 weight[0], weight[1], weight[2], weight[3]),
+				 hist_weight_regs[i]);
+}
+
+static void hst_enable(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_hst_config *arg, bool en)
+{
+	if (en)	{
+		u32 hist_prop = rkisp1_ioread32(params_vdev, CIF_ISP_HIST_PROP);
+
+		hist_prop &= ~CIF_ISP_HIST_PROP_MODE_MASK;
+		hist_prop |= arg->mode;
+		isp_param_set_bits(params_vdev, CIF_ISP_HIST_PROP, hist_prop);
+	} else {
+		isp_param_clear_bits(params_vdev, CIF_ISP_HIST_PROP,
+				CIF_ISP_HIST_PROP_MODE_MASK);
+	}
+}
+
+static void afm_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_afc_config *arg)
+{
+	size_t num_of_win = min_t(size_t, ARRAY_SIZE(arg->afm_win),
+				  arg->num_afm_win);
+	u32 afm_ctrl = rkisp1_ioread32(params_vdev, CIF_ISP_AFM_CTRL);
+	unsigned int i;
+
+	/* Switch off to configure. */
+	isp_param_clear_bits(params_vdev, CIF_ISP_AFM_CTRL, CIF_ISP_AFM_ENA);
+
+	for (i = 0; i < num_of_win; i++) {
+		rkisp1_iowrite32(params_vdev,
+				 CIF_ISP_AFM_WINDOW_X(arg->afm_win[i].h_offs) |
+				 CIF_ISP_AFM_WINDOW_Y(arg->afm_win[i].v_offs),
+				 CIF_ISP_AFM_LT_A + i * 8);
+		rkisp1_iowrite32(params_vdev,
+				 CIF_ISP_AFM_WINDOW_X(arg->afm_win[i].h_size +
+						      arg->afm_win[i].h_offs) |
+				 CIF_ISP_AFM_WINDOW_Y(arg->afm_win[i].v_size +
+						      arg->afm_win[i].v_offs),
+				 CIF_ISP_AFM_RB_A + i * 8);
+	}
+	rkisp1_iowrite32(params_vdev, arg->thres, CIF_ISP_AFM_THRES);
+	rkisp1_iowrite32(params_vdev, arg->var_shift, CIF_ISP_AFM_VAR_SHIFT);
+	/* restore afm status */
+	rkisp1_iowrite32(params_vdev, afm_ctrl, CIF_ISP_AFM_CTRL);
+}
+
+static void ie_config(struct rkisp1_isp_params_vdev *params_vdev,
+		      const struct cifisp_ie_config *arg)
+{
+	u32 eff_ctrl;
+
+	eff_ctrl = rkisp1_ioread32(params_vdev, CIF_IMG_EFF_CTRL);
+	eff_ctrl &= ~CIF_IMG_EFF_CTRL_MODE_MASK;
+
+	if (params_vdev->quantization == V4L2_QUANTIZATION_FULL_RANGE)
+		eff_ctrl |= CIF_IMG_EFF_CTRL_YCBCR_FULL;
+
+	switch (arg->effect) {
+	case V4L2_COLORFX_SEPIA:
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_SEPIA;
+		break;
+	case V4L2_COLORFX_SET_CBCR:
+		rkisp1_iowrite32(params_vdev, arg->eff_tint, CIF_IMG_EFF_TINT);
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_SEPIA;
+		break;
+		/*
+		 * Color selection is similar to water color(AQUA):
+		 * grayscale + selected color w threshold
+		 */
+	case V4L2_COLORFX_AQUA:
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_COLOR_SEL;
+		rkisp1_iowrite32(params_vdev, arg->color_sel,
+				 CIF_IMG_EFF_COLOR_SEL);
+		break;
+	case V4L2_COLORFX_EMBOSS:
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_EMBOSS;
+		rkisp1_iowrite32(params_vdev, arg->eff_mat_1,
+				 CIF_IMG_EFF_MAT_1);
+		rkisp1_iowrite32(params_vdev, arg->eff_mat_2,
+				 CIF_IMG_EFF_MAT_2);
+		rkisp1_iowrite32(params_vdev, arg->eff_mat_3,
+				 CIF_IMG_EFF_MAT_3);
+		break;
+	case V4L2_COLORFX_SKETCH:
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_SKETCH;
+		rkisp1_iowrite32(params_vdev, arg->eff_mat_3,
+				 CIF_IMG_EFF_MAT_3);
+		rkisp1_iowrite32(params_vdev, arg->eff_mat_4,
+				 CIF_IMG_EFF_MAT_4);
+		rkisp1_iowrite32(params_vdev, arg->eff_mat_5,
+				 CIF_IMG_EFF_MAT_5);
+		break;
+	case V4L2_COLORFX_BW:
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_BLACKWHITE;
+		break;
+	case V4L2_COLORFX_NEGATIVE:
+		eff_ctrl |= CIF_IMG_EFF_CTRL_MODE_NEGATIVE;
+		break;
+	default:
+		break;
+	}
+
+	rkisp1_iowrite32(params_vdev, eff_ctrl, CIF_IMG_EFF_CTRL);
+}
+
+static void ie_enable(struct rkisp1_isp_params_vdev *params_vdev, bool en)
+{
+	if (en) {
+		isp_param_set_bits(params_vdev, CIF_ICCL, CIF_ICCL_IE_CLK);
+		rkisp1_iowrite32(params_vdev, CIF_IMG_EFF_CTRL_ENABLE,
+				 CIF_IMG_EFF_CTRL);
+		isp_param_set_bits(params_vdev, CIF_IMG_EFF_CTRL,
+				   CIF_IMG_EFF_CTRL_CFG_UPD);
+	} else {
+		isp_param_clear_bits(params_vdev, CIF_IMG_EFF_CTRL,
+				     CIF_IMG_EFF_CTRL_ENABLE);
+		isp_param_clear_bits(params_vdev, CIF_ICCL, CIF_ICCL_IE_CLK);
+	}
+}
+
+static void csm_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       bool full_range)
+{
+	static const u16 full_range_coeff[] = {
+		0x0026, 0x004b, 0x000f,
+		0x01ea, 0x01d6, 0x0040,
+		0x0040, 0x01ca, 0x01f6
+	};
+	static const u16 limited_range_coeff[] = {
+		0x0021, 0x0040, 0x000d,
+		0x01ed, 0x01db, 0x0038,
+		0x0038, 0x01d1, 0x01f7,
+	};
+	unsigned int i;
+
+	if (full_range) {
+		for (i = 0; i < ARRAY_SIZE(full_range_coeff); i++)
+			rkisp1_iowrite32(params_vdev, full_range_coeff[i],
+					 CIF_ISP_CC_COEFF_0 + i * 4);
+
+		isp_param_set_bits(params_vdev, CIF_ISP_CTRL,
+				   CIF_ISP_CTRL_ISP_CSM_Y_FULL_ENA |
+				   CIF_ISP_CTRL_ISP_CSM_C_FULL_ENA);
+	} else {
+		for (i = 0; i < ARRAY_SIZE(limited_range_coeff); i++)
+			rkisp1_iowrite32(params_vdev, limited_range_coeff[i],
+					 CIF_ISP_CC_COEFF_0 + i * 4);
+
+		isp_param_clear_bits(params_vdev, CIF_ISP_CTRL,
+				     CIF_ISP_CTRL_ISP_CSM_Y_FULL_ENA |
+				     CIF_ISP_CTRL_ISP_CSM_C_FULL_ENA);
+	}
+}
+
+/* ISP De-noise Pre-Filter(DPF) function */
+static void dpf_config(struct rkisp1_isp_params_vdev *params_vdev,
+		       const struct cifisp_dpf_config *arg)
+{
+	unsigned int isp_dpf_mode, spatial_coeff, i;
+
+	switch (arg->gain.mode) {
+	case CIFISP_DPF_GAIN_USAGE_NF_GAINS:
+		isp_dpf_mode = CIF_ISP_DPF_MODE_USE_NF_GAIN |
+				CIF_ISP_DPF_MODE_AWB_GAIN_COMP;
+		break;
+	case CIFISP_DPF_GAIN_USAGE_LSC_GAINS:
+		isp_dpf_mode = CIF_ISP_DPF_MODE_LSC_GAIN_COMP;
+		break;
+	case CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS:
+		isp_dpf_mode = CIF_ISP_DPF_MODE_USE_NF_GAIN |
+				CIF_ISP_DPF_MODE_AWB_GAIN_COMP |
+				CIF_ISP_DPF_MODE_LSC_GAIN_COMP;
+		break;
+	case CIFISP_DPF_GAIN_USAGE_AWB_GAINS:
+		isp_dpf_mode = CIF_ISP_DPF_MODE_AWB_GAIN_COMP;
+		break;
+	case CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS:
+		isp_dpf_mode = CIF_ISP_DPF_MODE_LSC_GAIN_COMP |
+				CIF_ISP_DPF_MODE_AWB_GAIN_COMP;
+		break;
+	case CIFISP_DPF_GAIN_USAGE_DISABLED:
+	default:
+		isp_dpf_mode = 0;
+		break;
+	}
+
+	if (arg->nll.scale_mode == CIFISP_NLL_SCALE_LOGARITHMIC)
+		isp_dpf_mode |= CIF_ISP_DPF_MODE_NLL_SEGMENTATION;
+	if (arg->rb_flt.fltsize == CIFISP_DPF_RB_FILTERSIZE_9x9)
+		isp_dpf_mode |= CIF_ISP_DPF_MODE_RB_FLTSIZE_9x9;
+	if (!arg->rb_flt.r_enable)
+		isp_dpf_mode |= CIF_ISP_DPF_MODE_R_FLT_DIS;
+	if (!arg->rb_flt.b_enable)
+		isp_dpf_mode |= CIF_ISP_DPF_MODE_B_FLT_DIS;
+	if (!arg->g_flt.gb_enable)
+		isp_dpf_mode |= CIF_ISP_DPF_MODE_GB_FLT_DIS;
+	if (!arg->g_flt.gr_enable)
+		isp_dpf_mode |= CIF_ISP_DPF_MODE_GR_FLT_DIS;
+
+	isp_param_set_bits(params_vdev, CIF_ISP_DPF_MODE, isp_dpf_mode);
+	rkisp1_iowrite32(params_vdev, arg->gain.nf_b_gain,
+			 CIF_ISP_DPF_NF_GAIN_B);
+	rkisp1_iowrite32(params_vdev, arg->gain.nf_r_gain,
+			 CIF_ISP_DPF_NF_GAIN_R);
+	rkisp1_iowrite32(params_vdev, arg->gain.nf_gb_gain,
+			 CIF_ISP_DPF_NF_GAIN_GB);
+	rkisp1_iowrite32(params_vdev, arg->gain.nf_gr_gain,
+			 CIF_ISP_DPF_NF_GAIN_GR);
+
+	for (i = 0; i < CIFISP_DPF_MAX_NLF_COEFFS; i++) {
+		rkisp1_iowrite32(params_vdev, arg->nll.coeff[i],
+				 CIF_ISP_DPF_NULL_COEFF_0 + i * 4);
+	}
+
+	spatial_coeff = arg->g_flt.spatial_coeff[0] |
+			(arg->g_flt.spatial_coeff[1] << 8) |
+			(arg->g_flt.spatial_coeff[2] << 16) |
+			(arg->g_flt.spatial_coeff[3] << 24);
+	rkisp1_iowrite32(params_vdev, spatial_coeff,
+			 CIF_ISP_DPF_S_WEIGHT_G_1_4);
+
+	spatial_coeff = arg->g_flt.spatial_coeff[4] |
+			(arg->g_flt.spatial_coeff[5] << 8);
+	rkisp1_iowrite32(params_vdev, spatial_coeff,
+			 CIF_ISP_DPF_S_WEIGHT_G_5_6);
+
+	spatial_coeff = arg->rb_flt.spatial_coeff[0] |
+			(arg->rb_flt.spatial_coeff[1] << 8) |
+			(arg->rb_flt.spatial_coeff[2] << 16) |
+			(arg->rb_flt.spatial_coeff[3] << 24);
+	rkisp1_iowrite32(params_vdev, spatial_coeff,
+			 CIF_ISP_DPF_S_WEIGHT_RB_1_4);
+
+	spatial_coeff = arg->rb_flt.spatial_coeff[4] |
+			(arg->rb_flt.spatial_coeff[5] << 8);
+	rkisp1_iowrite32(params_vdev, spatial_coeff,
+			CIF_ISP_DPF_S_WEIGHT_RB_5_6);
+}
+
+static void dpf_strength_config(struct rkisp1_isp_params_vdev *params_vdev,
+				const struct cifisp_dpf_strength_config *arg)
+{
+	rkisp1_iowrite32(params_vdev, arg->b, CIF_ISP_DPF_STRENGTH_B);
+	rkisp1_iowrite32(params_vdev, arg->g, CIF_ISP_DPF_STRENGTH_G);
+	rkisp1_iowrite32(params_vdev, arg->r, CIF_ISP_DPF_STRENGTH_R);
+}
+
+static __maybe_unused
+void __isp_isr_other_config(struct rkisp1_isp_params_vdev *params_vdev,
+			    const struct rkisp1_isp_params_cfg *new_params)
+{
+	unsigned int module_en_update, module_cfg_update, module_ens;
+
+	module_en_update = new_params->module_en_update;
+	module_cfg_update = new_params->module_cfg_update;
+	module_ens = new_params->module_ens;
+
+	if ((module_en_update & CIFISP_MODULE_DPCC) ||
+	    (module_cfg_update & CIFISP_MODULE_DPCC)) {
+		/*update dpc config */
+		if ((module_cfg_update & CIFISP_MODULE_DPCC))
+			dpcc_config(params_vdev,
+				    &new_params->others.dpcc_config);
+
+		if (module_en_update & CIFISP_MODULE_DPCC) {
+			if (!!(module_ens & CIFISP_MODULE_DPCC))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_DPCC_MODE,
+						   CIF_ISP_DPCC_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_DPCC_MODE,
+						     CIF_ISP_DPCC_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_BLS) ||
+	    (module_cfg_update & CIFISP_MODULE_BLS)) {
+		/* update bls config */
+		if ((module_cfg_update & CIFISP_MODULE_BLS))
+			bls_config(params_vdev, &new_params->others.bls_config);
+
+		if (module_en_update & CIFISP_MODULE_BLS) {
+			if (!!(module_ens & CIFISP_MODULE_BLS))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_BLS_CTRL,
+						   CIF_ISP_BLS_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_BLS_CTRL,
+						     CIF_ISP_BLS_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_SDG) ||
+	    (module_cfg_update & CIFISP_MODULE_SDG)) {
+		/* update sdg config */
+		if ((module_cfg_update & CIFISP_MODULE_SDG))
+			sdg_config(params_vdev, &new_params->others.sdg_config);
+
+		if (module_en_update & CIFISP_MODULE_SDG) {
+			if (!!(module_ens & CIFISP_MODULE_SDG))
+				isp_param_set_bits(params_vdev,
+						CIF_ISP_CTRL,
+						CIF_ISP_CTRL_ISP_GAMMA_IN_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						CIF_ISP_CTRL,
+						CIF_ISP_CTRL_ISP_GAMMA_IN_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_LSC) ||
+	    (module_cfg_update & CIFISP_MODULE_LSC)) {
+		/* update lsc config */
+		if ((module_cfg_update & CIFISP_MODULE_LSC))
+			lsc_config(params_vdev, &new_params->others.lsc_config);
+
+		if (module_en_update & CIFISP_MODULE_LSC) {
+			if (!!(module_ens & CIFISP_MODULE_LSC))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_LSC_CTRL,
+						   CIF_ISP_LSC_CTRL_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_LSC_CTRL,
+						     CIF_ISP_LSC_CTRL_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_AWB_GAIN) ||
+	    (module_cfg_update & CIFISP_MODULE_AWB_GAIN)) {
+		/* update awb gains */
+		if ((module_cfg_update & CIFISP_MODULE_AWB_GAIN))
+			awb_gain_config(params_vdev,
+					&new_params->others.awb_gain_config);
+
+		if (module_en_update & CIFISP_MODULE_AWB_GAIN) {
+			if (!!(module_ens & CIFISP_MODULE_AWB_GAIN))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_CTRL,
+						   CIF_ISP_CTRL_ISP_AWB_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_CTRL,
+						     CIF_ISP_CTRL_ISP_AWB_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_BDM) ||
+	    (module_cfg_update & CIFISP_MODULE_BDM)) {
+		/* update bdm config */
+		if ((module_cfg_update & CIFISP_MODULE_BDM))
+			bdm_config(params_vdev, &new_params->others.bdm_config);
+
+		if (module_en_update & CIFISP_MODULE_BDM) {
+			if (!!(module_ens & CIFISP_MODULE_BDM))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_DEMOSAIC,
+						   CIF_ISP_DEMOSAIC_BYPASS);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_DEMOSAIC,
+						     CIF_ISP_DEMOSAIC_BYPASS);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_FLT) ||
+	    (module_cfg_update & CIFISP_MODULE_FLT)) {
+		/* update filter config */
+		if ((module_cfg_update & CIFISP_MODULE_FLT))
+			flt_config(params_vdev, &new_params->others.flt_config);
+
+		if (module_en_update & CIFISP_MODULE_FLT) {
+			if (!!(module_ens & CIFISP_MODULE_FLT))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_FILT_MODE,
+						   CIF_ISP_FLT_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_FILT_MODE,
+						     CIF_ISP_FLT_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_CTK) ||
+	    (module_cfg_update & CIFISP_MODULE_CTK)) {
+		/* update ctk config */
+		if ((module_cfg_update & CIFISP_MODULE_CTK))
+			ctk_config(params_vdev, &new_params->others.ctk_config);
+
+		if (module_en_update & CIFISP_MODULE_CTK)
+			ctk_enable(params_vdev,
+				   !!(module_ens & CIFISP_MODULE_CTK));
+	}
+
+	if ((module_en_update & CIFISP_MODULE_GOC) ||
+	    (module_cfg_update & CIFISP_MODULE_GOC)) {
+		/* update goc config */
+		if ((module_cfg_update & CIFISP_MODULE_GOC))
+			goc_config(params_vdev, &new_params->others.goc_config);
+
+		if (module_en_update & CIFISP_MODULE_GOC) {
+			if (!!(module_ens & CIFISP_MODULE_GOC))
+				isp_param_set_bits(params_vdev,
+						CIF_ISP_CTRL,
+						CIF_ISP_CTRL_ISP_GAMMA_OUT_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						CIF_ISP_CTRL,
+						CIF_ISP_CTRL_ISP_GAMMA_OUT_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_CPROC) ||
+	    (module_cfg_update & CIFISP_MODULE_CPROC)) {
+		/* update cproc config */
+		if ((module_cfg_update & CIFISP_MODULE_CPROC)) {
+			cproc_config(params_vdev,
+				     &new_params->others.cproc_config);
+
+		}
+
+		if (module_en_update & CIFISP_MODULE_CPROC) {
+			if (!!(module_ens & CIFISP_MODULE_CPROC))
+				isp_param_set_bits(params_vdev,
+						   CIF_C_PROC_CTRL,
+						   CIF_C_PROC_CTR_ENABLE);
+			else
+				isp_param_clear_bits(params_vdev,
+						   CIF_C_PROC_CTRL,
+						   CIF_C_PROC_CTR_ENABLE);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_IE) ||
+	    (module_cfg_update & CIFISP_MODULE_IE)) {
+		/* update ie config */
+		if ((module_cfg_update & CIFISP_MODULE_IE))
+			ie_config(params_vdev, &new_params->others.ie_config);
+
+		if (module_en_update & CIFISP_MODULE_IE)
+			ie_enable(params_vdev,
+				   !!(module_ens & CIFISP_MODULE_IE));
+	}
+
+	if ((module_en_update & CIFISP_MODULE_DPF) ||
+	    (module_cfg_update & CIFISP_MODULE_DPF)) {
+		/* update dpf  config */
+		if ((module_cfg_update & CIFISP_MODULE_DPF))
+			dpf_config(params_vdev, &new_params->others.dpf_config);
+
+		if (module_en_update & CIFISP_MODULE_DPF) {
+			if (!!(module_ens & CIFISP_MODULE_DPF))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_DPF_MODE,
+						   CIF_ISP_DPF_MODE_EN);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_DPF_MODE,
+						     CIF_ISP_DPF_MODE_EN);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_DPF_STRENGTH) ||
+	    (module_cfg_update & CIFISP_MODULE_DPF_STRENGTH)) {
+		/* update dpf strength config */
+		dpf_strength_config(params_vdev,
+				    &new_params->others.dpf_strength_config);
+	}
+}
+
+static __maybe_unused
+void __isp_isr_meas_config(struct rkisp1_isp_params_vdev *params_vdev,
+			   struct  rkisp1_isp_params_cfg *new_params)
+{
+	unsigned int module_en_update, module_cfg_update, module_ens;
+
+	module_en_update = new_params->module_en_update;
+	module_cfg_update = new_params->module_cfg_update;
+	module_ens = new_params->module_ens;
+
+	if ((module_en_update & CIFISP_MODULE_AWB) ||
+	    (module_cfg_update & CIFISP_MODULE_AWB)) {
+		/* update awb config */
+		if ((module_cfg_update & CIFISP_MODULE_AWB))
+			awb_meas_config(params_vdev,
+					&new_params->meas.awb_meas_config);
+
+		if (module_en_update & CIFISP_MODULE_AWB)
+			awb_meas_enable(params_vdev,
+					&new_params->meas.awb_meas_config,
+					!!(module_ens & CIFISP_MODULE_AWB));
+	}
+
+	if ((module_en_update & CIFISP_MODULE_AFC) ||
+	    (module_cfg_update & CIFISP_MODULE_AFC)) {
+		/* update afc config */
+		if ((module_cfg_update & CIFISP_MODULE_AFC))
+			afm_config(params_vdev, &new_params->meas.afc_config);
+
+		if (module_en_update & CIFISP_MODULE_AFC) {
+			if (!!(module_ens & CIFISP_MODULE_AFC))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_AFM_CTRL,
+						   CIF_ISP_AFM_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_AFM_CTRL,
+						     CIF_ISP_AFM_ENA);
+		}
+	}
+
+	if ((module_en_update & CIFISP_MODULE_HST) ||
+	    (module_cfg_update & CIFISP_MODULE_HST)) {
+		/* update hst config */
+		if ((module_cfg_update & CIFISP_MODULE_HST))
+			hst_config(params_vdev, &new_params->meas.hst_config);
+
+		if (module_en_update & CIFISP_MODULE_HST)
+			hst_enable(params_vdev,
+				   &new_params->meas.hst_config,
+				   !!(module_ens & CIFISP_MODULE_HST));
+	}
+
+	if ((module_en_update & CIFISP_MODULE_AEC) ||
+	    (module_cfg_update & CIFISP_MODULE_AEC)) {
+		/* update aec config */
+		if ((module_cfg_update & CIFISP_MODULE_AEC))
+			aec_config(params_vdev, &new_params->meas.aec_config);
+
+		if (module_en_update & CIFISP_MODULE_AEC) {
+			if (!!(module_ens & CIFISP_MODULE_AEC))
+				isp_param_set_bits(params_vdev,
+						   CIF_ISP_EXP_CTRL,
+						   CIF_ISP_EXP_ENA);
+			else
+				isp_param_clear_bits(params_vdev,
+						     CIF_ISP_EXP_CTRL,
+						     CIF_ISP_EXP_ENA);
+		}
+	}
+}
+
+void rkisp1_params_isr(struct rkisp1_isp_params_vdev *params_vdev, u32 isp_mis)
+{
+	struct rkisp1_isp_params_cfg *new_params;
+	struct rkisp1_buffer *cur_buf = NULL;
+	unsigned int cur_frame_id = -1;
+
+	cur_frame_id =
+		atomic_read(&params_vdev->dev->isp_sdev.frm_sync_seq) - 1;
+
+	spin_lock(&params_vdev->config_lock);
+	if (!params_vdev->streamon) {
+		spin_unlock(&params_vdev->config_lock);
+		return;
+	}
+
+	/* get one empty buffer */
+	if (!list_empty(&params_vdev->params))
+		cur_buf = list_first_entry(&params_vdev->params,
+					   struct rkisp1_buffer, queue);
+	spin_unlock(&params_vdev->config_lock);
+
+	if (!cur_buf)
+		return;
+
+	new_params = (struct rkisp1_isp_params_cfg *)(cur_buf->vaddr[0]);
+
+	if (isp_mis & CIF_ISP_FRAME) {
+		u32 isp_ctrl;
+
+		__isp_isr_other_config(params_vdev, new_params);
+		__isp_isr_meas_config(params_vdev, new_params);
+
+		/* update shadow register immediately */
+		isp_ctrl = rkisp1_ioread32(params_vdev, CIF_ISP_CTRL);
+		isp_ctrl |= CIF_ISP_CTRL_ISP_CFG_UPD;
+		rkisp1_iowrite32(params_vdev, isp_ctrl, CIF_ISP_CTRL);
+
+		spin_lock(&params_vdev->config_lock);
+		list_del(&cur_buf->queue);
+		spin_unlock(&params_vdev->config_lock);
+
+		cur_buf->vb.sequence = cur_frame_id;
+		vb2_buffer_done(&cur_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
+	}
+}
+
+static const struct cifisp_awb_meas_config awb_params_default_config = {
+	{
+		0, 0, RKISP1_DEFAULT_WIDTH, RKISP1_DEFAULT_HEIGHT
+	},
+	CIFISP_AWB_MODE_YCBCR, 200, 30, 20, 20, 0, 128, 128
+};
+
+static const struct cifisp_aec_config aec_params_default_config = {
+	CIFISP_EXP_MEASURING_MODE_0,
+	CIFISP_EXP_CTRL_AUTOSTOP_0,
+	{
+		RKISP1_DEFAULT_WIDTH >> 2, RKISP1_DEFAULT_HEIGHT >> 2,
+		RKISP1_DEFAULT_WIDTH >> 1, RKISP1_DEFAULT_HEIGHT >> 1
+	}
+};
+
+static const struct cifisp_hst_config hst_params_default_config = {
+	CIFISP_HISTOGRAM_MODE_RGB_COMBINED,
+	3,
+	{
+		RKISP1_DEFAULT_WIDTH >> 2, RKISP1_DEFAULT_HEIGHT >> 2,
+		RKISP1_DEFAULT_WIDTH >> 1, RKISP1_DEFAULT_HEIGHT >> 1
+	},
+	{
+		0, /* To be filled in with 0x01 at runtime. */
+	}
+};
+
+static const struct cifisp_afc_config afc_params_default_config = {
+	1,
+	{
+		{
+			300, 225, 200, 150
+		}
+	},
+	4,
+	14
+};
+
+static
+void rkisp1_params_config_parameter(struct rkisp1_isp_params_vdev *params_vdev)
+{
+	struct cifisp_hst_config hst = hst_params_default_config;
+
+	spin_lock(&params_vdev->config_lock);
+
+	awb_meas_config(params_vdev, &awb_params_default_config);
+	awb_meas_enable(params_vdev, &awb_params_default_config, true);
+
+	aec_config(params_vdev, &aec_params_default_config);
+	isp_param_set_bits(params_vdev, CIF_ISP_EXP_CTRL, CIF_ISP_EXP_ENA);
+
+	afm_config(params_vdev, &afc_params_default_config);
+	isp_param_set_bits(params_vdev, CIF_ISP_AFM_CTRL, CIF_ISP_AFM_ENA);
+
+	memset(hst.hist_weight, 0x01, sizeof(hst.hist_weight));
+	hst_config(params_vdev, &hst);
+	isp_param_set_bits(params_vdev, CIF_ISP_HIST_PROP,
+			   ~CIF_ISP_HIST_PROP_MODE_MASK |
+			   hst_params_default_config.mode);
+
+	/* set the  range */
+	if (params_vdev->quantization == V4L2_QUANTIZATION_FULL_RANGE)
+		csm_config(params_vdev, true);
+	else
+		csm_config(params_vdev, false);
+
+	/* override the default things */
+	__isp_isr_other_config(params_vdev, &params_vdev->cur_params);
+	__isp_isr_meas_config(params_vdev, &params_vdev->cur_params);
+
+	spin_unlock(&params_vdev->config_lock);
+}
+
+/* Not called when the camera active, thus not isr protection. */
+void rkisp1_params_configure_isp(struct rkisp1_isp_params_vdev *params_vdev,
+			  struct ispsd_in_fmt *in_fmt,
+			  enum v4l2_quantization quantization)
+{
+	params_vdev->quantization = quantization;
+	params_vdev->raw_type = in_fmt->bayer_pat;
+	rkisp1_params_config_parameter(params_vdev);
+}
+
+/* Not called when the camera active, thus not isr protection. */
+void rkisp1_params_disable_isp(struct rkisp1_isp_params_vdev *params_vdev)
+{
+	isp_param_clear_bits(params_vdev, CIF_ISP_DPCC_MODE, CIF_ISP_DPCC_ENA);
+	isp_param_clear_bits(params_vdev, CIF_ISP_LSC_CTRL,
+			     CIF_ISP_LSC_CTRL_ENA);
+	isp_param_clear_bits(params_vdev, CIF_ISP_BLS_CTRL, CIF_ISP_BLS_ENA);
+	isp_param_clear_bits(params_vdev, CIF_ISP_CTRL,
+			     CIF_ISP_CTRL_ISP_GAMMA_IN_ENA);
+	isp_param_clear_bits(params_vdev, CIF_ISP_CTRL,
+			     CIF_ISP_CTRL_ISP_GAMMA_OUT_ENA);
+	isp_param_clear_bits(params_vdev, CIF_ISP_DEMOSAIC,
+			     CIF_ISP_DEMOSAIC_BYPASS);
+	isp_param_clear_bits(params_vdev, CIF_ISP_FILT_MODE, CIF_ISP_FLT_ENA);
+	awb_meas_enable(params_vdev, NULL, false);
+	isp_param_clear_bits(params_vdev, CIF_ISP_CTRL,
+			     CIF_ISP_CTRL_ISP_AWB_ENA);
+	isp_param_clear_bits(params_vdev, CIF_ISP_EXP_CTRL, CIF_ISP_EXP_ENA);
+	ctk_enable(params_vdev, false);
+	isp_param_clear_bits(params_vdev, CIF_C_PROC_CTRL,
+			     CIF_C_PROC_CTR_ENABLE);
+	hst_enable(params_vdev, NULL, false);
+	isp_param_clear_bits(params_vdev, CIF_ISP_AFM_CTRL, CIF_ISP_AFM_ENA);
+	ie_enable(params_vdev, false);
+	isp_param_clear_bits(params_vdev, CIF_ISP_DPF_MODE,
+			     CIF_ISP_DPF_MODE_EN);
+}
+
+static int rkisp1_params_enum_fmt_meta_out(struct file *file, void *priv,
+					   struct v4l2_fmtdesc *f)
+{
+	struct video_device *video = video_devdata(file);
+	struct rkisp1_isp_params_vdev *params_vdev = video_get_drvdata(video);
+
+	if (f->index > 0 || f->type != video->queue->type)
+		return -EINVAL;
+
+	f->pixelformat = params_vdev->vdev_fmt.fmt.meta.dataformat;
+
+	return 0;
+}
+
+static int rkisp1_params_g_fmt_meta_out(struct file *file, void *fh,
+					struct v4l2_format *f)
+{
+	struct video_device *video = video_devdata(file);
+	struct rkisp1_isp_params_vdev *params_vdev = video_get_drvdata(video);
+	struct v4l2_meta_format *meta = &f->fmt.meta;
+
+	if (f->type != video->queue->type)
+		return -EINVAL;
+
+	memset(meta, 0, sizeof(*meta));
+	meta->dataformat = params_vdev->vdev_fmt.fmt.meta.dataformat;
+	meta->buffersize = params_vdev->vdev_fmt.fmt.meta.buffersize;
+
+	return 0;
+}
+
+static int rkisp1_params_querycap(struct file *file,
+				  void *priv, struct v4l2_capability *cap)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	strscpy(cap->driver, DRIVER_NAME, sizeof(cap->driver));
+	strscpy(cap->card, vdev->name, sizeof(cap->card));
+	strscpy(cap->bus_info, "platform: " DRIVER_NAME, sizeof(cap->bus_info));
+
+	return 0;
+}
+
+/* ISP params video device IOCTLs */
+static const struct v4l2_ioctl_ops rkisp1_params_ioctl = {
+	.vidioc_reqbufs = vb2_ioctl_reqbufs,
+	.vidioc_querybuf = vb2_ioctl_querybuf,
+	.vidioc_create_bufs = vb2_ioctl_create_bufs,
+	.vidioc_qbuf = vb2_ioctl_qbuf,
+	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_streamon = vb2_ioctl_streamon,
+	.vidioc_streamoff = vb2_ioctl_streamoff,
+	.vidioc_enum_fmt_meta_out = rkisp1_params_enum_fmt_meta_out,
+	.vidioc_g_fmt_meta_out = rkisp1_params_g_fmt_meta_out,
+	.vidioc_s_fmt_meta_out = rkisp1_params_g_fmt_meta_out,
+	.vidioc_try_fmt_meta_out = rkisp1_params_g_fmt_meta_out,
+	.vidioc_querycap = rkisp1_params_querycap,
+	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
+	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+static int rkisp1_params_vb2_queue_setup(struct vb2_queue *vq,
+					 unsigned int *num_buffers,
+					 unsigned int *num_planes,
+					 unsigned int sizes[],
+					 struct device *alloc_devs[])
+{
+	struct rkisp1_isp_params_vdev *params_vdev = vq->drv_priv;
+
+	*num_buffers = clamp_t(u32, *num_buffers,
+			       RKISP1_ISP_PARAMS_REQ_BUFS_MIN,
+			       RKISP1_ISP_PARAMS_REQ_BUFS_MAX);
+
+	*num_planes = 1;
+
+	sizes[0] = sizeof(struct rkisp1_isp_params_cfg);
+
+	INIT_LIST_HEAD(&params_vdev->params);
+	params_vdev->first_params = true;
+
+	return 0;
+}
+
+static void rkisp1_params_vb2_buf_queue(struct vb2_buffer *vb)
+{
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct rkisp1_buffer *params_buf = to_rkisp1_buffer(vbuf);
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct rkisp1_isp_params_vdev *params_vdev = vq->drv_priv;
+	struct rkisp1_isp_params_cfg *new_params;
+	unsigned long flags;
+
+	unsigned int cur_frame_id = -1;
+
+	cur_frame_id =
+		atomic_read(&params_vdev->dev->isp_sdev.frm_sync_seq) - 1;
+
+	if (params_vdev->first_params) {
+		new_params = (struct rkisp1_isp_params_cfg *)
+			(vb2_plane_vaddr(vb, 0));
+		vbuf->sequence = cur_frame_id;
+		vb2_buffer_done(&params_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
+		params_vdev->first_params = false;
+		params_vdev->cur_params = *new_params;
+		return;
+	}
+
+	params_buf->vaddr[0] = vb2_plane_vaddr(vb, 0);
+	spin_lock_irqsave(&params_vdev->config_lock, flags);
+	list_add_tail(&params_buf->queue, &params_vdev->params);
+	spin_unlock_irqrestore(&params_vdev->config_lock, flags);
+}
+
+static int rkisp1_params_vb2_buf_prepare(struct vb2_buffer *vb)
+{
+	if (vb2_plane_size(vb, 0) < sizeof(struct rkisp1_isp_params_cfg))
+		return -EINVAL;
+
+	vb2_set_plane_payload(vb, 0, sizeof(struct rkisp1_isp_params_cfg));
+
+	return 0;
+}
+
+static void rkisp1_params_vb2_stop_streaming(struct vb2_queue *vq)
+{
+	struct rkisp1_isp_params_vdev *params_vdev = vq->drv_priv;
+	struct rkisp1_buffer *buf;
+	unsigned long flags;
+	unsigned int i;
+
+	/* stop params input firstly */
+	spin_lock_irqsave(&params_vdev->config_lock, flags);
+	params_vdev->streamon = false;
+	spin_unlock_irqrestore(&params_vdev->config_lock, flags);
+
+	for (i = 0; i < RKISP1_ISP_PARAMS_REQ_BUFS_MAX; i++) {
+		spin_lock_irqsave(&params_vdev->config_lock, flags);
+		if (!list_empty(&params_vdev->params)) {
+			buf = list_first_entry(&params_vdev->params,
+					       struct rkisp1_buffer, queue);
+			list_del(&buf->queue);
+			spin_unlock_irqrestore(&params_vdev->config_lock,
+					       flags);
+		} else {
+			spin_unlock_irqrestore(&params_vdev->config_lock,
+					       flags);
+			break;
+		}
+
+		if (buf)
+			vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
+		buf = NULL;
+	}
+}
+
+static int
+rkisp1_params_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
+{
+	struct rkisp1_isp_params_vdev *params_vdev = queue->drv_priv;
+	unsigned long flags;
+
+	spin_lock_irqsave(&params_vdev->config_lock, flags);
+	params_vdev->streamon = true;
+	spin_unlock_irqrestore(&params_vdev->config_lock, flags);
+
+	return 0;
+}
+
+static struct vb2_ops rkisp1_params_vb2_ops = {
+	.queue_setup = rkisp1_params_vb2_queue_setup,
+	.wait_prepare = vb2_ops_wait_prepare,
+	.wait_finish = vb2_ops_wait_finish,
+	.buf_queue = rkisp1_params_vb2_buf_queue,
+	.buf_prepare = rkisp1_params_vb2_buf_prepare,
+	.start_streaming = rkisp1_params_vb2_start_streaming,
+	.stop_streaming = rkisp1_params_vb2_stop_streaming,
+
+};
+
+struct v4l2_file_operations rkisp1_params_fops = {
+	.mmap = vb2_fop_mmap,
+	.unlocked_ioctl = video_ioctl2,
+	.poll = vb2_fop_poll,
+	.open = v4l2_fh_open,
+	.release = vb2_fop_release
+};
+
+static int
+rkisp1_params_init_vb2_queue(struct vb2_queue *q,
+			     struct rkisp1_isp_params_vdev *params_vdev)
+{
+	struct rkisp1_vdev_node *node;
+
+	node = queue_to_node(q);
+
+	q->type = V4L2_BUF_TYPE_META_OUTPUT;
+	q->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
+	q->drv_priv = params_vdev;
+	q->ops = &rkisp1_params_vb2_ops;
+	q->mem_ops = &vb2_vmalloc_memops;
+	q->buf_struct_size = sizeof(struct rkisp1_buffer);
+	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	q->lock = &node->vlock;
+
+	return vb2_queue_init(q);
+}
+
+static void rkisp1_init_params_vdev(struct rkisp1_isp_params_vdev *params_vdev)
+{
+	params_vdev->vdev_fmt.fmt.meta.dataformat =
+		V4L2_META_FMT_RK_ISP1_PARAMS;
+	params_vdev->vdev_fmt.fmt.meta.buffersize =
+		sizeof(struct rkisp1_isp_params_cfg);
+}
+
+int rkisp1_register_params_vdev(struct rkisp1_isp_params_vdev *params_vdev,
+				struct v4l2_device *v4l2_dev,
+				struct rkisp1_device *dev)
+{
+	struct rkisp1_vdev_node *node = &params_vdev->vnode;
+	struct video_device *vdev = &node->vdev;
+	int ret;
+
+	params_vdev->dev = dev;
+	mutex_init(&node->vlock);
+	spin_lock_init(&params_vdev->config_lock);
+
+	strscpy(vdev->name, "rkisp1-input-params", sizeof(vdev->name));
+
+	video_set_drvdata(vdev, params_vdev);
+	vdev->ioctl_ops = &rkisp1_params_ioctl;
+	vdev->fops = &rkisp1_params_fops;
+	vdev->release = video_device_release_empty;
+	/*
+	 * Provide a mutex to v4l2 core. It will be used
+	 * to protect all fops and v4l2 ioctls.
+	 */
+	vdev->lock = &node->vlock;
+	vdev->v4l2_dev = v4l2_dev;
+	vdev->queue = &node->buf_queue;
+	vdev->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_META_OUTPUT;
+	vdev->vfl_dir = VFL_DIR_TX;
+	rkisp1_params_init_vb2_queue(vdev->queue, params_vdev);
+	rkisp1_init_params_vdev(params_vdev);
+	video_set_drvdata(vdev, params_vdev);
+
+	node->pad.flags = MEDIA_PAD_FL_SOURCE;
+	ret = media_entity_pads_init(&vdev->entity, 1, &node->pad);
+	if (ret < 0)
+		goto err_release_queue;
+	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
+	if (ret < 0) {
+		dev_err(&vdev->dev,
+			"could not register Video for Linux device\n");
+		goto err_cleanup_media_entity;
+	}
+	return 0;
+err_cleanup_media_entity:
+	media_entity_cleanup(&vdev->entity);
+err_release_queue:
+	vb2_queue_release(vdev->queue);
+	return ret;
+}
+
+void rkisp1_unregister_params_vdev(struct rkisp1_isp_params_vdev *params_vdev)
+{
+	struct rkisp1_vdev_node *node = &params_vdev->vnode;
+	struct video_device *vdev = &node->vdev;
+
+	video_unregister_device(vdev);
+	media_entity_cleanup(&vdev->entity);
+	vb2_queue_release(vdev->queue);
+}
diff --git a/drivers/media/platform/rockchip/isp1/isp_params.h b/drivers/media/platform/rockchip/isp1/isp_params.h
new file mode 100644
index 000000000000..95e5a778125b
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/isp_params.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#ifndef _RKISP1_ISP_H
+#define _RKISP1_ISP_H
+
+#include <linux/rkisp1-config.h>
+
+#include "common.h"
+
+/*
+ * struct rkisp1_isp_subdev - ISP input parameters device
+ *
+ * @cur_params: Current ISP parameters
+ * @first_params: the first params should take effect immediately
+ */
+struct rkisp1_isp_params_vdev {
+	struct rkisp1_vdev_node vnode;
+	struct rkisp1_device *dev;
+
+	spinlock_t config_lock;
+	struct list_head params;
+	struct rkisp1_isp_params_cfg cur_params;
+	struct v4l2_format vdev_fmt;
+	bool streamon;
+	bool first_params;
+
+	enum v4l2_quantization quantization;
+	enum rkisp1_fmt_raw_pat_type raw_type;
+};
+
+/* config params before ISP streaming */
+void rkisp1_params_configure_isp(struct rkisp1_isp_params_vdev *params_vdev,
+				 struct ispsd_in_fmt *in_fmt,
+				 enum v4l2_quantization quantization);
+void rkisp1_params_disable_isp(struct rkisp1_isp_params_vdev *params_vdev);
+
+int rkisp1_register_params_vdev(struct rkisp1_isp_params_vdev *params_vdev,
+				struct v4l2_device *v4l2_dev,
+				struct rkisp1_device *dev);
+
+void rkisp1_unregister_params_vdev(struct rkisp1_isp_params_vdev *params_vdev);
+
+void rkisp1_params_isr(struct rkisp1_isp_params_vdev *params_vdev, u32 isp_mis);
+
+#endif /* _RKISP1_ISP_H */
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (6 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 07/14] media: rkisp1: add ISP1 params driver Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-07 15:27   ` Sakari Ailus
  2019-08-07 15:36   ` Sakari Ailus
  2019-07-30 18:42 ` [PATCH v8 10/14] dt-bindings: Document the Rockchip ISP1 bindings Helen Koike
                   ` (7 subsequent siblings)
  15 siblings, 2 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Allon Huang, Jacob Chen,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, Helen Koike, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

Add the core driver for rockchip isp1.

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
Signed-off-by: Jacob Chen <cc@rock-chips.com>
Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
Signed-off-by: Tomasz Figa <tfiga@chromium.org>
[fixed compilation and run time errors regarding new v4l2 async API]
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
[Add missing module device table]
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
- add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
- Fix compilation and runtime errors due to bitrotting
The code has bit-rotten since March 2018, fix compilation errors.
The new V4L2 async notifier API requires notifiers to be initialized by
a call to v4l2_async_notifier_init() before being used, do so.
- Add missing module device table
- use clk_bulk framework
- add missing notifiers cleanups
- s/strlcpy/strscpy
- normalize bus_info name
- fix s_stream error path, stream_cnt wans't being decremented properly
- use devm_platform_ioremap_resource() helper
- s/deice/device
- redesign: remove mipi/csi subdevice, sensors connect directly to the
isp subdevice in the media topology now.
- remove "saved_state" member from rkisp1_stream struct
- Reverse the order of MIs
- Simplify MI interrupt handling
Rather than adding unnecessary indirection, just use stream index to
handle MI interrupt enable/disable/clear, since the stream index matches
the order of bits now, thanks to previous patch. While at it, remove
some dead code.
- code styling and checkpatch fixes

 drivers/media/platform/Kconfig                |  12 +
 drivers/media/platform/Makefile               |   1 +
 drivers/media/platform/rockchip/isp1/Makefile |   7 +
 drivers/media/platform/rockchip/isp1/common.h | 101 +++
 drivers/media/platform/rockchip/isp1/dev.c    | 675 ++++++++++++++++++
 drivers/media/platform/rockchip/isp1/dev.h    |  97 +++
 6 files changed, 893 insertions(+)
 create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
 create mode 100644 drivers/media/platform/rockchip/isp1/common.h
 create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
 create mode 100644 drivers/media/platform/rockchip/isp1/dev.h

diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index 89555f9a813f..e0e98937c565 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -106,6 +106,18 @@ config VIDEO_QCOM_CAMSS
 	select VIDEOBUF2_DMA_SG
 	select V4L2_FWNODE
 
+config VIDEO_ROCKCHIP_ISP1
+	tristate "Rockchip Image Signal Processing v1 Unit driver"
+	depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
+	depends on ARCH_ROCKCHIP || COMPILE_TEST
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEOBUF2_VMALLOC
+	select V4L2_FWNODE
+	select PHY_ROCKCHIP_DPHY
+	default n
+	---help---
+	  Support for ISP1 on the rockchip SoC.
+
 config VIDEO_S3C_CAMIF
 	tristate "Samsung S3C24XX/S3C64XX SoC Camera Interface driver"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 7cbbd925124c..f9fcf8e7c513 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -69,6 +69,7 @@ obj-$(CONFIG_VIDEO_RENESAS_FDP1)	+= rcar_fdp1.o
 obj-$(CONFIG_VIDEO_RENESAS_JPU)		+= rcar_jpu.o
 obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
 
+obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1)	+= rockchip/isp1/
 obj-$(CONFIG_VIDEO_ROCKCHIP_RGA)	+= rockchip/rga/
 
 obj-y	+= omap/
diff --git a/drivers/media/platform/rockchip/isp1/Makefile b/drivers/media/platform/rockchip/isp1/Makefile
new file mode 100644
index 000000000000..72706e80fc8b
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/Makefile
@@ -0,0 +1,7 @@
+obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1) += 	rockchip-isp1.o
+rockchip-isp1-objs 	   += 	rkisp1.o \
+				dev.o \
+				regs.o \
+				isp_stats.o \
+				isp_params.o \
+				capture.o
diff --git a/drivers/media/platform/rockchip/isp1/common.h b/drivers/media/platform/rockchip/isp1/common.h
new file mode 100644
index 000000000000..606ce2793546
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/common.h
@@ -0,0 +1,101 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#ifndef _RKISP1_COMMON_H
+#define _RKISP1_COMMON_H
+
+#include <linux/mutex.h>
+#include <media/media-device.h>
+#include <media/media-entity.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/videobuf2-v4l2.h>
+
+#define RKISP1_DEFAULT_WIDTH		800
+#define RKISP1_DEFAULT_HEIGHT		600
+
+#define RKISP1_MAX_STREAM		2
+#define RKISP1_STREAM_MP		0
+#define RKISP1_STREAM_SP		1
+
+#define RKISP1_PLANE_Y			0
+#define RKISP1_PLANE_CB			1
+#define RKISP1_PLANE_CR			2
+
+enum rkisp1_sd_type {
+	RKISP1_SD_SENSOR,
+	RKISP1_SD_PHY_CSI,
+	RKISP1_SD_VCM,
+	RKISP1_SD_FLASH,
+	RKISP1_SD_MAX,
+};
+
+/* One structure per video node */
+struct rkisp1_vdev_node {
+	struct vb2_queue buf_queue;
+	/* vfd lock */
+	struct mutex vlock;
+	struct video_device vdev;
+	struct media_pad pad;
+};
+
+enum rkisp1_fmt_pix_type {
+	FMT_YUV,
+	FMT_RGB,
+	FMT_BAYER,
+	FMT_JPEG,
+	FMT_MAX
+};
+
+enum rkisp1_fmt_raw_pat_type {
+	RAW_RGGB = 0,
+	RAW_GRBG,
+	RAW_GBRG,
+	RAW_BGGR,
+};
+
+struct rkisp1_buffer {
+	struct vb2_v4l2_buffer vb;
+	struct list_head queue;
+	union {
+		u32 buff_addr[VIDEO_MAX_PLANES];
+		void *vaddr[VIDEO_MAX_PLANES];
+	};
+};
+
+struct rkisp1_dummy_buffer {
+	void *vaddr;
+	dma_addr_t dma_addr;
+	u32 size;
+};
+
+extern int rkisp1_debug;
+
+static inline
+struct rkisp1_vdev_node *vdev_to_node(struct video_device *vdev)
+{
+	return container_of(vdev, struct rkisp1_vdev_node, vdev);
+}
+
+static inline struct rkisp1_vdev_node *queue_to_node(struct vb2_queue *q)
+{
+	return container_of(q, struct rkisp1_vdev_node, buf_queue);
+}
+
+static inline struct rkisp1_buffer *to_rkisp1_buffer(struct vb2_v4l2_buffer *vb)
+{
+	return container_of(vb, struct rkisp1_buffer, vb);
+}
+
+static inline struct vb2_queue *to_vb2_queue(struct file *file)
+{
+	struct rkisp1_vdev_node *vnode = video_drvdata(file);
+
+	return &vnode->buf_queue;
+}
+
+#endif /* _RKISP1_COMMON_H */
diff --git a/drivers/media/platform/rockchip/isp1/dev.c b/drivers/media/platform/rockchip/isp1/dev.c
new file mode 100644
index 000000000000..2b4a67e1a3b5
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/dev.c
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#include <linux/clk.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/of_platform.h>
+#include <linux/pm_runtime.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/phy/phy.h>
+#include <linux/phy/phy-mipi-dphy.h>
+
+#include "common.h"
+#include "regs.h"
+
+struct isp_match_data {
+	const char * const *clks;
+	int size;
+};
+
+struct sensor_async_subdev {
+	struct v4l2_async_subdev asd;
+	struct v4l2_mbus_config mbus;
+	unsigned int lanes;
+};
+
+int rkisp1_debug;
+module_param_named(debug, rkisp1_debug, int, 0644);
+MODULE_PARM_DESC(debug, "Debug level (0-1)");
+
+/**************************** pipeline operations******************************/
+
+static int __isp_pipeline_prepare(struct rkisp1_pipeline *p,
+				  struct media_entity *me)
+{
+	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
+	struct v4l2_subdev *sd;
+	unsigned int i;
+
+	p->num_subdevs = 0;
+	memset(p->subdevs, 0, sizeof(p->subdevs));
+
+	while (1) {
+		struct media_pad *pad = NULL;
+
+		/* Find remote source pad */
+		for (i = 0; i < me->num_pads; i++) {
+			struct media_pad *spad = &me->pads[i];
+
+			if (!(spad->flags & MEDIA_PAD_FL_SINK))
+				continue;
+			pad = media_entity_remote_pad(spad);
+			if (pad)
+				break;
+		}
+
+		if (!pad)
+			break;
+
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		if (sd != &dev->isp_sdev.sd)
+			p->subdevs[p->num_subdevs++] = sd;
+
+		me = &sd->entity;
+		if (me->num_pads == 1)
+			break;
+	}
+	return 0;
+}
+
+static int __subdev_set_power(struct v4l2_subdev *sd, int on)
+{
+	int ret;
+
+	if (!sd)
+		return -ENXIO;
+
+	ret = v4l2_subdev_call(sd, core, s_power, on);
+
+	return ret != -ENOIOCTLCMD ? ret : 0;
+}
+
+static int __isp_pipeline_s_power(struct rkisp1_pipeline *p, bool on)
+{
+	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
+	int i, ret;
+
+	if (on) {
+		__subdev_set_power(&dev->isp_sdev.sd, true);
+
+		for (i = p->num_subdevs - 1; i >= 0; --i) {
+			ret = __subdev_set_power(p->subdevs[i], true);
+			if (ret < 0 && ret != -ENXIO)
+				goto err_power_off;
+		}
+	} else {
+		for (i = 0; i < p->num_subdevs; ++i)
+			__subdev_set_power(p->subdevs[i], false);
+
+		__subdev_set_power(&dev->isp_sdev.sd, false);
+	}
+
+	return 0;
+
+err_power_off:
+	for (++i; i < p->num_subdevs; ++i)
+		__subdev_set_power(p->subdevs[i], false);
+	__subdev_set_power(&dev->isp_sdev.sd, true);
+	return ret;
+}
+
+static int rkisp1_pipeline_open(struct rkisp1_pipeline *p,
+				struct media_entity *me,
+				bool prepare)
+{
+	int ret;
+
+	if (WARN_ON(!p || !me))
+		return -EINVAL;
+	if (atomic_inc_return(&p->power_cnt) > 1)
+		return 0;
+
+	/* go through media graphic and get subdevs */
+	if (prepare)
+		__isp_pipeline_prepare(p, me);
+
+	if (!p->num_subdevs)
+		return -EINVAL;
+
+	ret = __isp_pipeline_s_power(p, 1);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static int rkisp1_pipeline_close(struct rkisp1_pipeline *p)
+{
+	int ret;
+
+	if (atomic_dec_return(&p->power_cnt) > 0)
+		return 0;
+	ret = __isp_pipeline_s_power(p, 0);
+
+	return ret == -ENXIO ? 0 : ret;
+}
+
+/*
+ * stream-on order: isp_subdev, mipi dphy, sensor
+ * stream-off order: mipi dphy, sensor, isp_subdev
+ */
+static int rkisp1_pipeline_set_stream(struct rkisp1_pipeline *p, bool on)
+{
+	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
+	int i, ret;
+
+	if ((on && atomic_inc_return(&p->stream_cnt) > 1) ||
+	    (!on && atomic_dec_return(&p->stream_cnt) > 0))
+		return 0;
+
+	if (on) {
+		ret = v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream,
+				       true);
+		if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV) {
+			v4l2_err(&dev->v4l2_dev,
+				 "s_stream failed on subdevice %s (%d)\n",
+				 dev->isp_sdev.sd.name,
+				 ret);
+			atomic_dec(&p->stream_cnt);
+			return ret;
+		}
+	}
+
+	/* phy -> sensor */
+	for (i = 0; i < p->num_subdevs; ++i) {
+		ret = v4l2_subdev_call(p->subdevs[i], video, s_stream, on);
+		if (on && ret < 0 && ret != -ENOIOCTLCMD && ret != -ENODEV)
+			goto err_stream_off;
+	}
+
+	if (!on)
+		v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
+
+	return 0;
+
+err_stream_off:
+	for (--i; i >= 0; --i)
+		v4l2_subdev_call(p->subdevs[i], video, s_stream, false);
+	v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
+	atomic_dec(&p->stream_cnt);
+	return ret;
+}
+
+/***************************** media controller *******************************/
+/* See http://opensource.rock-chips.com/wiki_Rockchip-isp1 for Topology */
+
+static int rkisp1_create_links(struct rkisp1_device *dev)
+{
+	struct media_entity *source, *sink;
+	struct rkisp1_sensor *sensor;
+	unsigned int flags, pad;
+	int ret;
+
+	/* sensor links(or mipi-phy) */
+	list_for_each_entry(sensor, &dev->sensors, list) {
+		for (pad = 0; pad < sensor->sd->entity.num_pads; pad++)
+			if (sensor->sd->entity.pads[pad].flags &
+				MEDIA_PAD_FL_SOURCE)
+				break;
+
+		if (pad == sensor->sd->entity.num_pads) {
+			dev_err(dev->dev,
+				"failed to find src pad for %s\n",
+				sensor->sd->name);
+
+			return -ENXIO;
+		}
+
+		ret = media_create_pad_link(
+				&sensor->sd->entity, pad,
+				&dev->isp_sdev.sd.entity,
+				RKISP1_ISP_PAD_SINK,
+				list_is_first(&sensor->list, &dev->sensors) ?
+				MEDIA_LNK_FL_ENABLED : 0);
+		if (ret) {
+			dev_err(dev->dev,
+				"failed to create link for %s\n",
+				sensor->sd->name);
+			return ret;
+		}
+	}
+
+	/* params links */
+	source = &dev->params_vdev.vnode.vdev.entity;
+	sink = &dev->isp_sdev.sd.entity;
+	flags = MEDIA_LNK_FL_ENABLED;
+	ret = media_create_pad_link(source, 0, sink,
+				       RKISP1_ISP_PAD_SINK_PARAMS, flags);
+	if (ret < 0)
+		return ret;
+
+	/* create isp internal links */
+	/* SP links */
+	source = &dev->isp_sdev.sd.entity;
+	sink = &dev->stream[RKISP1_STREAM_SP].vnode.vdev.entity;
+	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
+				       sink, 0, flags);
+	if (ret < 0)
+		return ret;
+
+	/* MP links */
+	source = &dev->isp_sdev.sd.entity;
+	sink = &dev->stream[RKISP1_STREAM_MP].vnode.vdev.entity;
+	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
+				       sink, 0, flags);
+	if (ret < 0)
+		return ret;
+
+	/* 3A stats links */
+	source = &dev->isp_sdev.sd.entity;
+	sink = &dev->stats_vdev.vnode.vdev.entity;
+	return media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_STATS,
+					sink, 0, flags);
+}
+
+static int subdev_notifier_bound(struct v4l2_async_notifier *notifier,
+				 struct v4l2_subdev *sd,
+				 struct v4l2_async_subdev *asd)
+{
+	struct rkisp1_device *isp_dev = container_of(notifier,
+						     struct rkisp1_device,
+						     notifier);
+	struct sensor_async_subdev *s_asd = container_of(asd,
+					struct sensor_async_subdev, asd);
+	struct rkisp1_sensor *sensor;
+
+	sensor = devm_kzalloc(isp_dev->dev, sizeof(*sensor), GFP_KERNEL);
+	if (!sensor)
+		return -ENOMEM;
+
+	sensor->lanes = s_asd->lanes;
+	sensor->mbus = s_asd->mbus;
+	sensor->sd = sd;
+	sensor->dphy = devm_phy_get(isp_dev->dev, "dphy");
+	if (IS_ERR(sensor->dphy)) {
+		if (PTR_ERR(sensor->dphy) != -EPROBE_DEFER)
+			dev_err(isp_dev->dev, "Couldn't get the MIPI D-PHY\n");
+		return PTR_ERR(sensor->dphy);
+	}
+	phy_init(sensor->dphy);
+
+	list_add(&sensor->list, &isp_dev->sensors);
+
+	return 0;
+}
+
+static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
+					  struct v4l2_subdev *sd)
+{
+	struct rkisp1_sensor *sensor;
+
+	list_for_each_entry(sensor, &dev->sensors, list)
+		if (sensor->sd == sd)
+			return sensor;
+
+	return NULL;
+}
+
+static void subdev_notifier_unbind(struct v4l2_async_notifier *notifier,
+				   struct v4l2_subdev *sd,
+				   struct v4l2_async_subdev *asd)
+{
+	struct rkisp1_device *isp_dev = container_of(notifier,
+						     struct rkisp1_device,
+						     notifier);
+	struct rkisp1_sensor *sensor = sd_to_sensor(isp_dev, sd);
+
+	/* TODO: check if a lock is required here */
+	list_del(&sensor->list);
+
+	phy_exit(sensor->dphy);
+}
+
+static int subdev_notifier_complete(struct v4l2_async_notifier *notifier)
+{
+	struct rkisp1_device *dev = container_of(notifier, struct rkisp1_device,
+						 notifier);
+	int ret;
+
+	mutex_lock(&dev->media_dev.graph_mutex);
+	ret = rkisp1_create_links(dev);
+	if (ret < 0)
+		goto unlock;
+	ret = v4l2_device_register_subdev_nodes(&dev->v4l2_dev);
+	if (ret < 0)
+		goto unlock;
+
+	v4l2_info(&dev->v4l2_dev, "Async subdev notifier completed\n");
+
+unlock:
+	mutex_unlock(&dev->media_dev.graph_mutex);
+	return ret;
+}
+
+static int rkisp1_fwnode_parse(struct device *dev,
+			       struct v4l2_fwnode_endpoint *vep,
+			       struct v4l2_async_subdev *asd)
+{
+	struct sensor_async_subdev *s_asd =
+			container_of(asd, struct sensor_async_subdev, asd);
+
+	if (vep->bus_type != V4L2_MBUS_CSI2_DPHY) {
+		dev_err(dev, "Only CSI2 bus type is currently supported\n");
+		return -EINVAL;
+	}
+
+	if (vep->base.port != 0) {
+		dev_err(dev, "The ISP has only port 0\n");
+		return -EINVAL;
+	}
+
+	s_asd->mbus.type = vep->bus_type;
+	s_asd->mbus.flags = vep->bus.mipi_csi2.flags;
+	s_asd->lanes = vep->bus.mipi_csi2.num_data_lanes;
+
+	switch (vep->bus.mipi_csi2.num_data_lanes) {
+	case 1:
+		s_asd->mbus.flags |= V4L2_MBUS_CSI2_1_LANE;
+		break;
+	case 2:
+		s_asd->mbus.flags |= V4L2_MBUS_CSI2_2_LANE;
+		break;
+	case 3:
+		s_asd->mbus.flags |= V4L2_MBUS_CSI2_3_LANE;
+		break;
+	case 4:
+		s_asd->mbus.flags |= V4L2_MBUS_CSI2_4_LANE;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_async_notifier_operations subdev_notifier_ops = {
+	.bound = subdev_notifier_bound,
+	.unbind = subdev_notifier_unbind,
+	.complete = subdev_notifier_complete,
+};
+
+static int isp_subdev_notifier(struct rkisp1_device *isp_dev)
+{
+	struct v4l2_async_notifier *ntf = &isp_dev->notifier;
+	struct device *dev = isp_dev->dev;
+	int ret;
+
+	v4l2_async_notifier_init(ntf);
+
+	ret = v4l2_async_notifier_parse_fwnode_endpoints_by_port(
+		dev, ntf, sizeof(struct sensor_async_subdev), 0,
+		rkisp1_fwnode_parse);
+	if (ret < 0)
+		return ret;
+
+	if (list_empty(&ntf->asd_list))
+		return -ENODEV;	/* no endpoint */
+
+	ntf->ops = &subdev_notifier_ops;
+
+	return v4l2_async_notifier_register(&isp_dev->v4l2_dev, ntf);
+}
+
+/***************************** platform device *******************************/
+
+static int rkisp1_register_platform_subdevs(struct rkisp1_device *dev)
+{
+	int ret;
+
+	ret = rkisp1_register_isp_subdev(dev, &dev->v4l2_dev);
+	if (ret < 0)
+		return ret;
+
+	ret = rkisp1_register_stream_vdevs(dev);
+	if (ret < 0)
+		goto err_unreg_isp_subdev;
+
+	ret = rkisp1_register_stats_vdev(&dev->stats_vdev, &dev->v4l2_dev, dev);
+	if (ret < 0)
+		goto err_unreg_stream_vdev;
+
+	ret = rkisp1_register_params_vdev(&dev->params_vdev, &dev->v4l2_dev,
+					  dev);
+	if (ret < 0)
+		goto err_unreg_stats_vdev;
+
+	ret = isp_subdev_notifier(dev);
+	if (ret < 0) {
+		v4l2_err(&dev->v4l2_dev,
+			 "Failed to register subdev notifier(%d)\n", ret);
+		goto err_unreg_params_vdev;
+	}
+
+	return 0;
+err_unreg_params_vdev:
+	rkisp1_unregister_params_vdev(&dev->params_vdev);
+err_unreg_stats_vdev:
+	rkisp1_unregister_stats_vdev(&dev->stats_vdev);
+err_unreg_stream_vdev:
+	rkisp1_unregister_stream_vdevs(dev);
+err_unreg_isp_subdev:
+	rkisp1_unregister_isp_subdev(dev);
+	return ret;
+}
+
+static const char * const rk3399_isp_clks[] = {
+	"clk_isp",
+	"aclk_isp",
+	"hclk_isp",
+	"aclk_isp_wrap",
+	"hclk_isp_wrap",
+};
+
+static const char * const rk3288_isp_clks[] = {
+	"clk_isp",
+	"aclk_isp",
+	"hclk_isp",
+	"pclk_isp_in",
+	"sclk_isp_jpe",
+};
+
+static const struct isp_match_data rk3288_isp_clk_data = {
+	.clks = rk3288_isp_clks,
+	.size = ARRAY_SIZE(rk3288_isp_clks),
+};
+
+static const struct isp_match_data rk3399_isp_clk_data = {
+	.clks = rk3399_isp_clks,
+	.size = ARRAY_SIZE(rk3399_isp_clks),
+};
+
+static const struct of_device_id rkisp1_plat_of_match[] = {
+	{
+		.compatible = "rockchip,rk3288-cif-isp",
+		.data = &rk3288_isp_clk_data,
+	}, {
+		.compatible = "rockchip,rk3399-cif-isp",
+		.data = &rk3399_isp_clk_data,
+	},
+	{},
+};
+MODULE_DEVICE_TABLE(of, rkisp1_plat_of_match);
+
+static irqreturn_t rkisp1_irq_handler(int irq, void *ctx)
+{
+	struct device *dev = ctx;
+	struct rkisp1_device *rkisp1_dev = dev_get_drvdata(dev);
+	unsigned int mis_val;
+
+	mis_val = readl(rkisp1_dev->base_addr + CIF_ISP_MIS);
+	if (mis_val)
+		rkisp1_isp_isr(mis_val, rkisp1_dev);
+
+	mis_val = readl(rkisp1_dev->base_addr + CIF_MIPI_MIS);
+	if (mis_val)
+		rkisp1_mipi_isr(mis_val, rkisp1_dev);
+
+	mis_val = readl(rkisp1_dev->base_addr + CIF_MI_MIS);
+	if (mis_val)
+		rkisp1_mi_isr(mis_val, rkisp1_dev);
+
+	return IRQ_HANDLED;
+}
+
+static int rkisp1_plat_probe(struct platform_device *pdev)
+{
+	struct device_node *node = pdev->dev.of_node;
+	const struct isp_match_data *clk_data;
+	const struct of_device_id *match;
+	struct device *dev = &pdev->dev;
+	struct rkisp1_device *isp_dev;
+	struct v4l2_device *v4l2_dev;
+	unsigned int i;
+	int ret, irq;
+
+	match = of_match_node(rkisp1_plat_of_match, node);
+	isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
+	if (!isp_dev)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&isp_dev->sensors);
+
+	dev_set_drvdata(dev, isp_dev);
+	isp_dev->dev = dev;
+
+	isp_dev->base_addr = devm_platform_ioremap_resource(pdev, 0);
+	if (IS_ERR(isp_dev->base_addr))
+		return PTR_ERR(isp_dev->base_addr);
+
+	irq = platform_get_irq(pdev, 0);
+	if (irq < 0)
+		return irq;
+
+	ret = devm_request_irq(dev, irq, rkisp1_irq_handler, IRQF_SHARED,
+			       dev_driver_string(dev), dev);
+	if (ret < 0) {
+		dev_err(dev, "request irq failed: %d\n", ret);
+		return ret;
+	}
+
+	isp_dev->irq = irq;
+	clk_data = match->data;
+
+	for (i = 0; i < clk_data->size; i++)
+		isp_dev->clks[i].id = clk_data->clks[i];
+	ret = devm_clk_bulk_get(dev, clk_data->size, isp_dev->clks);
+	if (ret)
+		return ret;
+	isp_dev->clk_size = clk_data->size;
+
+	atomic_set(&isp_dev->pipe.power_cnt, 0);
+	atomic_set(&isp_dev->pipe.stream_cnt, 0);
+	isp_dev->pipe.open = rkisp1_pipeline_open;
+	isp_dev->pipe.close = rkisp1_pipeline_close;
+	isp_dev->pipe.set_stream = rkisp1_pipeline_set_stream;
+
+	rkisp1_stream_init(isp_dev, RKISP1_STREAM_SP);
+	rkisp1_stream_init(isp_dev, RKISP1_STREAM_MP);
+
+	strscpy(isp_dev->media_dev.model, "rkisp1",
+		sizeof(isp_dev->media_dev.model));
+	isp_dev->media_dev.dev = &pdev->dev;
+	strscpy(isp_dev->media_dev.bus_info,
+		"platform: " DRIVER_NAME, sizeof(isp_dev->media_dev.bus_info));
+	media_device_init(&isp_dev->media_dev);
+
+	v4l2_dev = &isp_dev->v4l2_dev;
+	v4l2_dev->mdev = &isp_dev->media_dev;
+	strscpy(v4l2_dev->name, "rkisp1", sizeof(v4l2_dev->name));
+	v4l2_ctrl_handler_init(&isp_dev->ctrl_handler, 5);
+	v4l2_dev->ctrl_handler = &isp_dev->ctrl_handler;
+
+	ret = v4l2_device_register(isp_dev->dev, &isp_dev->v4l2_dev);
+	if (ret < 0)
+		return ret;
+
+	ret = media_device_register(&isp_dev->media_dev);
+	if (ret < 0) {
+		v4l2_err(v4l2_dev, "Failed to register media device: %d\n",
+			 ret);
+		goto err_unreg_v4l2_dev;
+	}
+
+	/* create & register platefom subdev (from of_node) */
+	ret = rkisp1_register_platform_subdevs(isp_dev);
+	if (ret < 0)
+		goto err_unreg_media_dev;
+
+	pm_runtime_enable(&pdev->dev);
+
+	return 0;
+
+err_unreg_media_dev:
+	media_device_unregister(&isp_dev->media_dev);
+err_unreg_v4l2_dev:
+	v4l2_device_unregister(&isp_dev->v4l2_dev);
+	return ret;
+}
+
+static int rkisp1_plat_remove(struct platform_device *pdev)
+{
+	struct rkisp1_device *isp_dev = platform_get_drvdata(pdev);
+
+	pm_runtime_disable(&pdev->dev);
+	media_device_unregister(&isp_dev->media_dev);
+	v4l2_async_notifier_unregister(&isp_dev->notifier);
+	v4l2_async_notifier_cleanup(&isp_dev->notifier);
+	v4l2_device_unregister(&isp_dev->v4l2_dev);
+	rkisp1_unregister_params_vdev(&isp_dev->params_vdev);
+	rkisp1_unregister_stats_vdev(&isp_dev->stats_vdev);
+	rkisp1_unregister_stream_vdevs(isp_dev);
+	rkisp1_unregister_isp_subdev(isp_dev);
+
+	return 0;
+}
+
+static int __maybe_unused rkisp1_runtime_suspend(struct device *dev)
+{
+	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
+
+	clk_bulk_disable_unprepare(isp_dev->clk_size, isp_dev->clks);
+	return pinctrl_pm_select_sleep_state(dev);
+}
+
+static int __maybe_unused rkisp1_runtime_resume(struct device *dev)
+{
+	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
+	int ret;
+
+	ret = pinctrl_pm_select_default_state(dev);
+	if (ret < 0)
+		return ret;
+	ret = clk_bulk_prepare_enable(isp_dev->clk_size, isp_dev->clks);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
+static const struct dev_pm_ops rkisp1_plat_pm_ops = {
+	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+				pm_runtime_force_resume)
+	SET_RUNTIME_PM_OPS(rkisp1_runtime_suspend, rkisp1_runtime_resume, NULL)
+};
+
+static struct platform_driver rkisp1_plat_drv = {
+	.driver = {
+		.name = DRIVER_NAME,
+		.of_match_table = of_match_ptr(rkisp1_plat_of_match),
+		.pm = &rkisp1_plat_pm_ops,
+	},
+	.probe = rkisp1_plat_probe,
+	.remove = rkisp1_plat_remove,
+};
+
+module_platform_driver(rkisp1_plat_drv);
+MODULE_AUTHOR("Rockchip Camera/ISP team");
+MODULE_DESCRIPTION("Rockchip ISP1 platform driver");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/media/platform/rockchip/isp1/dev.h b/drivers/media/platform/rockchip/isp1/dev.h
new file mode 100644
index 000000000000..f7cbee316523
--- /dev/null
+++ b/drivers/media/platform/rockchip/isp1/dev.h
@@ -0,0 +1,97 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
+/*
+ * Rockchip isp1 driver
+ *
+ * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
+ */
+
+#ifndef _RKISP1_DEV_H
+#define _RKISP1_DEV_H
+
+#include <linux/clk.h>
+
+#include "capture.h"
+#include "rkisp1.h"
+#include "isp_params.h"
+#include "isp_stats.h"
+
+#define DRIVER_NAME "rkisp1"
+#define ISP_VDEV_NAME DRIVER_NAME  "_ispdev"
+#define SP_VDEV_NAME DRIVER_NAME   "_selfpath"
+#define MP_VDEV_NAME DRIVER_NAME   "_mainpath"
+#define DMA_VDEV_NAME DRIVER_NAME  "_dmapath"
+
+#define GRP_ID_SENSOR			BIT(0)
+#define GRP_ID_MIPIPHY			BIT(1)
+#define GRP_ID_ISP			BIT(2)
+#define GRP_ID_ISP_MP			BIT(3)
+#define GRP_ID_ISP_SP			BIT(4)
+
+#define RKISP1_MAX_BUS_CLK	8
+#define RKISP1_MAX_SENSOR	2
+#define RKISP1_MAX_PIPELINE	4
+
+/*
+ * struct rkisp1_pipeline - An ISP hardware pipeline
+ *
+ * Capture device call other devices via pipeline
+ *
+ * @num_subdevs: number of linked subdevs
+ * @power_cnt: pipeline power count
+ * @stream_cnt: stream power count
+ */
+struct rkisp1_pipeline {
+	struct media_pipeline pipe;
+	int num_subdevs;
+	atomic_t power_cnt;
+	atomic_t stream_cnt;
+	struct v4l2_subdev *subdevs[RKISP1_MAX_PIPELINE];
+	int (*open)(struct rkisp1_pipeline *p,
+		    struct media_entity *me, bool prepare);
+	int (*close)(struct rkisp1_pipeline *p);
+	int (*set_stream)(struct rkisp1_pipeline *p, bool on);
+};
+
+/*
+ * struct rkisp1_sensor - Sensor information
+ * @mbus: media bus configuration
+ */
+struct rkisp1_sensor {
+	struct v4l2_subdev *sd;
+	struct v4l2_mbus_config mbus;
+	unsigned int lanes;
+	struct phy *dphy;
+	struct list_head list;
+};
+
+/*
+ * struct rkisp1_device - ISP platform device
+ * @base_addr: base register address
+ * @active_sensor: sensor in-use, set when streaming on
+ * @isp_sdev: ISP sub-device
+ * @rkisp1_stream: capture video device
+ * @stats_vdev: ISP statistics output device
+ * @params_vdev: ISP input parameters device
+ */
+struct rkisp1_device {
+	void __iomem *base_addr;
+	int irq;
+	struct device *dev;
+	unsigned int clk_size;
+	struct clk_bulk_data clks[RKISP1_MAX_BUS_CLK];
+	struct v4l2_device v4l2_dev;
+	struct v4l2_ctrl_handler ctrl_handler;
+	struct media_device media_dev;
+	struct v4l2_async_notifier notifier;
+	struct v4l2_subdev *subdevs[RKISP1_SD_MAX];
+	struct rkisp1_sensor *active_sensor;
+	struct list_head sensors;
+	struct rkisp1_isp_subdev isp_sdev;
+	struct rkisp1_stream stream[RKISP1_MAX_STREAM];
+	struct rkisp1_isp_stats_vdev stats_vdev;
+	struct rkisp1_isp_params_vdev params_vdev;
+	struct rkisp1_pipeline pipe;
+	struct vb2_alloc_ctx *alloc_ctx;
+};
+
+#endif
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 10/14] dt-bindings: Document the Rockchip ISP1 bindings
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (7 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-16  0:21   ` Laurent Pinchart
  2019-07-30 18:42 ` [PATCH v8 11/14] dt-bindings: Document the Rockchip MIPI RX D-PHY bindings Helen Koike
                   ` (6 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Rob Herring,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

Add DT bindings documentation for Rockchip ISP1

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Reviewed-by: Rob Herring <robh@kernel.org>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- update document with new design and tested example

 .../bindings/media/rockchip-isp1.txt          | 71 +++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt

diff --git a/Documentation/devicetree/bindings/media/rockchip-isp1.txt b/Documentation/devicetree/bindings/media/rockchip-isp1.txt
new file mode 100644
index 000000000000..a97fef0f189f
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/rockchip-isp1.txt
@@ -0,0 +1,71 @@
+Rockchip SoC Image Signal Processing unit v1
+----------------------------------------------
+
+Rockchip ISP1 is the Camera interface for the Rockchip series of SoCs
+which contains image processing, scaling, and compression funcitons.
+
+Required properties:
+- compatible: value should be one of the following
+	"rockchip,rk3288-cif-isp";
+	"rockchip,rk3399-cif-isp";
+- reg : offset and length of the register set for the device.
+- interrupts: should contain ISP interrupt.
+- clocks: phandle to the required clocks.
+- clock-names: required clock name.
+- iommus: required a iommu node.
+- phys: the phandle for the PHY port
+- phy-names: must contain "dphy"
+
+port node
+-------------------
+
+The device node should contain one 'ports' child node, with children 'port'
+with child 'endpoint'.
+nodes, according to the bindings defined in Documentation/devicetree/bindings/
+media/video-interfaces.txt.
+
+- endpoint(mipi):
+	- remote-endpoint: Connecting to Rockchip MIPI-DPHY,
+		which is defined in rockchip-mipi-dphy.txt.
+
+The port node must contain at least one endpoint, either parallel or mipi.
+It could have multiple endpoints, but please note the hardware don't support
+two sensors work at a time, they are supposed to work asynchronously.
+
+Device node example
+-------------------
+
+	isp0: isp0@ff910000 {
+		compatible = "rockchip,rk3399-cif-isp";
+		reg = <0x0 0xff910000 0x0 0x4000>;
+		interrupts = <GIC_SPI 43 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru SCLK_ISP0>,
+			 <&cru ACLK_ISP0>, <&cru ACLK_ISP0_WRAPPER>,
+			 <&cru HCLK_ISP0>, <&cru HCLK_ISP0_WRAPPER>;
+		clock-names = "clk_isp",
+			      "aclk_isp", "aclk_isp_wrap",
+			      "hclk_isp", "hclk_isp_wrap";
+		power-domains = <&power RK3399_PD_ISP0>;
+		iommus = <&isp0_mmu>;
+		phys = <&dphy>;
+		phy-names = "dphy";
+
+		ports {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			port@0 {
+				mipi_in_wcam: endpoint@0 {
+					reg = <0>;
+					remote-endpoint = <&wcam_out>;
+					data-lanes = <1 2>;
+				};
+
+				mipi_in_ucam: endpoint@1 {
+					reg = <1>;
+					remote-endpoint = <&ucam_out>;
+					data-lanes = <1>;
+				};
+			};
+		};
+	};
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 11/14] dt-bindings: Document the Rockchip MIPI RX D-PHY bindings
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (8 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 10/14] dt-bindings: Document the Rockchip ISP1 bindings Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-15 18:14   ` Laurent Pinchart
  2019-07-30 18:42 ` [PATCH v8 12/14] arm64: dts: rockchip: add isp0 node for rk3399 Helen Koike
                   ` (5 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Rob Herring,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

From: Jacob Chen <jacob2.chen@rock-chips.com>

Add DT bindings documentation for Rockchip MIPI D-PHY RX

Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
Reviewed-by: Rob Herring <robh@kernel.org>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- updated doc with new design and tested example

 .../bindings/media/rockchip-mipi-dphy.txt     | 38 +++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt

diff --git a/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt b/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
new file mode 100644
index 000000000000..2305d44d92db
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
@@ -0,0 +1,38 @@
+Rockchip SoC MIPI RX D-PHY
+-------------------------------------------------------------
+
+Required properties:
+- compatible: value should be one of the following
+	"rockchip,rk3288-mipi-dphy"
+	"rockchip,rk3399-mipi-dphy"
+- clocks : list of clock specifiers, corresponding to entries in
+	clock-names property;
+- clock-names: required clock name.
+- #phy-cells: Number of cells in a PHY specifier; Should be 0.
+
+MIPI RX D-PHY use registers in "general register files", it
+should be a child of the GRF.
+
+Optional properties:
+- reg: offset and length of the register set for the device.
+- rockchip,grf: MIPI TX1RX1 D-PHY not only has its own register but also
+		the GRF, so it is only necessary for MIPI TX1RX1 D-PHY.
+
+Device node example
+-------------------
+
+grf: syscon@ff770000 {
+	compatible = "rockchip,rk3399-grf", "syscon", "simple-mfd";
+
+...
+
+	dphy: mipi-dphy {
+		compatible = "rockchip,rk3399-mipi-dphy";
+		clocks = <&cru SCLK_MIPIDPHY_REF>,
+			<&cru SCLK_DPHY_RX0_CFG>,
+			<&cru PCLK_VIO_GRF>;
+		clock-names = "dphy-ref", "dphy-cfg", "grf";
+		power-domains = <&power RK3399_PD_VIO>;
+		#phy-cells = <0>;
+	};
+};
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 12/14] arm64: dts: rockchip: add isp0 node for rk3399
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (9 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 11/14] dt-bindings: Document the Rockchip MIPI RX D-PHY bindings Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-07-30 18:42 ` [PATCH v8 13/14] arm64: dts: rockchip: add rx0 mipi-phy " Helen Koike
                   ` (4 subsequent siblings)
  15 siblings, 0 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

From: Shunqian Zheng <zhengsq@rock-chips.com>

rk3399 have two ISP, but we havn't test isp1, so just add isp0 at present.

Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- add phy properties
- add ports

 arch/arm64/boot/dts/rockchip/rk3399.dtsi | 25 ++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3399.dtsi b/arch/arm64/boot/dts/rockchip/rk3399.dtsi
index cede1ad81be2..776d2bd48c06 100644
--- a/arch/arm64/boot/dts/rockchip/rk3399.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3399.dtsi
@@ -1698,6 +1698,31 @@
 		status = "disabled";
 	};
 
+	isp0: isp0@ff910000 {
+		compatible = "rockchip,rk3399-cif-isp";
+		reg = <0x0 0xff910000 0x0 0x4000>;
+		interrupts = <GIC_SPI 43 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru SCLK_ISP0>,
+			 <&cru ACLK_ISP0>, <&cru ACLK_ISP0_WRAPPER>,
+			 <&cru HCLK_ISP0>, <&cru HCLK_ISP0_WRAPPER>;
+		clock-names = "clk_isp",
+			      "aclk_isp", "aclk_isp_wrap",
+			      "hclk_isp", "hclk_isp_wrap";
+		power-domains = <&power RK3399_PD_ISP0>;
+		iommus = <&isp0_mmu>;
+		phys = <&mipi_dphy_rx0>;
+		phy-names = "dphy";
+
+		ports {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			port@0 {
+				reg = <0>;
+			};
+		};
+	};
+
 	isp0_mmu: iommu@ff914000 {
 		compatible = "rockchip,iommu";
 		reg = <0x0 0xff914000 0x0 0x100>, <0x0 0xff915000 0x0 0x100>;
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 13/14] arm64: dts: rockchip: add rx0 mipi-phy for rk3399
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (10 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 12/14] arm64: dts: rockchip: add isp0 node for rk3399 Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-07-30 18:42 ` [PATCH v8 14/14] MAINTAINERS: add entry for Rockchip ISP1 driver Helen Koike
                   ` (3 subsequent siblings)
  15 siblings, 0 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

From: Shunqian Zheng <zhengsq@rock-chips.com>

It's a Designware MIPI D-PHY, used for ISP0 in rk3399.

Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
[update for upstream]
Signed-off-by: Helen Koike <helen.koike@collabora.com>

---

Changes in v8: None
Changes in v7:
- add phy-cells

 arch/arm64/boot/dts/rockchip/rk3399.dtsi | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3399.dtsi b/arch/arm64/boot/dts/rockchip/rk3399.dtsi
index 776d2bd48c06..3630d95e5cd8 100644
--- a/arch/arm64/boot/dts/rockchip/rk3399.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3399.dtsi
@@ -1385,6 +1385,17 @@
 			status = "disabled";
 		};
 
+		mipi_dphy_rx0: mipi-dphy-rx0 {
+			compatible = "rockchip,rk3399-mipi-dphy";
+			clocks = <&cru SCLK_MIPIDPHY_REF>,
+				<&cru SCLK_DPHY_RX0_CFG>,
+				<&cru PCLK_VIO_GRF>;
+			clock-names = "dphy-ref", "dphy-cfg", "grf";
+			power-domains = <&power RK3399_PD_VIO>;
+			#phy-cells = <0>;
+			status = "disabled";
+		};
+
 		u2phy0: usb2-phy@e450 {
 			compatible = "rockchip,rk3399-usb2phy";
 			reg = <0xe450 0x10>;
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* [PATCH v8 14/14] MAINTAINERS: add entry for Rockchip ISP1 driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (11 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 13/14] arm64: dts: rockchip: add rx0 mipi-phy " Helen Koike
@ 2019-07-30 18:42 ` Helen Koike
  2019-08-15 13:56   ` Laurent Pinchart
  2019-07-30 20:15 ` [PATCH v8 00/14] Rockchip ISP1 Driver Hans Verkuil
                   ` (2 subsequent siblings)
  15 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 18:42 UTC (permalink / raw)
  To: linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Helen Koike, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

Add MAINTAINERS entry for the rockchip isp1 driver.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
---

Changes in v8: None
Changes in v7: None

 MAINTAINERS | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 6426db5198f0..7f38abcb4114 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13743,6 +13743,14 @@ F:	drivers/hid/hid-roccat*
 F:	include/linux/hid-roccat*
 F:	Documentation/ABI/*/sysfs-driver-hid-roccat*
 
+ROCKCHIP ISP V1 DRIVER
+M:	Helen Koike <helen.koike@collabora.com>
+L:	linux-media@vger.kernel.org
+S:	Maintained
+F:	drivers/media/platform/rockchip/isp1/
+F:	Documentation/devicetree/bindings/media/rockchip-isp1.txt
+F:	Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
+
 ROCKCHIP RASTER 2D GRAPHIC ACCELERATION UNIT DRIVER
 M:	Jacob chen <jacob2.chen@rock-chips.com>
 L:	linux-media@vger.kernel.org
-- 
2.22.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (12 preceding siblings ...)
  2019-07-30 18:42 ` [PATCH v8 14/14] MAINTAINERS: add entry for Rockchip ISP1 driver Helen Koike
@ 2019-07-30 20:15 ` Hans Verkuil
  2019-07-30 20:50   ` Helen Koike
  2019-08-07 15:37 ` Sakari Ailus
  2019-08-09 18:45 ` Manivannan Sadhasivam
  15 siblings, 1 reply; 76+ messages in thread
From: Hans Verkuil @ 2019-07-30 20:15 UTC (permalink / raw)
  To: Helen Koike, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

On 7/30/19 8:42 PM, Helen Koike wrote:
> Hello,
> 
> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
> rk3399 SoC.
> 
> I didn't change much from the last version, just applying the
> suggestions made in the previous one.
> 
> This patchset is also available at:
> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
> 
> Libcamera patched to work with this version:
> https://gitlab.collabora.com/koike/libcamera
> (also sent to the mailing list)
> 
> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
> Scarlet Chromebook.
> 
> Known issues (same as in v7):
> -------------
> - Reloading the module doesn't work (there is some missing cleanup when
> unloading)
> - When capturing in bayer format, changing the size doesn't seem to
> affect the image.
> - crop needs more tests
> - v4l2-compliance error:
>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

Can you mail me the full v4l2-compliance output?

Regards,

	Hans

> It seems that if controls are supported, v4l2-compliance says that
> controls of type 'Image Processing Controls' are mandatory, is this
> correct?
> - It seems there are still some issues with interrupts, but I couldn't
> isolate them yet.
> 
> Previous changelog:
> -------------------
> 
> changes in V6:
>   - add mipi txrx phy support
>   - remove bool and enum from uapi header
>   - add buf_prepare op
>   - correct some spelling problems
>   - return all queued buffers when starting stream failed
> 
> changes in V5: Sync with local changes,
>   - fix the SP height limit
>   - speed up the second stream capture
>   - the second stream can't force sync for rsz when start/stop streaming
>   - add frame id to param vb2 buf
>   - enable luminance maximum threshold
> 
> changes in V4:
>   - fix some bugs during development
>   - move quantization settings to rkisp1 subdev
>   - correct some spelling problems
>   - describe ports in dt-binding documents
> 
> changes in V3:
>   - add some comments
>   - fix wrong use of v4l2_async_subdev_notifier_register
>   - optimize two paths capture at a time
>   - remove compose
>   - re-struct headers
>   - add a tmp wiki page: http://opensource.rock-chips.com/wiki_Rockchip-isp1
> 
> changes in V2:
>   mipi-phy:
>     - use async probing
>     - make it be a child device of the GRF
>   isp:
>     - add dummy buffer
>     - change the way to get bus configuration, which make it possible to
>             add parallel sensor support in the future(without mipi-phy driver).
> 
> ------------------
> 
> Changes in v8:
> - Add SPDX in the header
> - Remove emacs configs
> - Fix doc style
> - Remove boiler plate license text
> 
> Changes in v7:
> - s/IPU3/RK_ISP1
> - s/correspond/corresponding
> - s/use/uses
> - s/docuemnt/document
> - Fix checkpatch errors (lines over 80 and SPDX)
> - Add TODO to improve docs
> - Migrate dphy specific code from
> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
> to drivers/phy/rockchip/phy-rockchip-dphy.c
> - Drop support for rk3288
> - Drop support for dphy txrx
> - code styling and checkpatch fixes
> - fixed warning because of unknown entity type
> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> and default values
> - fix typo riksp1/rkisp1
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now. As a consequence, remove the
> hack in mipidphy_g_mbus_config() where information from the sensor was
> being propagated through the topology.
> - From the old dphy:
>         * cache get_remote_sensor() in s_stream
>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> - Replace stream state with a boolean
> - code styling and checkpatch fixes
> - fix stop_stream (return after calling stop, do not reenable the stream)
> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> - s/intput/input
> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> reused by the capture
> - s/strlcpy/strscpy
> - sort out the locks in isp stats
> - code styling and checkpatch fixes
> - s/strlcpy/strscpy
> - s/strcpy/strscpy
> - fix config lsc error
> LSC data table size is 17x17, but when configuring data to ISP,
> should be aligned to 18x17. That means every last data of last
> line should be filled with 0, and not filled with the data of
> next line.
> - Update new ISP parameters immediately
> For those sub modules that have shadow registers in core isp, the
> new programing parameters would not be active if both
> CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT and CFG_UPD are not set. Now
> we configure CFG_UPD to force update the shadow registers when new
> ISP parameters are configured.
> - fix some ISP parameters config error
> Some ISP parameter config functions may override the old enable
> bit value, because the enable bits of these modules are in the
> same registers with parameters. So we should save the old enable
> bits firstly.
> - code styling and checkpatch fixes
> - s/strlcpy/strscpy
> - Fix v4l2-compliance issues:
>         * remove input ioctls
> media api can be used to define the topology, this input api is not
> required. Besides it, if an input is enumerated, v4l2-compliance is not
> happy with G_FMT returning the default colorspace instead of something
> more specific.
>         * return the pixelformat to the userspace
> G_/S_/TRY_ FORMAT should return a valid pixelformat to the user, even if
> the user gave an invalid one
>         * add missing default colorspace and ycbcr
>         * fix wrong pixformat in mp_fmts[] table
>         * add buf type check in s_/g_selection
>         * queue_setup - check sizes
>         * normalize bus_info name
>         * fix field any v4l2-compliance -s complain - set field none
>         when streaming
> - Fix compiling error: s/vidioc_enum_fmt_vid_cap_mplane/vidioc_enum_fmt_vid_cap
> - Replace stream state with a boolean
> The rkisp1_state enum consists only of 3 entries, where 1 is completely
> unused and the other two respectively mean not streaming or streaming.
> Replace it with a boolean called "streaming".
> - Simplify MI interrupt handling
> Rather than adding unnecessary indirection, just use stream index to
> handle MI interrupt enable/disable/clear, since the stream index matches
> the order of bits now, thanks to previous patch. While at it, remove
> some dead code.
> - code styling and checkpatch fixes
> - add link_validate: don't allow a link with bayer/non-bayer mismatch
> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
> - Fix compilation and runtime errors due to bitrotting
> The code has bit-rotten since March 2018, fix compilation errors.
> The new V4L2 async notifier API requires notifiers to be initialized by
> a call to v4l2_async_notifier_init() before being used, do so.
> - Add missing module device table
> - use clk_bulk framework
> - add missing notifiers cleanups
> - s/strlcpy/strscpy
> - normalize bus_info name
> - fix s_stream error path, stream_cnt wans't being decremented properly
> - use devm_platform_ioremap_resource() helper
> - s/deice/device
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now.
> - remove "saved_state" member from rkisp1_stream struct
> - Reverse the order of MIs
> - Simplify MI interrupt handling
> Rather than adding unnecessary indirection, just use stream index to
> handle MI interrupt enable/disable/clear, since the stream index matches
> the order of bits now, thanks to previous patch. While at it, remove
> some dead code.
> - code styling and checkpatch fixes
> - update document with new design and tested example
> - updated doc with new design and tested example
> - add phy properties
> - add ports
> - add phy-cells
> 
> Helen Koike (1):
>   MAINTAINERS: add entry for Rockchip ISP1 driver
> 
> Jacob Chen (9):
>   media: doc: add document for rkisp1 meta buffer format
>   media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
>   media: rkisp1: add Rockchip ISP1 subdev driver
>   media: rkisp1: add ISP1 statistics driver
>   media: rkisp1: add ISP1 params driver
>   media: rkisp1: add capture device driver
>   media: rkisp1: add rockchip isp1 core driver
>   dt-bindings: Document the Rockchip ISP1 bindings
>   dt-bindings: Document the Rockchip MIPI RX D-PHY bindings
> 
> Jeffy Chen (1):
>   media: rkisp1: Add user space ABI definitions
> 
> Shunqian Zheng (3):
>   media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
>   arm64: dts: rockchip: add isp0 node for rk3399
>   arm64: dts: rockchip: add rx0 mipi-phy for rk3399
> 
>  .../bindings/media/rockchip-isp1.txt          |   71 +
>  .../bindings/media/rockchip-mipi-dphy.txt     |   38 +
>  Documentation/media/uapi/v4l/meta-formats.rst |    2 +
>  .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    |   23 +
>  .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      |   22 +
>  MAINTAINERS                                   |    8 +
>  arch/arm64/boot/dts/rockchip/rk3399.dtsi      |   36 +
>  drivers/media/platform/Kconfig                |   12 +
>  drivers/media/platform/Makefile               |    1 +
>  drivers/media/platform/rockchip/isp1/Makefile |    7 +
>  .../media/platform/rockchip/isp1/capture.c    | 1754 +++++++++++++++++
>  .../media/platform/rockchip/isp1/capture.h    |  164 ++
>  drivers/media/platform/rockchip/isp1/common.h |  101 +
>  drivers/media/platform/rockchip/isp1/dev.c    |  675 +++++++
>  drivers/media/platform/rockchip/isp1/dev.h    |   97 +
>  .../media/platform/rockchip/isp1/isp_params.c | 1604 +++++++++++++++
>  .../media/platform/rockchip/isp1/isp_params.h |   50 +
>  .../media/platform/rockchip/isp1/isp_stats.c  |  508 +++++
>  .../media/platform/rockchip/isp1/isp_stats.h  |   60 +
>  drivers/media/platform/rockchip/isp1/regs.c   |  223 +++
>  drivers/media/platform/rockchip/isp1/regs.h   | 1525 ++++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 ++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |    2 +
>  drivers/phy/rockchip/Kconfig                  |    8 +
>  drivers/phy/rockchip/Makefile                 |    1 +
>  drivers/phy/rockchip/phy-rockchip-dphy.c      |  408 ++++
>  include/uapi/linux/rkisp1-config.h            |  816 ++++++++
>  include/uapi/linux/videodev2.h                |    4 +
>  29 files changed, 9617 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt
>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
>  create mode 100644 include/uapi/linux/rkisp1-config.h
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-30 20:15 ` [PATCH v8 00/14] Rockchip ISP1 Driver Hans Verkuil
@ 2019-07-30 20:50   ` Helen Koike
  2019-07-31  0:08     ` Helen Koike
  0 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-30 20:50 UTC (permalink / raw)
  To: Hans Verkuil, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, hans.verkuil,
	laurent.pinchart, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media



On 7/30/19 5:15 PM, Hans Verkuil wrote:
> On 7/30/19 8:42 PM, Helen Koike wrote:
>> Hello,
>>
>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>> rk3399 SoC.
>>
>> I didn't change much from the last version, just applying the
>> suggestions made in the previous one.
>>
>> This patchset is also available at:
>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>
>> Libcamera patched to work with this version:
>> https://gitlab.collabora.com/koike/libcamera
>> (also sent to the mailing list)
>>
>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>> Scarlet Chromebook.
>>
>> Known issues (same as in v7):
>> -------------
>> - Reloading the module doesn't work (there is some missing cleanup when
>> unloading)
>> - When capturing in bayer format, changing the size doesn't seem to
>> affect the image.
>> - crop needs more tests
>> - v4l2-compliance error:
>>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
> 
> Can you mail me the full v4l2-compliance output?

Sure, please check here: http://ix.io/1Q5u
I updated v4l-utils with the latest version and I re-ran bootstrap/configure/make,
but for some reason the hash from the link above is not the latest commit, probably some
old configuration somewhere. I'll resend this log as soon as I get v4l2-compliance
properly updated.

Thanks
Helen

> 
> Regards,
> 
> 	Hans
> 
>> It seems that if controls are supported, v4l2-compliance says that
>> controls of type 'Image Processing Controls' are mandatory, is this
>> correct?
>> - It seems there are still some issues with interrupts, but I couldn't
>> isolate them yet.
>>
>> Previous changelog:
>> -------------------
>>
>> changes in V6:
>>   - add mipi txrx phy support
>>   - remove bool and enum from uapi header
>>   - add buf_prepare op
>>   - correct some spelling problems
>>   - return all queued buffers when starting stream failed
>>
>> changes in V5: Sync with local changes,
>>   - fix the SP height limit
>>   - speed up the second stream capture
>>   - the second stream can't force sync for rsz when start/stop streaming
>>   - add frame id to param vb2 buf
>>   - enable luminance maximum threshold
>>
>> changes in V4:
>>   - fix some bugs during development
>>   - move quantization settings to rkisp1 subdev
>>   - correct some spelling problems
>>   - describe ports in dt-binding documents
>>
>> changes in V3:
>>   - add some comments
>>   - fix wrong use of v4l2_async_subdev_notifier_register
>>   - optimize two paths capture at a time
>>   - remove compose
>>   - re-struct headers
>>   - add a tmp wiki page: http://opensource.rock-chips.com/wiki_Rockchip-isp1
>>
>> changes in V2:
>>   mipi-phy:
>>     - use async probing
>>     - make it be a child device of the GRF
>>   isp:
>>     - add dummy buffer
>>     - change the way to get bus configuration, which make it possible to
>>             add parallel sensor support in the future(without mipi-phy driver).
>>
>> ------------------
>>
>> Changes in v8:
>> - Add SPDX in the header
>> - Remove emacs configs
>> - Fix doc style
>> - Remove boiler plate license text
>>
>> Changes in v7:
>> - s/IPU3/RK_ISP1
>> - s/correspond/corresponding
>> - s/use/uses
>> - s/docuemnt/document
>> - Fix checkpatch errors (lines over 80 and SPDX)
>> - Add TODO to improve docs
>> - Migrate dphy specific code from
>> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
>> to drivers/phy/rockchip/phy-rockchip-dphy.c
>> - Drop support for rk3288
>> - Drop support for dphy txrx
>> - code styling and checkpatch fixes
>> - fixed warning because of unknown entity type
>> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
>> and default values
>> - fix typo riksp1/rkisp1
>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>> isp subdevice in the media topology now. As a consequence, remove the
>> hack in mipidphy_g_mbus_config() where information from the sensor was
>> being propagated through the topology.
>> - From the old dphy:
>>         * cache get_remote_sensor() in s_stream
>>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
>> - Replace stream state with a boolean
>> - code styling and checkpatch fixes
>> - fix stop_stream (return after calling stop, do not reenable the stream)
>> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
>> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
>> - s/intput/input
>> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
>> reused by the capture
>> - s/strlcpy/strscpy
>> - sort out the locks in isp stats
>> - code styling and checkpatch fixes
>> - s/strlcpy/strscpy
>> - s/strcpy/strscpy
>> - fix config lsc error
>> LSC data table size is 17x17, but when configuring data to ISP,
>> should be aligned to 18x17. That means every last data of last
>> line should be filled with 0, and not filled with the data of
>> next line.
>> - Update new ISP parameters immediately
>> For those sub modules that have shadow registers in core isp, the
>> new programing parameters would not be active if both
>> CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT and CFG_UPD are not set. Now
>> we configure CFG_UPD to force update the shadow registers when new
>> ISP parameters are configured.
>> - fix some ISP parameters config error
>> Some ISP parameter config functions may override the old enable
>> bit value, because the enable bits of these modules are in the
>> same registers with parameters. So we should save the old enable
>> bits firstly.
>> - code styling and checkpatch fixes
>> - s/strlcpy/strscpy
>> - Fix v4l2-compliance issues:
>>         * remove input ioctls
>> media api can be used to define the topology, this input api is not
>> required. Besides it, if an input is enumerated, v4l2-compliance is not
>> happy with G_FMT returning the default colorspace instead of something
>> more specific.
>>         * return the pixelformat to the userspace
>> G_/S_/TRY_ FORMAT should return a valid pixelformat to the user, even if
>> the user gave an invalid one
>>         * add missing default colorspace and ycbcr
>>         * fix wrong pixformat in mp_fmts[] table
>>         * add buf type check in s_/g_selection
>>         * queue_setup - check sizes
>>         * normalize bus_info name
>>         * fix field any v4l2-compliance -s complain - set field none
>>         when streaming
>> - Fix compiling error: s/vidioc_enum_fmt_vid_cap_mplane/vidioc_enum_fmt_vid_cap
>> - Replace stream state with a boolean
>> The rkisp1_state enum consists only of 3 entries, where 1 is completely
>> unused and the other two respectively mean not streaming or streaming.
>> Replace it with a boolean called "streaming".
>> - Simplify MI interrupt handling
>> Rather than adding unnecessary indirection, just use stream index to
>> handle MI interrupt enable/disable/clear, since the stream index matches
>> the order of bits now, thanks to previous patch. While at it, remove
>> some dead code.
>> - code styling and checkpatch fixes
>> - add link_validate: don't allow a link with bayer/non-bayer mismatch
>> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
>> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
>> - Fix compilation and runtime errors due to bitrotting
>> The code has bit-rotten since March 2018, fix compilation errors.
>> The new V4L2 async notifier API requires notifiers to be initialized by
>> a call to v4l2_async_notifier_init() before being used, do so.
>> - Add missing module device table
>> - use clk_bulk framework
>> - add missing notifiers cleanups
>> - s/strlcpy/strscpy
>> - normalize bus_info name
>> - fix s_stream error path, stream_cnt wans't being decremented properly
>> - use devm_platform_ioremap_resource() helper
>> - s/deice/device
>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>> isp subdevice in the media topology now.
>> - remove "saved_state" member from rkisp1_stream struct
>> - Reverse the order of MIs
>> - Simplify MI interrupt handling
>> Rather than adding unnecessary indirection, just use stream index to
>> handle MI interrupt enable/disable/clear, since the stream index matches
>> the order of bits now, thanks to previous patch. While at it, remove
>> some dead code.
>> - code styling and checkpatch fixes
>> - update document with new design and tested example
>> - updated doc with new design and tested example
>> - add phy properties
>> - add ports
>> - add phy-cells
>>
>> Helen Koike (1):
>>   MAINTAINERS: add entry for Rockchip ISP1 driver
>>
>> Jacob Chen (9):
>>   media: doc: add document for rkisp1 meta buffer format
>>   media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
>>   media: rkisp1: add Rockchip ISP1 subdev driver
>>   media: rkisp1: add ISP1 statistics driver
>>   media: rkisp1: add ISP1 params driver
>>   media: rkisp1: add capture device driver
>>   media: rkisp1: add rockchip isp1 core driver
>>   dt-bindings: Document the Rockchip ISP1 bindings
>>   dt-bindings: Document the Rockchip MIPI RX D-PHY bindings
>>
>> Jeffy Chen (1):
>>   media: rkisp1: Add user space ABI definitions
>>
>> Shunqian Zheng (3):
>>   media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
>>   arm64: dts: rockchip: add isp0 node for rk3399
>>   arm64: dts: rockchip: add rx0 mipi-phy for rk3399
>>
>>  .../bindings/media/rockchip-isp1.txt          |   71 +
>>  .../bindings/media/rockchip-mipi-dphy.txt     |   38 +
>>  Documentation/media/uapi/v4l/meta-formats.rst |    2 +
>>  .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    |   23 +
>>  .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      |   22 +
>>  MAINTAINERS                                   |    8 +
>>  arch/arm64/boot/dts/rockchip/rk3399.dtsi      |   36 +
>>  drivers/media/platform/Kconfig                |   12 +
>>  drivers/media/platform/Makefile               |    1 +
>>  drivers/media/platform/rockchip/isp1/Makefile |    7 +
>>  .../media/platform/rockchip/isp1/capture.c    | 1754 +++++++++++++++++
>>  .../media/platform/rockchip/isp1/capture.h    |  164 ++
>>  drivers/media/platform/rockchip/isp1/common.h |  101 +
>>  drivers/media/platform/rockchip/isp1/dev.c    |  675 +++++++
>>  drivers/media/platform/rockchip/isp1/dev.h    |   97 +
>>  .../media/platform/rockchip/isp1/isp_params.c | 1604 +++++++++++++++
>>  .../media/platform/rockchip/isp1/isp_params.h |   50 +
>>  .../media/platform/rockchip/isp1/isp_stats.c  |  508 +++++
>>  .../media/platform/rockchip/isp1/isp_stats.h  |   60 +
>>  drivers/media/platform/rockchip/isp1/regs.c   |  223 +++
>>  drivers/media/platform/rockchip/isp1/regs.h   | 1525 ++++++++++++++
>>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 ++++++++++++
>>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>>  drivers/media/v4l2-core/v4l2-ioctl.c          |    2 +
>>  drivers/phy/rockchip/Kconfig                  |    8 +
>>  drivers/phy/rockchip/Makefile                 |    1 +
>>  drivers/phy/rockchip/phy-rockchip-dphy.c      |  408 ++++
>>  include/uapi/linux/rkisp1-config.h            |  816 ++++++++
>>  include/uapi/linux/videodev2.h                |    4 +
>>  29 files changed, 9617 insertions(+)
>>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt
>>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
>>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
>>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
>>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
>>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
>>  create mode 100644 include/uapi/linux/rkisp1-config.h
>>
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-30 20:50   ` Helen Koike
@ 2019-07-31  0:08     ` Helen Koike
  2019-07-31  4:29       ` Hans Verkuil
  0 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-07-31  0:08 UTC (permalink / raw)
  To: Hans Verkuil, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, zhengsq, jeffy.chen,
	zyc, linux-kernel, tfiga, hans.verkuil, laurent.pinchart,
	sakari.ailus, mchehab, ezequiel, linux-arm-kernel, linux-media



On 7/30/19 5:50 PM, Helen Koike wrote:
> 
> 
> On 7/30/19 5:15 PM, Hans Verkuil wrote:
>> On 7/30/19 8:42 PM, Helen Koike wrote:
>>> Hello,
>>>
>>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>>> rk3399 SoC.
>>>
>>> I didn't change much from the last version, just applying the
>>> suggestions made in the previous one.
>>>
>>> This patchset is also available at:
>>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>>
>>> Libcamera patched to work with this version:
>>> https://gitlab.collabora.com/koike/libcamera
>>> (also sent to the mailing list)
>>>
>>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>>> Scarlet Chromebook.
>>>
>>> Known issues (same as in v7):
>>> -------------
>>> - Reloading the module doesn't work (there is some missing cleanup when
>>> unloading)
>>> - When capturing in bayer format, changing the size doesn't seem to
>>> affect the image.
>>> - crop needs more tests
>>> - v4l2-compliance error:
>>>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>>
>> Can you mail me the full v4l2-compliance output?
> 
> Sure, please check here: http://ix.io/1Q5u
> I updated v4l-utils with the latest version and I re-ran bootstrap/configure/make,
> but for some reason the hash from the link above is not the latest commit, probably some
> old configuration somewhere. I'll resend this log as soon as I get v4l2-compliance
> properly updated.

Please see the output of v4l2-compliance here with an updated v4l-utils: http://ix.io/1Q6A

> 
> Thanks
> Helen
> 
>>
>> Regards,
>>
>> 	Hans
>>
>>> It seems that if controls are supported, v4l2-compliance says that
>>> controls of type 'Image Processing Controls' are mandatory, is this
>>> correct?
>>> - It seems there are still some issues with interrupts, but I couldn't
>>> isolate them yet.
>>>
>>> Previous changelog:
>>> -------------------
>>>
>>> changes in V6:
>>>   - add mipi txrx phy support
>>>   - remove bool and enum from uapi header
>>>   - add buf_prepare op
>>>   - correct some spelling problems
>>>   - return all queued buffers when starting stream failed
>>>
>>> changes in V5: Sync with local changes,
>>>   - fix the SP height limit
>>>   - speed up the second stream capture
>>>   - the second stream can't force sync for rsz when start/stop streaming
>>>   - add frame id to param vb2 buf
>>>   - enable luminance maximum threshold
>>>
>>> changes in V4:
>>>   - fix some bugs during development
>>>   - move quantization settings to rkisp1 subdev
>>>   - correct some spelling problems
>>>   - describe ports in dt-binding documents
>>>
>>> changes in V3:
>>>   - add some comments
>>>   - fix wrong use of v4l2_async_subdev_notifier_register
>>>   - optimize two paths capture at a time
>>>   - remove compose
>>>   - re-struct headers
>>>   - add a tmp wiki page: http://opensource.rock-chips.com/wiki_Rockchip-isp1
>>>
>>> changes in V2:
>>>   mipi-phy:
>>>     - use async probing
>>>     - make it be a child device of the GRF
>>>   isp:
>>>     - add dummy buffer
>>>     - change the way to get bus configuration, which make it possible to
>>>             add parallel sensor support in the future(without mipi-phy driver).
>>>
>>> ------------------
>>>
>>> Changes in v8:
>>> - Add SPDX in the header
>>> - Remove emacs configs
>>> - Fix doc style
>>> - Remove boiler plate license text
>>>
>>> Changes in v7:
>>> - s/IPU3/RK_ISP1
>>> - s/correspond/corresponding
>>> - s/use/uses
>>> - s/docuemnt/document
>>> - Fix checkpatch errors (lines over 80 and SPDX)
>>> - Add TODO to improve docs
>>> - Migrate dphy specific code from
>>> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
>>> to drivers/phy/rockchip/phy-rockchip-dphy.c
>>> - Drop support for rk3288
>>> - Drop support for dphy txrx
>>> - code styling and checkpatch fixes
>>> - fixed warning because of unknown entity type
>>> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
>>> and default values
>>> - fix typo riksp1/rkisp1
>>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>>> isp subdevice in the media topology now. As a consequence, remove the
>>> hack in mipidphy_g_mbus_config() where information from the sensor was
>>> being propagated through the topology.
>>> - From the old dphy:
>>>         * cache get_remote_sensor() in s_stream
>>>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
>>> - Replace stream state with a boolean
>>> - code styling and checkpatch fixes
>>> - fix stop_stream (return after calling stop, do not reenable the stream)
>>> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
>>> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
>>> - s/intput/input
>>> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
>>> reused by the capture
>>> - s/strlcpy/strscpy
>>> - sort out the locks in isp stats
>>> - code styling and checkpatch fixes
>>> - s/strlcpy/strscpy
>>> - s/strcpy/strscpy
>>> - fix config lsc error
>>> LSC data table size is 17x17, but when configuring data to ISP,
>>> should be aligned to 18x17. That means every last data of last
>>> line should be filled with 0, and not filled with the data of
>>> next line.
>>> - Update new ISP parameters immediately
>>> For those sub modules that have shadow registers in core isp, the
>>> new programing parameters would not be active if both
>>> CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT and CFG_UPD are not set. Now
>>> we configure CFG_UPD to force update the shadow registers when new
>>> ISP parameters are configured.
>>> - fix some ISP parameters config error
>>> Some ISP parameter config functions may override the old enable
>>> bit value, because the enable bits of these modules are in the
>>> same registers with parameters. So we should save the old enable
>>> bits firstly.
>>> - code styling and checkpatch fixes
>>> - s/strlcpy/strscpy
>>> - Fix v4l2-compliance issues:
>>>         * remove input ioctls
>>> media api can be used to define the topology, this input api is not
>>> required. Besides it, if an input is enumerated, v4l2-compliance is not
>>> happy with G_FMT returning the default colorspace instead of something
>>> more specific.
>>>         * return the pixelformat to the userspace
>>> G_/S_/TRY_ FORMAT should return a valid pixelformat to the user, even if
>>> the user gave an invalid one
>>>         * add missing default colorspace and ycbcr
>>>         * fix wrong pixformat in mp_fmts[] table
>>>         * add buf type check in s_/g_selection
>>>         * queue_setup - check sizes
>>>         * normalize bus_info name
>>>         * fix field any v4l2-compliance -s complain - set field none
>>>         when streaming
>>> - Fix compiling error: s/vidioc_enum_fmt_vid_cap_mplane/vidioc_enum_fmt_vid_cap
>>> - Replace stream state with a boolean
>>> The rkisp1_state enum consists only of 3 entries, where 1 is completely
>>> unused and the other two respectively mean not streaming or streaming.
>>> Replace it with a boolean called "streaming".
>>> - Simplify MI interrupt handling
>>> Rather than adding unnecessary indirection, just use stream index to
>>> handle MI interrupt enable/disable/clear, since the stream index matches
>>> the order of bits now, thanks to previous patch. While at it, remove
>>> some dead code.
>>> - code styling and checkpatch fixes
>>> - add link_validate: don't allow a link with bayer/non-bayer mismatch
>>> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
>>> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
>>> - Fix compilation and runtime errors due to bitrotting
>>> The code has bit-rotten since March 2018, fix compilation errors.
>>> The new V4L2 async notifier API requires notifiers to be initialized by
>>> a call to v4l2_async_notifier_init() before being used, do so.
>>> - Add missing module device table
>>> - use clk_bulk framework
>>> - add missing notifiers cleanups
>>> - s/strlcpy/strscpy
>>> - normalize bus_info name
>>> - fix s_stream error path, stream_cnt wans't being decremented properly
>>> - use devm_platform_ioremap_resource() helper
>>> - s/deice/device
>>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>>> isp subdevice in the media topology now.
>>> - remove "saved_state" member from rkisp1_stream struct
>>> - Reverse the order of MIs
>>> - Simplify MI interrupt handling
>>> Rather than adding unnecessary indirection, just use stream index to
>>> handle MI interrupt enable/disable/clear, since the stream index matches
>>> the order of bits now, thanks to previous patch. While at it, remove
>>> some dead code.
>>> - code styling and checkpatch fixes
>>> - update document with new design and tested example
>>> - updated doc with new design and tested example
>>> - add phy properties
>>> - add ports
>>> - add phy-cells
>>>
>>> Helen Koike (1):
>>>   MAINTAINERS: add entry for Rockchip ISP1 driver
>>>
>>> Jacob Chen (9):
>>>   media: doc: add document for rkisp1 meta buffer format
>>>   media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
>>>   media: rkisp1: add Rockchip ISP1 subdev driver
>>>   media: rkisp1: add ISP1 statistics driver
>>>   media: rkisp1: add ISP1 params driver
>>>   media: rkisp1: add capture device driver
>>>   media: rkisp1: add rockchip isp1 core driver
>>>   dt-bindings: Document the Rockchip ISP1 bindings
>>>   dt-bindings: Document the Rockchip MIPI RX D-PHY bindings
>>>
>>> Jeffy Chen (1):
>>>   media: rkisp1: Add user space ABI definitions
>>>
>>> Shunqian Zheng (3):
>>>   media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
>>>   arm64: dts: rockchip: add isp0 node for rk3399
>>>   arm64: dts: rockchip: add rx0 mipi-phy for rk3399
>>>
>>>  .../bindings/media/rockchip-isp1.txt          |   71 +
>>>  .../bindings/media/rockchip-mipi-dphy.txt     |   38 +
>>>  Documentation/media/uapi/v4l/meta-formats.rst |    2 +
>>>  .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    |   23 +
>>>  .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      |   22 +
>>>  MAINTAINERS                                   |    8 +
>>>  arch/arm64/boot/dts/rockchip/rk3399.dtsi      |   36 +
>>>  drivers/media/platform/Kconfig                |   12 +
>>>  drivers/media/platform/Makefile               |    1 +
>>>  drivers/media/platform/rockchip/isp1/Makefile |    7 +
>>>  .../media/platform/rockchip/isp1/capture.c    | 1754 +++++++++++++++++
>>>  .../media/platform/rockchip/isp1/capture.h    |  164 ++
>>>  drivers/media/platform/rockchip/isp1/common.h |  101 +
>>>  drivers/media/platform/rockchip/isp1/dev.c    |  675 +++++++
>>>  drivers/media/platform/rockchip/isp1/dev.h    |   97 +
>>>  .../media/platform/rockchip/isp1/isp_params.c | 1604 +++++++++++++++
>>>  .../media/platform/rockchip/isp1/isp_params.h |   50 +
>>>  .../media/platform/rockchip/isp1/isp_stats.c  |  508 +++++
>>>  .../media/platform/rockchip/isp1/isp_stats.h  |   60 +
>>>  drivers/media/platform/rockchip/isp1/regs.c   |  223 +++
>>>  drivers/media/platform/rockchip/isp1/regs.h   | 1525 ++++++++++++++
>>>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 ++++++++++++
>>>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>>>  drivers/media/v4l2-core/v4l2-ioctl.c          |    2 +
>>>  drivers/phy/rockchip/Kconfig                  |    8 +
>>>  drivers/phy/rockchip/Makefile                 |    1 +
>>>  drivers/phy/rockchip/phy-rockchip-dphy.c      |  408 ++++
>>>  include/uapi/linux/rkisp1-config.h            |  816 ++++++++
>>>  include/uapi/linux/videodev2.h                |    4 +
>>>  29 files changed, 9617 insertions(+)
>>>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt
>>>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
>>>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
>>>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.c
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.h
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.c
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.h
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.c
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.h
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>>>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
>>>  create mode 100644 include/uapi/linux/rkisp1-config.h
>>>
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-31  0:08     ` Helen Koike
@ 2019-07-31  4:29       ` Hans Verkuil
  2019-07-31  4:33         ` Hans Verkuil
  0 siblings, 1 reply; 76+ messages in thread
From: Hans Verkuil @ 2019-07-31  4:29 UTC (permalink / raw)
  To: Helen Koike, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, zhengsq, jeffy.chen,
	zyc, linux-kernel, tfiga, hans.verkuil, laurent.pinchart,
	sakari.ailus, mchehab, ezequiel, linux-arm-kernel, linux-media

On 7/31/19 2:08 AM, Helen Koike wrote:
> 
> 
> On 7/30/19 5:50 PM, Helen Koike wrote:
>>
>>
>> On 7/30/19 5:15 PM, Hans Verkuil wrote:
>>> On 7/30/19 8:42 PM, Helen Koike wrote:
>>>> Hello,
>>>>
>>>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>>>> rk3399 SoC.
>>>>
>>>> I didn't change much from the last version, just applying the
>>>> suggestions made in the previous one.
>>>>
>>>> This patchset is also available at:
>>>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>>>
>>>> Libcamera patched to work with this version:
>>>> https://gitlab.collabora.com/koike/libcamera
>>>> (also sent to the mailing list)
>>>>
>>>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>>>> Scarlet Chromebook.
>>>>
>>>> Known issues (same as in v7):
>>>> -------------
>>>> - Reloading the module doesn't work (there is some missing cleanup when
>>>> unloading)
>>>> - When capturing in bayer format, changing the size doesn't seem to
>>>> affect the image.
>>>> - crop needs more tests
>>>> - v4l2-compliance error:
>>>>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
>>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>>>
>>> Can you mail me the full v4l2-compliance output?
>>
>> Sure, please check here: http://ix.io/1Q5u
>> I updated v4l-utils with the latest version and I re-ran bootstrap/configure/make,
>> but for some reason the hash from the link above is not the latest commit, probably some
>> old configuration somewhere. I'll resend this log as soon as I get v4l2-compliance
>> properly updated.
> 
> Please see the output of v4l2-compliance here with an updated v4l-utils: http://ix.io/1Q6A

So this FAIL is for /dev/v4l-subdev0 (rkisp1-isp-subdev).

What is weird that this subdev does not appear to have controls at all.

What is the output of 'v4l2-ctl -d /dev/v4l-subdev0 -l'? And if it lists
controls, then why?

If you run 'v4l2-compliance -u /dev/v4l-subdev0', do you get a fail as
well?

BTW, note that struct rkisp1_isp_subdev has a ctrl_handler field that
isn't used at all.

Regards,

	Hans

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-31  4:29       ` Hans Verkuil
@ 2019-07-31  4:33         ` Hans Verkuil
  2019-07-31  4:55           ` Hans Verkuil
  0 siblings, 1 reply; 76+ messages in thread
From: Hans Verkuil @ 2019-07-31  4:33 UTC (permalink / raw)
  To: Helen Koike, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, zhengsq, jeffy.chen,
	zyc, linux-kernel, tfiga, hans.verkuil, laurent.pinchart,
	sakari.ailus, mchehab, ezequiel, linux-arm-kernel, linux-media

On 7/31/19 6:29 AM, Hans Verkuil wrote:
> On 7/31/19 2:08 AM, Helen Koike wrote:
>>
>>
>> On 7/30/19 5:50 PM, Helen Koike wrote:
>>>
>>>
>>> On 7/30/19 5:15 PM, Hans Verkuil wrote:
>>>> On 7/30/19 8:42 PM, Helen Koike wrote:
>>>>> Hello,
>>>>>
>>>>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>>>>> rk3399 SoC.
>>>>>
>>>>> I didn't change much from the last version, just applying the
>>>>> suggestions made in the previous one.
>>>>>
>>>>> This patchset is also available at:
>>>>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>>>>
>>>>> Libcamera patched to work with this version:
>>>>> https://gitlab.collabora.com/koike/libcamera
>>>>> (also sent to the mailing list)
>>>>>
>>>>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>>>>> Scarlet Chromebook.
>>>>>
>>>>> Known issues (same as in v7):
>>>>> -------------
>>>>> - Reloading the module doesn't work (there is some missing cleanup when
>>>>> unloading)
>>>>> - When capturing in bayer format, changing the size doesn't seem to
>>>>> affect the image.
>>>>> - crop needs more tests
>>>>> - v4l2-compliance error:
>>>>>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
>>>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>>>>
>>>> Can you mail me the full v4l2-compliance output?
>>>
>>> Sure, please check here: http://ix.io/1Q5u
>>> I updated v4l-utils with the latest version and I re-ran bootstrap/configure/make,
>>> but for some reason the hash from the link above is not the latest commit, probably some
>>> old configuration somewhere. I'll resend this log as soon as I get v4l2-compliance
>>> properly updated.
>>
>> Please see the output of v4l2-compliance here with an updated v4l-utils: http://ix.io/1Q6A
> 
> So this FAIL is for /dev/v4l-subdev0 (rkisp1-isp-subdev).
> 
> What is weird that this subdev does not appear to have controls at all.
> 
> What is the output of 'v4l2-ctl -d /dev/v4l-subdev0 -l'? And if it lists
> controls, then why?
> 
> If you run 'v4l2-compliance -u /dev/v4l-subdev0', do you get a fail as
> well?

I see the same issue with v4l-subdev1, but I see no "Media Driver Info"
in the v4l2-compliance output for that subdev. That's strange. It would
be good to know why that's happening.

Regards,

	Hans

> 
> BTW, note that struct rkisp1_isp_subdev has a ctrl_handler field that
> isn't used at all.
> 
> Regards,
> 
> 	Hans
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-31  4:33         ` Hans Verkuil
@ 2019-07-31  4:55           ` Hans Verkuil
  2019-07-31 14:42             ` Helen Koike
  0 siblings, 1 reply; 76+ messages in thread
From: Hans Verkuil @ 2019-07-31  4:55 UTC (permalink / raw)
  To: Helen Koike, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, zhengsq, jeffy.chen,
	zyc, linux-kernel, tfiga, hans.verkuil, laurent.pinchart,
	sakari.ailus, mchehab, ezequiel, linux-arm-kernel, linux-media

On 7/31/19 6:33 AM, Hans Verkuil wrote:
> On 7/31/19 6:29 AM, Hans Verkuil wrote:
>> On 7/31/19 2:08 AM, Helen Koike wrote:
>>>
>>>
>>> On 7/30/19 5:50 PM, Helen Koike wrote:
>>>>
>>>>
>>>> On 7/30/19 5:15 PM, Hans Verkuil wrote:
>>>>> On 7/30/19 8:42 PM, Helen Koike wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>>>>>> rk3399 SoC.
>>>>>>
>>>>>> I didn't change much from the last version, just applying the
>>>>>> suggestions made in the previous one.
>>>>>>
>>>>>> This patchset is also available at:
>>>>>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>>>>>
>>>>>> Libcamera patched to work with this version:
>>>>>> https://gitlab.collabora.com/koike/libcamera
>>>>>> (also sent to the mailing list)
>>>>>>
>>>>>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>>>>>> Scarlet Chromebook.
>>>>>>
>>>>>> Known issues (same as in v7):
>>>>>> -------------
>>>>>> - Reloading the module doesn't work (there is some missing cleanup when
>>>>>> unloading)
>>>>>> - When capturing in bayer format, changing the size doesn't seem to
>>>>>> affect the image.
>>>>>> - crop needs more tests
>>>>>> - v4l2-compliance error:
>>>>>>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
>>>>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>>>>>
>>>>> Can you mail me the full v4l2-compliance output?
>>>>
>>>> Sure, please check here: http://ix.io/1Q5u
>>>> I updated v4l-utils with the latest version and I re-ran bootstrap/configure/make,
>>>> but for some reason the hash from the link above is not the latest commit, probably some
>>>> old configuration somewhere. I'll resend this log as soon as I get v4l2-compliance
>>>> properly updated.
>>>
>>> Please see the output of v4l2-compliance here with an updated v4l-utils: http://ix.io/1Q6A
>>
>> So this FAIL is for /dev/v4l-subdev0 (rkisp1-isp-subdev).
>>
>> What is weird that this subdev does not appear to have controls at all.
>>
>> What is the output of 'v4l2-ctl -d /dev/v4l-subdev0 -l'? And if it lists
>> controls, then why?
>>
>> If you run 'v4l2-compliance -u /dev/v4l-subdev0', do you get a fail as
>> well?
> 
> I see the same issue with v4l-subdev1, but I see no "Media Driver Info"
> in the v4l2-compliance output for that subdev. That's strange. It would
> be good to know why that's happening.

It looks to be some parenting issue: v4l2-compliance expects to find
a mediaX directory in /sys/dev/char/81\:Y/device/ where 81:Y is the major/minor
of /dev/v4l-subdev1.

Because is this mi_get_media_fd() cannot find the media device for the subdev
in v4l2-compliance.

Regards,

	Hans

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-31  4:55           ` Hans Verkuil
@ 2019-07-31 14:42             ` Helen Koike
  0 siblings, 0 replies; 76+ messages in thread
From: Helen Koike @ 2019-07-31 14:42 UTC (permalink / raw)
  To: Hans Verkuil, linux-rockchip
  Cc: devicetree, eddie.cai.linux, kernel, heiko, zhengsq, jeffy.chen,
	zyc, linux-kernel, tfiga, hans.verkuil, laurent.pinchart,
	sakari.ailus, mchehab, ezequiel, linux-arm-kernel, linux-media



On 7/31/19 1:55 AM, Hans Verkuil wrote:
> On 7/31/19 6:33 AM, Hans Verkuil wrote:
>> On 7/31/19 6:29 AM, Hans Verkuil wrote:
>>> On 7/31/19 2:08 AM, Helen Koike wrote:
>>>>
>>>>
>>>> On 7/30/19 5:50 PM, Helen Koike wrote:
>>>>>
>>>>>
>>>>> On 7/30/19 5:15 PM, Hans Verkuil wrote:
>>>>>> On 7/30/19 8:42 PM, Helen Koike wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>>>>>>> rk3399 SoC.
>>>>>>>
>>>>>>> I didn't change much from the last version, just applying the
>>>>>>> suggestions made in the previous one.
>>>>>>>
>>>>>>> This patchset is also available at:
>>>>>>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>>>>>>
>>>>>>> Libcamera patched to work with this version:
>>>>>>> https://gitlab.collabora.com/koike/libcamera
>>>>>>> (also sent to the mailing list)
>>>>>>>
>>>>>>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>>>>>>> Scarlet Chromebook.
>>>>>>>
>>>>>>> Known issues (same as in v7):
>>>>>>> -------------
>>>>>>> - Reloading the module doesn't work (there is some missing cleanup when
>>>>>>> unloading)
>>>>>>> - When capturing in bayer format, changing the size doesn't seem to
>>>>>>> affect the image.
>>>>>>> - crop needs more tests
>>>>>>> - v4l2-compliance error:
>>>>>>>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
>>>>>>> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
>>>>>>
>>>>>> Can you mail me the full v4l2-compliance output?
>>>>>
>>>>> Sure, please check here: http://ix.io/1Q5u
>>>>> I updated v4l-utils with the latest version and I re-ran bootstrap/configure/make,
>>>>> but for some reason the hash from the link above is not the latest commit, probably some
>>>>> old configuration somewhere. I'll resend this log as soon as I get v4l2-compliance
>>>>> properly updated.
>>>>
>>>> Please see the output of v4l2-compliance here with an updated v4l-utils: http://ix.io/1Q6A
>>>
>>> So this FAIL is for /dev/v4l-subdev0 (rkisp1-isp-subdev).
>>>
>>> What is weird that this subdev does not appear to have controls at all.
>>>
>>> What is the output of 'v4l2-ctl -d /dev/v4l-subdev0 -l'? And if it lists
>>> controls, then why?

root@rockpi:~# v4l2-ctl -d /dev/v4l-subdev0 -l

Image Processing Controls

                     pixel_rate 0x009f0902 (int64)  : min=1 max=2147483647 step=1 default=1 value=1 flags=read-only

root@rockpi:~# v4l2-ctl -d /dev/v4l-subdev1 -l

Image Processing Controls

                     pixel_rate 0x009f0902 (int64)  : min=1 max=2147483647 step=1 default=1 value=1 flags=read-only

It seems that ISP heritage the control from the sensor driver. I believe it
happens because isp_dev->ctrl_handler was assigned to the v4l2_dev object:

dev.c:  v4l2_ctrl_handler_init(&isp_dev->ctrl_handler, 5);
dev.c:  v4l2_dev->ctrl_handler = &isp_dev->ctrl_handler;


>>>
>>> If you run 'v4l2-compliance -u /dev/v4l-subdev0', do you get a fail as
>>> well?

Yes, but if I remove the ctrl_handler field from the struct rkisp1_isp_subdev
I don't get the error on /dev/v4l-subdev0 (ISP) anymore, I just get the error
on /dev/v4l-subdev1 (the sensor).

Here is the output of v4l2-compliance -m /dev/media0 without the
ctrl_handler field: http://ix.io/1Q9N

>>
>> I see the same issue with v4l-subdev1, but I see no "Media Driver Info"
>> in the v4l2-compliance output for that subdev. That's strange. It would
>> be good to know why that's happening.
> 
> It looks to be some parenting issue: v4l2-compliance expects to find
> a mediaX directory in /sys/dev/char/81\:Y/device/ where 81:Y is the major/minor
> of /dev/v4l-subdev1.
> 
> Because is this mi_get_media_fd() cannot find the media device for the subdev
> in v4l2-compliance.

So from my understanding this seems to be an issue in the sensor driver that
I'm using and not in the ISP (to be verified).
I'll submit the next version without the ctrl_handler field in struct rkisp1_isp_subdev,
is that ok?

Thanks
Helen


> 
> Regards,
> 
> 	Hans
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-07-30 18:42 ` [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver Helen Koike
@ 2019-08-06 18:51   ` Helen Koike
  2019-08-07 10:39     ` Hans Verkuil
  2019-08-08  9:14   ` Sakari Ailus
  2019-08-16  0:13   ` Laurent Pinchart
  2 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-08-06 18:51 UTC (permalink / raw)
  To: hans.verkuil
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, laurent.pinchart, sakari.ailus, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Hans,

On 7/30/19 3:42 PM, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add the subdev driver for rockchip isp1.
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> [fixed unknown entity type / switched to PIXEL_RATE]
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - fixed warning because of unknown entity type
> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> and default values
> - fix typo riksp1/rkisp1
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now. As a consequence, remove the
> hack in mipidphy_g_mbus_config() where information from the sensor was
> being propagated through the topology.
> - From the old dphy:
>         * cache get_remote_sensor() in s_stream
>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> - Replace stream state with a boolean
> - code styling and checkpatch fixes
> - fix stop_stream (return after calling stop, do not reenable the stream)
> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> - s/intput/input
> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> reused by the capture
> 
>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>  2 files changed, 1397 insertions(+)
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> 
> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> new file mode 100644
> index 000000000000..6d0c0ffb5e03
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> @@ -0,0 +1,1286 @@
> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#include <linux/iopoll.h>
> +#include <linux/phy/phy.h>
> +#include <linux/phy/phy-mipi-dphy.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/videodev2.h>
> +#include <linux/vmalloc.h>
> +#include <media/v4l2-event.h>
> +
> +#include "common.h"
> +#include "regs.h"
> +
> +#define CIF_ISP_INPUT_W_MAX		4032
> +#define CIF_ISP_INPUT_H_MAX		3024
> +#define CIF_ISP_INPUT_W_MIN		32
> +#define CIF_ISP_INPUT_H_MIN		32
> +#define CIF_ISP_OUTPUT_W_MAX		CIF_ISP_INPUT_W_MAX
> +#define CIF_ISP_OUTPUT_H_MAX		CIF_ISP_INPUT_H_MAX
> +#define CIF_ISP_OUTPUT_W_MIN		CIF_ISP_INPUT_W_MIN
> +#define CIF_ISP_OUTPUT_H_MIN		CIF_ISP_INPUT_H_MIN
> +
> +/*
> + * NOTE: MIPI controller and input MUX are also configured in this file,
> + * because ISP Subdev is not only describe ISP submodule(input size,format,
> + * output size, format), but also a virtual route device.
> + */
> +
> +/*
> + * There are many variables named with format/frame in below code,
> + * please see here for their meaning.
> + *
> + * Cropping regions of ISP
> + *
> + * +---------------------------------------------------------+
> + * | Sensor image                                            |
> + * | +---------------------------------------------------+   |
> + * | | ISP_ACQ (for black level)                         |   |
> + * | | in_frm                                            |   |
> + * | | +--------------------------------------------+    |   |
> + * | | |    ISP_OUT                                 |    |   |
> + * | | |    in_crop                                 |    |   |
> + * | | |    +---------------------------------+     |    |   |
> + * | | |    |   ISP_IS                        |     |    |   |
> + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
> + * | | |    +---------------------------------+     |    |   |
> + * | | +--------------------------------------------+    |   |
> + * | +---------------------------------------------------+   |
> + * +---------------------------------------------------------+
> + */
> +
> +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
> +}
> +
> +/* Get sensor by enabled media link */
> +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
> +{
> +	struct media_pad *local, *remote;
> +	struct media_entity *sensor_me;
> +
> +	local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
> +	remote = media_entity_remote_pad(local);
> +	if (!remote) {
> +		v4l2_warn(sd, "No link between isp and sensor\n");
> +		return NULL;
> +	}
> +
> +	sensor_me = media_entity_remote_pad(local)->entity;
> +	return media_entity_to_v4l2_subdev(sensor_me);
> +}
> +
> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
> +					       struct v4l2_subdev *sd)
> +{
> +	struct rkisp1_sensor *sensor;
> +
> +	list_for_each_entry(sensor, &dev->sensors, list)
> +		if (sensor->sd == sd)
> +			return sensor;
> +
> +	return NULL;
> +}
> +
> +/****************  register operations ****************/
> +
> +/*
> + * Image Stabilization.
> + * This should only be called when configuring CIF
> + * or at the frame end interrupt
> + */
> +static void rkisp1_config_ism(struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
> +	u32 val;
> +
> +	writel(0, base + CIF_ISP_IS_RECENTER);
> +	writel(0, base + CIF_ISP_IS_MAX_DX);
> +	writel(0, base + CIF_ISP_IS_MAX_DY);
> +	writel(0, base + CIF_ISP_IS_DISPLACE);
> +	writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
> +	writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
> +	writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
> +	writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
> +
> +	/* IS(Image Stabilization) is always on, working as output crop */
> +	writel(1, base + CIF_ISP_IS_CTRL);
> +	val = readl(base + CIF_ISP_CTRL);
> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD;
> +	writel(val, base + CIF_ISP_CTRL);
> +}
> +
> +/*
> + * configure isp blocks with input format, size......
> + */
> +static int rkisp1_config_isp(struct rkisp1_device *dev)
> +{
> +	u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
> +	struct v4l2_rect *out_crop, *in_crop;
> +	void __iomem *base = dev->base_addr;
> +	struct v4l2_mbus_framefmt *in_frm;
> +	struct ispsd_out_fmt *out_fmt;
> +	struct rkisp1_sensor *sensor;
> +	struct ispsd_in_fmt *in_fmt;
> +
> +	sensor = dev->active_sensor;
> +	in_frm = &dev->isp_sdev.in_frm;
> +	in_fmt = &dev->isp_sdev.in_fmt;
> +	out_fmt = &dev->isp_sdev.out_fmt;
> +	out_crop = &dev->isp_sdev.out_crop;
> +	in_crop = &dev->isp_sdev.in_crop;
> +
> +	if (in_fmt->fmt_type == FMT_BAYER) {
> +		acq_mult = 1;
> +		if (out_fmt->fmt_type == FMT_BAYER) {
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl =
> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
> +			else
> +				isp_ctrl =
> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
> +		} else {
> +			writel(CIF_ISP_DEMOSAIC_TH(0xc),
> +			       base + CIF_ISP_DEMOSAIC);
> +
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
> +			else
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
> +		}
> +	} else if (in_fmt->fmt_type == FMT_YUV) {
> +		acq_mult = 2;
> +		if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +			isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> +		} else {
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
> +			else
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> +
> +		}
> +
> +		irq_mask |= CIF_ISP_DATA_LOSS;
> +	}
> +
> +	/* Set up input acquisition properties */
> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
> +			signal = CIF_ISP_ACQ_PROP_POS_EDGE;
> +	}
> +
> +	if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
> +			signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
> +
> +		if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
> +			signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
> +	}
> +
> +	writel(isp_ctrl, base + CIF_ISP_CTRL);
> +	writel(signal | in_fmt->yuv_seq |
> +	       CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
> +	       CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
> +	writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
> +
> +	/* Acquisition Size */
> +	writel(0, base + CIF_ISP_ACQ_H_OFFS);
> +	writel(0, base + CIF_ISP_ACQ_V_OFFS);
> +	writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
> +	writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
> +
> +	/* ISP Out Area */
> +	writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
> +	writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
> +	writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
> +	writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
> +
> +	/* interrupt mask */
> +	irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
> +		    CIF_ISP_FRAME_IN;
> +	writel(irq_mask, base + CIF_ISP_IMSC);
> +
> +	if (out_fmt->fmt_type == FMT_BAYER)
> +		rkisp1_params_disable_isp(&dev->params_vdev);
> +	else
> +		rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
> +					    dev->isp_sdev.quantization);
> +
> +	return 0;
> +}
> +
> +static int rkisp1_config_dvp(struct rkisp1_device *dev)
> +{
> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> +	void __iomem *base = dev->base_addr;
> +	u32 val, input_sel;
> +
> +	switch (in_fmt->bus_width) {
> +	case 8:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
> +		break;
> +	case 10:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
> +		break;
> +	case 12:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
> +		break;
> +	default:
> +		v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
> +		return -EINVAL;
> +	}
> +
> +	val = readl(base + CIF_ISP_ACQ_PROP);
> +	writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
> +
> +	return 0;
> +}
> +
> +static int rkisp1_config_mipi(struct rkisp1_device *dev)
> +{
> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	void __iomem *base = dev->base_addr;
> +	unsigned int lanes;
> +	u32 mipi_ctrl;
> +
> +	/*
> +	 * sensor->mbus is set in isp or d-phy notifier_bound function
> +	 */
> +	switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
> +	case V4L2_MBUS_CSI2_4_LANE:
> +		lanes = 4;
> +		break;
> +	case V4L2_MBUS_CSI2_3_LANE:
> +		lanes = 3;
> +		break;
> +	case V4L2_MBUS_CSI2_2_LANE:
> +		lanes = 2;
> +		break;
> +	case V4L2_MBUS_CSI2_1_LANE:
> +		lanes = 1;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
> +		    CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
> +		    CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
> +		    CIF_MIPI_CTRL_CLOCKLANE_ENA;
> +
> +	writel(mipi_ctrl, base + CIF_MIPI_CTRL);
> +
> +	/* Configure Data Type and Virtual Channel */
> +	writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
> +	       base + CIF_MIPI_IMG_DATA_SEL);
> +
> +	/* Clear MIPI interrupts */
> +	writel(~0, base + CIF_MIPI_ICR);
> +	/*
> +	 * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
> +	 * isp bus may be dead when switch isp.
> +	 */
> +	writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
> +	       CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
> +	       base + CIF_MIPI_IMSC);
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
> +		 "  MIPI_IMG_DATA_SEL 0x%08x\n"
> +		 "  MIPI_STATUS 0x%08x\n"
> +		 "  MIPI_IMSC 0x%08x\n",
> +		 readl(base + CIF_MIPI_CTRL),
> +		 readl(base + CIF_MIPI_IMG_DATA_SEL),
> +		 readl(base + CIF_MIPI_STATUS),
> +		 readl(base + CIF_MIPI_IMSC));
> +
> +	return 0;
> +}
> +
> +/* Configure MUX */
> +static int rkisp1_config_path(struct rkisp1_device *dev)
> +{
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
> +	int ret = 0;
> +
> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		ret = rkisp1_config_dvp(dev);
> +		dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
> +	} else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +		ret = rkisp1_config_mipi(dev);
> +		dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
> +	}
> +
> +	writel(dpcl, dev->base_addr + CIF_VI_DPCL);
> +
> +	return ret;
> +}
> +
> +/* Hareware configure Entry */
> +static int rkisp1_config_cif(struct rkisp1_device *dev)
> +{
> +	u32 cif_id;
> +	int ret;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	cif_id = readl(dev->base_addr + CIF_VI_ID);
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
> +
> +	ret = rkisp1_config_isp(dev);
> +	if (ret < 0)
> +		return ret;
> +	ret = rkisp1_config_path(dev);
> +	if (ret < 0)
> +		return ret;
> +	rkisp1_config_ism(dev);
> +
> +	return 0;
> +}
> +
> +/* Mess register operations to stop isp */
> +static int rkisp1_isp_stop(struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	/*
> +	 * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
> +	 * Stop ISP(isp) ->wait for ISP isp off
> +	 */
> +	/* stop and clear MI, MIPI, and ISP interrupts */
> +	writel(0, base + CIF_MIPI_IMSC);
> +	writel(~0, base + CIF_MIPI_ICR);
> +
> +	writel(0, base + CIF_ISP_IMSC);
> +	writel(~0, base + CIF_ISP_ICR);
> +
> +	writel(0, base + CIF_MI_IMSC);
> +	writel(~0, base + CIF_MI_ICR);
> +	val = readl(base + CIF_MIPI_CTRL);
> +	writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
> +	/* stop ISP */
> +	val = readl(base + CIF_ISP_CTRL);
> +	val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
> +	writel(val, base + CIF_ISP_CTRL);
> +
> +	val = readl(base + CIF_ISP_CTRL);
> +	writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
> +
> +	readx_poll_timeout(readl, base + CIF_ISP_RIS,
> +			   val, val & CIF_ISP_OFF, 20, 100);
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		"streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming,
> +		 readl(base + CIF_MI_CTRL),
> +		 readl(base + CIF_ISP_CTRL),
> +		 readl(base + CIF_MIPI_CTRL));
> +
> +	writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
> +	writel(0x0, base + CIF_IRCL);
> +
> +	return 0;
> +}
> +
> +/* Mess register operations to start isp */
> +static int rkisp1_isp_start(struct rkisp1_device *dev)
> +{
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	/* Activate MIPI */
> +	if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +		val = readl(base + CIF_MIPI_CTRL);
> +		writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
> +	}
> +	/* Activate ISP */
> +	val = readl(base + CIF_ISP_CTRL);
> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
> +	       CIF_ISP_CTRL_ISP_INFORM_ENABLE;
> +	writel(val, base + CIF_ISP_CTRL);
> +
> +	/* XXX: Is the 1000us too long?
> +	 * CIF spec says to wait for sufficient time after enabling
> +	 * the MIPI interface and before starting the sensor output.
> +	 */
> +	usleep_range(1000, 1200);
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
> +		 "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming,
> +		 readl(base + CIF_MI_CTRL),
> +		 readl(base + CIF_ISP_CTRL),
> +		 readl(base + CIF_MIPI_CTRL));
> +
> +	return 0;
> +}
> +
> +static void rkisp1_config_clk(struct rkisp1_device *dev)
> +{
> +	u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
> +		  CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
> +		  CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
> +
> +	writel(val, dev->base_addr + CIF_ICCL);
> +}
> +
> +/***************************** isp sub-devs *******************************/
> +
> +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
> +	{
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCBYCR,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_YVYU8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCRYCB,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_UYVY8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CBYCRY,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_VYUY8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CRYCBY,
> +		.bus_width	= 16,
> +	},
> +};
> +
> +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
> +	{
> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_2X8,
> +		.fmt_type	= FMT_YUV,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	},
> +};
> +
> +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
> +{
> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
> +	const struct ispsd_in_fmt *fmt;
> +
> +	for (i = 0; i < array_size; i++) {
> +		fmt = &rkisp1_isp_input_formats[i];
> +		if (fmt->mbus_code == mbus_code)
> +			return fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
> +{
> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
> +	const struct ispsd_out_fmt *fmt;
> +
> +	for (i = 0; i < array_size; i++) {
> +		fmt = &rkisp1_isp_output_formats[i];
> +		if (fmt->mbus_code == mbus_code)
> +			return fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
> +					struct v4l2_subdev_pad_config *cfg,
> +					struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	unsigned int i = code->index;
> +
> +	if ((code->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> +		if (i > 0)
> +			return -EINVAL;
> +		code->code = MEDIA_BUS_FMT_FIXED;
> +		return 0;
> +	}
> +
> +	if (code->pad == RKISP1_ISP_PAD_SINK) {
> +		if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
> +			return -EINVAL;
> +		code->code = rkisp1_isp_input_formats[i].mbus_code;
> +	} else {
> +		if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
> +			return -EINVAL;
> +		code->code = rkisp1_isp_output_formats[i].mbus_code;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
> +				     struct v4l2_subdev_pad_config *cfg)
> +{
> +	struct v4l2_rect *mf_in_crop, *mf_out_crop;
> +	struct v4l2_mbus_framefmt *mf_in, *mf_out;
> +
> +	mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
> +	mf_in->width = RKISP1_DEFAULT_WIDTH;
> +	mf_in->height = RKISP1_DEFAULT_HEIGHT;
> +	mf_in->field = V4L2_FIELD_NONE;
> +	mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
> +
> +	mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
> +	mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
> +	mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
> +	mf_in_crop->left = 0;
> +	mf_in_crop->top = 0;
> +
> +	mf_out = v4l2_subdev_get_try_format(sd, cfg,
> +					    RKISP1_ISP_PAD_SOURCE_PATH);
> +	*mf_out = *mf_in;
> +	mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
> +	mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +
> +	mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
> +					       RKISP1_ISP_PAD_SOURCE_PATH);
> +	*mf_out_crop = *mf_in_crop;
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_format *fmt)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> +
> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
> +		/*
> +		 * NOTE: setting a format here doesn't make much sense
> +		 * but v4l2-compliance complains
> +		 */
> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;

As I had mentioned to you, this is called for the isp pads connected to the
DMA engines for statistics and parameters (meta data).

If I remove those, I get the following errors:

Sub-Device ioctls (Sink Pad 1):
        test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
                fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
                fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
        test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
        test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
        test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
                fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
                fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
        test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
        test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
        test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)

Here is the full log: http://ix.io/1QNt

Is this a bug in v4l2-compliance?

Thanks
Helen

> +		fmt->format.field = V4L2_FIELD_NONE;
> +		return 0;
> +	}
> +
> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> +		fmt->format = *mf;
> +		return 0;
> +	}
> +
> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> +		*mf = isp_sd->in_frm;
> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		/* format of source pad */
> +		*mf = isp_sd->in_frm;
> +		mf->code = isp_sd->out_fmt.mbus_code;
> +		/* window size of source pad */
> +		mf->width = isp_sd->out_crop.width;
> +		mf->height = isp_sd->out_crop.height;
> +		mf->quantization = isp_sd->quantization;
> +	}
> +	mf->field = V4L2_FIELD_NONE;
> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
> +				  unsigned int pad,
> +				  struct v4l2_mbus_framefmt *fmt)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> +	const struct ispsd_out_fmt *out_fmt;
> +	const struct ispsd_in_fmt *in_fmt;
> +
> +	switch (pad) {
> +	case RKISP1_ISP_PAD_SINK:
> +		in_fmt = find_in_fmt(fmt->code);
> +		if (in_fmt)
> +			fmt->code = in_fmt->mbus_code;
> +		else
> +			fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
> +		fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
> +				      CIF_ISP_INPUT_W_MAX);
> +		fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
> +				      CIF_ISP_INPUT_H_MAX);
> +		break;
> +	case RKISP1_ISP_PAD_SOURCE_PATH:
> +		out_fmt = find_out_fmt(fmt->code);
> +		if (out_fmt)
> +			fmt->code = out_fmt->mbus_code;
> +		else
> +			fmt->code = rkisp1_isp_output_formats[0].mbus_code;
> +		/* window size is set in s_selection */
> +		fmt->width  = isp_sd->out_crop.width;
> +		fmt->height = isp_sd->out_crop.height;
> +		/* full range by default */
> +		if (!fmt->quantization)
> +			fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +		break;
> +	}
> +
> +	fmt->field = V4L2_FIELD_NONE;
> +}
> +
> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_format *fmt)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> +
> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
> +		return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
> +
> +	rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
> +
> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		struct v4l2_mbus_framefmt *try_mf;
> +
> +		try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> +		*try_mf = *mf;
> +		return 0;
> +	}
> +
> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> +		const struct ispsd_in_fmt *in_fmt;
> +
> +		in_fmt = find_in_fmt(mf->code);
> +		isp_sd->in_fmt = *in_fmt;
> +		isp_sd->in_frm = *mf;
> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		const struct ispsd_out_fmt *out_fmt;
> +
> +		/* Ignore width/height */
> +		out_fmt = find_out_fmt(mf->code);
> +		isp_sd->out_fmt = *out_fmt;
> +		/*
> +		 * It is quantization for output,
> +		 * isp use bt601 limit-range in internal
> +		 */
> +		isp_sd->quantization = mf->quantization;
> +	}
> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
> +	struct v4l2_rect in_crop = isp_sd->in_crop;
> +	struct v4l2_rect *input = &sel->r;
> +
> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		in_frm = *v4l2_subdev_get_try_format(sd, cfg,
> +						     RKISP1_ISP_PAD_SINK);
> +		in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
> +						    RKISP1_ISP_PAD_SINK);
> +	}
> +
> +	input->left = ALIGN(input->left, 2);
> +	input->width = ALIGN(input->width, 2);
> +
> +	if (sel->pad == RKISP1_ISP_PAD_SINK) {
> +		input->left = clamp_t(u32, input->left, 0, in_frm.width);
> +		input->top = clamp_t(u32, input->top, 0, in_frm.height);
> +		input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
> +				       in_frm.width - input->left);
> +		input->height = clamp_t(u32, input->height,
> +					CIF_ISP_INPUT_H_MIN,
> +					in_frm.height - input->top);
> +	} else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		input->left = clamp_t(u32, input->left, 0, in_crop.width);
> +		input->top = clamp_t(u32, input->top, 0, in_crop.height);
> +		input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
> +				       in_crop.width - input->left);
> +		input->height = clamp_t(u32, input->height,
> +					CIF_ISP_OUTPUT_H_MIN,
> +					in_crop.height - input->top);
> +	}
> +}
> +
> +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_pad_config *cfg,
> +				       struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt *frm;
> +	struct v4l2_rect *rect;
> +
> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> +	    sel->pad != RKISP1_ISP_PAD_SINK)
> +		return -EINVAL;
> +
> +	switch (sel->target) {
> +	case V4L2_SEL_TGT_CROP_BOUNDS:
> +		if (sel->pad == RKISP1_ISP_PAD_SINK) {
> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +				frm = v4l2_subdev_get_try_format(sd, cfg,
> +								 sel->pad);
> +			else
> +				frm = &isp_sd->in_frm;
> +
> +			sel->r.height = frm->height;
> +			sel->r.width = frm->width;
> +			sel->r.left = 0;
> +			sel->r.top = 0;
> +		} else {
> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +				rect = v4l2_subdev_get_try_crop(sd, cfg,
> +							RKISP1_ISP_PAD_SINK);
> +			else
> +				rect = &isp_sd->in_crop;
> +			sel->r = *rect;
> +		}
> +		break;
> +	case V4L2_SEL_TGT_CROP:
> +		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +			rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> +		else if (sel->pad == RKISP1_ISP_PAD_SINK)
> +			rect = &isp_sd->in_crop;
> +		else
> +			rect = &isp_sd->out_crop;
> +		sel->r = *rect;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_pad_config *cfg,
> +				       struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct rkisp1_device *dev = sd_to_isp_dev(sd);
> +
> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> +	    sel->pad != RKISP1_ISP_PAD_SINK)
> +		return -EINVAL;
> +	if (sel->target != V4L2_SEL_TGT_CROP)
> +		return -EINVAL;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
> +		 sel->r.left, sel->r.top, sel->r.width, sel->r.height);
> +	rkisp1_isp_sd_try_crop(sd, cfg, sel);
> +
> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		struct v4l2_rect *try_sel =
> +			v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> +
> +		*try_sel = sel->r;
> +		return 0;
> +	}
> +
> +	if (sel->pad == RKISP1_ISP_PAD_SINK)
> +		isp_sd->in_crop = sel->r;
> +	else
> +		isp_sd->out_crop = sel->r;
> +
> +	return 0;
> +}
> +
> +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
> +				    struct rkisp1_sensor *sensor)
> +{
> +	union phy_configure_opts opts = { 0 };
> +	struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
> +	struct v4l2_ctrl *pixel_rate;
> +	s64 pixel_clock;
> +
> +	pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
> +				    V4L2_CID_PIXEL_RATE);
> +	if (!pixel_rate) {
> +		v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
> +		return -EPIPE;
> +	}
> +
> +	pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
> +	if (!pixel_clock) {
> +		v4l2_err(sensor->sd, "Invalid pixel rate value\n");
> +		return -EINVAL;
> +	}
> +
> +	phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
> +					 sensor->lanes, cfg);
> +	phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
> +	phy_configure(sensor->dphy, &opts);
> +	phy_power_on(sensor->dphy);
> +
> +	return 0;
> +}
> +
> +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
> +{
> +	phy_power_off(sensor->dphy);
> +}
> +
> +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct v4l2_subdev *sensor_sd;
> +	int ret = 0;
> +
> +	if (!on) {
> +		ret = rkisp1_isp_stop(isp_dev);
> +		if (ret < 0)
> +			return ret;
> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> +		return 0;
> +	}
> +
> +	sensor_sd = get_remote_sensor(sd);
> +	if (!sensor_sd)
> +		return -ENODEV;
> +
> +	isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
> +	if (!isp_dev->active_sensor)
> +		return -ENODEV;
> +
> +	atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
> +	ret = rkisp1_config_cif(isp_dev);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* TODO: support other interfaces */
> +	if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
> +		return -EINVAL;
> +
> +	ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
> +				       isp_dev->active_sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = rkisp1_isp_start(isp_dev);
> +	if (ret)
> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> +
> +	return ret;
> +}
> +
> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	int ret;
> +
> +	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> +
> +	if (on) {
> +		ret = pm_runtime_get_sync(isp_dev->dev);
> +		if (ret < 0)
> +			return ret;
> +
> +		rkisp1_config_clk(isp_dev);
> +	} else {
> +		ret = pm_runtime_put(isp_dev->dev);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_subdev_link_validate(struct media_link *link)
> +{
> +	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
> +		return 0;
> +
> +	return v4l2_subdev_link_validate(link);
> +}
> +
> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
> +					struct media_link *link,
> +					struct v4l2_subdev_format *source_fmt,
> +					struct v4l2_subdev_format *sink_fmt)
> +{
> +	if (source_fmt->format.code != sink_fmt->format.code)
> +		return -EINVAL;
> +
> +	/* Crop is available */
> +	if (source_fmt->format.width < sink_fmt->format.width ||
> +	    source_fmt->format.height < sink_fmt->format.height)
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> +{
> +	struct v4l2_event event = {
> +		.type = V4L2_EVENT_FRAME_SYNC,
> +		.u.frame_sync.frame_sequence =
> +			atomic_inc_return(&isp->frm_sync_seq) - 1,
> +	};
> +	v4l2_event_queue(isp->sd.devnode, &event);
> +}
> +
> +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> +				  struct v4l2_event_subscription *sub)
> +{
> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> +		return -EINVAL;
> +
> +	/* Line number. For now only zero accepted. */
> +	if (sub->id != 0)
> +		return -EINVAL;
> +
> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> +}
> +
> +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
> +	.enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
> +	.get_selection = rkisp1_isp_sd_get_selection,
> +	.set_selection = rkisp1_isp_sd_set_selection,
> +	.init_cfg = rkisp1_isp_sd_init_config,
> +	.get_fmt = rkisp1_isp_sd_get_fmt,
> +	.set_fmt = rkisp1_isp_sd_set_fmt,
> +	.link_validate = rkisp1_subdev_fmt_link_validate,
> +};
> +
> +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
> +	.link_validate = rkisp1_subdev_link_validate,
> +};
> +
> +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
> +	.s_stream = rkisp1_isp_sd_s_stream,
> +};
> +
> +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
> +	.subscribe_event = rkisp1_isp_sd_subs_evt,
> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> +	.s_power = rkisp1_isp_sd_s_power,
> +};
> +
> +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {
> +	.core = &rkisp1_isp_core_ops,
> +	.video = &rkisp1_isp_sd_video_ops,
> +	.pad = &rkisp1_isp_sd_pad_ops,
> +};
> +
> +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
> +{
> +	struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
> +	struct v4l2_rect *in_crop = &isp_sd->in_crop;
> +	struct v4l2_rect *out_crop = &isp_sd->out_crop;
> +	struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
> +	struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
> +
> +	*in_fmt = rkisp1_isp_input_formats[0];
> +	in_frm->width = RKISP1_DEFAULT_WIDTH;
> +	in_frm->height = RKISP1_DEFAULT_HEIGHT;
> +	in_frm->code = in_fmt->mbus_code;
> +
> +	in_crop->width = in_frm->width;
> +	in_crop->height = in_frm->height;
> +	in_crop->left = 0;
> +	in_crop->top = 0;
> +
> +	/* propagate to source */
> +	*out_crop = *in_crop;
> +	*out_fmt = rkisp1_isp_output_formats[0];
> +	isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +}
> +
> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> +			       struct v4l2_device *v4l2_dev)
> +{
> +	struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
> +	struct v4l2_subdev *sd = &isp_sdev->sd;
> +	int ret;
> +
> +	v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	sd->entity.ops = &rkisp1_isp_sd_media_ops;
> +	snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
> +
> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> +	ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
> +				     isp_sdev->pads);
> +	if (ret < 0)
> +		return ret;
> +
> +	sd->owner = THIS_MODULE;
> +	v4l2_set_subdevdata(sd, isp_dev);
> +
> +	sd->grp_id = GRP_ID_ISP;
> +	ret = v4l2_device_register_subdev(v4l2_dev, sd);
> +	if (ret < 0) {
> +		v4l2_err(sd, "Failed to register isp subdev\n");
> +		goto err_cleanup_media_entity;
> +	}
> +
> +	rkisp1_isp_sd_init_default_fmt(isp_sdev);
> +
> +	return 0;
> +err_cleanup_media_entity:
> +	media_entity_cleanup(&sd->entity);
> +	return ret;
> +}
> +
> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
> +{
> +	struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
> +
> +	v4l2_device_unregister_subdev(sd);
> +	media_entity_cleanup(&sd->entity);
> +}
> +
> +/****************  Interrupter Handler ****************/
> +
> +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
> +{
> +	struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	writel(~0, base + CIF_MIPI_ICR);
> +
> +	/*
> +	 * Disable DPHY errctrl interrupt, because this dphy
> +	 * erctrl signal is asserted until the next changes
> +	 * of line state. This time is may be too long and cpu
> +	 * is hold in this interrupt.
> +	 */
> +	if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
> +		val = readl(base + CIF_MIPI_IMSC);
> +		writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
> +		dev->isp_sdev.dphy_errctrl_disabled = true;
> +	}
> +
> +	/*
> +	 * Enable DPHY errctrl interrupt again, if mipi have receive
> +	 * the whole frame without any error.
> +	 */
> +	if (mis == CIF_MIPI_FRAME_END) {
> +		/*
> +		 * Enable DPHY errctrl interrupt again, if mipi have receive
> +		 * the whole frame without any error.
> +		 */
> +		if (dev->isp_sdev.dphy_errctrl_disabled) {
> +			val = readl(base + CIF_MIPI_IMSC);
> +			val |= CIF_MIPI_ERR_CTRL(0x0f);
> +			writel(val, base + CIF_MIPI_IMSC);
> +			dev->isp_sdev.dphy_errctrl_disabled = false;
> +		}
> +	} else {
> +		v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
> +	}
> +}
> +
> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	unsigned int isp_mis_tmp = 0;
> +	unsigned int isp_err = 0;
> +
> +	/* start edge of v_sync */
> +	if (isp_mis & CIF_ISP_V_START) {
> +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> +
> +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_V_START)
> +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> +				 isp_mis_tmp);
> +	}
> +
> +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> +		/* Clear pic_size_error */
> +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> +		isp_err = readl(base + CIF_ISP_ERR);
> +		v4l2_err(&dev->v4l2_dev,
> +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> +		writel(isp_err, base + CIF_ISP_ERR_CLR);
> +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> +		/* Clear data_loss */
> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> +	}
> +
> +	/* sampled input frame is complete */
> +	if (isp_mis & CIF_ISP_FRAME_IN) {
> +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> +				 isp_mis_tmp);
> +	}
> +
> +	/* frame was completely put out */
> +	if (isp_mis & CIF_ISP_FRAME) {
> +		u32 isp_ris = 0;
> +		/* Clear Frame In (ISP) */
> +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_FRAME)
> +			v4l2_err(&dev->v4l2_dev,
> +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> +
> +		isp_ris = readl(base + CIF_ISP_RIS);
> +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> +	}
> +
> +	/*
> +	 * Then update changed configs. Some of them involve
> +	 * lot of register writes. Do those only one per frame.
> +	 * Do the updates in the order of the processing flow.
> +	 */
> +	rkisp1_params_isr(&dev->params_vdev, isp_mis);
> +}
> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
> new file mode 100644
> index 000000000000..b0366e354ec2
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
> @@ -0,0 +1,111 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#ifndef _RKISP1_H
> +#define _RKISP1_H
> +
> +#include <linux/platform_device.h>
> +#include <media/v4l2-fwnode.h>
> +
> +#include "common.h"
> +
> +struct rkisp1_stream;
> +
> +/*
> + * struct ispsd_in_fmt - ISP intput-pad format
> + *
> + * Translate mbus_code to hardware format values
> + *
> + * @bus_width: used for parallel
> + */
> +struct ispsd_in_fmt {
> +	u32 mbus_code;
> +	u8 fmt_type;
> +	u32 mipi_dt;
> +	u32 yuv_seq;
> +	enum rkisp1_fmt_raw_pat_type bayer_pat;
> +	u8 bus_width;
> +};
> +
> +struct ispsd_out_fmt {
> +	u32 mbus_code;
> +	u8 fmt_type;
> +};
> +
> +struct rkisp1_ie_config {
> +	unsigned int effect;
> +};
> +
> +enum rkisp1_isp_pad {
> +	RKISP1_ISP_PAD_SINK,
> +	RKISP1_ISP_PAD_SINK_PARAMS,
> +	RKISP1_ISP_PAD_SOURCE_PATH,
> +	RKISP1_ISP_PAD_SOURCE_STATS,
> +	RKISP1_ISP_PAD_MAX
> +};
> +
> +/*
> + * struct rkisp1_isp_subdev - ISP sub-device
> + *
> + * See Cropping regions of ISP in rkisp1.c for details
> + * @in_frm: input size, don't have to be equal to sensor size
> + * @in_fmt: input format
> + * @in_crop: crop for sink pad
> + * @out_fmt: output format
> + * @out_crop: output size
> + *
> + * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)
> + * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
> + * @quantization: output quantization
> + */
> +struct rkisp1_isp_subdev {
> +	struct v4l2_subdev sd;
> +	struct media_pad pads[RKISP1_ISP_PAD_MAX];
> +	struct v4l2_ctrl_handler ctrl_handler;
> +	struct v4l2_mbus_framefmt in_frm;
> +	struct ispsd_in_fmt in_fmt;
> +	struct v4l2_rect in_crop;
> +	struct ispsd_out_fmt out_fmt;
> +	struct v4l2_rect out_crop;
> +	bool dphy_errctrl_disabled;
> +	atomic_t frm_sync_seq;
> +	enum v4l2_quantization quantization;
> +};
> +
> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> +			       struct v4l2_device *v4l2_dev);
> +
> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
> +
> +void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
> +
> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
> +
> +static inline
> +struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->out_fmt;
> +}
> +
> +static inline
> +struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->in_fmt;
> +}
> +
> +static inline
> +struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->out_crop;
> +}
> +
> +static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct rkisp1_isp_subdev, sd);
> +}
> +
> +#endif /* _RKISP1_H */
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-06 18:51   ` Helen Koike
@ 2019-08-07 10:39     ` Hans Verkuil
  2019-08-15 19:35       ` Laurent Pinchart
  0 siblings, 1 reply; 76+ messages in thread
From: Hans Verkuil @ 2019-08-07 10:39 UTC (permalink / raw)
  To: Helen Koike, hans.verkuil
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, laurent.pinchart, sakari.ailus, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

On 8/6/19 8:51 PM, Helen Koike wrote:
> Hi Hans,
> 
> On 7/30/19 3:42 PM, Helen Koike wrote:
>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>
>> Add the subdev driver for rockchip isp1.
>>
>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
>> Signed-off-by: Jacob Chen <cc@rock-chips.com>
>> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>> [fixed unknown entity type / switched to PIXEL_RATE]
>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>> [update for upstream]
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>
>> ---
>>
>> Changes in v8: None
>> Changes in v7:
>> - fixed warning because of unknown entity type
>> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
>> and default values
>> - fix typo riksp1/rkisp1
>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>> isp subdevice in the media topology now. As a consequence, remove the
>> hack in mipidphy_g_mbus_config() where information from the sensor was
>> being propagated through the topology.
>> - From the old dphy:
>>         * cache get_remote_sensor() in s_stream
>>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
>> - Replace stream state with a boolean
>> - code styling and checkpatch fixes
>> - fix stop_stream (return after calling stop, do not reenable the stream)
>> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
>> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
>> - s/intput/input
>> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
>> reused by the capture
>>
>>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>>  2 files changed, 1397 insertions(+)
>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>>
>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
>> new file mode 100644
>> index 000000000000..6d0c0ffb5e03
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
>> @@ -0,0 +1,1286 @@

<snip>

>> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
>> +				 struct v4l2_subdev_pad_config *cfg,
>> +				 struct v4l2_subdev_format *fmt)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>> +
>> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
>> +		/*
>> +		 * NOTE: setting a format here doesn't make much sense
>> +		 * but v4l2-compliance complains
>> +		 */
>> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
>> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> 
> As I had mentioned to you, this is called for the isp pads connected to the
> DMA engines for statistics and parameters (meta data).
> 
> If I remove those, I get the following errors:
> 
> Sub-Device ioctls (Sink Pad 1):
>         test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
>                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
>                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
>         test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
>         test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
>         test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
>                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
>                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
>         test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
>         test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
>         test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
> 
> Here is the full log: http://ix.io/1QNt
> 
> Is this a bug in v4l2-compliance?

Yes and no :-)

Currently v4l2-compliance assumes that only video is transferred over a media bus.
But that's not the case here, and testing the code field doesn't help v4l2-compliance
since MEDIA_BUS_FMT_FIXED is also still used by some older subdev drivers for video.

I think we need a new bus format: MEDIA_BUS_FMT_FIXED_METADATA. Then v4l2-compliance
can tell it apart from the regular fixed video bus format.

If I do a 'git grep MEDIA_BUS_FMT_FIXED' then I see that it is also in use by vsp1
for histogram information, so that should also be converted to use the new FIXED_METADATA
format, although that might be too late (there might be userspace complications).

Regards,

	Hans

> 
> Thanks
> Helen
> 
>> +		fmt->format.field = V4L2_FIELD_NONE;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>> +		fmt->format = *mf;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>> +		*mf = isp_sd->in_frm;
>> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		/* format of source pad */
>> +		*mf = isp_sd->in_frm;
>> +		mf->code = isp_sd->out_fmt.mbus_code;
>> +		/* window size of source pad */
>> +		mf->width = isp_sd->out_crop.width;
>> +		mf->height = isp_sd->out_crop.height;
>> +		mf->quantization = isp_sd->quantization;
>> +	}
>> +	mf->field = V4L2_FIELD_NONE;
>> +
>> +	return 0;
>> +}

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-07-30 18:42 ` [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver Helen Koike
@ 2019-08-07 13:05   ` Sakari Ailus
  2019-08-07 13:37     ` Helen Koike
  0 siblings, 1 reply; 76+ messages in thread
From: Sakari Ailus @ 2019-08-07 13:05 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

Thanks for the patchset.

On Tue, Jul 30, 2019 at 03:42:46PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> [migrate to phy framework]
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8:
> - Remove boiler plate license text
> 
> Changes in v7:
> - Migrate dphy specific code from
> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
> to drivers/phy/rockchip/phy-rockchip-dphy.c
> - Drop support for rk3288
> - Drop support for dphy txrx
> - code styling and checkpatch fixes
> 
>  drivers/phy/rockchip/Kconfig             |   8 +
>  drivers/phy/rockchip/Makefile            |   1 +
>  drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
>  3 files changed, 417 insertions(+)
>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
> 
> diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
> index c454c90cd99e..afd072f135e6 100644
> --- a/drivers/phy/rockchip/Kconfig
> +++ b/drivers/phy/rockchip/Kconfig
> @@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
>  	help
>  	  Enable this to support the Rockchip Display Port PHY.
>  
> +config PHY_ROCKCHIP_DPHY
> +	tristate "Rockchip MIPI Synopsys DPHY driver"
> +	depends on ARCH_ROCKCHIP && OF

How about (...) || COMPILE_TEST ?

> +	select GENERIC_PHY_MIPI_DPHY
> +	select GENERIC_PHY
> +	help
> +	  Enable this to support the Rockchip MIPI Synopsys DPHY.
> +
>  config PHY_ROCKCHIP_EMMC
>  	tristate "Rockchip EMMC PHY Driver"
>  	depends on ARCH_ROCKCHIP && OF
> diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
> index fd21cbaf40dd..f62e9010bcaf 100644
> --- a/drivers/phy/rockchip/Makefile
> +++ b/drivers/phy/rockchip/Makefile
> @@ -1,5 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0
>  obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
> +obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
>  obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
> diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
> new file mode 100644
> index 000000000000..3a29976c2dff
> --- /dev/null
> +++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
> @@ -0,0 +1,408 @@
> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +/*
> + * Rockchip MIPI Synopsys DPHY driver
> + *
> + * Based on:
> + *
> + * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
> + * Author: Yakir Yang <ykk@@rock-chips.com>
> + */
> +
> +#include <linux/clk.h>
> +#include <linux/io.h>
> +#include <linux/mfd/syscon.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_device.h>
> +#include <linux/phy/phy.h>
> +#include <linux/phy/phy-mipi-dphy.h>
> +#include <linux/platform_device.h>
> +#include <linux/regmap.h>
> +
> +#define RK3399_GRF_SOC_CON9	0x6224
> +#define RK3399_GRF_SOC_CON21	0x6254
> +#define RK3399_GRF_SOC_CON22	0x6258
> +#define RK3399_GRF_SOC_CON23	0x625c
> +#define RK3399_GRF_SOC_CON24	0x6260
> +#define RK3399_GRF_SOC_CON25	0x6264
> +#define RK3399_GRF_SOC_STATUS1	0xe2a4
> +
> +#define CLOCK_LANE_HS_RX_CONTROL		0x34
> +#define LANE0_HS_RX_CONTROL			0x44
> +#define LANE1_HS_RX_CONTROL			0x54
> +#define LANE2_HS_RX_CONTROL			0x84
> +#define LANE3_HS_RX_CONTROL			0x94
> +#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
> +
> +#define MAX_DPHY_CLK 8
> +
> +#define PHY_TESTEN_ADDR			(0x1 << 16)
> +#define PHY_TESTEN_DATA			(0x0 << 16)
> +#define PHY_TESTCLK			(0x1 << 1)
> +#define PHY_TESTCLR			(0x1 << 0)
> +#define THS_SETTLE_COUNTER_THRESHOLD	0x04
> +
> +#define HIWORD_UPDATE(val, mask, shift) \
> +	((val) << (shift) | (mask) << ((shift) + 16))
> +
> +#define GRF_SOC_CON12                           0x0274
> +
> +#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
> +#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
> +
> +#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
> +#define GRF_EDP_PHY_SIDDQ_ON                    0
> +#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)
> +
> +struct hsfreq_range {
> +	u32 range_h;
> +	u8 cfg_bit;
> +};
> +
> +static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
> +	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
> +	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
> +	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
> +	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
> +	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
> +	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
> +	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
> +	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
> +	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
> +	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}
> +};
> +
> +static const char * const rk3399_mipidphy_clks[] = {
> +	"dphy-ref",
> +	"dphy-cfg",
> +	"grf",
> +};
> +
> +enum dphy_reg_id {
> +	GRF_DPHY_RX0_TURNDISABLE = 0,
> +	GRF_DPHY_RX0_FORCERXMODE,
> +	GRF_DPHY_RX0_FORCETXSTOPMODE,
> +	GRF_DPHY_RX0_ENABLE,
> +	GRF_DPHY_RX0_TESTCLR,
> +	GRF_DPHY_RX0_TESTCLK,
> +	GRF_DPHY_RX0_TESTEN,
> +	GRF_DPHY_RX0_TESTDIN,
> +	GRF_DPHY_RX0_TURNREQUEST,
> +	GRF_DPHY_RX0_TESTDOUT,
> +	GRF_DPHY_TX0_TURNDISABLE,
> +	GRF_DPHY_TX0_FORCERXMODE,
> +	GRF_DPHY_TX0_FORCETXSTOPMODE,
> +	GRF_DPHY_TX0_TURNREQUEST,
> +	GRF_DPHY_TX1RX1_TURNDISABLE,
> +	GRF_DPHY_TX1RX1_FORCERXMODE,
> +	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
> +	GRF_DPHY_TX1RX1_ENABLE,
> +	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
> +	GRF_DPHY_TX1RX1_BASEDIR,
> +	GRF_DPHY_TX1RX1_ENABLECLK,
> +	GRF_DPHY_TX1RX1_TURNREQUEST,
> +	GRF_DPHY_RX1_SRC_SEL,
> +	/* rk3288 only */
> +	GRF_CON_DISABLE_ISP,
> +	GRF_CON_ISP_DPHY_SEL,
> +	GRF_DSI_CSI_TESTBUS_SEL,
> +	GRF_DVP_V18SEL,
> +	/* below is for rk3399 only */
> +	GRF_DPHY_RX0_CLK_INV_SEL,
> +	GRF_DPHY_RX1_CLK_INV_SEL,
> +};
> +
> +struct dphy_reg {
> +	u32 offset;
> +	u32 mask;
> +	u32 shift;
> +};
> +
> +#define PHY_REG(_offset, _width, _shift) \
> +	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
> +
> +static const struct dphy_reg rk3399_grf_dphy_regs[] = {
> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
> +	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
> +	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),
> +};
> +
> +struct dphy_drv_data {
> +	const char * const *clks;
> +	int num_clks;
> +	const struct hsfreq_range *hsfreq_ranges;
> +	int num_hsfreq_ranges;
> +	const struct dphy_reg *regs;
> +};
> +
> +struct rockchip_dphy {
> +	struct device *dev;
> +	struct regmap *grf;
> +	const struct dphy_reg *grf_regs;
> +	struct clk_bulk_data clks[MAX_DPHY_CLK];
> +
> +	const struct dphy_drv_data *drv_data;
> +	struct phy_configure_opts_mipi_dphy config;
> +};
> +
> +static inline void write_grf_reg(struct rockchip_dphy *priv,
> +				 int index, u8 value)
> +{
> +	const struct dphy_reg *reg = &priv->grf_regs[index];
> +	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
> +
> +	WARN_ON(!reg->offset);
> +	regmap_write(priv->grf, reg->offset, val);
> +}
> +
> +static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
> +			     u8 test_code, u8 test_data)
> +{
> +	/*
> +	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
> +	 * is latched internally as the current test code. Test data is
> +	 * programmed internally by rising edge on TESTCLK.
> +	 */
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> +}
> +
> +/* should be move to power_on */
> +static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
> +{
> +	const struct dphy_drv_data *drv_data = priv->drv_data;
> +	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
> +	struct phy_configure_opts_mipi_dphy *config = &priv->config;
> +	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
> +	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
> +
> +	do_div(data_rate_mbps, 1000 * 1000);
> +
> +	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
> +		__func__, config->lanes, data_rate_mbps);
> +
> +	for (i = 0; i < num_hsfreq_ranges; i++) {
> +		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
> +			hsfreq = hsfreq_ranges[i].cfg_bit;
> +			break;
> +		}
> +	}
> +
> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
> +
> +	/* Disable lan turn around, which is ignored in receive mode */
> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
> +
> +	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
> +
> +	/* dphy start */
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
> +	usleep_range(100, 150);
> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
> +	usleep_range(100, 150);
> +
> +	/* set clock lane */
> +	/* HS hsfreq_range & lane 0  settle bypass */
> +	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
> +	/* HS RX Control of lane0 */
> +	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
> +	/* HS RX Control of lane1 */
> +	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
> +	/* HS RX Control of lane2 */
> +	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
> +	/* HS RX Control of lane3 */
> +	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);
> +	/* HS RX Data Lanes Settle State Time Control */
> +	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
> +			 THS_SETTLE_COUNTER_THRESHOLD);
> +
> +	/* Normal operation */
> +	mipidphy0_wr_reg(priv, 0x0, 0);
> +
> +	return 0;
> +}
> +
> +static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
> +{
> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> +	int ret;
> +
> +	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */
> +	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
> +	if (ret)
> +		return ret;
> +
> +	memcpy(&priv->config, opts, sizeof(priv->config));

You could to:

	priv->config = *opts;

Up to you. Some people like memcpy(). :-)

> +
> +	return 0;
> +}
> +
> +static int rockchip_dphy_power_on(struct phy *phy)
> +{
> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> +	int ret;
> +
> +	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
> +	if (ret)
> +		return ret;
> +
> +	return mipidphy_rx_stream_on(priv);
> +}
> +
> +static int rockchip_dphy_power_off(struct phy *phy)
> +{
> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> +
> +	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
> +	return 0;
> +}
> +
> +static int rockchip_dphy_init(struct phy *phy)
> +{
> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> +	int ret;
> +
> +	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);

return ...;

> +	if (ret)
> +		return ret;
> +	return 0;
> +}
> +
> +static int rockchip_dphy_exit(struct phy *phy)
> +{
> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> +
> +	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
> +	return 0;
> +}
> +
> +static const struct phy_ops rockchip_dphy_ops = {
> +	.power_on	= rockchip_dphy_power_on,
> +	.power_off	= rockchip_dphy_power_off,
> +	.init		= rockchip_dphy_init,
> +	.exit		= rockchip_dphy_exit,
> +	.configure	= rockchip_dphy_configure,
> +	.owner		= THIS_MODULE,
> +};
> +
> +static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
> +	.clks = rk3399_mipidphy_clks,
> +	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
> +	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
> +	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
> +	.regs = rk3399_grf_dphy_regs,

Do you expect to support more of the similar PHY(s) --- are there such? If
not, you could put these in the code that uses them.

> +};
> +
> +static const struct of_device_id rockchip_dphy_dt_ids[] = {
> +	{
> +		.compatible = "rockchip,rk3399-mipi-dphy",
> +		.data = &rk3399_mipidphy_drv_data,
> +	},
> +	{}
> +};
> +MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
> +
> +static int rockchip_dphy_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct device_node *np = dev->of_node;
> +	const struct dphy_drv_data *drv_data;
> +	struct phy_provider *phy_provider;
> +	const struct of_device_id *of_id;
> +	struct rockchip_dphy *priv;
> +	struct regmap *grf;
> +	struct phy *phy;
> +	unsigned int i;
> +	int ret;
> +
> +	if (!dev->parent || !dev->parent->of_node)
> +		return -ENODEV;
> +
> +	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
> +		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");
> +		return -EINVAL;
> +	}
> +
> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +	priv->dev = dev;
> +
> +	grf = syscon_node_to_regmap(dev->parent->of_node);
> +	if (IS_ERR(grf)) {
> +		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
> +						      "rockchip,grf");
> +		if (IS_ERR(grf)) {
> +			dev_err(dev, "Can't find GRF syscon\n");
> +			return -ENODEV;
> +		}
> +	}
> +	priv->grf = grf;
> +
> +	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
> +	if (!of_id)
> +		return -EINVAL;
> +
> +	drv_data = of_id->data;
> +	priv->grf_regs = drv_data->regs;
> +	priv->drv_data = drv_data;
> +	for (i = 0; i < drv_data->num_clks; i++)
> +		priv->clks[i].id = drv_data->clks[i];
> +	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
> +	if (ret)
> +		return ret;
> +
> +	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
> +	if (IS_ERR(phy)) {
> +		dev_err(dev, "failed to create phy\n");
> +		return PTR_ERR(phy);
> +	}
> +	phy_set_drvdata(phy, priv);
> +
> +	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
> +
> +	return PTR_ERR_OR_ZERO(phy_provider);
> +}
> +
> +static struct platform_driver rockchip_dphy_driver = {
> +	.probe = rockchip_dphy_probe,
> +	.driver = {
> +		.name	= "rockchip-mipi-dphy",
> +		.of_match_table = rockchip_dphy_dt_ids,
> +	},
> +};
> +module_platform_driver(rockchip_dphy_driver);
> +
> +MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
> +MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
> +MODULE_LICENSE("Dual MIT/GPL");

-- 
Kind regards,

Sakari Ailus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 02/14] media: doc: add document for rkisp1 meta buffer format
  2019-07-30 18:42 ` [PATCH v8 02/14] media: doc: add document for " Helen Koike
@ 2019-08-07 13:09   ` Sakari Ailus
  2019-08-15 13:51   ` Laurent Pinchart
  1 sibling, 0 replies; 76+ messages in thread
From: Sakari Ailus @ 2019-08-07 13:09 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Jacob Chen,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga,
	linux-rockchip, hans.verkuil, laurent.pinchart, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Tue, Jul 30, 2019 at 03:42:44PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> This commit add document for rkisp1 meta buffer format
> 
> Signed-off-by: Jacob Chen <jacob-chen@rock-chips.com>
> Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8:
> - Add SPDX in the header
> - Remove emacs configs
> - Fix doc style
> 
> Changes in v7:
> - s/correspond/corresponding
> - s/use/uses
> - s/docuemnt/document
> 
>  Documentation/media/uapi/v4l/meta-formats.rst |  2 ++
>  .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    | 23 +++++++++++++++++++
>  .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      | 22 ++++++++++++++++++
>  3 files changed, 47 insertions(+)
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
> 
> diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
> index b10ca9ee3968..5de621fea3cc 100644
> --- a/Documentation/media/uapi/v4l/meta-formats.rst
> +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> @@ -24,3 +24,5 @@ These formats are used for the :ref:`metadata` interface only.
>      pixfmt-meta-uvc
>      pixfmt-meta-vsp1-hgo
>      pixfmt-meta-vsp1-hgt
> +    pixfmt-meta-rkisp1-params
> +    pixfmt-meta-rkisp1-stat
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
> new file mode 100644
> index 000000000000..103b5cb79b7c
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
> @@ -0,0 +1,23 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +
> +.. _v4l2-meta-fmt-rkisp1-params:
> +
> +============================
> +V4L2_META_FMT_RK_ISP1_PARAMS
> +============================
> +
> +Rockchip ISP1 Parameters Data
> +
> +Description
> +===========
> +
> +This format describes input parameters for the Rockchip ISP1.
> +
> +It uses c-struct :c:type:`rkisp1_isp_params_cfg`, which is defined in
> +the ``linux/rkisp1-config.h`` header file.

Do you have an insight on in which order the device executes the processing
steps for the image data? The pipeline may not be entirely linear either
(statistics, for instance). This should be included in the documentation.

> +
> +The parameters consist of multiple modules.
> +The module won't be updated if the corresponding bit was not set in module_*_update.
> +
> +.. kernel-doc:: include/uapi/linux/rkisp1-config.h
> +   :functions: rkisp1_isp_params_cfg
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
> new file mode 100644
> index 000000000000..4ad303f96421
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
> @@ -0,0 +1,22 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +
> +.. _v4l2-meta-fmt-rkisp1-stat:
> +
> +=============================
> +V4L2_META_FMT_RK_ISP1_STAT_3A
> +=============================
> +
> +
> +Rockchip ISP1 Statistics Data
> +
> +Description
> +===========
> +
> +This format describes image color statistics information generated by the Rockchip
> +ISP1.
> +
> +It uses c-struct :c:type:`rkisp1_stat_buffer`, which is defined in
> +the ``linux/rkisp1-config.h`` header file.
> +
> +.. kernel-doc:: include/uapi/linux/rkisp1-config.h
> +   :functions: rkisp1_stat_buffer

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-08-07 13:05   ` Sakari Ailus
@ 2019-08-07 13:37     ` Helen Koike
  2019-08-15 17:54       ` Laurent Pinchart
  0 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-08-07 13:37 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Sakari,

thanks for your review,

On 8/7/19 10:05 AM, Sakari Ailus wrote:
> Hi Helen,
> 
> Thanks for the patchset.
> 
> On Tue, Jul 30, 2019 at 03:42:46PM -0300, Helen Koike wrote:
>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>
>> This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver
>>
>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>> [migrate to phy framework]
>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>> [update for upstream]
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>
>> ---
>>
>> Changes in v8:
>> - Remove boiler plate license text
>>
>> Changes in v7:
>> - Migrate dphy specific code from
>> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
>> to drivers/phy/rockchip/phy-rockchip-dphy.c
>> - Drop support for rk3288
>> - Drop support for dphy txrx
>> - code styling and checkpatch fixes
>>
>>  drivers/phy/rockchip/Kconfig             |   8 +
>>  drivers/phy/rockchip/Makefile            |   1 +
>>  drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
>>  3 files changed, 417 insertions(+)
>>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
>>
>> diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
>> index c454c90cd99e..afd072f135e6 100644
>> --- a/drivers/phy/rockchip/Kconfig
>> +++ b/drivers/phy/rockchip/Kconfig
>> @@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
>>  	help
>>  	  Enable this to support the Rockchip Display Port PHY.
>>  
>> +config PHY_ROCKCHIP_DPHY
>> +	tristate "Rockchip MIPI Synopsys DPHY driver"
>> +	depends on ARCH_ROCKCHIP && OF
> 
> How about (...) || COMPILE_TEST ?
> 
>> +	select GENERIC_PHY_MIPI_DPHY
>> +	select GENERIC_PHY
>> +	help
>> +	  Enable this to support the Rockchip MIPI Synopsys DPHY.
>> +
>>  config PHY_ROCKCHIP_EMMC
>>  	tristate "Rockchip EMMC PHY Driver"
>>  	depends on ARCH_ROCKCHIP && OF
>> diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
>> index fd21cbaf40dd..f62e9010bcaf 100644
>> --- a/drivers/phy/rockchip/Makefile
>> +++ b/drivers/phy/rockchip/Makefile
>> @@ -1,5 +1,6 @@
>>  # SPDX-License-Identifier: GPL-2.0
>>  obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
>> +obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
>>  obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
>>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
>>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
>> diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
>> new file mode 100644
>> index 000000000000..3a29976c2dff
>> --- /dev/null
>> +++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
>> @@ -0,0 +1,408 @@
>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>> +/*
>> + * Rockchip MIPI Synopsys DPHY driver
>> + *
>> + * Based on:
>> + *
>> + * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
>> + * Author: Yakir Yang <ykk@@rock-chips.com>
>> + */
>> +
>> +#include <linux/clk.h>
>> +#include <linux/io.h>
>> +#include <linux/mfd/syscon.h>
>> +#include <linux/module.h>
>> +#include <linux/of.h>
>> +#include <linux/of_device.h>
>> +#include <linux/phy/phy.h>
>> +#include <linux/phy/phy-mipi-dphy.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/regmap.h>
>> +
>> +#define RK3399_GRF_SOC_CON9	0x6224
>> +#define RK3399_GRF_SOC_CON21	0x6254
>> +#define RK3399_GRF_SOC_CON22	0x6258
>> +#define RK3399_GRF_SOC_CON23	0x625c
>> +#define RK3399_GRF_SOC_CON24	0x6260
>> +#define RK3399_GRF_SOC_CON25	0x6264
>> +#define RK3399_GRF_SOC_STATUS1	0xe2a4
>> +
>> +#define CLOCK_LANE_HS_RX_CONTROL		0x34
>> +#define LANE0_HS_RX_CONTROL			0x44
>> +#define LANE1_HS_RX_CONTROL			0x54
>> +#define LANE2_HS_RX_CONTROL			0x84
>> +#define LANE3_HS_RX_CONTROL			0x94
>> +#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
>> +
>> +#define MAX_DPHY_CLK 8
>> +
>> +#define PHY_TESTEN_ADDR			(0x1 << 16)
>> +#define PHY_TESTEN_DATA			(0x0 << 16)
>> +#define PHY_TESTCLK			(0x1 << 1)
>> +#define PHY_TESTCLR			(0x1 << 0)
>> +#define THS_SETTLE_COUNTER_THRESHOLD	0x04
>> +
>> +#define HIWORD_UPDATE(val, mask, shift) \
>> +	((val) << (shift) | (mask) << ((shift) + 16))
>> +
>> +#define GRF_SOC_CON12                           0x0274
>> +
>> +#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
>> +#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
>> +
>> +#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
>> +#define GRF_EDP_PHY_SIDDQ_ON                    0
>> +#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)
>> +
>> +struct hsfreq_range {
>> +	u32 range_h;
>> +	u8 cfg_bit;
>> +};
>> +
>> +static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
>> +	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
>> +	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
>> +	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
>> +	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
>> +	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
>> +	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
>> +	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
>> +	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
>> +	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
>> +	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}
>> +};
>> +
>> +static const char * const rk3399_mipidphy_clks[] = {
>> +	"dphy-ref",
>> +	"dphy-cfg",
>> +	"grf",
>> +};
>> +
>> +enum dphy_reg_id {
>> +	GRF_DPHY_RX0_TURNDISABLE = 0,
>> +	GRF_DPHY_RX0_FORCERXMODE,
>> +	GRF_DPHY_RX0_FORCETXSTOPMODE,
>> +	GRF_DPHY_RX0_ENABLE,
>> +	GRF_DPHY_RX0_TESTCLR,
>> +	GRF_DPHY_RX0_TESTCLK,
>> +	GRF_DPHY_RX0_TESTEN,
>> +	GRF_DPHY_RX0_TESTDIN,
>> +	GRF_DPHY_RX0_TURNREQUEST,
>> +	GRF_DPHY_RX0_TESTDOUT,
>> +	GRF_DPHY_TX0_TURNDISABLE,
>> +	GRF_DPHY_TX0_FORCERXMODE,
>> +	GRF_DPHY_TX0_FORCETXSTOPMODE,
>> +	GRF_DPHY_TX0_TURNREQUEST,
>> +	GRF_DPHY_TX1RX1_TURNDISABLE,
>> +	GRF_DPHY_TX1RX1_FORCERXMODE,
>> +	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
>> +	GRF_DPHY_TX1RX1_ENABLE,
>> +	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
>> +	GRF_DPHY_TX1RX1_BASEDIR,
>> +	GRF_DPHY_TX1RX1_ENABLECLK,
>> +	GRF_DPHY_TX1RX1_TURNREQUEST,
>> +	GRF_DPHY_RX1_SRC_SEL,
>> +	/* rk3288 only */
>> +	GRF_CON_DISABLE_ISP,
>> +	GRF_CON_ISP_DPHY_SEL,
>> +	GRF_DSI_CSI_TESTBUS_SEL,
>> +	GRF_DVP_V18SEL,
>> +	/* below is for rk3399 only */
>> +	GRF_DPHY_RX0_CLK_INV_SEL,
>> +	GRF_DPHY_RX1_CLK_INV_SEL,
>> +};
>> +
>> +struct dphy_reg {
>> +	u32 offset;
>> +	u32 mask;
>> +	u32 shift;
>> +};
>> +
>> +#define PHY_REG(_offset, _width, _shift) \
>> +	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
>> +
>> +static const struct dphy_reg rk3399_grf_dphy_regs[] = {
>> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
>> +	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
>> +	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
>> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
>> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
>> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
>> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
>> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
>> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
>> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
>> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
>> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
>> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
>> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
>> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
>> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
>> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
>> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
>> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
>> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
>> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
>> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
>> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
>> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
>> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),
>> +};
>> +
>> +struct dphy_drv_data {
>> +	const char * const *clks;
>> +	int num_clks;
>> +	const struct hsfreq_range *hsfreq_ranges;
>> +	int num_hsfreq_ranges;
>> +	const struct dphy_reg *regs;
>> +};
>> +
>> +struct rockchip_dphy {
>> +	struct device *dev;
>> +	struct regmap *grf;
>> +	const struct dphy_reg *grf_regs;
>> +	struct clk_bulk_data clks[MAX_DPHY_CLK];
>> +
>> +	const struct dphy_drv_data *drv_data;
>> +	struct phy_configure_opts_mipi_dphy config;
>> +};
>> +
>> +static inline void write_grf_reg(struct rockchip_dphy *priv,
>> +				 int index, u8 value)
>> +{
>> +	const struct dphy_reg *reg = &priv->grf_regs[index];
>> +	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
>> +
>> +	WARN_ON(!reg->offset);
>> +	regmap_write(priv->grf, reg->offset, val);
>> +}
>> +
>> +static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
>> +			     u8 test_code, u8 test_data)
>> +{
>> +	/*
>> +	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
>> +	 * is latched internally as the current test code. Test data is
>> +	 * programmed internally by rising edge on TESTCLK.
>> +	 */
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
>> +}
>> +
>> +/* should be move to power_on */
>> +static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
>> +{
>> +	const struct dphy_drv_data *drv_data = priv->drv_data;
>> +	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
>> +	struct phy_configure_opts_mipi_dphy *config = &priv->config;
>> +	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
>> +	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
>> +
>> +	do_div(data_rate_mbps, 1000 * 1000);
>> +
>> +	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
>> +		__func__, config->lanes, data_rate_mbps);
>> +
>> +	for (i = 0; i < num_hsfreq_ranges; i++) {
>> +		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
>> +			hsfreq = hsfreq_ranges[i].cfg_bit;
>> +			break;
>> +		}
>> +	}
>> +
>> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
>> +
>> +	/* Disable lan turn around, which is ignored in receive mode */
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
>> +
>> +	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
>> +
>> +	/* dphy start */
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
>> +	usleep_range(100, 150);
>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
>> +	usleep_range(100, 150);
>> +
>> +	/* set clock lane */
>> +	/* HS hsfreq_range & lane 0  settle bypass */
>> +	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
>> +	/* HS RX Control of lane0 */
>> +	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
>> +	/* HS RX Control of lane1 */
>> +	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
>> +	/* HS RX Control of lane2 */
>> +	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
>> +	/* HS RX Control of lane3 */
>> +	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);
>> +	/* HS RX Data Lanes Settle State Time Control */
>> +	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
>> +			 THS_SETTLE_COUNTER_THRESHOLD);
>> +
>> +	/* Normal operation */
>> +	mipidphy0_wr_reg(priv, 0x0, 0);
>> +
>> +	return 0;
>> +}
>> +
>> +static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
>> +{
>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>> +	int ret;
>> +
>> +	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */
>> +	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
>> +	if (ret)
>> +		return ret;
>> +
>> +	memcpy(&priv->config, opts, sizeof(priv->config));
> 
> You could to:
> 
> 	priv->config = *opts;
> 
> Up to you. Some people like memcpy(). :-)

your way is better thanks!

> 
>> +
>> +	return 0;
>> +}
>> +
>> +static int rockchip_dphy_power_on(struct phy *phy)
>> +{
>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>> +	int ret;
>> +
>> +	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return mipidphy_rx_stream_on(priv);
>> +}
>> +
>> +static int rockchip_dphy_power_off(struct phy *phy)
>> +{
>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>> +
>> +	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
>> +	return 0;
>> +}
>> +
>> +static int rockchip_dphy_init(struct phy *phy)
>> +{
>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>> +	int ret;
>> +
>> +	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);
> 
> return ...;
> 
>> +	if (ret)
>> +		return ret;
>> +	return 0;
>> +}
>> +
>> +static int rockchip_dphy_exit(struct phy *phy)
>> +{
>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>> +
>> +	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
>> +	return 0;
>> +}
>> +
>> +static const struct phy_ops rockchip_dphy_ops = {
>> +	.power_on	= rockchip_dphy_power_on,
>> +	.power_off	= rockchip_dphy_power_off,
>> +	.init		= rockchip_dphy_init,
>> +	.exit		= rockchip_dphy_exit,
>> +	.configure	= rockchip_dphy_configure,
>> +	.owner		= THIS_MODULE,
>> +};
>> +
>> +static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
>> +	.clks = rk3399_mipidphy_clks,
>> +	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
>> +	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
>> +	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
>> +	.regs = rk3399_grf_dphy_regs,
> 
> Do you expect to support more of the similar PHY(s) --- are there such? If
> not, you could put these in the code that uses them.

Yes, for rk3288 in the future.

Regards,
Helen

> 
>> +};
>> +
>> +static const struct of_device_id rockchip_dphy_dt_ids[] = {
>> +	{
>> +		.compatible = "rockchip,rk3399-mipi-dphy",
>> +		.data = &rk3399_mipidphy_drv_data,
>> +	},
>> +	{}
>> +};
>> +MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
>> +
>> +static int rockchip_dphy_probe(struct platform_device *pdev)
>> +{
>> +	struct device *dev = &pdev->dev;
>> +	struct device_node *np = dev->of_node;
>> +	const struct dphy_drv_data *drv_data;
>> +	struct phy_provider *phy_provider;
>> +	const struct of_device_id *of_id;
>> +	struct rockchip_dphy *priv;
>> +	struct regmap *grf;
>> +	struct phy *phy;
>> +	unsigned int i;
>> +	int ret;
>> +
>> +	if (!dev->parent || !dev->parent->of_node)
>> +		return -ENODEV;
>> +
>> +	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
>> +		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
>> +	if (!priv)
>> +		return -ENOMEM;
>> +	priv->dev = dev;
>> +
>> +	grf = syscon_node_to_regmap(dev->parent->of_node);
>> +	if (IS_ERR(grf)) {
>> +		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
>> +						      "rockchip,grf");
>> +		if (IS_ERR(grf)) {
>> +			dev_err(dev, "Can't find GRF syscon\n");
>> +			return -ENODEV;
>> +		}
>> +	}
>> +	priv->grf = grf;
>> +
>> +	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
>> +	if (!of_id)
>> +		return -EINVAL;
>> +
>> +	drv_data = of_id->data;
>> +	priv->grf_regs = drv_data->regs;
>> +	priv->drv_data = drv_data;
>> +	for (i = 0; i < drv_data->num_clks; i++)
>> +		priv->clks[i].id = drv_data->clks[i];
>> +	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
>> +	if (ret)
>> +		return ret;
>> +
>> +	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
>> +	if (IS_ERR(phy)) {
>> +		dev_err(dev, "failed to create phy\n");
>> +		return PTR_ERR(phy);
>> +	}
>> +	phy_set_drvdata(phy, priv);
>> +
>> +	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
>> +
>> +	return PTR_ERR_OR_ZERO(phy_provider);
>> +}
>> +
>> +static struct platform_driver rockchip_dphy_driver = {
>> +	.probe = rockchip_dphy_probe,
>> +	.driver = {
>> +		.name	= "rockchip-mipi-dphy",
>> +		.of_match_table = rockchip_dphy_dt_ids,
>> +	},
>> +};
>> +module_platform_driver(rockchip_dphy_driver);
>> +
>> +MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
>> +MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
>> +MODULE_LICENSE("Dual MIT/GPL");
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver
  2019-07-30 18:42 ` [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver Helen Koike
@ 2019-08-07 15:27   ` Sakari Ailus
  2019-08-08 21:59     ` Helen Koike
  2019-08-07 15:36   ` Sakari Ailus
  1 sibling, 1 reply; 76+ messages in thread
From: Sakari Ailus @ 2019-08-07 15:27 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Tue, Jul 30, 2019 at 03:42:51PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add the core driver for rockchip isp1.
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> [fixed compilation and run time errors regarding new v4l2 async API]
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> [Add missing module device table]
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
> - Fix compilation and runtime errors due to bitrotting
> The code has bit-rotten since March 2018, fix compilation errors.
> The new V4L2 async notifier API requires notifiers to be initialized by
> a call to v4l2_async_notifier_init() before being used, do so.
> - Add missing module device table
> - use clk_bulk framework
> - add missing notifiers cleanups
> - s/strlcpy/strscpy
> - normalize bus_info name
> - fix s_stream error path, stream_cnt wans't being decremented properly
> - use devm_platform_ioremap_resource() helper
> - s/deice/device
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now.
> - remove "saved_state" member from rkisp1_stream struct
> - Reverse the order of MIs
> - Simplify MI interrupt handling
> Rather than adding unnecessary indirection, just use stream index to
> handle MI interrupt enable/disable/clear, since the stream index matches
> the order of bits now, thanks to previous patch. While at it, remove
> some dead code.
> - code styling and checkpatch fixes
> 
>  drivers/media/platform/Kconfig                |  12 +
>  drivers/media/platform/Makefile               |   1 +
>  drivers/media/platform/rockchip/isp1/Makefile |   7 +
>  drivers/media/platform/rockchip/isp1/common.h | 101 +++
>  drivers/media/platform/rockchip/isp1/dev.c    | 675 ++++++++++++++++++
>  drivers/media/platform/rockchip/isp1/dev.h    |  97 +++
>  6 files changed, 893 insertions(+)
>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
> 
> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> index 89555f9a813f..e0e98937c565 100644
> --- a/drivers/media/platform/Kconfig
> +++ b/drivers/media/platform/Kconfig
> @@ -106,6 +106,18 @@ config VIDEO_QCOM_CAMSS
>  	select VIDEOBUF2_DMA_SG
>  	select V4L2_FWNODE
>  
> +config VIDEO_ROCKCHIP_ISP1
> +	tristate "Rockchip Image Signal Processing v1 Unit driver"
> +	depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
> +	depends on ARCH_ROCKCHIP || COMPILE_TEST
> +	select VIDEOBUF2_DMA_CONTIG
> +	select VIDEOBUF2_VMALLOC
> +	select V4L2_FWNODE
> +	select PHY_ROCKCHIP_DPHY
> +	default n
> +	---help---
> +	  Support for ISP1 on the rockchip SoC.
> +
>  config VIDEO_S3C_CAMIF
>  	tristate "Samsung S3C24XX/S3C64XX SoC Camera Interface driver"
>  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API
> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> index 7cbbd925124c..f9fcf8e7c513 100644
> --- a/drivers/media/platform/Makefile
> +++ b/drivers/media/platform/Makefile
> @@ -69,6 +69,7 @@ obj-$(CONFIG_VIDEO_RENESAS_FDP1)	+= rcar_fdp1.o
>  obj-$(CONFIG_VIDEO_RENESAS_JPU)		+= rcar_jpu.o
>  obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
>  
> +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1)	+= rockchip/isp1/
>  obj-$(CONFIG_VIDEO_ROCKCHIP_RGA)	+= rockchip/rga/
>  
>  obj-y	+= omap/
> diff --git a/drivers/media/platform/rockchip/isp1/Makefile b/drivers/media/platform/rockchip/isp1/Makefile
> new file mode 100644
> index 000000000000..72706e80fc8b
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/Makefile
> @@ -0,0 +1,7 @@
> +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1) += 	rockchip-isp1.o
> +rockchip-isp1-objs 	   += 	rkisp1.o \
> +				dev.o \
> +				regs.o \
> +				isp_stats.o \
> +				isp_params.o \
> +				capture.o
> diff --git a/drivers/media/platform/rockchip/isp1/common.h b/drivers/media/platform/rockchip/isp1/common.h
> new file mode 100644
> index 000000000000..606ce2793546
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/common.h
> @@ -0,0 +1,101 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#ifndef _RKISP1_COMMON_H
> +#define _RKISP1_COMMON_H
> +
> +#include <linux/mutex.h>
> +#include <media/media-device.h>
> +#include <media/media-entity.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/videobuf2-v4l2.h>
> +
> +#define RKISP1_DEFAULT_WIDTH		800
> +#define RKISP1_DEFAULT_HEIGHT		600
> +
> +#define RKISP1_MAX_STREAM		2
> +#define RKISP1_STREAM_MP		0
> +#define RKISP1_STREAM_SP		1
> +
> +#define RKISP1_PLANE_Y			0
> +#define RKISP1_PLANE_CB			1
> +#define RKISP1_PLANE_CR			2
> +
> +enum rkisp1_sd_type {
> +	RKISP1_SD_SENSOR,
> +	RKISP1_SD_PHY_CSI,
> +	RKISP1_SD_VCM,
> +	RKISP1_SD_FLASH,
> +	RKISP1_SD_MAX,
> +};

I wonder if this is a leftover from the driver development time. Same goes
for the subdevs field in struct rkisp1_device.

> +
> +/* One structure per video node */
> +struct rkisp1_vdev_node {
> +	struct vb2_queue buf_queue;
> +	/* vfd lock */
> +	struct mutex vlock;
> +	struct video_device vdev;
> +	struct media_pad pad;
> +};
> +
> +enum rkisp1_fmt_pix_type {
> +	FMT_YUV,
> +	FMT_RGB,
> +	FMT_BAYER,
> +	FMT_JPEG,
> +	FMT_MAX
> +};
> +
> +enum rkisp1_fmt_raw_pat_type {
> +	RAW_RGGB = 0,
> +	RAW_GRBG,
> +	RAW_GBRG,
> +	RAW_BGGR,
> +};
> +
> +struct rkisp1_buffer {
> +	struct vb2_v4l2_buffer vb;
> +	struct list_head queue;
> +	union {
> +		u32 buff_addr[VIDEO_MAX_PLANES];
> +		void *vaddr[VIDEO_MAX_PLANES];
> +	};
> +};
> +
> +struct rkisp1_dummy_buffer {
> +	void *vaddr;
> +	dma_addr_t dma_addr;
> +	u32 size;
> +};
> +
> +extern int rkisp1_debug;
> +
> +static inline
> +struct rkisp1_vdev_node *vdev_to_node(struct video_device *vdev)
> +{
> +	return container_of(vdev, struct rkisp1_vdev_node, vdev);
> +}
> +
> +static inline struct rkisp1_vdev_node *queue_to_node(struct vb2_queue *q)
> +{
> +	return container_of(q, struct rkisp1_vdev_node, buf_queue);
> +}
> +
> +static inline struct rkisp1_buffer *to_rkisp1_buffer(struct vb2_v4l2_buffer *vb)
> +{
> +	return container_of(vb, struct rkisp1_buffer, vb);
> +}
> +
> +static inline struct vb2_queue *to_vb2_queue(struct file *file)
> +{
> +	struct rkisp1_vdev_node *vnode = video_drvdata(file);
> +
> +	return &vnode->buf_queue;
> +}
> +
> +#endif /* _RKISP1_COMMON_H */
> diff --git a/drivers/media/platform/rockchip/isp1/dev.c b/drivers/media/platform/rockchip/isp1/dev.c
> new file mode 100644
> index 000000000000..2b4a67e1a3b5
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/dev.c
> @@ -0,0 +1,675 @@
> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#include <linux/clk.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_graph.h>
> +#include <linux/of_platform.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/phy/phy.h>
> +#include <linux/phy/phy-mipi-dphy.h>
> +
> +#include "common.h"
> +#include "regs.h"
> +
> +struct isp_match_data {
> +	const char * const *clks;
> +	int size;

unsigned int

> +};
> +
> +struct sensor_async_subdev {
> +	struct v4l2_async_subdev asd;
> +	struct v4l2_mbus_config mbus;
> +	unsigned int lanes;
> +};
> +
> +int rkisp1_debug;
> +module_param_named(debug, rkisp1_debug, int, 0644);
> +MODULE_PARM_DESC(debug, "Debug level (0-1)");

Have you thought of using dynamic debug instead?

> +
> +/**************************** pipeline operations******************************/
> +
> +static int __isp_pipeline_prepare(struct rkisp1_pipeline *p,
> +				  struct media_entity *me)
> +{
> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
> +	struct v4l2_subdev *sd;
> +	unsigned int i;
> +
> +	p->num_subdevs = 0;
> +	memset(p->subdevs, 0, sizeof(p->subdevs));
> +
> +	while (1) {
> +		struct media_pad *pad = NULL;
> +
> +		/* Find remote source pad */
> +		for (i = 0; i < me->num_pads; i++) {
> +			struct media_pad *spad = &me->pads[i];
> +
> +			if (!(spad->flags & MEDIA_PAD_FL_SINK))
> +				continue;
> +			pad = media_entity_remote_pad(spad);
> +			if (pad)
> +				break;
> +		}
> +
> +		if (!pad)
> +			break;
> +
> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> +		if (sd != &dev->isp_sdev.sd)
> +			p->subdevs[p->num_subdevs++] = sd;

How do you make sure you don't overrun the array?

Instead, I'd avoid maintaining the array in the first place --- the same
information is available from the MC framework data structures --- see e.g.
the omap3isp driver.

> +
> +		me = &sd->entity;
> +		if (me->num_pads == 1)
> +			break;
> +	}
> +	return 0;
> +}
> +
> +static int __subdev_set_power(struct v4l2_subdev *sd, int on)
> +{
> +	int ret;
> +
> +	if (!sd)
> +		return -ENXIO;
> +
> +	ret = v4l2_subdev_call(sd, core, s_power, on);
> +
> +	return ret != -ENOIOCTLCMD ? ret : 0;
> +}
> +
> +static int __isp_pipeline_s_power(struct rkisp1_pipeline *p, bool on)

Could you instead use v4l2_pipeline_pm_use()?

> +{
> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
> +	int i, ret;
> +
> +	if (on) {
> +		__subdev_set_power(&dev->isp_sdev.sd, true);
> +
> +		for (i = p->num_subdevs - 1; i >= 0; --i) {
> +			ret = __subdev_set_power(p->subdevs[i], true);
> +			if (ret < 0 && ret != -ENXIO)
> +				goto err_power_off;
> +		}
> +	} else {
> +		for (i = 0; i < p->num_subdevs; ++i)
> +			__subdev_set_power(p->subdevs[i], false);
> +
> +		__subdev_set_power(&dev->isp_sdev.sd, false);
> +	}
> +
> +	return 0;
> +
> +err_power_off:
> +	for (++i; i < p->num_subdevs; ++i)
> +		__subdev_set_power(p->subdevs[i], false);
> +	__subdev_set_power(&dev->isp_sdev.sd, true);
> +	return ret;
> +}
> +
> +static int rkisp1_pipeline_open(struct rkisp1_pipeline *p,
> +				struct media_entity *me,
> +				bool prepare)
> +{
> +	int ret;
> +
> +	if (WARN_ON(!p || !me))
> +		return -EINVAL;
> +	if (atomic_inc_return(&p->power_cnt) > 1)
> +		return 0;
> +
> +	/* go through media graphic and get subdevs */
> +	if (prepare)
> +		__isp_pipeline_prepare(p, me);
> +
> +	if (!p->num_subdevs)
> +		return -EINVAL;
> +
> +	ret = __isp_pipeline_s_power(p, 1);
> +	if (ret < 0)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int rkisp1_pipeline_close(struct rkisp1_pipeline *p)
> +{
> +	int ret;
> +
> +	if (atomic_dec_return(&p->power_cnt) > 0)
> +		return 0;
> +	ret = __isp_pipeline_s_power(p, 0);
> +
> +	return ret == -ENXIO ? 0 : ret;
> +}
> +
> +/*
> + * stream-on order: isp_subdev, mipi dphy, sensor
> + * stream-off order: mipi dphy, sensor, isp_subdev
> + */
> +static int rkisp1_pipeline_set_stream(struct rkisp1_pipeline *p, bool on)
> +{
> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
> +	int i, ret;
> +
> +	if ((on && atomic_inc_return(&p->stream_cnt) > 1) ||
> +	    (!on && atomic_dec_return(&p->stream_cnt) > 0))
> +		return 0;
> +
> +	if (on) {
> +		ret = v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream,
> +				       true);
> +		if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV) {
> +			v4l2_err(&dev->v4l2_dev,
> +				 "s_stream failed on subdevice %s (%d)\n",
> +				 dev->isp_sdev.sd.name,
> +				 ret);
> +			atomic_dec(&p->stream_cnt);
> +			return ret;
> +		}
> +	}
> +
> +	/* phy -> sensor */
> +	for (i = 0; i < p->num_subdevs; ++i) {
> +		ret = v4l2_subdev_call(p->subdevs[i], video, s_stream, on);
> +		if (on && ret < 0 && ret != -ENOIOCTLCMD && ret != -ENODEV)
> +			goto err_stream_off;

You should stop after the first external sub-device.

It seems even the omap3isp driver doesn't do that. It's not easy to spot
such issues indeed.

> +	}
> +
> +	if (!on)
> +		v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
> +
> +	return 0;
> +
> +err_stream_off:
> +	for (--i; i >= 0; --i)
> +		v4l2_subdev_call(p->subdevs[i], video, s_stream, false);
> +	v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
> +	atomic_dec(&p->stream_cnt);
> +	return ret;
> +}
> +
> +/***************************** media controller *******************************/
> +/* See http://opensource.rock-chips.com/wiki_Rockchip-isp1 for Topology */

The host appears to be down, or there's a routing problem. Unless this is
fixed, having such a URL here doesn't do much good. :-I

> +
> +static int rkisp1_create_links(struct rkisp1_device *dev)
> +{
> +	struct media_entity *source, *sink;
> +	struct rkisp1_sensor *sensor;
> +	unsigned int flags, pad;
> +	int ret;
> +
> +	/* sensor links(or mipi-phy) */
> +	list_for_each_entry(sensor, &dev->sensors, list) {
> +		for (pad = 0; pad < sensor->sd->entity.num_pads; pad++)
> +			if (sensor->sd->entity.pads[pad].flags &
> +				MEDIA_PAD_FL_SOURCE)
> +				break;

Could you use media_entity_get_fwnode_pad() instead?

> +
> +		if (pad == sensor->sd->entity.num_pads) {
> +			dev_err(dev->dev,
> +				"failed to find src pad for %s\n",
> +				sensor->sd->name);
> +
> +			return -ENXIO;
> +		}
> +
> +		ret = media_create_pad_link(
> +				&sensor->sd->entity, pad,
> +				&dev->isp_sdev.sd.entity,
> +				RKISP1_ISP_PAD_SINK,
> +				list_is_first(&sensor->list, &dev->sensors) ?
> +				MEDIA_LNK_FL_ENABLED : 0);
> +		if (ret) {
> +			dev_err(dev->dev,
> +				"failed to create link for %s\n",
> +				sensor->sd->name);
> +			return ret;
> +		}
> +	}
> +
> +	/* params links */
> +	source = &dev->params_vdev.vnode.vdev.entity;
> +	sink = &dev->isp_sdev.sd.entity;
> +	flags = MEDIA_LNK_FL_ENABLED;
> +	ret = media_create_pad_link(source, 0, sink,
> +				       RKISP1_ISP_PAD_SINK_PARAMS, flags);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* create isp internal links */
> +	/* SP links */
> +	source = &dev->isp_sdev.sd.entity;
> +	sink = &dev->stream[RKISP1_STREAM_SP].vnode.vdev.entity;
> +	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
> +				       sink, 0, flags);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* MP links */
> +	source = &dev->isp_sdev.sd.entity;
> +	sink = &dev->stream[RKISP1_STREAM_MP].vnode.vdev.entity;
> +	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
> +				       sink, 0, flags);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* 3A stats links */
> +	source = &dev->isp_sdev.sd.entity;
> +	sink = &dev->stats_vdev.vnode.vdev.entity;
> +	return media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_STATS,
> +					sink, 0, flags);

Indentation. Same for the calls to the same function above.

> +}
> +
> +static int subdev_notifier_bound(struct v4l2_async_notifier *notifier,
> +				 struct v4l2_subdev *sd,
> +				 struct v4l2_async_subdev *asd)
> +{
> +	struct rkisp1_device *isp_dev = container_of(notifier,
> +						     struct rkisp1_device,
> +						     notifier);
> +	struct sensor_async_subdev *s_asd = container_of(asd,
> +					struct sensor_async_subdev, asd);
> +	struct rkisp1_sensor *sensor;
> +
> +	sensor = devm_kzalloc(isp_dev->dev, sizeof(*sensor), GFP_KERNEL);
> +	if (!sensor)
> +		return -ENOMEM;
> +
> +	sensor->lanes = s_asd->lanes;
> +	sensor->mbus = s_asd->mbus;
> +	sensor->sd = sd;
> +	sensor->dphy = devm_phy_get(isp_dev->dev, "dphy");
> +	if (IS_ERR(sensor->dphy)) {
> +		if (PTR_ERR(sensor->dphy) != -EPROBE_DEFER)
> +			dev_err(isp_dev->dev, "Couldn't get the MIPI D-PHY\n");
> +		return PTR_ERR(sensor->dphy);
> +	}
> +	phy_init(sensor->dphy);
> +
> +	list_add(&sensor->list, &isp_dev->sensors);

In general, maintaining the information on the external subdevs on your own
adds complexity to the driver. You can get the information when you need it
from the data structures maintained by MC (see e.g. the omap3isp driver for
examples).

> +
> +	return 0;
> +}
> +
> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
> +					  struct v4l2_subdev *sd)
> +{
> +	struct rkisp1_sensor *sensor;
> +
> +	list_for_each_entry(sensor, &dev->sensors, list)
> +		if (sensor->sd == sd)
> +			return sensor;
> +
> +	return NULL;
> +}
> +
> +static void subdev_notifier_unbind(struct v4l2_async_notifier *notifier,
> +				   struct v4l2_subdev *sd,
> +				   struct v4l2_async_subdev *asd)
> +{
> +	struct rkisp1_device *isp_dev = container_of(notifier,
> +						     struct rkisp1_device,
> +						     notifier);
> +	struct rkisp1_sensor *sensor = sd_to_sensor(isp_dev, sd);
> +
> +	/* TODO: check if a lock is required here */
> +	list_del(&sensor->list);
> +
> +	phy_exit(sensor->dphy);
> +}
> +
> +static int subdev_notifier_complete(struct v4l2_async_notifier *notifier)
> +{
> +	struct rkisp1_device *dev = container_of(notifier, struct rkisp1_device,
> +						 notifier);
> +	int ret;
> +
> +	mutex_lock(&dev->media_dev.graph_mutex);
> +	ret = rkisp1_create_links(dev);
> +	if (ret < 0)
> +		goto unlock;
> +	ret = v4l2_device_register_subdev_nodes(&dev->v4l2_dev);
> +	if (ret < 0)
> +		goto unlock;
> +
> +	v4l2_info(&dev->v4l2_dev, "Async subdev notifier completed\n");
> +
> +unlock:
> +	mutex_unlock(&dev->media_dev.graph_mutex);
> +	return ret;
> +}
> +
> +static int rkisp1_fwnode_parse(struct device *dev,
> +			       struct v4l2_fwnode_endpoint *vep,
> +			       struct v4l2_async_subdev *asd)
> +{
> +	struct sensor_async_subdev *s_asd =
> +			container_of(asd, struct sensor_async_subdev, asd);
> +
> +	if (vep->bus_type != V4L2_MBUS_CSI2_DPHY) {
> +		dev_err(dev, "Only CSI2 bus type is currently supported\n");
> +		return -EINVAL;
> +	}
> +
> +	if (vep->base.port != 0) {
> +		dev_err(dev, "The ISP has only port 0\n");
> +		return -EINVAL;
> +	}
> +
> +	s_asd->mbus.type = vep->bus_type;
> +	s_asd->mbus.flags = vep->bus.mipi_csi2.flags;
> +	s_asd->lanes = vep->bus.mipi_csi2.num_data_lanes;
> +
> +	switch (vep->bus.mipi_csi2.num_data_lanes) {
> +	case 1:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_1_LANE;
> +		break;
> +	case 2:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_2_LANE;
> +		break;
> +	case 3:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_3_LANE;
> +		break;
> +	case 4:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_4_LANE;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_async_notifier_operations subdev_notifier_ops = {
> +	.bound = subdev_notifier_bound,
> +	.unbind = subdev_notifier_unbind,
> +	.complete = subdev_notifier_complete,
> +};
> +
> +static int isp_subdev_notifier(struct rkisp1_device *isp_dev)
> +{
> +	struct v4l2_async_notifier *ntf = &isp_dev->notifier;
> +	struct device *dev = isp_dev->dev;
> +	int ret;
> +
> +	v4l2_async_notifier_init(ntf);
> +
> +	ret = v4l2_async_notifier_parse_fwnode_endpoints_by_port(
> +		dev, ntf, sizeof(struct sensor_async_subdev), 0,
> +		rkisp1_fwnode_parse);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (list_empty(&ntf->asd_list))
> +		return -ENODEV;	/* no endpoint */
> +
> +	ntf->ops = &subdev_notifier_ops;
> +
> +	return v4l2_async_notifier_register(&isp_dev->v4l2_dev, ntf);
> +}
> +
> +/***************************** platform device *******************************/
> +
> +static int rkisp1_register_platform_subdevs(struct rkisp1_device *dev)
> +{
> +	int ret;
> +
> +	ret = rkisp1_register_isp_subdev(dev, &dev->v4l2_dev);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = rkisp1_register_stream_vdevs(dev);
> +	if (ret < 0)
> +		goto err_unreg_isp_subdev;
> +
> +	ret = rkisp1_register_stats_vdev(&dev->stats_vdev, &dev->v4l2_dev, dev);
> +	if (ret < 0)
> +		goto err_unreg_stream_vdev;
> +
> +	ret = rkisp1_register_params_vdev(&dev->params_vdev, &dev->v4l2_dev,
> +					  dev);
> +	if (ret < 0)
> +		goto err_unreg_stats_vdev;
> +
> +	ret = isp_subdev_notifier(dev);
> +	if (ret < 0) {
> +		v4l2_err(&dev->v4l2_dev,
> +			 "Failed to register subdev notifier(%d)\n", ret);
> +		goto err_unreg_params_vdev;
> +	}
> +
> +	return 0;
> +err_unreg_params_vdev:
> +	rkisp1_unregister_params_vdev(&dev->params_vdev);
> +err_unreg_stats_vdev:
> +	rkisp1_unregister_stats_vdev(&dev->stats_vdev);
> +err_unreg_stream_vdev:
> +	rkisp1_unregister_stream_vdevs(dev);
> +err_unreg_isp_subdev:
> +	rkisp1_unregister_isp_subdev(dev);
> +	return ret;
> +}
> +
> +static const char * const rk3399_isp_clks[] = {
> +	"clk_isp",
> +	"aclk_isp",
> +	"hclk_isp",
> +	"aclk_isp_wrap",
> +	"hclk_isp_wrap",
> +};
> +
> +static const char * const rk3288_isp_clks[] = {
> +	"clk_isp",
> +	"aclk_isp",
> +	"hclk_isp",
> +	"pclk_isp_in",
> +	"sclk_isp_jpe",
> +};
> +
> +static const struct isp_match_data rk3288_isp_clk_data = {
> +	.clks = rk3288_isp_clks,
> +	.size = ARRAY_SIZE(rk3288_isp_clks),
> +};
> +
> +static const struct isp_match_data rk3399_isp_clk_data = {
> +	.clks = rk3399_isp_clks,
> +	.size = ARRAY_SIZE(rk3399_isp_clks),
> +};
> +
> +static const struct of_device_id rkisp1_plat_of_match[] = {
> +	{
> +		.compatible = "rockchip,rk3288-cif-isp",
> +		.data = &rk3288_isp_clk_data,
> +	}, {
> +		.compatible = "rockchip,rk3399-cif-isp",
> +		.data = &rk3399_isp_clk_data,
> +	},
> +	{},
> +};
> +MODULE_DEVICE_TABLE(of, rkisp1_plat_of_match);
> +
> +static irqreturn_t rkisp1_irq_handler(int irq, void *ctx)
> +{
> +	struct device *dev = ctx;
> +	struct rkisp1_device *rkisp1_dev = dev_get_drvdata(dev);
> +	unsigned int mis_val;
> +
> +	mis_val = readl(rkisp1_dev->base_addr + CIF_ISP_MIS);
> +	if (mis_val)
> +		rkisp1_isp_isr(mis_val, rkisp1_dev);
> +
> +	mis_val = readl(rkisp1_dev->base_addr + CIF_MIPI_MIS);
> +	if (mis_val)
> +		rkisp1_mipi_isr(mis_val, rkisp1_dev);
> +
> +	mis_val = readl(rkisp1_dev->base_addr + CIF_MI_MIS);
> +	if (mis_val)
> +		rkisp1_mi_isr(mis_val, rkisp1_dev);
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static int rkisp1_plat_probe(struct platform_device *pdev)
> +{
> +	struct device_node *node = pdev->dev.of_node;
> +	const struct isp_match_data *clk_data;
> +	const struct of_device_id *match;
> +	struct device *dev = &pdev->dev;
> +	struct rkisp1_device *isp_dev;
> +	struct v4l2_device *v4l2_dev;
> +	unsigned int i;
> +	int ret, irq;
> +
> +	match = of_match_node(rkisp1_plat_of_match, node);
> +	isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
> +	if (!isp_dev)
> +		return -ENOMEM;
> +
> +	INIT_LIST_HEAD(&isp_dev->sensors);
> +
> +	dev_set_drvdata(dev, isp_dev);
> +	isp_dev->dev = dev;
> +
> +	isp_dev->base_addr = devm_platform_ioremap_resource(pdev, 0);
> +	if (IS_ERR(isp_dev->base_addr))
> +		return PTR_ERR(isp_dev->base_addr);
> +
> +	irq = platform_get_irq(pdev, 0);
> +	if (irq < 0)
> +		return irq;
> +
> +	ret = devm_request_irq(dev, irq, rkisp1_irq_handler, IRQF_SHARED,
> +			       dev_driver_string(dev), dev);
> +	if (ret < 0) {
> +		dev_err(dev, "request irq failed: %d\n", ret);
> +		return ret;
> +	}
> +
> +	isp_dev->irq = irq;
> +	clk_data = match->data;
> +
> +	for (i = 0; i < clk_data->size; i++)
> +		isp_dev->clks[i].id = clk_data->clks[i];
> +	ret = devm_clk_bulk_get(dev, clk_data->size, isp_dev->clks);
> +	if (ret)
> +		return ret;
> +	isp_dev->clk_size = clk_data->size;
> +
> +	atomic_set(&isp_dev->pipe.power_cnt, 0);
> +	atomic_set(&isp_dev->pipe.stream_cnt, 0);
> +	isp_dev->pipe.open = rkisp1_pipeline_open;
> +	isp_dev->pipe.close = rkisp1_pipeline_close;
> +	isp_dev->pipe.set_stream = rkisp1_pipeline_set_stream;
> +
> +	rkisp1_stream_init(isp_dev, RKISP1_STREAM_SP);
> +	rkisp1_stream_init(isp_dev, RKISP1_STREAM_MP);
> +
> +	strscpy(isp_dev->media_dev.model, "rkisp1",
> +		sizeof(isp_dev->media_dev.model));
> +	isp_dev->media_dev.dev = &pdev->dev;
> +	strscpy(isp_dev->media_dev.bus_info,
> +		"platform: " DRIVER_NAME, sizeof(isp_dev->media_dev.bus_info));
> +	media_device_init(&isp_dev->media_dev);
> +
> +	v4l2_dev = &isp_dev->v4l2_dev;
> +	v4l2_dev->mdev = &isp_dev->media_dev;
> +	strscpy(v4l2_dev->name, "rkisp1", sizeof(v4l2_dev->name));
> +	v4l2_ctrl_handler_init(&isp_dev->ctrl_handler, 5);
> +	v4l2_dev->ctrl_handler = &isp_dev->ctrl_handler;
> +
> +	ret = v4l2_device_register(isp_dev->dev, &isp_dev->v4l2_dev);
> +	if (ret < 0)

Once you've initialised the control handler, you'll need to free it in case
of an error. I.e. add one more label for that purpose near the end.

> +		return ret;
> +
> +	ret = media_device_register(&isp_dev->media_dev);
> +	if (ret < 0) {
> +		v4l2_err(v4l2_dev, "Failed to register media device: %d\n",
> +			 ret);
> +		goto err_unreg_v4l2_dev;
> +	}
> +
> +	/* create & register platefom subdev (from of_node) */
> +	ret = rkisp1_register_platform_subdevs(isp_dev);
> +	if (ret < 0)
> +		goto err_unreg_media_dev;
> +
> +	pm_runtime_enable(&pdev->dev);
> +
> +	return 0;
> +
> +err_unreg_media_dev:
> +	media_device_unregister(&isp_dev->media_dev);
> +err_unreg_v4l2_dev:
> +	v4l2_device_unregister(&isp_dev->v4l2_dev);
> +	return ret;
> +}
> +
> +static int rkisp1_plat_remove(struct platform_device *pdev)
> +{
> +	struct rkisp1_device *isp_dev = platform_get_drvdata(pdev);
> +
> +	pm_runtime_disable(&pdev->dev);
> +	media_device_unregister(&isp_dev->media_dev);
> +	v4l2_async_notifier_unregister(&isp_dev->notifier);
> +	v4l2_async_notifier_cleanup(&isp_dev->notifier);
> +	v4l2_device_unregister(&isp_dev->v4l2_dev);
> +	rkisp1_unregister_params_vdev(&isp_dev->params_vdev);
> +	rkisp1_unregister_stats_vdev(&isp_dev->stats_vdev);
> +	rkisp1_unregister_stream_vdevs(isp_dev);
> +	rkisp1_unregister_isp_subdev(isp_dev);
> +
> +	return 0;
> +}
> +
> +static int __maybe_unused rkisp1_runtime_suspend(struct device *dev)
> +{
> +	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
> +
> +	clk_bulk_disable_unprepare(isp_dev->clk_size, isp_dev->clks);
> +	return pinctrl_pm_select_sleep_state(dev);
> +}
> +
> +static int __maybe_unused rkisp1_runtime_resume(struct device *dev)
> +{
> +	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
> +	int ret;
> +
> +	ret = pinctrl_pm_select_default_state(dev);
> +	if (ret < 0)
> +		return ret;
> +	ret = clk_bulk_prepare_enable(isp_dev->clk_size, isp_dev->clks);
> +	if (ret < 0)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static const struct dev_pm_ops rkisp1_plat_pm_ops = {
> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> +				pm_runtime_force_resume)
> +	SET_RUNTIME_PM_OPS(rkisp1_runtime_suspend, rkisp1_runtime_resume, NULL)
> +};
> +
> +static struct platform_driver rkisp1_plat_drv = {
> +	.driver = {
> +		.name = DRIVER_NAME,
> +		.of_match_table = of_match_ptr(rkisp1_plat_of_match),
> +		.pm = &rkisp1_plat_pm_ops,
> +	},
> +	.probe = rkisp1_plat_probe,
> +	.remove = rkisp1_plat_remove,
> +};
> +
> +module_platform_driver(rkisp1_plat_drv);
> +MODULE_AUTHOR("Rockchip Camera/ISP team");
> +MODULE_DESCRIPTION("Rockchip ISP1 platform driver");
> +MODULE_LICENSE("Dual BSD/GPL");

BSD or MIT?

> diff --git a/drivers/media/platform/rockchip/isp1/dev.h b/drivers/media/platform/rockchip/isp1/dev.h
> new file mode 100644
> index 000000000000..f7cbee316523
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/dev.h
> @@ -0,0 +1,97 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#ifndef _RKISP1_DEV_H
> +#define _RKISP1_DEV_H
> +
> +#include <linux/clk.h>
> +
> +#include "capture.h"
> +#include "rkisp1.h"
> +#include "isp_params.h"
> +#include "isp_stats.h"
> +
> +#define DRIVER_NAME "rkisp1"
> +#define ISP_VDEV_NAME DRIVER_NAME  "_ispdev"
> +#define SP_VDEV_NAME DRIVER_NAME   "_selfpath"
> +#define MP_VDEV_NAME DRIVER_NAME   "_mainpath"
> +#define DMA_VDEV_NAME DRIVER_NAME  "_dmapath"
> +
> +#define GRP_ID_SENSOR			BIT(0)
> +#define GRP_ID_MIPIPHY			BIT(1)
> +#define GRP_ID_ISP			BIT(2)
> +#define GRP_ID_ISP_MP			BIT(3)
> +#define GRP_ID_ISP_SP			BIT(4)
> +
> +#define RKISP1_MAX_BUS_CLK	8
> +#define RKISP1_MAX_SENSOR	2
> +#define RKISP1_MAX_PIPELINE	4
> +
> +/*
> + * struct rkisp1_pipeline - An ISP hardware pipeline
> + *
> + * Capture device call other devices via pipeline
> + *
> + * @num_subdevs: number of linked subdevs
> + * @power_cnt: pipeline power count
> + * @stream_cnt: stream power count
> + */
> +struct rkisp1_pipeline {
> +	struct media_pipeline pipe;
> +	int num_subdevs;
> +	atomic_t power_cnt;
> +	atomic_t stream_cnt;
> +	struct v4l2_subdev *subdevs[RKISP1_MAX_PIPELINE];
> +	int (*open)(struct rkisp1_pipeline *p,
> +		    struct media_entity *me, bool prepare);
> +	int (*close)(struct rkisp1_pipeline *p);
> +	int (*set_stream)(struct rkisp1_pipeline *p, bool on);
> +};
> +
> +/*
> + * struct rkisp1_sensor - Sensor information
> + * @mbus: media bus configuration
> + */
> +struct rkisp1_sensor {
> +	struct v4l2_subdev *sd;
> +	struct v4l2_mbus_config mbus;
> +	unsigned int lanes;
> +	struct phy *dphy;
> +	struct list_head list;
> +};

You seem to also have struct sensor_async_subdev that appears to contain
similar information. Would it be possible to unify the two?

This would appear to allow you getting rid of functions such as
sd_to_sensor, for instance.

> +
> +/*
> + * struct rkisp1_device - ISP platform device
> + * @base_addr: base register address
> + * @active_sensor: sensor in-use, set when streaming on
> + * @isp_sdev: ISP sub-device
> + * @rkisp1_stream: capture video device
> + * @stats_vdev: ISP statistics output device
> + * @params_vdev: ISP input parameters device
> + */
> +struct rkisp1_device {
> +	void __iomem *base_addr;
> +	int irq;
> +	struct device *dev;
> +	unsigned int clk_size;
> +	struct clk_bulk_data clks[RKISP1_MAX_BUS_CLK];
> +	struct v4l2_device v4l2_dev;
> +	struct v4l2_ctrl_handler ctrl_handler;
> +	struct media_device media_dev;
> +	struct v4l2_async_notifier notifier;
> +	struct v4l2_subdev *subdevs[RKISP1_SD_MAX];
> +	struct rkisp1_sensor *active_sensor;
> +	struct list_head sensors;
> +	struct rkisp1_isp_subdev isp_sdev;
> +	struct rkisp1_stream stream[RKISP1_MAX_STREAM];
> +	struct rkisp1_isp_stats_vdev stats_vdev;
> +	struct rkisp1_isp_params_vdev params_vdev;
> +	struct rkisp1_pipeline pipe;
> +	struct vb2_alloc_ctx *alloc_ctx;
> +};
> +
> +#endif

-- 
Regards,

Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver
  2019-07-30 18:42 ` [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver Helen Koike
  2019-08-07 15:27   ` Sakari Ailus
@ 2019-08-07 15:36   ` Sakari Ailus
  1 sibling, 0 replies; 76+ messages in thread
From: Sakari Ailus @ 2019-08-07 15:36 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Tue, Jul 30, 2019 at 03:42:51PM -0300, Helen Koike wrote:

...

> +static int rkisp1_fwnode_parse(struct device *dev,
> +			       struct v4l2_fwnode_endpoint *vep,
> +			       struct v4l2_async_subdev *asd)
> +{
> +	struct sensor_async_subdev *s_asd =
> +			container_of(asd, struct sensor_async_subdev, asd);
> +
> +	if (vep->bus_type != V4L2_MBUS_CSI2_DPHY) {
> +		dev_err(dev, "Only CSI2 bus type is currently supported\n");
> +		return -EINVAL;
> +	}
> +
> +	if (vep->base.port != 0) {
> +		dev_err(dev, "The ISP has only port 0\n");
> +		return -EINVAL;
> +	}
> +
> +	s_asd->mbus.type = vep->bus_type;
> +	s_asd->mbus.flags = vep->bus.mipi_csi2.flags;
> +	s_asd->lanes = vep->bus.mipi_csi2.num_data_lanes;
> +
> +	switch (vep->bus.mipi_csi2.num_data_lanes) {
> +	case 1:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_1_LANE;
> +		break;
> +	case 2:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_2_LANE;
> +		break;
> +	case 3:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_3_LANE;
> +		break;
> +	case 4:
> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_4_LANE;
> +		break;

Could you use struct v4l2_fwnode_endpoint directly? The mbus config is a
legacy struct from bygone times and I'd like to avoid using it in new
drivers.

> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct v4l2_async_notifier_operations subdev_notifier_ops = {
> +	.bound = subdev_notifier_bound,
> +	.unbind = subdev_notifier_unbind,
> +	.complete = subdev_notifier_complete,
> +};
> +
> +static int isp_subdev_notifier(struct rkisp1_device *isp_dev)
> +{
> +	struct v4l2_async_notifier *ntf = &isp_dev->notifier;
> +	struct device *dev = isp_dev->dev;
> +	int ret;
> +
> +	v4l2_async_notifier_init(ntf);
> +
> +	ret = v4l2_async_notifier_parse_fwnode_endpoints_by_port(
> +		dev, ntf, sizeof(struct sensor_async_subdev), 0,
> +		rkisp1_fwnode_parse);

I know these functions aren't old but there's a better alternative. See
e.g. isp_parse_of_endpoints in drivers/media/platform/omap3isp/isp.c or
cio2_parse_firmware in drivers/media/pci/intel/ipu3/ipu3-cio2.c.

> +	if (ret < 0)
> +		return ret;
> +
> +	if (list_empty(&ntf->asd_list))
> +		return -ENODEV;	/* no endpoint */
> +
> +	ntf->ops = &subdev_notifier_ops;
> +
> +	return v4l2_async_notifier_register(&isp_dev->v4l2_dev, ntf);
> +}

-- 
Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (13 preceding siblings ...)
  2019-07-30 20:15 ` [PATCH v8 00/14] Rockchip ISP1 Driver Hans Verkuil
@ 2019-08-07 15:37 ` Sakari Ailus
  2019-08-07 17:57   ` Helen Koike
  2019-08-09 18:45 ` Manivannan Sadhasivam
  15 siblings, 1 reply; 76+ messages in thread
From: Sakari Ailus @ 2019-08-07 15:37 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, laurent.pinchart, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

On Tue, Jul 30, 2019 at 03:42:42PM -0300, Helen Koike wrote:
> Hello,
> 
> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
> rk3399 SoC.
> 
> I didn't change much from the last version, just applying the
> suggestions made in the previous one.
> 
> This patchset is also available at:
> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
> 
> Libcamera patched to work with this version:
> https://gitlab.collabora.com/koike/libcamera
> (also sent to the mailing list)
> 
> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
> Scarlet Chromebook.

Could you also post media-ctl -p printout e.g. as a reply to the cover
letter?

Thanks.

-- 
Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-08-07 15:37 ` Sakari Ailus
@ 2019-08-07 17:57   ` Helen Koike
  0 siblings, 0 replies; 76+ messages in thread
From: Helen Koike @ 2019-08-07 17:57 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, laurent.pinchart, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media



On 8/7/19 12:37 PM, Sakari Ailus wrote:
> On Tue, Jul 30, 2019 at 03:42:42PM -0300, Helen Koike wrote:
>> Hello,
>>
>> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
>> rk3399 SoC.
>>
>> I didn't change much from the last version, just applying the
>> suggestions made in the previous one.
>>
>> This patchset is also available at:
>> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
>>
>> Libcamera patched to work with this version:
>> https://gitlab.collabora.com/koike/libcamera
>> (also sent to the mailing list)
>>
>> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
>> Scarlet Chromebook.
> 
> Could you also post media-ctl -p printout e.g. as a reply to the cover
> letter?
> 
> Thanks.
> 

Yes, I had posted in v7 and I forgot to add it in subsequent cover letters:

media-ctl --print-dot -> file available at: http://ix.io/1NIH

root@rockpi:~# media-ctl -p
Media controller API version 5.3.0

Media device information
------------------------
driver          rkisp1
model           rkisp1
serial
bus info        platform: rkisp1
hw revision     0x0
driver version  5.3.0

Device topology
- entity 1: rkisp1-isp-subdev (4 pads, 5 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:SBGGR10_1X10/800x600 field:none
                 crop.bounds:(0,0)/800x600
                 crop:(0,0)/800x600]
                <- "ov5647 4-0036":0 [ENABLED]
        pad1: Sink
                [fmt:FIXED/800x600 field:none]
                <- "rkisp1-input-params":0 [ENABLED]
        pad2: Source
                [fmt:YUYV8_2X8/800x600 field:none
                 crop.bounds:(0,0)/800x600
                 crop:(0,0)/800x600]
                -> "rkisp1_selfpath":0 [ENABLED]
                -> "rkisp1_mainpath":0 [ENABLED]
        pad3: Source
                [fmt:FIXED/800x600 field:none]
                -> "rkisp1-statistics":0 [ENABLED]

- entity 6: rkisp1_mainpath (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "rkisp1-isp-subdev":2 [ENABLED]

- entity 10: rkisp1_selfpath (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "rkisp1-isp-subdev":2 [ENABLED]

- entity 14: rkisp1-statistics (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Sink
                <- "rkisp1-isp-subdev":3 [ENABLED]

- entity 18: rkisp1-input-params (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Source
                -> "rkisp1-isp-subdev":1 [ENABLED]

- entity 22: ov5647 4-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev1
        pad0: Source
                [fmt:SBGGR8_1X8/1280x960 field:none]
                -> "rkisp1-isp-subdev":0 [ENABLED]


Thanks
Helen

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-07-30 18:42 ` [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver Helen Koike
  2019-08-06 18:51   ` Helen Koike
@ 2019-08-08  9:14   ` Sakari Ailus
  2019-08-15  0:58     ` Helen Koike
  2019-08-16  0:13   ` Laurent Pinchart
  2 siblings, 1 reply; 76+ messages in thread
From: Sakari Ailus @ 2019-08-08  9:14 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add the subdev driver for rockchip isp1.
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> [fixed unknown entity type / switched to PIXEL_RATE]
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - fixed warning because of unknown entity type
> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> and default values
> - fix typo riksp1/rkisp1
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now. As a consequence, remove the
> hack in mipidphy_g_mbus_config() where information from the sensor was
> being propagated through the topology.
> - From the old dphy:
>         * cache get_remote_sensor() in s_stream
>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> - Replace stream state with a boolean
> - code styling and checkpatch fixes
> - fix stop_stream (return after calling stop, do not reenable the stream)
> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> - s/intput/input
> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> reused by the capture
> 
>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>  2 files changed, 1397 insertions(+)
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> 
> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> new file mode 100644
> index 000000000000..6d0c0ffb5e03
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> @@ -0,0 +1,1286 @@
> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#include <linux/iopoll.h>
> +#include <linux/phy/phy.h>
> +#include <linux/phy/phy-mipi-dphy.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/videodev2.h>
> +#include <linux/vmalloc.h>
> +#include <media/v4l2-event.h>
> +
> +#include "common.h"
> +#include "regs.h"
> +
> +#define CIF_ISP_INPUT_W_MAX		4032
> +#define CIF_ISP_INPUT_H_MAX		3024
> +#define CIF_ISP_INPUT_W_MIN		32
> +#define CIF_ISP_INPUT_H_MIN		32
> +#define CIF_ISP_OUTPUT_W_MAX		CIF_ISP_INPUT_W_MAX
> +#define CIF_ISP_OUTPUT_H_MAX		CIF_ISP_INPUT_H_MAX
> +#define CIF_ISP_OUTPUT_W_MIN		CIF_ISP_INPUT_W_MIN
> +#define CIF_ISP_OUTPUT_H_MIN		CIF_ISP_INPUT_H_MIN
> +
> +/*
> + * NOTE: MIPI controller and input MUX are also configured in this file,
> + * because ISP Subdev is not only describe ISP submodule(input size,format,
> + * output size, format), but also a virtual route device.
> + */
> +
> +/*
> + * There are many variables named with format/frame in below code,
> + * please see here for their meaning.
> + *
> + * Cropping regions of ISP
> + *
> + * +---------------------------------------------------------+
> + * | Sensor image                                            |
> + * | +---------------------------------------------------+   |
> + * | | ISP_ACQ (for black level)                         |   |
> + * | | in_frm                                            |   |
> + * | | +--------------------------------------------+    |   |
> + * | | |    ISP_OUT                                 |    |   |
> + * | | |    in_crop                                 |    |   |
> + * | | |    +---------------------------------+     |    |   |
> + * | | |    |   ISP_IS                        |     |    |   |
> + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
> + * | | |    +---------------------------------+     |    |   |
> + * | | +--------------------------------------------+    |   |
> + * | +---------------------------------------------------+   |
> + * +---------------------------------------------------------+
> + */
> +
> +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
> +}
> +
> +/* Get sensor by enabled media link */
> +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
> +{
> +	struct media_pad *local, *remote;
> +	struct media_entity *sensor_me;
> +
> +	local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
> +	remote = media_entity_remote_pad(local);
> +	if (!remote) {
> +		v4l2_warn(sd, "No link between isp and sensor\n");
> +		return NULL;
> +	}
> +
> +	sensor_me = media_entity_remote_pad(local)->entity;
> +	return media_entity_to_v4l2_subdev(sensor_me);
> +}
> +
> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
> +					       struct v4l2_subdev *sd)

Indentation.

> +{
> +	struct rkisp1_sensor *sensor;
> +
> +	list_for_each_entry(sensor, &dev->sensors, list)
> +		if (sensor->sd == sd)
> +			return sensor;
> +
> +	return NULL;
> +}
> +
> +/****************  register operations ****************/
> +
> +/*
> + * Image Stabilization.
> + * This should only be called when configuring CIF
> + * or at the frame end interrupt
> + */
> +static void rkisp1_config_ism(struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
> +	u32 val;
> +
> +	writel(0, base + CIF_ISP_IS_RECENTER);
> +	writel(0, base + CIF_ISP_IS_MAX_DX);
> +	writel(0, base + CIF_ISP_IS_MAX_DY);
> +	writel(0, base + CIF_ISP_IS_DISPLACE);
> +	writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
> +	writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
> +	writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
> +	writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
> +
> +	/* IS(Image Stabilization) is always on, working as output crop */
> +	writel(1, base + CIF_ISP_IS_CTRL);
> +	val = readl(base + CIF_ISP_CTRL);
> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD;
> +	writel(val, base + CIF_ISP_CTRL);
> +}
> +
> +/*
> + * configure isp blocks with input format, size......
> + */
> +static int rkisp1_config_isp(struct rkisp1_device *dev)
> +{
> +	u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
> +	struct v4l2_rect *out_crop, *in_crop;
> +	void __iomem *base = dev->base_addr;
> +	struct v4l2_mbus_framefmt *in_frm;
> +	struct ispsd_out_fmt *out_fmt;
> +	struct rkisp1_sensor *sensor;
> +	struct ispsd_in_fmt *in_fmt;
> +
> +	sensor = dev->active_sensor;
> +	in_frm = &dev->isp_sdev.in_frm;
> +	in_fmt = &dev->isp_sdev.in_fmt;
> +	out_fmt = &dev->isp_sdev.out_fmt;
> +	out_crop = &dev->isp_sdev.out_crop;
> +	in_crop = &dev->isp_sdev.in_crop;
> +
> +	if (in_fmt->fmt_type == FMT_BAYER) {
> +		acq_mult = 1;
> +		if (out_fmt->fmt_type == FMT_BAYER) {
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl =
> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
> +			else
> +				isp_ctrl =
> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
> +		} else {
> +			writel(CIF_ISP_DEMOSAIC_TH(0xc),
> +			       base + CIF_ISP_DEMOSAIC);
> +
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
> +			else
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
> +		}
> +	} else if (in_fmt->fmt_type == FMT_YUV) {
> +		acq_mult = 2;
> +		if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +			isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> +		} else {
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
> +			else
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> +
> +		}
> +
> +		irq_mask |= CIF_ISP_DATA_LOSS;
> +	}
> +
> +	/* Set up input acquisition properties */
> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
> +			signal = CIF_ISP_ACQ_PROP_POS_EDGE;
> +	}
> +
> +	if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
> +			signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
> +
> +		if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
> +			signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
> +	}
> +
> +	writel(isp_ctrl, base + CIF_ISP_CTRL);
> +	writel(signal | in_fmt->yuv_seq |
> +	       CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
> +	       CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
> +	writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
> +
> +	/* Acquisition Size */
> +	writel(0, base + CIF_ISP_ACQ_H_OFFS);
> +	writel(0, base + CIF_ISP_ACQ_V_OFFS);
> +	writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
> +	writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
> +
> +	/* ISP Out Area */
> +	writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
> +	writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
> +	writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
> +	writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
> +
> +	/* interrupt mask */
> +	irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
> +		    CIF_ISP_FRAME_IN;
> +	writel(irq_mask, base + CIF_ISP_IMSC);
> +
> +	if (out_fmt->fmt_type == FMT_BAYER)
> +		rkisp1_params_disable_isp(&dev->params_vdev);
> +	else
> +		rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
> +					    dev->isp_sdev.quantization);
> +
> +	return 0;
> +}
> +
> +static int rkisp1_config_dvp(struct rkisp1_device *dev)
> +{
> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> +	void __iomem *base = dev->base_addr;
> +	u32 val, input_sel;
> +
> +	switch (in_fmt->bus_width) {
> +	case 8:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
> +		break;
> +	case 10:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
> +		break;
> +	case 12:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
> +		break;
> +	default:
> +		v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
> +		return -EINVAL;
> +	}
> +
> +	val = readl(base + CIF_ISP_ACQ_PROP);
> +	writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
> +
> +	return 0;
> +}
> +
> +static int rkisp1_config_mipi(struct rkisp1_device *dev)
> +{
> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	void __iomem *base = dev->base_addr;
> +	unsigned int lanes;
> +	u32 mipi_ctrl;
> +
> +	/*
> +	 * sensor->mbus is set in isp or d-phy notifier_bound function
> +	 */
> +	switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
> +	case V4L2_MBUS_CSI2_4_LANE:
> +		lanes = 4;
> +		break;
> +	case V4L2_MBUS_CSI2_3_LANE:
> +		lanes = 3;
> +		break;
> +	case V4L2_MBUS_CSI2_2_LANE:
> +		lanes = 2;
> +		break;
> +	case V4L2_MBUS_CSI2_1_LANE:
> +		lanes = 1;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
> +		    CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
> +		    CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
> +		    CIF_MIPI_CTRL_CLOCKLANE_ENA;
> +
> +	writel(mipi_ctrl, base + CIF_MIPI_CTRL);
> +
> +	/* Configure Data Type and Virtual Channel */
> +	writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
> +	       base + CIF_MIPI_IMG_DATA_SEL);
> +
> +	/* Clear MIPI interrupts */
> +	writel(~0, base + CIF_MIPI_ICR);
> +	/*
> +	 * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
> +	 * isp bus may be dead when switch isp.
> +	 */
> +	writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
> +	       CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
> +	       base + CIF_MIPI_IMSC);
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
> +		 "  MIPI_IMG_DATA_SEL 0x%08x\n"
> +		 "  MIPI_STATUS 0x%08x\n"
> +		 "  MIPI_IMSC 0x%08x\n",
> +		 readl(base + CIF_MIPI_CTRL),
> +		 readl(base + CIF_MIPI_IMG_DATA_SEL),
> +		 readl(base + CIF_MIPI_STATUS),
> +		 readl(base + CIF_MIPI_IMSC));
> +
> +	return 0;
> +}
> +
> +/* Configure MUX */
> +static int rkisp1_config_path(struct rkisp1_device *dev)
> +{
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
> +	int ret = 0;
> +
> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		ret = rkisp1_config_dvp(dev);
> +		dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
> +	} else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +		ret = rkisp1_config_mipi(dev);
> +		dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
> +	}
> +
> +	writel(dpcl, dev->base_addr + CIF_VI_DPCL);
> +
> +	return ret;
> +}
> +
> +/* Hareware configure Entry */
> +static int rkisp1_config_cif(struct rkisp1_device *dev)
> +{
> +	u32 cif_id;
> +	int ret;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	cif_id = readl(dev->base_addr + CIF_VI_ID);
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
> +
> +	ret = rkisp1_config_isp(dev);
> +	if (ret < 0)
> +		return ret;
> +	ret = rkisp1_config_path(dev);
> +	if (ret < 0)
> +		return ret;
> +	rkisp1_config_ism(dev);
> +
> +	return 0;
> +}
> +
> +/* Mess register operations to stop isp */
> +static int rkisp1_isp_stop(struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	/*
> +	 * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
> +	 * Stop ISP(isp) ->wait for ISP isp off
> +	 */
> +	/* stop and clear MI, MIPI, and ISP interrupts */
> +	writel(0, base + CIF_MIPI_IMSC);
> +	writel(~0, base + CIF_MIPI_ICR);
> +
> +	writel(0, base + CIF_ISP_IMSC);
> +	writel(~0, base + CIF_ISP_ICR);
> +
> +	writel(0, base + CIF_MI_IMSC);
> +	writel(~0, base + CIF_MI_ICR);
> +	val = readl(base + CIF_MIPI_CTRL);
> +	writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
> +	/* stop ISP */
> +	val = readl(base + CIF_ISP_CTRL);
> +	val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
> +	writel(val, base + CIF_ISP_CTRL);
> +
> +	val = readl(base + CIF_ISP_CTRL);
> +	writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
> +
> +	readx_poll_timeout(readl, base + CIF_ISP_RIS,
> +			   val, val & CIF_ISP_OFF, 20, 100);
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		"streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming,
> +		 readl(base + CIF_MI_CTRL),
> +		 readl(base + CIF_ISP_CTRL),
> +		 readl(base + CIF_MIPI_CTRL));
> +
> +	writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
> +	writel(0x0, base + CIF_IRCL);
> +
> +	return 0;
> +}
> +
> +/* Mess register operations to start isp */
> +static int rkisp1_isp_start(struct rkisp1_device *dev)
> +{
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	/* Activate MIPI */
> +	if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +		val = readl(base + CIF_MIPI_CTRL);
> +		writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
> +	}
> +	/* Activate ISP */
> +	val = readl(base + CIF_ISP_CTRL);
> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
> +	       CIF_ISP_CTRL_ISP_INFORM_ENABLE;
> +	writel(val, base + CIF_ISP_CTRL);
> +
> +	/* XXX: Is the 1000us too long?
> +	 * CIF spec says to wait for sufficient time after enabling
> +	 * the MIPI interface and before starting the sensor output.
> +	 */
> +	usleep_range(1000, 1200);
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
> +		 "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming,
> +		 readl(base + CIF_MI_CTRL),
> +		 readl(base + CIF_ISP_CTRL),
> +		 readl(base + CIF_MIPI_CTRL));
> +
> +	return 0;
> +}
> +
> +static void rkisp1_config_clk(struct rkisp1_device *dev)
> +{
> +	u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
> +		  CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
> +		  CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
> +
> +	writel(val, dev->base_addr + CIF_ICCL);
> +}
> +
> +/***************************** isp sub-devs *******************************/
> +
> +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
> +	{
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCBYCR,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_YVYU8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCRYCB,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_UYVY8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CBYCRY,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_VYUY8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CRYCBY,
> +		.bus_width	= 16,
> +	},
> +};
> +
> +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
> +	{
> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_2X8,
> +		.fmt_type	= FMT_YUV,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	},
> +};
> +
> +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
> +{
> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);

I think it'd be nicer to just use ARRAY_SIZE(...) in the condition. Same
below.

> +	const struct ispsd_in_fmt *fmt;
> +
> +	for (i = 0; i < array_size; i++) {
> +		fmt = &rkisp1_isp_input_formats[i];
> +		if (fmt->mbus_code == mbus_code)
> +			return fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
> +{
> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
> +	const struct ispsd_out_fmt *fmt;
> +
> +	for (i = 0; i < array_size; i++) {
> +		fmt = &rkisp1_isp_output_formats[i];
> +		if (fmt->mbus_code == mbus_code)
> +			return fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
> +					struct v4l2_subdev_pad_config *cfg,
> +					struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	unsigned int i = code->index;
> +
> +	if ((code->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> +		if (i > 0)
> +			return -EINVAL;
> +		code->code = MEDIA_BUS_FMT_FIXED;
> +		return 0;
> +	}
> +
> +	if (code->pad == RKISP1_ISP_PAD_SINK) {
> +		if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
> +			return -EINVAL;
> +		code->code = rkisp1_isp_input_formats[i].mbus_code;
> +	} else {
> +		if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
> +			return -EINVAL;
> +		code->code = rkisp1_isp_output_formats[i].mbus_code;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
> +				     struct v4l2_subdev_pad_config *cfg)
> +{
> +	struct v4l2_rect *mf_in_crop, *mf_out_crop;
> +	struct v4l2_mbus_framefmt *mf_in, *mf_out;
> +
> +	mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
> +	mf_in->width = RKISP1_DEFAULT_WIDTH;
> +	mf_in->height = RKISP1_DEFAULT_HEIGHT;
> +	mf_in->field = V4L2_FIELD_NONE;
> +	mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
> +
> +	mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
> +	mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
> +	mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
> +	mf_in_crop->left = 0;
> +	mf_in_crop->top = 0;
> +
> +	mf_out = v4l2_subdev_get_try_format(sd, cfg,
> +					    RKISP1_ISP_PAD_SOURCE_PATH);
> +	*mf_out = *mf_in;
> +	mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
> +	mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +
> +	mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
> +					       RKISP1_ISP_PAD_SOURCE_PATH);
> +	*mf_out_crop = *mf_in_crop;
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_format *fmt)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> +
> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
> +		/*
> +		 * NOTE: setting a format here doesn't make much sense
> +		 * but v4l2-compliance complains
> +		 */
> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> +		fmt->format.field = V4L2_FIELD_NONE;
> +		return 0;
> +	}
> +
> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> +		fmt->format = *mf;
> +		return 0;
> +	}
> +
> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> +		*mf = isp_sd->in_frm;
> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		/* format of source pad */
> +		*mf = isp_sd->in_frm;
> +		mf->code = isp_sd->out_fmt.mbus_code;
> +		/* window size of source pad */
> +		mf->width = isp_sd->out_crop.width;
> +		mf->height = isp_sd->out_crop.height;
> +		mf->quantization = isp_sd->quantization;
> +	}
> +	mf->field = V4L2_FIELD_NONE;
> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
> +				  unsigned int pad,
> +				  struct v4l2_mbus_framefmt *fmt)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> +	const struct ispsd_out_fmt *out_fmt;
> +	const struct ispsd_in_fmt *in_fmt;
> +
> +	switch (pad) {
> +	case RKISP1_ISP_PAD_SINK:
> +		in_fmt = find_in_fmt(fmt->code);
> +		if (in_fmt)
> +			fmt->code = in_fmt->mbus_code;
> +		else
> +			fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
> +		fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
> +				      CIF_ISP_INPUT_W_MAX);
> +		fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
> +				      CIF_ISP_INPUT_H_MAX);
> +		break;
> +	case RKISP1_ISP_PAD_SOURCE_PATH:
> +		out_fmt = find_out_fmt(fmt->code);
> +		if (out_fmt)
> +			fmt->code = out_fmt->mbus_code;
> +		else
> +			fmt->code = rkisp1_isp_output_formats[0].mbus_code;
> +		/* window size is set in s_selection */
> +		fmt->width  = isp_sd->out_crop.width;
> +		fmt->height = isp_sd->out_crop.height;
> +		/* full range by default */
> +		if (!fmt->quantization)
> +			fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +		break;
> +	}
> +
> +	fmt->field = V4L2_FIELD_NONE;
> +}
> +
> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_format *fmt)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> +

Note that for sub-device nodes, the driver is itself responsible for
serialising the access to its data structures.

> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
> +		return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
> +
> +	rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
> +
> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		struct v4l2_mbus_framefmt *try_mf;
> +
> +		try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> +		*try_mf = *mf;
> +		return 0;
> +	}
> +
> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> +		const struct ispsd_in_fmt *in_fmt;
> +
> +		in_fmt = find_in_fmt(mf->code);
> +		isp_sd->in_fmt = *in_fmt;
> +		isp_sd->in_frm = *mf;
> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		const struct ispsd_out_fmt *out_fmt;
> +
> +		/* Ignore width/height */
> +		out_fmt = find_out_fmt(mf->code);
> +		isp_sd->out_fmt = *out_fmt;
> +		/*
> +		 * It is quantization for output,
> +		 * isp use bt601 limit-range in internal
> +		 */
> +		isp_sd->quantization = mf->quantization;
> +	}
> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
> +	struct v4l2_rect in_crop = isp_sd->in_crop;
> +	struct v4l2_rect *input = &sel->r;
> +
> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		in_frm = *v4l2_subdev_get_try_format(sd, cfg,
> +						     RKISP1_ISP_PAD_SINK);
> +		in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
> +						    RKISP1_ISP_PAD_SINK);
> +	}
> +
> +	input->left = ALIGN(input->left, 2);
> +	input->width = ALIGN(input->width, 2);
> +
> +	if (sel->pad == RKISP1_ISP_PAD_SINK) {
> +		input->left = clamp_t(u32, input->left, 0, in_frm.width);
> +		input->top = clamp_t(u32, input->top, 0, in_frm.height);
> +		input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
> +				       in_frm.width - input->left);
> +		input->height = clamp_t(u32, input->height,
> +					CIF_ISP_INPUT_H_MIN,
> +					in_frm.height - input->top);
> +	} else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		input->left = clamp_t(u32, input->left, 0, in_crop.width);
> +		input->top = clamp_t(u32, input->top, 0, in_crop.height);
> +		input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
> +				       in_crop.width - input->left);
> +		input->height = clamp_t(u32, input->height,
> +					CIF_ISP_OUTPUT_H_MIN,
> +					in_crop.height - input->top);
> +	}
> +}
> +
> +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_pad_config *cfg,
> +				       struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt *frm;
> +	struct v4l2_rect *rect;
> +
> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> +	    sel->pad != RKISP1_ISP_PAD_SINK)
> +		return -EINVAL;
> +
> +	switch (sel->target) {
> +	case V4L2_SEL_TGT_CROP_BOUNDS:
> +		if (sel->pad == RKISP1_ISP_PAD_SINK) {
> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +				frm = v4l2_subdev_get_try_format(sd, cfg,
> +								 sel->pad);
> +			else
> +				frm = &isp_sd->in_frm;
> +
> +			sel->r.height = frm->height;
> +			sel->r.width = frm->width;
> +			sel->r.left = 0;
> +			sel->r.top = 0;
> +		} else {
> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +				rect = v4l2_subdev_get_try_crop(sd, cfg,
> +							RKISP1_ISP_PAD_SINK);
> +			else
> +				rect = &isp_sd->in_crop;
> +			sel->r = *rect;
> +		}
> +		break;
> +	case V4L2_SEL_TGT_CROP:
> +		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +			rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> +		else if (sel->pad == RKISP1_ISP_PAD_SINK)
> +			rect = &isp_sd->in_crop;
> +		else
> +			rect = &isp_sd->out_crop;
> +		sel->r = *rect;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_pad_config *cfg,
> +				       struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct rkisp1_device *dev = sd_to_isp_dev(sd);
> +
> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> +	    sel->pad != RKISP1_ISP_PAD_SINK)
> +		return -EINVAL;
> +	if (sel->target != V4L2_SEL_TGT_CROP)
> +		return -EINVAL;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
> +		 sel->r.left, sel->r.top, sel->r.width, sel->r.height);
> +	rkisp1_isp_sd_try_crop(sd, cfg, sel);
> +
> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		struct v4l2_rect *try_sel =
> +			v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> +
> +		*try_sel = sel->r;
> +		return 0;
> +	}
> +
> +	if (sel->pad == RKISP1_ISP_PAD_SINK)
> +		isp_sd->in_crop = sel->r;
> +	else
> +		isp_sd->out_crop = sel->r;
> +
> +	return 0;
> +}
> +
> +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
> +				    struct rkisp1_sensor *sensor)
> +{
> +	union phy_configure_opts opts = { 0 };
> +	struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
> +	struct v4l2_ctrl *pixel_rate;
> +	s64 pixel_clock;
> +
> +	pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
> +				    V4L2_CID_PIXEL_RATE);
> +	if (!pixel_rate) {
> +		v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
> +		return -EPIPE;
> +	}
> +
> +	pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
> +	if (!pixel_clock) {
> +		v4l2_err(sensor->sd, "Invalid pixel rate value\n");
> +		return -EINVAL;
> +	}
> +
> +	phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
> +					 sensor->lanes, cfg);
> +	phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
> +	phy_configure(sensor->dphy, &opts);
> +	phy_power_on(sensor->dphy);
> +
> +	return 0;
> +}
> +
> +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
> +{
> +	phy_power_off(sensor->dphy);
> +}
> +
> +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct v4l2_subdev *sensor_sd;
> +	int ret = 0;
> +
> +	if (!on) {
> +		ret = rkisp1_isp_stop(isp_dev);
> +		if (ret < 0)
> +			return ret;
> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> +		return 0;
> +	}
> +
> +	sensor_sd = get_remote_sensor(sd);
> +	if (!sensor_sd)
> +		return -ENODEV;
> +
> +	isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
> +	if (!isp_dev->active_sensor)
> +		return -ENODEV;
> +
> +	atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
> +	ret = rkisp1_config_cif(isp_dev);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* TODO: support other interfaces */
> +	if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
> +		return -EINVAL;
> +
> +	ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
> +				       isp_dev->active_sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = rkisp1_isp_start(isp_dev);
> +	if (ret)
> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> +
> +	return ret;
> +}
> +
> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)

If you support runtime PM, you shouldn't implement the s_power op.

You'll still need to call s_power on external subdevs though.

> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	int ret;
> +
> +	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> +
> +	if (on) {
> +		ret = pm_runtime_get_sync(isp_dev->dev);
> +		if (ret < 0)
> +			return ret;
> +
> +		rkisp1_config_clk(isp_dev);
> +	} else {
> +		ret = pm_runtime_put(isp_dev->dev);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_subdev_link_validate(struct media_link *link)
> +{
> +	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)

Is this test correct? The source is the source end of the link, i.e. the
video node.

How about the links that end in a video node?

> +		return 0;
> +
> +	return v4l2_subdev_link_validate(link);
> +}
> +
> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
> +					struct media_link *link,
> +					struct v4l2_subdev_format *source_fmt,
> +					struct v4l2_subdev_format *sink_fmt)
> +{
> +	if (source_fmt->format.code != sink_fmt->format.code)
> +		return -EINVAL;
> +
> +	/* Crop is available */
> +	if (source_fmt->format.width < sink_fmt->format.width ||
> +	    source_fmt->format.height < sink_fmt->format.height)
> +		return -EINVAL;
> +

Could you use v4l2_subdev_link_validate_default()?

> +	return 0;
> +}
> +
> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> +{
> +	struct v4l2_event event = {
> +		.type = V4L2_EVENT_FRAME_SYNC,
> +		.u.frame_sync.frame_sequence =
> +			atomic_inc_return(&isp->frm_sync_seq) - 1,
> +	};
> +	v4l2_event_queue(isp->sd.devnode, &event);
> +}
> +
> +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> +				  struct v4l2_event_subscription *sub)
> +{
> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> +		return -EINVAL;
> +
> +	/* Line number. For now only zero accepted. */
> +	if (sub->id != 0)
> +		return -EINVAL;
> +
> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> +}
> +
> +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
> +	.enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
> +	.get_selection = rkisp1_isp_sd_get_selection,
> +	.set_selection = rkisp1_isp_sd_set_selection,
> +	.init_cfg = rkisp1_isp_sd_init_config,
> +	.get_fmt = rkisp1_isp_sd_get_fmt,
> +	.set_fmt = rkisp1_isp_sd_set_fmt,
> +	.link_validate = rkisp1_subdev_fmt_link_validate,
> +};
> +
> +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
> +	.link_validate = rkisp1_subdev_link_validate,
> +};
> +
> +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
> +	.s_stream = rkisp1_isp_sd_s_stream,
> +};
> +
> +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
> +	.subscribe_event = rkisp1_isp_sd_subs_evt,
> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> +	.s_power = rkisp1_isp_sd_s_power,
> +};
> +
> +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {

const

> +	.core = &rkisp1_isp_core_ops,
> +	.video = &rkisp1_isp_sd_video_ops,
> +	.pad = &rkisp1_isp_sd_pad_ops,
> +};
> +
> +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
> +{
> +	struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
> +	struct v4l2_rect *in_crop = &isp_sd->in_crop;
> +	struct v4l2_rect *out_crop = &isp_sd->out_crop;
> +	struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
> +	struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
> +
> +	*in_fmt = rkisp1_isp_input_formats[0];
> +	in_frm->width = RKISP1_DEFAULT_WIDTH;
> +	in_frm->height = RKISP1_DEFAULT_HEIGHT;
> +	in_frm->code = in_fmt->mbus_code;
> +
> +	in_crop->width = in_frm->width;
> +	in_crop->height = in_frm->height;
> +	in_crop->left = 0;
> +	in_crop->top = 0;
> +
> +	/* propagate to source */
> +	*out_crop = *in_crop;
> +	*out_fmt = rkisp1_isp_output_formats[0];
> +	isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +}
> +
> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> +			       struct v4l2_device *v4l2_dev)
> +{
> +	struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
> +	struct v4l2_subdev *sd = &isp_sdev->sd;
> +	int ret;
> +
> +	v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	sd->entity.ops = &rkisp1_isp_sd_media_ops;
> +	snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");

strscpy()

> +
> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> +	ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
> +				     isp_sdev->pads);
> +	if (ret < 0)
> +		return ret;
> +
> +	sd->owner = THIS_MODULE;
> +	v4l2_set_subdevdata(sd, isp_dev);
> +
> +	sd->grp_id = GRP_ID_ISP;
> +	ret = v4l2_device_register_subdev(v4l2_dev, sd);
> +	if (ret < 0) {
> +		v4l2_err(sd, "Failed to register isp subdev\n");
> +		goto err_cleanup_media_entity;
> +	}
> +
> +	rkisp1_isp_sd_init_default_fmt(isp_sdev);
> +
> +	return 0;

A newline would be nice here.

> +err_cleanup_media_entity:
> +	media_entity_cleanup(&sd->entity);

And here.

> +	return ret;
> +}
> +
> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
> +{
> +	struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
> +
> +	v4l2_device_unregister_subdev(sd);
> +	media_entity_cleanup(&sd->entity);
> +}
> +
> +/****************  Interrupter Handler ****************/
> +
> +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
> +{
> +	struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	writel(~0, base + CIF_MIPI_ICR);
> +
> +	/*
> +	 * Disable DPHY errctrl interrupt, because this dphy
> +	 * erctrl signal is asserted until the next changes
> +	 * of line state. This time is may be too long and cpu
> +	 * is hold in this interrupt.
> +	 */
> +	if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
> +		val = readl(base + CIF_MIPI_IMSC);
> +		writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
> +		dev->isp_sdev.dphy_errctrl_disabled = true;
> +	}
> +
> +	/*
> +	 * Enable DPHY errctrl interrupt again, if mipi have receive
> +	 * the whole frame without any error.
> +	 */
> +	if (mis == CIF_MIPI_FRAME_END) {
> +		/*
> +		 * Enable DPHY errctrl interrupt again, if mipi have receive
> +		 * the whole frame without any error.
> +		 */
> +		if (dev->isp_sdev.dphy_errctrl_disabled) {
> +			val = readl(base + CIF_MIPI_IMSC);
> +			val |= CIF_MIPI_ERR_CTRL(0x0f);
> +			writel(val, base + CIF_MIPI_IMSC);
> +			dev->isp_sdev.dphy_errctrl_disabled = false;
> +		}
> +	} else {
> +		v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
> +	}
> +}
> +
> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	unsigned int isp_mis_tmp = 0;
> +	unsigned int isp_err = 0;
> +
> +	/* start edge of v_sync */
> +	if (isp_mis & CIF_ISP_V_START) {
> +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> +
> +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_V_START)
> +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> +				 isp_mis_tmp);
> +	}
> +
> +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {

Extra parentheses.

> +		/* Clear pic_size_error */
> +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> +		isp_err = readl(base + CIF_ISP_ERR);
> +		v4l2_err(&dev->v4l2_dev,
> +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> +		writel(isp_err, base + CIF_ISP_ERR_CLR);
> +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> +		/* Clear data_loss */
> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> +	}
> +
> +	/* sampled input frame is complete */
> +	if (isp_mis & CIF_ISP_FRAME_IN) {
> +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> +				 isp_mis_tmp);
> +	}
> +
> +	/* frame was completely put out */
> +	if (isp_mis & CIF_ISP_FRAME) {
> +		u32 isp_ris = 0;
> +		/* Clear Frame In (ISP) */
> +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_FRAME)
> +			v4l2_err(&dev->v4l2_dev,
> +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> +
> +		isp_ris = readl(base + CIF_ISP_RIS);
> +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> +	}
> +
> +	/*
> +	 * Then update changed configs. Some of them involve
> +	 * lot of register writes. Do those only one per frame.
> +	 * Do the updates in the order of the processing flow.
> +	 */
> +	rkisp1_params_isr(&dev->params_vdev, isp_mis);
> +}
> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
> new file mode 100644
> index 000000000000..b0366e354ec2
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
> @@ -0,0 +1,111 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#ifndef _RKISP1_H
> +#define _RKISP1_H
> +
> +#include <linux/platform_device.h>
> +#include <media/v4l2-fwnode.h>
> +
> +#include "common.h"
> +
> +struct rkisp1_stream;
> +
> +/*
> + * struct ispsd_in_fmt - ISP intput-pad format
> + *
> + * Translate mbus_code to hardware format values
> + *
> + * @bus_width: used for parallel
> + */
> +struct ispsd_in_fmt {
> +	u32 mbus_code;
> +	u8 fmt_type;
> +	u32 mipi_dt;
> +	u32 yuv_seq;
> +	enum rkisp1_fmt_raw_pat_type bayer_pat;
> +	u8 bus_width;
> +};
> +
> +struct ispsd_out_fmt {
> +	u32 mbus_code;
> +	u8 fmt_type;
> +};
> +
> +struct rkisp1_ie_config {
> +	unsigned int effect;
> +};
> +
> +enum rkisp1_isp_pad {
> +	RKISP1_ISP_PAD_SINK,
> +	RKISP1_ISP_PAD_SINK_PARAMS,
> +	RKISP1_ISP_PAD_SOURCE_PATH,
> +	RKISP1_ISP_PAD_SOURCE_STATS,
> +	RKISP1_ISP_PAD_MAX
> +};
> +
> +/*
> + * struct rkisp1_isp_subdev - ISP sub-device
> + *
> + * See Cropping regions of ISP in rkisp1.c for details
> + * @in_frm: input size, don't have to be equal to sensor size
> + * @in_fmt: input format
> + * @in_crop: crop for sink pad
> + * @out_fmt: output format
> + * @out_crop: output size
> + *
> + * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)
> + * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
> + * @quantization: output quantization
> + */
> +struct rkisp1_isp_subdev {
> +	struct v4l2_subdev sd;
> +	struct media_pad pads[RKISP1_ISP_PAD_MAX];
> +	struct v4l2_ctrl_handler ctrl_handler;
> +	struct v4l2_mbus_framefmt in_frm;
> +	struct ispsd_in_fmt in_fmt;
> +	struct v4l2_rect in_crop;
> +	struct ispsd_out_fmt out_fmt;
> +	struct v4l2_rect out_crop;
> +	bool dphy_errctrl_disabled;
> +	atomic_t frm_sync_seq;
> +	enum v4l2_quantization quantization;
> +};
> +
> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> +			       struct v4l2_device *v4l2_dev);
> +
> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
> +
> +void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
> +
> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
> +
> +static inline
> +struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->out_fmt;
> +}
> +
> +static inline
> +struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->in_fmt;
> +}
> +
> +static inline
> +struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->out_crop;
> +}

I'd just use the struct fields directly in the code as it's easier to
figure out which field in the struct is being accessed.

> +
> +static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct rkisp1_isp_subdev, sd);
> +}
> +
> +#endif /* _RKISP1_H */

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 06/14] media: rkisp1: add ISP1 statistics driver
  2019-07-30 18:42 ` [PATCH v8 06/14] media: rkisp1: add ISP1 statistics driver Helen Koike
@ 2019-08-08  9:37   ` Sakari Ailus
  0 siblings, 0 replies; 76+ messages in thread
From: Sakari Ailus @ 2019-08-08  9:37 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Tue, Jul 30, 2019 at 03:42:48PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add the capture video driver for rockchip isp1 statistics block.
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - s/strlcpy/strscpy
> - sort out the locks in isp stats
> - code styling and checkpatch fixes
> 
>  .../media/platform/rockchip/isp1/isp_stats.c  | 508 ++++++++++++++++++
>  .../media/platform/rockchip/isp1/isp_stats.h  |  60 +++
>  2 files changed, 568 insertions(+)
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h
> 
> diff --git a/drivers/media/platform/rockchip/isp1/isp_stats.c b/drivers/media/platform/rockchip/isp1/isp_stats.c
> new file mode 100644
> index 000000000000..01d947867c70
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/isp_stats.c
> @@ -0,0 +1,508 @@
> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#include <media/v4l2-common.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/videobuf2-core.h>
> +#include <media/videobuf2-vmalloc.h>	/* for ISP statistics */
> +
> +#include "dev.h"
> +#include "regs.h"
> +
> +#define RKISP1_ISP_STATS_REQ_BUFS_MIN 2
> +#define RKISP1_ISP_STATS_REQ_BUFS_MAX 8
> +
> +static int rkisp1_stats_enum_fmt_meta_cap(struct file *file, void *priv,
> +					  struct v4l2_fmtdesc *f)
> +{
> +	struct video_device *video = video_devdata(file);
> +	struct rkisp1_isp_stats_vdev *stats_vdev = video_get_drvdata(video);
> +
> +	if (f->index > 0 || f->type != video->queue->type)
> +		return -EINVAL;
> +
> +	f->pixelformat = stats_vdev->vdev_fmt.fmt.meta.dataformat;
> +	return 0;
> +}
> +
> +static int rkisp1_stats_g_fmt_meta_cap(struct file *file, void *priv,
> +				       struct v4l2_format *f)
> +{
> +	struct video_device *video = video_devdata(file);
> +	struct rkisp1_isp_stats_vdev *stats_vdev = video_get_drvdata(video);
> +	struct v4l2_meta_format *meta = &f->fmt.meta;
> +
> +	if (f->type != video->queue->type)
> +		return -EINVAL;
> +
> +	memset(meta, 0, sizeof(*meta));
> +	meta->dataformat = stats_vdev->vdev_fmt.fmt.meta.dataformat;
> +	meta->buffersize = stats_vdev->vdev_fmt.fmt.meta.buffersize;
> +
> +	return 0;
> +}
> +
> +static int rkisp1_stats_querycap(struct file *file,
> +				 void *priv, struct v4l2_capability *cap)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +
> +	strscpy(cap->driver, DRIVER_NAME, sizeof(cap->driver));
> +	strscpy(cap->card, vdev->name, sizeof(cap->card));
> +	strscpy(cap->bus_info, "platform: " DRIVER_NAME, sizeof(cap->bus_info));
> +
> +	return 0;
> +}
> +
> +/* ISP video device IOCTLs */
> +static const struct v4l2_ioctl_ops rkisp1_stats_ioctl = {
> +	.vidioc_reqbufs = vb2_ioctl_reqbufs,
> +	.vidioc_querybuf = vb2_ioctl_querybuf,
> +	.vidioc_create_bufs = vb2_ioctl_create_bufs,
> +	.vidioc_qbuf = vb2_ioctl_qbuf,
> +	.vidioc_dqbuf = vb2_ioctl_dqbuf,
> +	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> +	.vidioc_expbuf = vb2_ioctl_expbuf,
> +	.vidioc_streamon = vb2_ioctl_streamon,
> +	.vidioc_streamoff = vb2_ioctl_streamoff,
> +	.vidioc_enum_fmt_meta_cap = rkisp1_stats_enum_fmt_meta_cap,
> +	.vidioc_g_fmt_meta_cap = rkisp1_stats_g_fmt_meta_cap,
> +	.vidioc_s_fmt_meta_cap = rkisp1_stats_g_fmt_meta_cap,
> +	.vidioc_try_fmt_meta_cap = rkisp1_stats_g_fmt_meta_cap,
> +	.vidioc_querycap = rkisp1_stats_querycap,
> +	.vidioc_subscribe_event = v4l2_ctrl_subscribe_event,
> +	.vidioc_unsubscribe_event = v4l2_event_unsubscribe,

Do you have controls on the video nodes? If not, you can remove the two.

> +};
> +
> +struct v4l2_file_operations rkisp1_stats_fops = {

static const

> +	.mmap = vb2_fop_mmap,
> +	.unlocked_ioctl = video_ioctl2,
> +	.poll = vb2_fop_poll,
> +	.open = v4l2_fh_open,
> +	.release = vb2_fop_release
> +};
> +
> +static int rkisp1_stats_vb2_queue_setup(struct vb2_queue *vq,
> +					unsigned int *num_buffers,
> +					unsigned int *num_planes,
> +					unsigned int sizes[],
> +					struct device *alloc_devs[])
> +{
> +	struct rkisp1_isp_stats_vdev *stats_vdev = vq->drv_priv;
> +
> +	*num_planes = 1;
> +
> +	*num_buffers = clamp_t(u32, *num_buffers, RKISP1_ISP_STATS_REQ_BUFS_MIN,
> +			       RKISP1_ISP_STATS_REQ_BUFS_MAX);
> +
> +	sizes[0] = sizeof(struct rkisp1_stat_buffer);
> +
> +	INIT_LIST_HEAD(&stats_vdev->stat);
> +
> +	return 0;
> +}
> +
> +static void rkisp1_stats_vb2_buf_queue(struct vb2_buffer *vb)
> +{
> +	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> +	struct rkisp1_buffer *stats_buf = to_rkisp1_buffer(vbuf);
> +	struct vb2_queue *vq = vb->vb2_queue;
> +	struct rkisp1_isp_stats_vdev *stats_dev = vq->drv_priv;
> +
> +	stats_buf->vaddr[0] = vb2_plane_vaddr(vb, 0);
> +
> +	mutex_lock(&stats_dev->wq_lock);
> +	list_add_tail(&stats_buf->queue, &stats_dev->stat);
> +	mutex_unlock(&stats_dev->wq_lock);
> +}
> +
> +static int rkisp1_stats_vb2_buf_prepare(struct vb2_buffer *vb)
> +{
> +	if (vb2_plane_size(vb, 0) < sizeof(struct rkisp1_stat_buffer))
> +		return -EINVAL;
> +
> +	vb2_set_plane_payload(vb, 0, sizeof(struct rkisp1_stat_buffer));
> +
> +	return 0;
> +}
> +
> +static void rkisp1_stats_vb2_stop_streaming(struct vb2_queue *vq)
> +{
> +	struct rkisp1_isp_stats_vdev *stats_vdev = vq->drv_priv;
> +	struct rkisp1_buffer *buf;
> +	unsigned long flags;
> +	unsigned int i;
> +
> +	/* Make sure no new work queued in isr before draining wq */
> +	spin_lock_irqsave(&stats_vdev->irq_lock, flags);
> +	stats_vdev->streamon = false;
> +	spin_unlock_irqrestore(&stats_vdev->irq_lock, flags);
> +
> +	drain_workqueue(stats_vdev->readout_wq);
> +
> +	mutex_lock(&stats_vdev->wq_lock);
> +	for (i = 0; i < RKISP1_ISP_STATS_REQ_BUFS_MAX; i++) {
> +		if (list_empty(&stats_vdev->stat))
> +			break;
> +		buf = list_first_entry(&stats_vdev->stat,
> +				       struct rkisp1_buffer, queue);
> +		list_del(&buf->queue);
> +		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
> +	}
> +	mutex_unlock(&stats_vdev->wq_lock);
> +}
> +
> +static int
> +rkisp1_stats_vb2_start_streaming(struct vb2_queue *queue,
> +				 unsigned int count)
> +{
> +	struct rkisp1_isp_stats_vdev *stats_vdev = queue->drv_priv;
> +
> +	stats_vdev->streamon = true;
> +
> +	return 0;
> +}
> +
> +static struct vb2_ops rkisp1_stats_vb2_ops = {

const

> +	.queue_setup = rkisp1_stats_vb2_queue_setup,
> +	.buf_queue = rkisp1_stats_vb2_buf_queue,
> +	.buf_prepare = rkisp1_stats_vb2_buf_prepare,
> +	.wait_prepare = vb2_ops_wait_prepare,
> +	.wait_finish = vb2_ops_wait_finish,
> +	.stop_streaming = rkisp1_stats_vb2_stop_streaming,
> +	.start_streaming = rkisp1_stats_vb2_start_streaming,
> +};
> +
> +static int rkisp1_stats_init_vb2_queue(struct vb2_queue *q,
> +				       struct rkisp1_isp_stats_vdev *stats_vdev)
> +{
> +	struct rkisp1_vdev_node *node;
> +
> +	node = queue_to_node(q);
> +
> +	q->type = V4L2_BUF_TYPE_META_CAPTURE;
> +	q->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
> +	q->drv_priv = stats_vdev;
> +	q->ops = &rkisp1_stats_vb2_ops;
> +	q->mem_ops = &vb2_vmalloc_memops;
> +	q->buf_struct_size = sizeof(struct rkisp1_buffer);
> +	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> +	q->lock = &node->vlock;
> +
> +	return vb2_queue_init(q);
> +}
> +
> +static void rkisp1_stats_get_awb_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
> +				      struct rkisp1_stat_buffer *pbuf)
> +{
> +	/* Protect against concurrent access from ISR? */
> +	u32 reg_val;
> +
> +	pbuf->meas_type |= CIFISP_STAT_AWB;
> +	reg_val = readl(stats_vdev->dev->base_addr + CIF_ISP_AWB_WHITE_CNT);
> +	pbuf->params.awb.awb_mean[0].cnt = CIF_ISP_AWB_GET_PIXEL_CNT(reg_val);
> +	reg_val = readl(stats_vdev->dev->base_addr + CIF_ISP_AWB_MEAN);
> +
> +	pbuf->params.awb.awb_mean[0].mean_cr_or_r =
> +		CIF_ISP_AWB_GET_MEAN_CR_R(reg_val);
> +	pbuf->params.awb.awb_mean[0].mean_cb_or_b =
> +		CIF_ISP_AWB_GET_MEAN_CB_B(reg_val);
> +	pbuf->params.awb.awb_mean[0].mean_y_or_g =
> +		CIF_ISP_AWB_GET_MEAN_Y_G(reg_val);
> +}
> +
> +static void rkisp1_stats_get_aec_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
> +				      struct rkisp1_stat_buffer *pbuf)
> +{
> +	void __iomem *addr = stats_vdev->dev->base_addr + CIF_ISP_EXP_MEAN_00;
> +	unsigned int i;
> +
> +	pbuf->meas_type |= CIFISP_STAT_AUTOEXP;
> +	for (i = 0; i < CIFISP_AE_MEAN_MAX; i++)
> +		pbuf->params.ae.exp_mean[i] = (u8)readl(addr + i * 4);
> +}
> +
> +static void rkisp1_stats_get_afc_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
> +				      struct rkisp1_stat_buffer *pbuf)
> +{
> +	void __iomem *base_addr;
> +	struct cifisp_af_stat *af;
> +
> +	pbuf->meas_type = CIFISP_STAT_AFM_FIN;
> +
> +	af = &pbuf->params.af;
> +	base_addr = stats_vdev->dev->base_addr;
> +	af->window[0].sum = readl(base_addr + CIF_ISP_AFM_SUM_A);
> +	af->window[0].lum = readl(base_addr + CIF_ISP_AFM_LUM_A);
> +	af->window[1].sum = readl(base_addr + CIF_ISP_AFM_SUM_B);
> +	af->window[1].lum = readl(base_addr + CIF_ISP_AFM_LUM_B);
> +	af->window[2].sum = readl(base_addr + CIF_ISP_AFM_SUM_C);
> +	af->window[2].lum = readl(base_addr + CIF_ISP_AFM_LUM_C);
> +}
> +
> +static void rkisp1_stats_get_hst_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
> +				      struct rkisp1_stat_buffer *pbuf)
> +{
> +	void __iomem *addr = stats_vdev->dev->base_addr + CIF_ISP_HIST_BIN_0;
> +	unsigned int i;
> +
> +	pbuf->meas_type |= CIFISP_STAT_HIST;
> +	for (i = 0; i < CIFISP_HIST_BIN_N_MAX; i++)
> +		pbuf->params.hist.hist_bins[i] = readl(addr + (i * 4));
> +}
> +
> +static void rkisp1_stats_get_bls_meas(struct rkisp1_isp_stats_vdev *stats_vdev,
> +				      struct rkisp1_stat_buffer *pbuf)
> +{
> +	struct rkisp1_device *dev = stats_vdev->dev;
> +	const struct ispsd_in_fmt *in_fmt =
> +			rkisp1_get_ispsd_in_fmt(&dev->isp_sdev);
> +	void __iomem *base = stats_vdev->dev->base_addr;
> +	struct cifisp_bls_meas_val *bls_val;
> +
> +	bls_val = &pbuf->params.ae.bls_val;
> +	if (in_fmt->bayer_pat == RAW_BGGR) {
> +		bls_val->meas_b = readl(base + CIF_ISP_BLS_A_MEASURED);
> +		bls_val->meas_gb = readl(base + CIF_ISP_BLS_B_MEASURED);
> +		bls_val->meas_gr = readl(base + CIF_ISP_BLS_C_MEASURED);
> +		bls_val->meas_r = readl(base + CIF_ISP_BLS_D_MEASURED);
> +	} else if (in_fmt->bayer_pat == RAW_GBRG) {
> +		bls_val->meas_gb = readl(base + CIF_ISP_BLS_A_MEASURED);
> +		bls_val->meas_b = readl(base + CIF_ISP_BLS_B_MEASURED);
> +		bls_val->meas_r = readl(base + CIF_ISP_BLS_C_MEASURED);
> +		bls_val->meas_gr = readl(base + CIF_ISP_BLS_D_MEASURED);
> +	} else if (in_fmt->bayer_pat == RAW_GRBG) {
> +		bls_val->meas_gr = readl(base + CIF_ISP_BLS_A_MEASURED);
> +		bls_val->meas_r = readl(base + CIF_ISP_BLS_B_MEASURED);
> +		bls_val->meas_b = readl(base + CIF_ISP_BLS_C_MEASURED);
> +		bls_val->meas_gb = readl(base + CIF_ISP_BLS_D_MEASURED);
> +	} else if (in_fmt->bayer_pat == RAW_RGGB) {
> +		bls_val->meas_r = readl(base + CIF_ISP_BLS_A_MEASURED);
> +		bls_val->meas_gr = readl(base + CIF_ISP_BLS_B_MEASURED);
> +		bls_val->meas_gb = readl(base + CIF_ISP_BLS_C_MEASURED);
> +		bls_val->meas_b = readl(base + CIF_ISP_BLS_D_MEASURED);
> +	}
> +}
> +
> +static void
> +rkisp1_stats_send_measurement(struct rkisp1_isp_stats_vdev *stats_vdev,
> +			      struct rkisp1_isp_readout_work *meas_work)
> +{
> +	struct rkisp1_stat_buffer *cur_stat_buf;
> +	struct rkisp1_buffer *cur_buf = NULL;
> +	unsigned int cur_frame_id = -1;
> +
> +	cur_frame_id = atomic_read(&stats_vdev->dev->isp_sdev.frm_sync_seq) - 1;
> +	if (cur_frame_id != meas_work->frame_id) {
> +		v4l2_warn(stats_vdev->vnode.vdev.v4l2_dev,
> +			  "Measurement late(%d, %d)\n",
> +			  cur_frame_id, meas_work->frame_id);
> +		cur_frame_id = meas_work->frame_id;
> +	}
> +
> +	mutex_lock(&stats_vdev->wq_lock);
> +	/* get one empty buffer */
> +	if (!list_empty(&stats_vdev->stat)) {
> +		cur_buf = list_first_entry(&stats_vdev->stat,
> +					   struct rkisp1_buffer, queue);
> +		list_del(&cur_buf->queue);
> +	}
> +	mutex_unlock(&stats_vdev->wq_lock);
> +
> +	if (!cur_buf)
> +		return;
> +
> +	cur_stat_buf =
> +		(struct rkisp1_stat_buffer *)(cur_buf->vaddr[0]);
> +
> +	if (meas_work->isp_ris & CIF_ISP_AWB_DONE) {
> +		rkisp1_stats_get_awb_meas(stats_vdev, cur_stat_buf);
> +		cur_stat_buf->meas_type |= CIFISP_STAT_AWB;
> +	}
> +
> +	if (meas_work->isp_ris & CIF_ISP_AFM_FIN) {
> +		rkisp1_stats_get_afc_meas(stats_vdev, cur_stat_buf);
> +		cur_stat_buf->meas_type |= CIFISP_STAT_AFM_FIN;
> +	}
> +
> +	if (meas_work->isp_ris & CIF_ISP_EXP_END) {
> +		rkisp1_stats_get_aec_meas(stats_vdev, cur_stat_buf);
> +		rkisp1_stats_get_bls_meas(stats_vdev, cur_stat_buf);
> +		cur_stat_buf->meas_type |= CIFISP_STAT_AUTOEXP;
> +	}
> +
> +	if (meas_work->isp_ris & CIF_ISP_HIST_MEASURE_RDY) {
> +		rkisp1_stats_get_hst_meas(stats_vdev, cur_stat_buf);
> +		cur_stat_buf->meas_type |= CIFISP_STAT_HIST;
> +	}
> +
> +	vb2_set_plane_payload(&cur_buf->vb.vb2_buf, 0,
> +			      sizeof(struct rkisp1_stat_buffer));
> +	cur_buf->vb.sequence = cur_frame_id;
> +	cur_buf->vb.vb2_buf.timestamp = ktime_get_ns();

It might be better to get the timestamp in the beginning of the function
before acquiring the mutex which can take a relatively long time.

> +	vb2_buffer_done(&cur_buf->vb.vb2_buf, VB2_BUF_STATE_DONE);
> +}
> +
> +static void rkisp1_stats_readout_work(struct work_struct *work)
> +{
> +	struct rkisp1_isp_readout_work *readout_work = container_of(work,
> +						struct rkisp1_isp_readout_work,
> +						work);
> +	struct rkisp1_isp_stats_vdev *stats_vdev = readout_work->stats_vdev;
> +
> +	if (readout_work->readout == RKISP1_ISP_READOUT_MEAS)
> +		rkisp1_stats_send_measurement(stats_vdev, readout_work);
> +
> +	kfree(readout_work);
> +}
> +
> +int rkisp1_stats_isr(struct rkisp1_isp_stats_vdev *stats_vdev, u32 isp_ris)
> +{
> +	unsigned int cur_frame_id =
> +		atomic_read(&stats_vdev->dev->isp_sdev.frm_sync_seq) - 1;
> +	struct rkisp1_isp_readout_work *work;
> +	unsigned int isp_mis_tmp = 0;
> +#ifdef LOG_ISR_EXE_TIME
> +	ktime_t in_t = ktime_get();
> +#endif
> +
> +	spin_lock(&stats_vdev->irq_lock);
> +
> +	writel((CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN | CIF_ISP_EXP_END |
> +		CIF_ISP_HIST_MEASURE_RDY),
> +		stats_vdev->dev->base_addr + CIF_ISP_ICR);
> +
> +	isp_mis_tmp = readl(stats_vdev->dev->base_addr + CIF_ISP_MIS);
> +	if (isp_mis_tmp &
> +		(CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> +		 CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> +		v4l2_err(stats_vdev->vnode.vdev.v4l2_dev,
> +			 "isp icr 3A info err: 0x%x\n",
> +			 isp_mis_tmp);
> +
> +	if (!stats_vdev->streamon)
> +		goto unlock;
> +	if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN | CIF_ISP_EXP_END |
> +		CIF_ISP_HIST_MEASURE_RDY)) {
> +		work = (struct rkisp1_isp_readout_work *)
> +			kzalloc(sizeof(struct rkisp1_isp_readout_work),
> +				GFP_ATOMIC);
> +		if (work) {
> +			INIT_WORK(&work->work,
> +				  rkisp1_stats_readout_work);
> +			work->readout = RKISP1_ISP_READOUT_MEAS;
> +			work->stats_vdev = stats_vdev;
> +			work->frame_id = cur_frame_id;
> +			work->isp_ris = isp_ris;
> +			if (!queue_work(stats_vdev->readout_wq,
> +					&work->work))

Please use threaded interrupt handling instead.

> +				kfree(work);
> +		} else {
> +			v4l2_err(stats_vdev->vnode.vdev.v4l2_dev,
> +				 "Could not allocate work\n");
> +		}
> +	}
> +
> +#ifdef LOG_ISR_EXE_TIME
> +	if (isp_ris & (CIF_ISP_EXP_END | CIF_ISP_AWB_DONE |
> +		       CIF_ISP_FRAME | CIF_ISP_HIST_MEASURE_RDY)) {
> +		unsigned int diff_us =
> +				ktime_to_us(ktime_sub(ktime_get(), in_t));
> +
> +		if (diff_us > g_longest_isr_time)
> +			g_longest_isr_time = diff_us;
> +
> +		v4l2_info(stats_vdev->vnode.vdev.v4l2_dev,
> +			  "isp_isr time %d %d\n", diff_us, g_longest_isr_time);
> +	}
> +#endif

I'd remove this; it could be useful during development time but hardly any
longer.

> +
> +unlock:
> +	spin_unlock(&stats_vdev->irq_lock);
> +
> +	return 0;
> +}
> +
> +static void rkisp1_init_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev)
> +{
> +	stats_vdev->vdev_fmt.fmt.meta.dataformat =
> +		V4L2_META_FMT_RK_ISP1_STAT_3A;
> +	stats_vdev->vdev_fmt.fmt.meta.buffersize =
> +		sizeof(struct rkisp1_stat_buffer);
> +}
> +
> +int rkisp1_register_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev,
> +			       struct v4l2_device *v4l2_dev,
> +			       struct rkisp1_device *dev)
> +{
> +	struct rkisp1_vdev_node *node = &stats_vdev->vnode;
> +	struct video_device *vdev = &node->vdev;
> +	unsigned int ret;
> +
> +	stats_vdev->dev = dev;
> +	mutex_init(&stats_vdev->wq_lock);
> +	mutex_init(&node->vlock);
> +	INIT_LIST_HEAD(&stats_vdev->stat);
> +	spin_lock_init(&stats_vdev->irq_lock);
> +
> +	strlcpy(vdev->name, "rkisp1-statistics", sizeof(vdev->name));

strscpy()?

> +
> +	video_set_drvdata(vdev, stats_vdev);
> +	vdev->ioctl_ops = &rkisp1_stats_ioctl;
> +	vdev->fops = &rkisp1_stats_fops;
> +	vdev->release = video_device_release_empty;
> +	vdev->lock = &node->vlock;
> +	vdev->v4l2_dev = v4l2_dev;
> +	vdev->queue = &node->buf_queue;
> +	vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
> +	vdev->vfl_dir =  VFL_DIR_RX;
> +	rkisp1_stats_init_vb2_queue(vdev->queue, stats_vdev);
> +	rkisp1_init_stats_vdev(stats_vdev);
> +	video_set_drvdata(vdev, stats_vdev);
> +
> +	node->pad.flags = MEDIA_PAD_FL_SINK;
> +	ret = media_entity_pads_init(&vdev->entity, 1, &node->pad);
> +	if (ret < 0)
> +		goto err_release_queue;
> +
> +	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
> +	if (ret < 0) {
> +		dev_err(&vdev->dev,
> +			"could not register Video for Linux device\n");
> +		goto err_cleanup_media_entity;
> +	}
> +
> +	stats_vdev->readout_wq =
> +	    alloc_workqueue("measurement_queue",

Indentation.

> +			    WQ_UNBOUND | WQ_MEM_RECLAIM, 1);
> +
> +	if (!stats_vdev->readout_wq) {
> +		ret = -ENOMEM;
> +			goto err_unreg_vdev;

Same here.

> +	}
> +
> +	return 0;

Please leave an extra newline before the labels below, as well as before
return.

> +err_unreg_vdev:
> +	video_unregister_device(vdev);
> +err_cleanup_media_entity:
> +	media_entity_cleanup(&vdev->entity);
> +err_release_queue:
> +	vb2_queue_release(vdev->queue);

Also remember mutex_destroy() for the two mutexes.

> +	return ret;
> +}
> +
> +void rkisp1_unregister_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev)
> +{
> +	struct rkisp1_vdev_node *node = &stats_vdev->vnode;
> +	struct video_device *vdev = &node->vdev;
> +
> +	destroy_workqueue(stats_vdev->readout_wq);
> +	video_unregister_device(vdev);
> +	media_entity_cleanup(&vdev->entity);
> +	vb2_queue_release(vdev->queue);

mutex_destroy() here, too.

> +}
> diff --git a/drivers/media/platform/rockchip/isp1/isp_stats.h b/drivers/media/platform/rockchip/isp1/isp_stats.h
> new file mode 100644
> index 000000000000..b46c8537e1c7
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/isp_stats.h
> @@ -0,0 +1,60 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#ifndef _RKISP1_ISP_STATS_H
> +#define _RKISP1_ISP_STATS_H
> +
> +#include <linux/rkisp1-config.h>
> +
> +#include "common.h"
> +
> +struct rkisp1_isp_stats_vdev;
> +
> +enum rkisp1_isp_readout_cmd {
> +	RKISP1_ISP_READOUT_MEAS,
> +	RKISP1_ISP_READOUT_META,
> +};
> +
> +struct rkisp1_isp_readout_work {
> +	struct work_struct work;
> +	struct rkisp1_isp_stats_vdev *stats_vdev;
> +
> +	unsigned int frame_id;
> +	unsigned int isp_ris;
> +	enum rkisp1_isp_readout_cmd readout;
> +	struct vb2_buffer *vb;
> +};
> +
> +/*
> + * struct rkisp1_isp_stats_vdev - ISP Statistics device
> + *
> + * @irq_lock: buffer queue lock
> + * @stat: stats buffer list
> + * @readout_wq: workqueue for statistics information read
> + */
> +struct rkisp1_isp_stats_vdev {
> +	struct rkisp1_vdev_node vnode;
> +	struct rkisp1_device *dev;
> +
> +	spinlock_t irq_lock;
> +	struct list_head stat;
> +	struct v4l2_format vdev_fmt;
> +	bool streamon;
> +
> +	struct workqueue_struct *readout_wq;
> +	struct mutex wq_lock;
> +};
> +
> +int rkisp1_stats_isr(struct rkisp1_isp_stats_vdev *stats_vdev, u32 isp_ris);
> +
> +int rkisp1_register_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev,
> +			       struct v4l2_device *v4l2_dev,
> +			       struct rkisp1_device *dev);
> +
> +void rkisp1_unregister_stats_vdev(struct rkisp1_isp_stats_vdev *stats_vdev);
> +
> +#endif /* _RKISP1_ISP_STATS_H */

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver
  2019-08-07 15:27   ` Sakari Ailus
@ 2019-08-08 21:59     ` Helen Koike
  2019-08-09 12:05       ` Sakari Ailus
  0 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-08-08 21:59 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Sakari,

Thanks for your review, just some comments/questions below:

On 8/7/19 12:27 PM, Sakari Ailus wrote:
> Hi Helen,
> 
> On Tue, Jul 30, 2019 at 03:42:51PM -0300, Helen Koike wrote:
>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>
>> Add the core driver for rockchip isp1.
>>
>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
>> Signed-off-by: Jacob Chen <cc@rock-chips.com>
>> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>> [fixed compilation and run time errors regarding new v4l2 async API]
>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> [Add missing module device table]
>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>> [update for upstream]
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>
>> ---
>>
>> Changes in v8: None
>> Changes in v7:
>> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
>> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
>> - Fix compilation and runtime errors due to bitrotting
>> The code has bit-rotten since March 2018, fix compilation errors.
>> The new V4L2 async notifier API requires notifiers to be initialized by
>> a call to v4l2_async_notifier_init() before being used, do so.
>> - Add missing module device table
>> - use clk_bulk framework
>> - add missing notifiers cleanups
>> - s/strlcpy/strscpy
>> - normalize bus_info name
>> - fix s_stream error path, stream_cnt wans't being decremented properly
>> - use devm_platform_ioremap_resource() helper
>> - s/deice/device
>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>> isp subdevice in the media topology now.
>> - remove "saved_state" member from rkisp1_stream struct
>> - Reverse the order of MIs
>> - Simplify MI interrupt handling
>> Rather than adding unnecessary indirection, just use stream index to
>> handle MI interrupt enable/disable/clear, since the stream index matches
>> the order of bits now, thanks to previous patch. While at it, remove
>> some dead code.
>> - code styling and checkpatch fixes
>>
>>  drivers/media/platform/Kconfig                |  12 +
>>  drivers/media/platform/Makefile               |   1 +
>>  drivers/media/platform/rockchip/isp1/Makefile |   7 +
>>  drivers/media/platform/rockchip/isp1/common.h | 101 +++
>>  drivers/media/platform/rockchip/isp1/dev.c    | 675 ++++++++++++++++++
>>  drivers/media/platform/rockchip/isp1/dev.h    |  97 +++
>>  6 files changed, 893 insertions(+)
>>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
>>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
>>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
>>
>> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
>> index 89555f9a813f..e0e98937c565 100644
>> --- a/drivers/media/platform/Kconfig
>> +++ b/drivers/media/platform/Kconfig
>> @@ -106,6 +106,18 @@ config VIDEO_QCOM_CAMSS
>>  	select VIDEOBUF2_DMA_SG
>>  	select V4L2_FWNODE
>>  
>> +config VIDEO_ROCKCHIP_ISP1
>> +	tristate "Rockchip Image Signal Processing v1 Unit driver"
>> +	depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
>> +	depends on ARCH_ROCKCHIP || COMPILE_TEST
>> +	select VIDEOBUF2_DMA_CONTIG
>> +	select VIDEOBUF2_VMALLOC
>> +	select V4L2_FWNODE
>> +	select PHY_ROCKCHIP_DPHY
>> +	default n
>> +	---help---
>> +	  Support for ISP1 on the rockchip SoC.
>> +
>>  config VIDEO_S3C_CAMIF
>>  	tristate "Samsung S3C24XX/S3C64XX SoC Camera Interface driver"
>>  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API
>> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
>> index 7cbbd925124c..f9fcf8e7c513 100644
>> --- a/drivers/media/platform/Makefile
>> +++ b/drivers/media/platform/Makefile
>> @@ -69,6 +69,7 @@ obj-$(CONFIG_VIDEO_RENESAS_FDP1)	+= rcar_fdp1.o
>>  obj-$(CONFIG_VIDEO_RENESAS_JPU)		+= rcar_jpu.o
>>  obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
>>  
>> +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1)	+= rockchip/isp1/
>>  obj-$(CONFIG_VIDEO_ROCKCHIP_RGA)	+= rockchip/rga/
>>  
>>  obj-y	+= omap/
>> diff --git a/drivers/media/platform/rockchip/isp1/Makefile b/drivers/media/platform/rockchip/isp1/Makefile
>> new file mode 100644
>> index 000000000000..72706e80fc8b
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/Makefile
>> @@ -0,0 +1,7 @@
>> +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1) += 	rockchip-isp1.o
>> +rockchip-isp1-objs 	   += 	rkisp1.o \
>> +				dev.o \
>> +				regs.o \
>> +				isp_stats.o \
>> +				isp_params.o \
>> +				capture.o
>> diff --git a/drivers/media/platform/rockchip/isp1/common.h b/drivers/media/platform/rockchip/isp1/common.h
>> new file mode 100644
>> index 000000000000..606ce2793546
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/common.h
>> @@ -0,0 +1,101 @@
>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>> +/*
>> + * Rockchip isp1 driver
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#ifndef _RKISP1_COMMON_H
>> +#define _RKISP1_COMMON_H
>> +
>> +#include <linux/mutex.h>
>> +#include <media/media-device.h>
>> +#include <media/media-entity.h>
>> +#include <media/v4l2-ctrls.h>
>> +#include <media/v4l2-device.h>
>> +#include <media/videobuf2-v4l2.h>
>> +
>> +#define RKISP1_DEFAULT_WIDTH		800
>> +#define RKISP1_DEFAULT_HEIGHT		600
>> +
>> +#define RKISP1_MAX_STREAM		2
>> +#define RKISP1_STREAM_MP		0
>> +#define RKISP1_STREAM_SP		1
>> +
>> +#define RKISP1_PLANE_Y			0
>> +#define RKISP1_PLANE_CB			1
>> +#define RKISP1_PLANE_CR			2
>> +
>> +enum rkisp1_sd_type {
>> +	RKISP1_SD_SENSOR,
>> +	RKISP1_SD_PHY_CSI,
>> +	RKISP1_SD_VCM,
>> +	RKISP1_SD_FLASH,
>> +	RKISP1_SD_MAX,
>> +};
> 
> I wonder if this is a leftover from the driver development time. Same goes
> for the subdevs field in struct rkisp1_device.

It is a left over, I'm removing them for the next version, thanks.

> 
>> +
>> +/* One structure per video node */
>> +struct rkisp1_vdev_node {
>> +	struct vb2_queue buf_queue;
>> +	/* vfd lock */
>> +	struct mutex vlock;
>> +	struct video_device vdev;
>> +	struct media_pad pad;
>> +};
>> +
>> +enum rkisp1_fmt_pix_type {
>> +	FMT_YUV,
>> +	FMT_RGB,
>> +	FMT_BAYER,
>> +	FMT_JPEG,
>> +	FMT_MAX
>> +};
>> +
>> +enum rkisp1_fmt_raw_pat_type {
>> +	RAW_RGGB = 0,
>> +	RAW_GRBG,
>> +	RAW_GBRG,
>> +	RAW_BGGR,
>> +};
>> +
>> +struct rkisp1_buffer {
>> +	struct vb2_v4l2_buffer vb;
>> +	struct list_head queue;
>> +	union {
>> +		u32 buff_addr[VIDEO_MAX_PLANES];
>> +		void *vaddr[VIDEO_MAX_PLANES];
>> +	};
>> +};
>> +
>> +struct rkisp1_dummy_buffer {
>> +	void *vaddr;
>> +	dma_addr_t dma_addr;
>> +	u32 size;
>> +};
>> +
>> +extern int rkisp1_debug;
>> +
>> +static inline
>> +struct rkisp1_vdev_node *vdev_to_node(struct video_device *vdev)
>> +{
>> +	return container_of(vdev, struct rkisp1_vdev_node, vdev);
>> +}
>> +
>> +static inline struct rkisp1_vdev_node *queue_to_node(struct vb2_queue *q)
>> +{
>> +	return container_of(q, struct rkisp1_vdev_node, buf_queue);
>> +}
>> +
>> +static inline struct rkisp1_buffer *to_rkisp1_buffer(struct vb2_v4l2_buffer *vb)
>> +{
>> +	return container_of(vb, struct rkisp1_buffer, vb);
>> +}
>> +
>> +static inline struct vb2_queue *to_vb2_queue(struct file *file)
>> +{
>> +	struct rkisp1_vdev_node *vnode = video_drvdata(file);
>> +
>> +	return &vnode->buf_queue;
>> +}
>> +
>> +#endif /* _RKISP1_COMMON_H */
>> diff --git a/drivers/media/platform/rockchip/isp1/dev.c b/drivers/media/platform/rockchip/isp1/dev.c
>> new file mode 100644
>> index 000000000000..2b4a67e1a3b5
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/dev.c
>> @@ -0,0 +1,675 @@
>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>> +/*
>> + * Rockchip isp1 driver
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#include <linux/clk.h>
>> +#include <linux/interrupt.h>
>> +#include <linux/module.h>
>> +#include <linux/of.h>
>> +#include <linux/of_graph.h>
>> +#include <linux/of_platform.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/pinctrl/consumer.h>
>> +#include <linux/phy/phy.h>
>> +#include <linux/phy/phy-mipi-dphy.h>
>> +
>> +#include "common.h"
>> +#include "regs.h"
>> +
>> +struct isp_match_data {
>> +	const char * const *clks;
>> +	int size;
> 
> unsigned int

ack

> 
>> +};
>> +
>> +struct sensor_async_subdev {
>> +	struct v4l2_async_subdev asd;
>> +	struct v4l2_mbus_config mbus;
>> +	unsigned int lanes;
>> +};
>> +
>> +int rkisp1_debug;
>> +module_param_named(debug, rkisp1_debug, int, 0644);
>> +MODULE_PARM_DESC(debug, "Debug level (0-1)");
> 
> Have you thought of using dynamic debug instead?

right, this is being used in v4l2_dbg(), which I can replace by dev_dbg()
that is already covered by dynamic debug iirc.
Should I also replace v4l2_err() by dev_err() (I always get confused by
which log function I should use).

> 
>> +
>> +/**************************** pipeline operations******************************/
>> +
>> +static int __isp_pipeline_prepare(struct rkisp1_pipeline *p,
>> +				  struct media_entity *me)
>> +{
>> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
>> +	struct v4l2_subdev *sd;
>> +	unsigned int i;
>> +
>> +	p->num_subdevs = 0;
>> +	memset(p->subdevs, 0, sizeof(p->subdevs));
>> +
>> +	while (1) {
>> +		struct media_pad *pad = NULL;
>> +
>> +		/* Find remote source pad */
>> +		for (i = 0; i < me->num_pads; i++) {
>> +			struct media_pad *spad = &me->pads[i];
>> +
>> +			if (!(spad->flags & MEDIA_PAD_FL_SINK))
>> +				continue;
>> +			pad = media_entity_remote_pad(spad);
>> +			if (pad)
>> +				break;
>> +		}
>> +
>> +		if (!pad)
>> +			break;
>> +
>> +		sd = media_entity_to_v4l2_subdev(pad->entity);
>> +		if (sd != &dev->isp_sdev.sd)
>> +			p->subdevs[p->num_subdevs++] = sd;
> 
> How do you make sure you don't overrun the array?

Because the maximum path the topology can have is:
sensor->rkisp->capture

> 
> Instead, I'd avoid maintaining the array in the first place --- the same
> information is available from the MC framework data structures --- see e.g.
> the omap3isp driver.

If I understand correctly, omap3isp navigates through the topology in the same way,
but it just don't save in an array, but I reuse this information in other places,
mostly for power up/down (see below why I don't use v4l2_pipeline_pm_use())

> 
>> +
>> +		me = &sd->entity;
>> +		if (me->num_pads == 1)
>> +			break;
>> +	}
>> +	return 0;
>> +}
>> +
>> +static int __subdev_set_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	int ret;
>> +
>> +	if (!sd)
>> +		return -ENXIO;
>> +
>> +	ret = v4l2_subdev_call(sd, core, s_power, on);
>> +
>> +	return ret != -ENOIOCTLCMD ? ret : 0;
>> +}
>> +
>> +static int __isp_pipeline_s_power(struct rkisp1_pipeline *p, bool on)
> 
> Could you instead use v4l2_pipeline_pm_use()?

Unless I misunderstood this (which is very likely), v4l2_pipeline_pm_use() calls pipeline_pm_power(),
that applies power change to all entities in a pipeline, but if I have two sensors
connected to the ISP, one with link enabled and the other with a disabled link,
I don't want to power both sensors on, just the one participating in that stream. And
if I call v4l2_pipeline_pm_use() it will power on both, which is not what I want.

Does it make sense?

> 
>> +{
>> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
>> +	int i, ret;
>> +
>> +	if (on) {
>> +		__subdev_set_power(&dev->isp_sdev.sd, true);
>> +
>> +		for (i = p->num_subdevs - 1; i >= 0; --i) {
>> +			ret = __subdev_set_power(p->subdevs[i], true);
>> +			if (ret < 0 && ret != -ENXIO)
>> +				goto err_power_off;
>> +		}
>> +	} else {
>> +		for (i = 0; i < p->num_subdevs; ++i)
>> +			__subdev_set_power(p->subdevs[i], false);
>> +
>> +		__subdev_set_power(&dev->isp_sdev.sd, false);
>> +	}
>> +
>> +	return 0;
>> +
>> +err_power_off:
>> +	for (++i; i < p->num_subdevs; ++i)
>> +		__subdev_set_power(p->subdevs[i], false);
>> +	__subdev_set_power(&dev->isp_sdev.sd, true);
>> +	return ret;
>> +}
>> +
>> +static int rkisp1_pipeline_open(struct rkisp1_pipeline *p,
>> +				struct media_entity *me,
>> +				bool prepare)
>> +{
>> +	int ret;
>> +
>> +	if (WARN_ON(!p || !me))
>> +		return -EINVAL;
>> +	if (atomic_inc_return(&p->power_cnt) > 1)
>> +		return 0;
>> +
>> +	/* go through media graphic and get subdevs */
>> +	if (prepare)
>> +		__isp_pipeline_prepare(p, me);
>> +
>> +	if (!p->num_subdevs)
>> +		return -EINVAL;
>> +
>> +	ret = __isp_pipeline_s_power(p, 1);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_pipeline_close(struct rkisp1_pipeline *p)
>> +{
>> +	int ret;
>> +
>> +	if (atomic_dec_return(&p->power_cnt) > 0)
>> +		return 0;
>> +	ret = __isp_pipeline_s_power(p, 0);
>> +
>> +	return ret == -ENXIO ? 0 : ret;
>> +}
>> +
>> +/*
>> + * stream-on order: isp_subdev, mipi dphy, sensor
>> + * stream-off order: mipi dphy, sensor, isp_subdev
>> + */
>> +static int rkisp1_pipeline_set_stream(struct rkisp1_pipeline *p, bool on)
>> +{
>> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
>> +	int i, ret;
>> +
>> +	if ((on && atomic_inc_return(&p->stream_cnt) > 1) ||
>> +	    (!on && atomic_dec_return(&p->stream_cnt) > 0))
>> +		return 0;
>> +
>> +	if (on) {
>> +		ret = v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream,
>> +				       true);
>> +		if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV) {
>> +			v4l2_err(&dev->v4l2_dev,
>> +				 "s_stream failed on subdevice %s (%d)\n",
>> +				 dev->isp_sdev.sd.name,
>> +				 ret);
>> +			atomic_dec(&p->stream_cnt);
>> +			return ret;
>> +		}
>> +	}
>> +
>> +	/* phy -> sensor */
>> +	for (i = 0; i < p->num_subdevs; ++i) {
>> +		ret = v4l2_subdev_call(p->subdevs[i], video, s_stream, on);
>> +		if (on && ret < 0 && ret != -ENOIOCTLCMD && ret != -ENODEV)
>> +			goto err_stream_off;
> 
> You should stop after the first external sub-device.
> 
> It seems even the omap3isp driver doesn't do that. It's not easy to spot
> such issues indeed.

I'm not sure I understand, what do you call an external sub-device? Is the sensor
an external subdevice?

> 
>> +	}
>> +
>> +	if (!on)
>> +		v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
>> +
>> +	return 0;
>> +
>> +err_stream_off:
>> +	for (--i; i >= 0; --i)
>> +		v4l2_subdev_call(p->subdevs[i], video, s_stream, false);
>> +	v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
>> +	atomic_dec(&p->stream_cnt);
>> +	return ret;
>> +}
>> +
>> +/***************************** media controller *******************************/
>> +/* See http://opensource.rock-chips.com/wiki_Rockchip-isp1 for Topology */
> 
> The host appears to be down, or there's a routing problem. Unless this is
> fixed, having such a URL here doesn't do much good. :-I

This is a left over, sorry about that.
I can access the URL now. I'll try to get some of the docs and move to the kernel docs.

> 
>> +
>> +static int rkisp1_create_links(struct rkisp1_device *dev)
>> +{
>> +	struct media_entity *source, *sink;
>> +	struct rkisp1_sensor *sensor;
>> +	unsigned int flags, pad;
>> +	int ret;
>> +
>> +	/* sensor links(or mipi-phy) */
>> +	list_for_each_entry(sensor, &dev->sensors, list) {
>> +		for (pad = 0; pad < sensor->sd->entity.num_pads; pad++)
>> +			if (sensor->sd->entity.pads[pad].flags &
>> +				MEDIA_PAD_FL_SOURCE)
>> +				break;
> 
> Could you use media_entity_get_fwnode_pad() instead?

Yes, I didn't know about it actually, thanks for that, looks cleaner (I'll send in
the next version).

> 
>> +
>> +		if (pad == sensor->sd->entity.num_pads) {
>> +			dev_err(dev->dev,
>> +				"failed to find src pad for %s\n",
>> +				sensor->sd->name);
>> +
>> +			return -ENXIO;
>> +		}
>> +
>> +		ret = media_create_pad_link(
>> +				&sensor->sd->entity, pad,
>> +				&dev->isp_sdev.sd.entity,
>> +				RKISP1_ISP_PAD_SINK,
>> +				list_is_first(&sensor->list, &dev->sensors) ?
>> +				MEDIA_LNK_FL_ENABLED : 0);
>> +		if (ret) {
>> +			dev_err(dev->dev,
>> +				"failed to create link for %s\n",
>> +				sensor->sd->name);
>> +			return ret;
>> +		}
>> +	}
>> +
>> +	/* params links */
>> +	source = &dev->params_vdev.vnode.vdev.entity;
>> +	sink = &dev->isp_sdev.sd.entity;
>> +	flags = MEDIA_LNK_FL_ENABLED;
>> +	ret = media_create_pad_link(source, 0, sink,
>> +				       RKISP1_ISP_PAD_SINK_PARAMS, flags);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* create isp internal links */
>> +	/* SP links */
>> +	source = &dev->isp_sdev.sd.entity;
>> +	sink = &dev->stream[RKISP1_STREAM_SP].vnode.vdev.entity;
>> +	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
>> +				       sink, 0, flags);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* MP links */
>> +	source = &dev->isp_sdev.sd.entity;
>> +	sink = &dev->stream[RKISP1_STREAM_MP].vnode.vdev.entity;
>> +	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
>> +				       sink, 0, flags);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* 3A stats links */
>> +	source = &dev->isp_sdev.sd.entity;
>> +	sink = &dev->stats_vdev.vnode.vdev.entity;
>> +	return media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_STATS,
>> +					sink, 0, flags);
> 
> Indentation. Same for the calls to the same function above.

ack

> 
>> +}
>> +
>> +static int subdev_notifier_bound(struct v4l2_async_notifier *notifier,
>> +				 struct v4l2_subdev *sd,
>> +				 struct v4l2_async_subdev *asd)
>> +{
>> +	struct rkisp1_device *isp_dev = container_of(notifier,
>> +						     struct rkisp1_device,
>> +						     notifier);
>> +	struct sensor_async_subdev *s_asd = container_of(asd,
>> +					struct sensor_async_subdev, asd);
>> +	struct rkisp1_sensor *sensor;
>> +
>> +	sensor = devm_kzalloc(isp_dev->dev, sizeof(*sensor), GFP_KERNEL);
>> +	if (!sensor)
>> +		return -ENOMEM;
>> +
>> +	sensor->lanes = s_asd->lanes;
>> +	sensor->mbus = s_asd->mbus;
>> +	sensor->sd = sd;
>> +	sensor->dphy = devm_phy_get(isp_dev->dev, "dphy");
>> +	if (IS_ERR(sensor->dphy)) {
>> +		if (PTR_ERR(sensor->dphy) != -EPROBE_DEFER)
>> +			dev_err(isp_dev->dev, "Couldn't get the MIPI D-PHY\n");
>> +		return PTR_ERR(sensor->dphy);
>> +	}
>> +	phy_init(sensor->dphy);
>> +
>> +	list_add(&sensor->list, &isp_dev->sensors);
> 
> In general, maintaining the information on the external subdevs on your own
> adds complexity to the driver. You can get the information when you need it
> from the data structures maintained by MC (see e.g. the omap3isp driver for
> examples).
> 
>> +
>> +	return 0;
>> +}
>> +
>> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
>> +					  struct v4l2_subdev *sd)
>> +{
>> +	struct rkisp1_sensor *sensor;
>> +
>> +	list_for_each_entry(sensor, &dev->sensors, list)
>> +		if (sensor->sd == sd)
>> +			return sensor;
>> +
>> +	return NULL;
>> +}
>> +
>> +static void subdev_notifier_unbind(struct v4l2_async_notifier *notifier,
>> +				   struct v4l2_subdev *sd,
>> +				   struct v4l2_async_subdev *asd)
>> +{
>> +	struct rkisp1_device *isp_dev = container_of(notifier,
>> +						     struct rkisp1_device,
>> +						     notifier);
>> +	struct rkisp1_sensor *sensor = sd_to_sensor(isp_dev, sd);
>> +
>> +	/* TODO: check if a lock is required here */
>> +	list_del(&sensor->list);
>> +
>> +	phy_exit(sensor->dphy);
>> +}
>> +
>> +static int subdev_notifier_complete(struct v4l2_async_notifier *notifier)
>> +{
>> +	struct rkisp1_device *dev = container_of(notifier, struct rkisp1_device,
>> +						 notifier);
>> +	int ret;
>> +
>> +	mutex_lock(&dev->media_dev.graph_mutex);
>> +	ret = rkisp1_create_links(dev);
>> +	if (ret < 0)
>> +		goto unlock;
>> +	ret = v4l2_device_register_subdev_nodes(&dev->v4l2_dev);
>> +	if (ret < 0)
>> +		goto unlock;
>> +
>> +	v4l2_info(&dev->v4l2_dev, "Async subdev notifier completed\n");
>> +
>> +unlock:
>> +	mutex_unlock(&dev->media_dev.graph_mutex);
>> +	return ret;
>> +}
>> +
>> +static int rkisp1_fwnode_parse(struct device *dev,
>> +			       struct v4l2_fwnode_endpoint *vep,
>> +			       struct v4l2_async_subdev *asd)
>> +{
>> +	struct sensor_async_subdev *s_asd =
>> +			container_of(asd, struct sensor_async_subdev, asd);
>> +
>> +	if (vep->bus_type != V4L2_MBUS_CSI2_DPHY) {
>> +		dev_err(dev, "Only CSI2 bus type is currently supported\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (vep->base.port != 0) {
>> +		dev_err(dev, "The ISP has only port 0\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	s_asd->mbus.type = vep->bus_type;
>> +	s_asd->mbus.flags = vep->bus.mipi_csi2.flags;
>> +	s_asd->lanes = vep->bus.mipi_csi2.num_data_lanes;
>> +
>> +	switch (vep->bus.mipi_csi2.num_data_lanes) {
>> +	case 1:
>> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_1_LANE;
>> +		break;
>> +	case 2:
>> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_2_LANE;
>> +		break;
>> +	case 3:
>> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_3_LANE;
>> +		break;
>> +	case 4:
>> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_4_LANE;
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct v4l2_async_notifier_operations subdev_notifier_ops = {
>> +	.bound = subdev_notifier_bound,
>> +	.unbind = subdev_notifier_unbind,
>> +	.complete = subdev_notifier_complete,
>> +};
>> +
>> +static int isp_subdev_notifier(struct rkisp1_device *isp_dev)
>> +{
>> +	struct v4l2_async_notifier *ntf = &isp_dev->notifier;
>> +	struct device *dev = isp_dev->dev;
>> +	int ret;
>> +
>> +	v4l2_async_notifier_init(ntf);
>> +
>> +	ret = v4l2_async_notifier_parse_fwnode_endpoints_by_port(
>> +		dev, ntf, sizeof(struct sensor_async_subdev), 0,
>> +		rkisp1_fwnode_parse);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	if (list_empty(&ntf->asd_list))
>> +		return -ENODEV;	/* no endpoint */
>> +
>> +	ntf->ops = &subdev_notifier_ops;
>> +
>> +	return v4l2_async_notifier_register(&isp_dev->v4l2_dev, ntf);
>> +}
>> +
>> +/***************************** platform device *******************************/
>> +
>> +static int rkisp1_register_platform_subdevs(struct rkisp1_device *dev)
>> +{
>> +	int ret;
>> +
>> +	ret = rkisp1_register_isp_subdev(dev, &dev->v4l2_dev);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	ret = rkisp1_register_stream_vdevs(dev);
>> +	if (ret < 0)
>> +		goto err_unreg_isp_subdev;
>> +
>> +	ret = rkisp1_register_stats_vdev(&dev->stats_vdev, &dev->v4l2_dev, dev);
>> +	if (ret < 0)
>> +		goto err_unreg_stream_vdev;
>> +
>> +	ret = rkisp1_register_params_vdev(&dev->params_vdev, &dev->v4l2_dev,
>> +					  dev);
>> +	if (ret < 0)
>> +		goto err_unreg_stats_vdev;
>> +
>> +	ret = isp_subdev_notifier(dev);
>> +	if (ret < 0) {
>> +		v4l2_err(&dev->v4l2_dev,
>> +			 "Failed to register subdev notifier(%d)\n", ret);
>> +		goto err_unreg_params_vdev;
>> +	}
>> +
>> +	return 0;
>> +err_unreg_params_vdev:
>> +	rkisp1_unregister_params_vdev(&dev->params_vdev);
>> +err_unreg_stats_vdev:
>> +	rkisp1_unregister_stats_vdev(&dev->stats_vdev);
>> +err_unreg_stream_vdev:
>> +	rkisp1_unregister_stream_vdevs(dev);
>> +err_unreg_isp_subdev:
>> +	rkisp1_unregister_isp_subdev(dev);
>> +	return ret;
>> +}
>> +
>> +static const char * const rk3399_isp_clks[] = {
>> +	"clk_isp",
>> +	"aclk_isp",
>> +	"hclk_isp",
>> +	"aclk_isp_wrap",
>> +	"hclk_isp_wrap",
>> +};
>> +
>> +static const char * const rk3288_isp_clks[] = {
>> +	"clk_isp",
>> +	"aclk_isp",
>> +	"hclk_isp",
>> +	"pclk_isp_in",
>> +	"sclk_isp_jpe",
>> +};
>> +
>> +static const struct isp_match_data rk3288_isp_clk_data = {
>> +	.clks = rk3288_isp_clks,
>> +	.size = ARRAY_SIZE(rk3288_isp_clks),
>> +};
>> +
>> +static const struct isp_match_data rk3399_isp_clk_data = {
>> +	.clks = rk3399_isp_clks,
>> +	.size = ARRAY_SIZE(rk3399_isp_clks),
>> +};
>> +
>> +static const struct of_device_id rkisp1_plat_of_match[] = {
>> +	{
>> +		.compatible = "rockchip,rk3288-cif-isp",
>> +		.data = &rk3288_isp_clk_data,
>> +	}, {
>> +		.compatible = "rockchip,rk3399-cif-isp",
>> +		.data = &rk3399_isp_clk_data,
>> +	},
>> +	{},
>> +};
>> +MODULE_DEVICE_TABLE(of, rkisp1_plat_of_match);
>> +
>> +static irqreturn_t rkisp1_irq_handler(int irq, void *ctx)
>> +{
>> +	struct device *dev = ctx;
>> +	struct rkisp1_device *rkisp1_dev = dev_get_drvdata(dev);
>> +	unsigned int mis_val;
>> +
>> +	mis_val = readl(rkisp1_dev->base_addr + CIF_ISP_MIS);
>> +	if (mis_val)
>> +		rkisp1_isp_isr(mis_val, rkisp1_dev);
>> +
>> +	mis_val = readl(rkisp1_dev->base_addr + CIF_MIPI_MIS);
>> +	if (mis_val)
>> +		rkisp1_mipi_isr(mis_val, rkisp1_dev);
>> +
>> +	mis_val = readl(rkisp1_dev->base_addr + CIF_MI_MIS);
>> +	if (mis_val)
>> +		rkisp1_mi_isr(mis_val, rkisp1_dev);
>> +
>> +	return IRQ_HANDLED;
>> +}
>> +
>> +static int rkisp1_plat_probe(struct platform_device *pdev)
>> +{
>> +	struct device_node *node = pdev->dev.of_node;
>> +	const struct isp_match_data *clk_data;
>> +	const struct of_device_id *match;
>> +	struct device *dev = &pdev->dev;
>> +	struct rkisp1_device *isp_dev;
>> +	struct v4l2_device *v4l2_dev;
>> +	unsigned int i;
>> +	int ret, irq;
>> +
>> +	match = of_match_node(rkisp1_plat_of_match, node);
>> +	isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
>> +	if (!isp_dev)
>> +		return -ENOMEM;
>> +
>> +	INIT_LIST_HEAD(&isp_dev->sensors);
>> +
>> +	dev_set_drvdata(dev, isp_dev);
>> +	isp_dev->dev = dev;
>> +
>> +	isp_dev->base_addr = devm_platform_ioremap_resource(pdev, 0);
>> +	if (IS_ERR(isp_dev->base_addr))
>> +		return PTR_ERR(isp_dev->base_addr);
>> +
>> +	irq = platform_get_irq(pdev, 0);
>> +	if (irq < 0)
>> +		return irq;
>> +
>> +	ret = devm_request_irq(dev, irq, rkisp1_irq_handler, IRQF_SHARED,
>> +			       dev_driver_string(dev), dev);
>> +	if (ret < 0) {
>> +		dev_err(dev, "request irq failed: %d\n", ret);
>> +		return ret;
>> +	}
>> +
>> +	isp_dev->irq = irq;
>> +	clk_data = match->data;
>> +
>> +	for (i = 0; i < clk_data->size; i++)
>> +		isp_dev->clks[i].id = clk_data->clks[i];
>> +	ret = devm_clk_bulk_get(dev, clk_data->size, isp_dev->clks);
>> +	if (ret)
>> +		return ret;
>> +	isp_dev->clk_size = clk_data->size;
>> +
>> +	atomic_set(&isp_dev->pipe.power_cnt, 0);
>> +	atomic_set(&isp_dev->pipe.stream_cnt, 0);
>> +	isp_dev->pipe.open = rkisp1_pipeline_open;
>> +	isp_dev->pipe.close = rkisp1_pipeline_close;
>> +	isp_dev->pipe.set_stream = rkisp1_pipeline_set_stream;
>> +
>> +	rkisp1_stream_init(isp_dev, RKISP1_STREAM_SP);
>> +	rkisp1_stream_init(isp_dev, RKISP1_STREAM_MP);
>> +
>> +	strscpy(isp_dev->media_dev.model, "rkisp1",
>> +		sizeof(isp_dev->media_dev.model));
>> +	isp_dev->media_dev.dev = &pdev->dev;
>> +	strscpy(isp_dev->media_dev.bus_info,
>> +		"platform: " DRIVER_NAME, sizeof(isp_dev->media_dev.bus_info));
>> +	media_device_init(&isp_dev->media_dev);
>> +
>> +	v4l2_dev = &isp_dev->v4l2_dev;
>> +	v4l2_dev->mdev = &isp_dev->media_dev;
>> +	strscpy(v4l2_dev->name, "rkisp1", sizeof(v4l2_dev->name));
>> +	v4l2_ctrl_handler_init(&isp_dev->ctrl_handler, 5);
>> +	v4l2_dev->ctrl_handler = &isp_dev->ctrl_handler;
>> +
>> +	ret = v4l2_device_register(isp_dev->dev, &isp_dev->v4l2_dev);
>> +	if (ret < 0)
> 
> Once you've initialised the control handler, you'll need to free it in case
> of an error. I.e. add one more label for that purpose near the end.

control handler is not required, I'll remove it for the next version.

> 
>> +		return ret;
>> +
>> +	ret = media_device_register(&isp_dev->media_dev);
>> +	if (ret < 0) {
>> +		v4l2_err(v4l2_dev, "Failed to register media device: %d\n",
>> +			 ret);
>> +		goto err_unreg_v4l2_dev;
>> +	}
>> +
>> +	/* create & register platefom subdev (from of_node) */
>> +	ret = rkisp1_register_platform_subdevs(isp_dev);
>> +	if (ret < 0)
>> +		goto err_unreg_media_dev;
>> +
>> +	pm_runtime_enable(&pdev->dev);
>> +
>> +	return 0;
>> +
>> +err_unreg_media_dev:
>> +	media_device_unregister(&isp_dev->media_dev);
>> +err_unreg_v4l2_dev:
>> +	v4l2_device_unregister(&isp_dev->v4l2_dev);
>> +	return ret;
>> +}
>> +
>> +static int rkisp1_plat_remove(struct platform_device *pdev)
>> +{
>> +	struct rkisp1_device *isp_dev = platform_get_drvdata(pdev);
>> +
>> +	pm_runtime_disable(&pdev->dev);
>> +	media_device_unregister(&isp_dev->media_dev);
>> +	v4l2_async_notifier_unregister(&isp_dev->notifier);
>> +	v4l2_async_notifier_cleanup(&isp_dev->notifier);
>> +	v4l2_device_unregister(&isp_dev->v4l2_dev);
>> +	rkisp1_unregister_params_vdev(&isp_dev->params_vdev);
>> +	rkisp1_unregister_stats_vdev(&isp_dev->stats_vdev);
>> +	rkisp1_unregister_stream_vdevs(isp_dev);
>> +	rkisp1_unregister_isp_subdev(isp_dev);
>> +
>> +	return 0;
>> +}
>> +
>> +static int __maybe_unused rkisp1_runtime_suspend(struct device *dev)
>> +{
>> +	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
>> +
>> +	clk_bulk_disable_unprepare(isp_dev->clk_size, isp_dev->clks);
>> +	return pinctrl_pm_select_sleep_state(dev);
>> +}
>> +
>> +static int __maybe_unused rkisp1_runtime_resume(struct device *dev)
>> +{
>> +	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
>> +	int ret;
>> +
>> +	ret = pinctrl_pm_select_default_state(dev);
>> +	if (ret < 0)
>> +		return ret;
>> +	ret = clk_bulk_prepare_enable(isp_dev->clk_size, isp_dev->clks);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	return 0;
>> +}
>> +
>> +static const struct dev_pm_ops rkisp1_plat_pm_ops = {
>> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
>> +				pm_runtime_force_resume)
>> +	SET_RUNTIME_PM_OPS(rkisp1_runtime_suspend, rkisp1_runtime_resume, NULL)
>> +};
>> +
>> +static struct platform_driver rkisp1_plat_drv = {
>> +	.driver = {
>> +		.name = DRIVER_NAME,
>> +		.of_match_table = of_match_ptr(rkisp1_plat_of_match),
>> +		.pm = &rkisp1_plat_pm_ops,
>> +	},
>> +	.probe = rkisp1_plat_probe,
>> +	.remove = rkisp1_plat_remove,
>> +};
>> +
>> +module_platform_driver(rkisp1_plat_drv);
>> +MODULE_AUTHOR("Rockchip Camera/ISP team");
>> +MODULE_DESCRIPTION("Rockchip ISP1 platform driver");
>> +MODULE_LICENSE("Dual BSD/GPL");
> 
> BSD or MIT?

Thanks for spotting this, I'll verify.

> 
>> diff --git a/drivers/media/platform/rockchip/isp1/dev.h b/drivers/media/platform/rockchip/isp1/dev.h
>> new file mode 100644
>> index 000000000000..f7cbee316523
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/dev.h
>> @@ -0,0 +1,97 @@
>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>> +/*
>> + * Rockchip isp1 driver
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#ifndef _RKISP1_DEV_H
>> +#define _RKISP1_DEV_H
>> +
>> +#include <linux/clk.h>
>> +
>> +#include "capture.h"
>> +#include "rkisp1.h"
>> +#include "isp_params.h"
>> +#include "isp_stats.h"
>> +
>> +#define DRIVER_NAME "rkisp1"
>> +#define ISP_VDEV_NAME DRIVER_NAME  "_ispdev"
>> +#define SP_VDEV_NAME DRIVER_NAME   "_selfpath"
>> +#define MP_VDEV_NAME DRIVER_NAME   "_mainpath"
>> +#define DMA_VDEV_NAME DRIVER_NAME  "_dmapath"
>> +
>> +#define GRP_ID_SENSOR			BIT(0)
>> +#define GRP_ID_MIPIPHY			BIT(1)
>> +#define GRP_ID_ISP			BIT(2)
>> +#define GRP_ID_ISP_MP			BIT(3)
>> +#define GRP_ID_ISP_SP			BIT(4)
>> +
>> +#define RKISP1_MAX_BUS_CLK	8
>> +#define RKISP1_MAX_SENSOR	2
>> +#define RKISP1_MAX_PIPELINE	4
>> +
>> +/*
>> + * struct rkisp1_pipeline - An ISP hardware pipeline
>> + *
>> + * Capture device call other devices via pipeline
>> + *
>> + * @num_subdevs: number of linked subdevs
>> + * @power_cnt: pipeline power count
>> + * @stream_cnt: stream power count
>> + */
>> +struct rkisp1_pipeline {
>> +	struct media_pipeline pipe;
>> +	int num_subdevs;
>> +	atomic_t power_cnt;
>> +	atomic_t stream_cnt;
>> +	struct v4l2_subdev *subdevs[RKISP1_MAX_PIPELINE];
>> +	int (*open)(struct rkisp1_pipeline *p,
>> +		    struct media_entity *me, bool prepare);
>> +	int (*close)(struct rkisp1_pipeline *p);
>> +	int (*set_stream)(struct rkisp1_pipeline *p, bool on);
>> +};
>> +
>> +/*
>> + * struct rkisp1_sensor - Sensor information
>> + * @mbus: media bus configuration
>> + */
>> +struct rkisp1_sensor {
>> +	struct v4l2_subdev *sd;
>> +	struct v4l2_mbus_config mbus;
>> +	unsigned int lanes;
>> +	struct phy *dphy;
>> +	struct list_head list;
>> +};
> 
> You seem to also have struct sensor_async_subdev that appears to contain
> similar information. Would it be possible to unify the two?
> 
> This would appear to allow you getting rid of functions such as
> sd_to_sensor, for instance.

ack, I managed to get rid of this, and I don't even need to keep them
on a list, I'll submit in the next version.

Thanks a lot for your review
Helen

> 
>> +
>> +/*
>> + * struct rkisp1_device - ISP platform device
>> + * @base_addr: base register address
>> + * @active_sensor: sensor in-use, set when streaming on
>> + * @isp_sdev: ISP sub-device
>> + * @rkisp1_stream: capture video device
>> + * @stats_vdev: ISP statistics output device
>> + * @params_vdev: ISP input parameters device
>> + */
>> +struct rkisp1_device {
>> +	void __iomem *base_addr;
>> +	int irq;
>> +	struct device *dev;
>> +	unsigned int clk_size;
>> +	struct clk_bulk_data clks[RKISP1_MAX_BUS_CLK];
>> +	struct v4l2_device v4l2_dev;
>> +	struct v4l2_ctrl_handler ctrl_handler;
>> +	struct media_device media_dev;
>> +	struct v4l2_async_notifier notifier;
>> +	struct v4l2_subdev *subdevs[RKISP1_SD_MAX];
>> +	struct rkisp1_sensor *active_sensor;
>> +	struct list_head sensors;
>> +	struct rkisp1_isp_subdev isp_sdev;
>> +	struct rkisp1_stream stream[RKISP1_MAX_STREAM];
>> +	struct rkisp1_isp_stats_vdev stats_vdev;
>> +	struct rkisp1_isp_params_vdev params_vdev;
>> +	struct rkisp1_pipeline pipe;
>> +	struct vb2_alloc_ctx *alloc_ctx;
>> +};
>> +
>> +#endif
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver
  2019-08-08 21:59     ` Helen Koike
@ 2019-08-09 12:05       ` Sakari Ailus
  0 siblings, 0 replies; 76+ messages in thread
From: Sakari Ailus @ 2019-08-09 12:05 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Thu, Aug 08, 2019 at 06:59:47PM -0300, Helen Koike wrote:
> Hi Sakari,
> 
> Thanks for your review, just some comments/questions below:
> 
> On 8/7/19 12:27 PM, Sakari Ailus wrote:
> > Hi Helen,
> > 
> > On Tue, Jul 30, 2019 at 03:42:51PM -0300, Helen Koike wrote:
> >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> >>
> >> Add the core driver for rockchip isp1.
> >>
> >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> >> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> >> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> >> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> >> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> >> [fixed compilation and run time errors regarding new v4l2 async API]
> >> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >> [Add missing module device table]
> >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> >> [update for upstream]
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>
> >> ---
> >>
> >> Changes in v8: None
> >> Changes in v7:
> >> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
> >> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
> >> - Fix compilation and runtime errors due to bitrotting
> >> The code has bit-rotten since March 2018, fix compilation errors.
> >> The new V4L2 async notifier API requires notifiers to be initialized by
> >> a call to v4l2_async_notifier_init() before being used, do so.
> >> - Add missing module device table
> >> - use clk_bulk framework
> >> - add missing notifiers cleanups
> >> - s/strlcpy/strscpy
> >> - normalize bus_info name
> >> - fix s_stream error path, stream_cnt wans't being decremented properly
> >> - use devm_platform_ioremap_resource() helper
> >> - s/deice/device
> >> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> >> isp subdevice in the media topology now.
> >> - remove "saved_state" member from rkisp1_stream struct
> >> - Reverse the order of MIs
> >> - Simplify MI interrupt handling
> >> Rather than adding unnecessary indirection, just use stream index to
> >> handle MI interrupt enable/disable/clear, since the stream index matches
> >> the order of bits now, thanks to previous patch. While at it, remove
> >> some dead code.
> >> - code styling and checkpatch fixes
> >>
> >>  drivers/media/platform/Kconfig                |  12 +
> >>  drivers/media/platform/Makefile               |   1 +
> >>  drivers/media/platform/rockchip/isp1/Makefile |   7 +
> >>  drivers/media/platform/rockchip/isp1/common.h | 101 +++
> >>  drivers/media/platform/rockchip/isp1/dev.c    | 675 ++++++++++++++++++
> >>  drivers/media/platform/rockchip/isp1/dev.h    |  97 +++
> >>  6 files changed, 893 insertions(+)
> >>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
> >>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
> >>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
> >>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
> >>
> >> diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
> >> index 89555f9a813f..e0e98937c565 100644
> >> --- a/drivers/media/platform/Kconfig
> >> +++ b/drivers/media/platform/Kconfig
> >> @@ -106,6 +106,18 @@ config VIDEO_QCOM_CAMSS
> >>  	select VIDEOBUF2_DMA_SG
> >>  	select V4L2_FWNODE
> >>  
> >> +config VIDEO_ROCKCHIP_ISP1
> >> +	tristate "Rockchip Image Signal Processing v1 Unit driver"
> >> +	depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
> >> +	depends on ARCH_ROCKCHIP || COMPILE_TEST
> >> +	select VIDEOBUF2_DMA_CONTIG
> >> +	select VIDEOBUF2_VMALLOC
> >> +	select V4L2_FWNODE
> >> +	select PHY_ROCKCHIP_DPHY
> >> +	default n
> >> +	---help---
> >> +	  Support for ISP1 on the rockchip SoC.
> >> +
> >>  config VIDEO_S3C_CAMIF
> >>  	tristate "Samsung S3C24XX/S3C64XX SoC Camera Interface driver"
> >>  	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API
> >> diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
> >> index 7cbbd925124c..f9fcf8e7c513 100644
> >> --- a/drivers/media/platform/Makefile
> >> +++ b/drivers/media/platform/Makefile
> >> @@ -69,6 +69,7 @@ obj-$(CONFIG_VIDEO_RENESAS_FDP1)	+= rcar_fdp1.o
> >>  obj-$(CONFIG_VIDEO_RENESAS_JPU)		+= rcar_jpu.o
> >>  obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
> >>  
> >> +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1)	+= rockchip/isp1/
> >>  obj-$(CONFIG_VIDEO_ROCKCHIP_RGA)	+= rockchip/rga/
> >>  
> >>  obj-y	+= omap/
> >> diff --git a/drivers/media/platform/rockchip/isp1/Makefile b/drivers/media/platform/rockchip/isp1/Makefile
> >> new file mode 100644
> >> index 000000000000..72706e80fc8b
> >> --- /dev/null
> >> +++ b/drivers/media/platform/rockchip/isp1/Makefile
> >> @@ -0,0 +1,7 @@
> >> +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP1) += 	rockchip-isp1.o
> >> +rockchip-isp1-objs 	   += 	rkisp1.o \
> >> +				dev.o \
> >> +				regs.o \
> >> +				isp_stats.o \
> >> +				isp_params.o \
> >> +				capture.o
> >> diff --git a/drivers/media/platform/rockchip/isp1/common.h b/drivers/media/platform/rockchip/isp1/common.h
> >> new file mode 100644
> >> index 000000000000..606ce2793546
> >> --- /dev/null
> >> +++ b/drivers/media/platform/rockchip/isp1/common.h
> >> @@ -0,0 +1,101 @@
> >> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> >> +/*
> >> + * Rockchip isp1 driver
> >> + *
> >> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> >> + */
> >> +
> >> +#ifndef _RKISP1_COMMON_H
> >> +#define _RKISP1_COMMON_H
> >> +
> >> +#include <linux/mutex.h>
> >> +#include <media/media-device.h>
> >> +#include <media/media-entity.h>
> >> +#include <media/v4l2-ctrls.h>
> >> +#include <media/v4l2-device.h>
> >> +#include <media/videobuf2-v4l2.h>
> >> +
> >> +#define RKISP1_DEFAULT_WIDTH		800
> >> +#define RKISP1_DEFAULT_HEIGHT		600
> >> +
> >> +#define RKISP1_MAX_STREAM		2
> >> +#define RKISP1_STREAM_MP		0
> >> +#define RKISP1_STREAM_SP		1
> >> +
> >> +#define RKISP1_PLANE_Y			0
> >> +#define RKISP1_PLANE_CB			1
> >> +#define RKISP1_PLANE_CR			2
> >> +
> >> +enum rkisp1_sd_type {
> >> +	RKISP1_SD_SENSOR,
> >> +	RKISP1_SD_PHY_CSI,
> >> +	RKISP1_SD_VCM,
> >> +	RKISP1_SD_FLASH,
> >> +	RKISP1_SD_MAX,
> >> +};
> > 
> > I wonder if this is a leftover from the driver development time. Same goes
> > for the subdevs field in struct rkisp1_device.
> 
> It is a left over, I'm removing them for the next version, thanks.
> 
> > 
> >> +
> >> +/* One structure per video node */
> >> +struct rkisp1_vdev_node {
> >> +	struct vb2_queue buf_queue;
> >> +	/* vfd lock */
> >> +	struct mutex vlock;
> >> +	struct video_device vdev;
> >> +	struct media_pad pad;
> >> +};
> >> +
> >> +enum rkisp1_fmt_pix_type {
> >> +	FMT_YUV,
> >> +	FMT_RGB,
> >> +	FMT_BAYER,
> >> +	FMT_JPEG,
> >> +	FMT_MAX
> >> +};
> >> +
> >> +enum rkisp1_fmt_raw_pat_type {
> >> +	RAW_RGGB = 0,
> >> +	RAW_GRBG,
> >> +	RAW_GBRG,
> >> +	RAW_BGGR,
> >> +};
> >> +
> >> +struct rkisp1_buffer {
> >> +	struct vb2_v4l2_buffer vb;
> >> +	struct list_head queue;
> >> +	union {
> >> +		u32 buff_addr[VIDEO_MAX_PLANES];
> >> +		void *vaddr[VIDEO_MAX_PLANES];
> >> +	};
> >> +};
> >> +
> >> +struct rkisp1_dummy_buffer {
> >> +	void *vaddr;
> >> +	dma_addr_t dma_addr;
> >> +	u32 size;
> >> +};
> >> +
> >> +extern int rkisp1_debug;
> >> +
> >> +static inline
> >> +struct rkisp1_vdev_node *vdev_to_node(struct video_device *vdev)
> >> +{
> >> +	return container_of(vdev, struct rkisp1_vdev_node, vdev);
> >> +}
> >> +
> >> +static inline struct rkisp1_vdev_node *queue_to_node(struct vb2_queue *q)
> >> +{
> >> +	return container_of(q, struct rkisp1_vdev_node, buf_queue);
> >> +}
> >> +
> >> +static inline struct rkisp1_buffer *to_rkisp1_buffer(struct vb2_v4l2_buffer *vb)
> >> +{
> >> +	return container_of(vb, struct rkisp1_buffer, vb);
> >> +}
> >> +
> >> +static inline struct vb2_queue *to_vb2_queue(struct file *file)
> >> +{
> >> +	struct rkisp1_vdev_node *vnode = video_drvdata(file);
> >> +
> >> +	return &vnode->buf_queue;
> >> +}
> >> +
> >> +#endif /* _RKISP1_COMMON_H */
> >> diff --git a/drivers/media/platform/rockchip/isp1/dev.c b/drivers/media/platform/rockchip/isp1/dev.c
> >> new file mode 100644
> >> index 000000000000..2b4a67e1a3b5
> >> --- /dev/null
> >> +++ b/drivers/media/platform/rockchip/isp1/dev.c
> >> @@ -0,0 +1,675 @@
> >> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> >> +/*
> >> + * Rockchip isp1 driver
> >> + *
> >> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> >> + */
> >> +
> >> +#include <linux/clk.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/module.h>
> >> +#include <linux/of.h>
> >> +#include <linux/of_graph.h>
> >> +#include <linux/of_platform.h>
> >> +#include <linux/pm_runtime.h>
> >> +#include <linux/pinctrl/consumer.h>
> >> +#include <linux/phy/phy.h>
> >> +#include <linux/phy/phy-mipi-dphy.h>
> >> +
> >> +#include "common.h"
> >> +#include "regs.h"
> >> +
> >> +struct isp_match_data {
> >> +	const char * const *clks;
> >> +	int size;
> > 
> > unsigned int
> 
> ack
> 
> > 
> >> +};
> >> +
> >> +struct sensor_async_subdev {
> >> +	struct v4l2_async_subdev asd;
> >> +	struct v4l2_mbus_config mbus;
> >> +	unsigned int lanes;
> >> +};
> >> +
> >> +int rkisp1_debug;
> >> +module_param_named(debug, rkisp1_debug, int, 0644);
> >> +MODULE_PARM_DESC(debug, "Debug level (0-1)");
> > 
> > Have you thought of using dynamic debug instead?
> 
> right, this is being used in v4l2_dbg(), which I can replace by dev_dbg()
> that is already covered by dynamic debug iirc.
> Should I also replace v4l2_err() by dev_err() (I always get confused by
> which log function I should use).

In case you switch to the dev_*() macros, then yes. The corresponding
v4l2_*() macros are still an entirely valid interface to use. But today we
likely wouldn't add such macros; I usually ask driver developers using both
to switch to the dev_*() ones.

> 
> > 
> >> +
> >> +/**************************** pipeline operations******************************/
> >> +
> >> +static int __isp_pipeline_prepare(struct rkisp1_pipeline *p,
> >> +				  struct media_entity *me)
> >> +{
> >> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
> >> +	struct v4l2_subdev *sd;
> >> +	unsigned int i;
> >> +
> >> +	p->num_subdevs = 0;
> >> +	memset(p->subdevs, 0, sizeof(p->subdevs));
> >> +
> >> +	while (1) {
> >> +		struct media_pad *pad = NULL;
> >> +
> >> +		/* Find remote source pad */
> >> +		for (i = 0; i < me->num_pads; i++) {
> >> +			struct media_pad *spad = &me->pads[i];
> >> +
> >> +			if (!(spad->flags & MEDIA_PAD_FL_SINK))
> >> +				continue;
> >> +			pad = media_entity_remote_pad(spad);
> >> +			if (pad)
> >> +				break;
> >> +		}
> >> +
> >> +		if (!pad)
> >> +			break;
> >> +
> >> +		sd = media_entity_to_v4l2_subdev(pad->entity);
> >> +		if (sd != &dev->isp_sdev.sd)
> >> +			p->subdevs[p->num_subdevs++] = sd;
> > 
> > How do you make sure you don't overrun the array?
> 
> Because the maximum path the topology can have is:
> sensor->rkisp->capture

The sensor may consist of more than one sub-device, and there may be
bridges such as parallel/CSI-2 ones in between. The latter would be an
unlikely hardware configuration for this ISP though as it supports both.

External ISPs in front of the camera sensor are another possibility.

Basically you can't make expectations of the topology outside the scope of
your own device --- the topology of which is known.

> 
> > 
> > Instead, I'd avoid maintaining the array in the first place --- the same
> > information is available from the MC framework data structures --- see e.g.
> > the omap3isp driver.
> 
> If I understand correctly, omap3isp navigates through the topology in the same way,
> but it just don't save in an array, but I reuse this information in other places,
> mostly for power up/down (see below why I don't use v4l2_pipeline_pm_use())

Correct.

> 
> > 
> >> +
> >> +		me = &sd->entity;
> >> +		if (me->num_pads == 1)
> >> +			break;
> >> +	}
> >> +	return 0;
> >> +}
> >> +
> >> +static int __subdev_set_power(struct v4l2_subdev *sd, int on)
> >> +{
> >> +	int ret;
> >> +
> >> +	if (!sd)
> >> +		return -ENXIO;
> >> +
> >> +	ret = v4l2_subdev_call(sd, core, s_power, on);
> >> +
> >> +	return ret != -ENOIOCTLCMD ? ret : 0;
> >> +}
> >> +
> >> +static int __isp_pipeline_s_power(struct rkisp1_pipeline *p, bool on)
> > 
> > Could you instead use v4l2_pipeline_pm_use()?
> 
> Unless I misunderstood this (which is very likely), v4l2_pipeline_pm_use() calls pipeline_pm_power(),
> that applies power change to all entities in a pipeline, but if I have two sensors
> connected to the ISP, one with link enabled and the other with a disabled link,
> I don't want to power both sensors on, just the one participating in that stream. And
> if I call v4l2_pipeline_pm_use() it will power on both, which is not what I want.
> 
> Does it make sense?

The v4l2_pipeline_pm_use() only follows the active links. A simplistic
implementation could power on all the subdevs in the graph but no; the
v4l2_pipeline_pm_use() exists for this very purpose.

> 
> > 
> >> +{
> >> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
> >> +	int i, ret;
> >> +
> >> +	if (on) {
> >> +		__subdev_set_power(&dev->isp_sdev.sd, true);
> >> +
> >> +		for (i = p->num_subdevs - 1; i >= 0; --i) {
> >> +			ret = __subdev_set_power(p->subdevs[i], true);
> >> +			if (ret < 0 && ret != -ENXIO)
> >> +				goto err_power_off;
> >> +		}
> >> +	} else {
> >> +		for (i = 0; i < p->num_subdevs; ++i)
> >> +			__subdev_set_power(p->subdevs[i], false);
> >> +
> >> +		__subdev_set_power(&dev->isp_sdev.sd, false);
> >> +	}
> >> +
> >> +	return 0;
> >> +
> >> +err_power_off:
> >> +	for (++i; i < p->num_subdevs; ++i)
> >> +		__subdev_set_power(p->subdevs[i], false);
> >> +	__subdev_set_power(&dev->isp_sdev.sd, true);
> >> +	return ret;
> >> +}
> >> +
> >> +static int rkisp1_pipeline_open(struct rkisp1_pipeline *p,
> >> +				struct media_entity *me,
> >> +				bool prepare)
> >> +{
> >> +	int ret;
> >> +
> >> +	if (WARN_ON(!p || !me))
> >> +		return -EINVAL;
> >> +	if (atomic_inc_return(&p->power_cnt) > 1)
> >> +		return 0;
> >> +
> >> +	/* go through media graphic and get subdevs */
> >> +	if (prepare)
> >> +		__isp_pipeline_prepare(p, me);
> >> +
> >> +	if (!p->num_subdevs)
> >> +		return -EINVAL;
> >> +
> >> +	ret = __isp_pipeline_s_power(p, 1);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int rkisp1_pipeline_close(struct rkisp1_pipeline *p)
> >> +{
> >> +	int ret;
> >> +
> >> +	if (atomic_dec_return(&p->power_cnt) > 0)
> >> +		return 0;
> >> +	ret = __isp_pipeline_s_power(p, 0);
> >> +
> >> +	return ret == -ENXIO ? 0 : ret;
> >> +}
> >> +
> >> +/*
> >> + * stream-on order: isp_subdev, mipi dphy, sensor
> >> + * stream-off order: mipi dphy, sensor, isp_subdev
> >> + */
> >> +static int rkisp1_pipeline_set_stream(struct rkisp1_pipeline *p, bool on)
> >> +{
> >> +	struct rkisp1_device *dev = container_of(p, struct rkisp1_device, pipe);
> >> +	int i, ret;
> >> +
> >> +	if ((on && atomic_inc_return(&p->stream_cnt) > 1) ||
> >> +	    (!on && atomic_dec_return(&p->stream_cnt) > 0))
> >> +		return 0;
> >> +
> >> +	if (on) {
> >> +		ret = v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream,
> >> +				       true);
> >> +		if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV) {
> >> +			v4l2_err(&dev->v4l2_dev,
> >> +				 "s_stream failed on subdevice %s (%d)\n",
> >> +				 dev->isp_sdev.sd.name,
> >> +				 ret);
> >> +			atomic_dec(&p->stream_cnt);
> >> +			return ret;
> >> +		}
> >> +	}
> >> +
> >> +	/* phy -> sensor */
> >> +	for (i = 0; i < p->num_subdevs; ++i) {
> >> +		ret = v4l2_subdev_call(p->subdevs[i], video, s_stream, on);
> >> +		if (on && ret < 0 && ret != -ENOIOCTLCMD && ret != -ENODEV)
> >> +			goto err_stream_off;
> > 
> > You should stop after the first external sub-device.
> > 
> > It seems even the omap3isp driver doesn't do that. It's not easy to spot
> > such issues indeed.
> 
> I'm not sure I understand, what do you call an external sub-device? Is the sensor
> an external subdevice?

One that is not controlled by this driver.

> 
> > 
> >> +	}
> >> +
> >> +	if (!on)
> >> +		v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
> >> +
> >> +	return 0;
> >> +
> >> +err_stream_off:
> >> +	for (--i; i >= 0; --i)
> >> +		v4l2_subdev_call(p->subdevs[i], video, s_stream, false);
> >> +	v4l2_subdev_call(&dev->isp_sdev.sd, video, s_stream, false);
> >> +	atomic_dec(&p->stream_cnt);
> >> +	return ret;
> >> +}
> >> +
> >> +/***************************** media controller *******************************/
> >> +/* See http://opensource.rock-chips.com/wiki_Rockchip-isp1 for Topology */
> > 
> > The host appears to be down, or there's a routing problem. Unless this is
> > fixed, having such a URL here doesn't do much good. :-I
> 
> This is a left over, sorry about that.
> I can access the URL now. I'll try to get some of the docs and move to the kernel docs.

I wonder if the server is only powered on during the local office hours.
;-)

More seriously, I looked into this and it seems there's a routing problem
somewhere between my ISP and that server. From a few other places the
server is reachable.

> 
> > 
> >> +
> >> +static int rkisp1_create_links(struct rkisp1_device *dev)
> >> +{
> >> +	struct media_entity *source, *sink;
> >> +	struct rkisp1_sensor *sensor;
> >> +	unsigned int flags, pad;
> >> +	int ret;
> >> +
> >> +	/* sensor links(or mipi-phy) */
> >> +	list_for_each_entry(sensor, &dev->sensors, list) {
> >> +		for (pad = 0; pad < sensor->sd->entity.num_pads; pad++)
> >> +			if (sensor->sd->entity.pads[pad].flags &
> >> +				MEDIA_PAD_FL_SOURCE)
> >> +				break;
> > 
> > Could you use media_entity_get_fwnode_pad() instead?
> 
> Yes, I didn't know about it actually, thanks for that, looks cleaner (I'll send in
> the next version).
> 
> > 
> >> +
> >> +		if (pad == sensor->sd->entity.num_pads) {
> >> +			dev_err(dev->dev,
> >> +				"failed to find src pad for %s\n",
> >> +				sensor->sd->name);
> >> +
> >> +			return -ENXIO;
> >> +		}
> >> +
> >> +		ret = media_create_pad_link(
> >> +				&sensor->sd->entity, pad,
> >> +				&dev->isp_sdev.sd.entity,
> >> +				RKISP1_ISP_PAD_SINK,
> >> +				list_is_first(&sensor->list, &dev->sensors) ?
> >> +				MEDIA_LNK_FL_ENABLED : 0);
> >> +		if (ret) {
> >> +			dev_err(dev->dev,
> >> +				"failed to create link for %s\n",
> >> +				sensor->sd->name);
> >> +			return ret;
> >> +		}
> >> +	}
> >> +
> >> +	/* params links */
> >> +	source = &dev->params_vdev.vnode.vdev.entity;
> >> +	sink = &dev->isp_sdev.sd.entity;
> >> +	flags = MEDIA_LNK_FL_ENABLED;
> >> +	ret = media_create_pad_link(source, 0, sink,
> >> +				       RKISP1_ISP_PAD_SINK_PARAMS, flags);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	/* create isp internal links */
> >> +	/* SP links */
> >> +	source = &dev->isp_sdev.sd.entity;
> >> +	sink = &dev->stream[RKISP1_STREAM_SP].vnode.vdev.entity;
> >> +	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
> >> +				       sink, 0, flags);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	/* MP links */
> >> +	source = &dev->isp_sdev.sd.entity;
> >> +	sink = &dev->stream[RKISP1_STREAM_MP].vnode.vdev.entity;
> >> +	ret = media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_PATH,
> >> +				       sink, 0, flags);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	/* 3A stats links */
> >> +	source = &dev->isp_sdev.sd.entity;
> >> +	sink = &dev->stats_vdev.vnode.vdev.entity;
> >> +	return media_create_pad_link(source, RKISP1_ISP_PAD_SOURCE_STATS,
> >> +					sink, 0, flags);
> > 
> > Indentation. Same for the calls to the same function above.
> 
> ack
> 
> > 
> >> +}
> >> +
> >> +static int subdev_notifier_bound(struct v4l2_async_notifier *notifier,
> >> +				 struct v4l2_subdev *sd,
> >> +				 struct v4l2_async_subdev *asd)
> >> +{
> >> +	struct rkisp1_device *isp_dev = container_of(notifier,
> >> +						     struct rkisp1_device,
> >> +						     notifier);
> >> +	struct sensor_async_subdev *s_asd = container_of(asd,
> >> +					struct sensor_async_subdev, asd);
> >> +	struct rkisp1_sensor *sensor;
> >> +
> >> +	sensor = devm_kzalloc(isp_dev->dev, sizeof(*sensor), GFP_KERNEL);
> >> +	if (!sensor)
> >> +		return -ENOMEM;
> >> +
> >> +	sensor->lanes = s_asd->lanes;
> >> +	sensor->mbus = s_asd->mbus;
> >> +	sensor->sd = sd;
> >> +	sensor->dphy = devm_phy_get(isp_dev->dev, "dphy");
> >> +	if (IS_ERR(sensor->dphy)) {
> >> +		if (PTR_ERR(sensor->dphy) != -EPROBE_DEFER)
> >> +			dev_err(isp_dev->dev, "Couldn't get the MIPI D-PHY\n");
> >> +		return PTR_ERR(sensor->dphy);
> >> +	}
> >> +	phy_init(sensor->dphy);
> >> +
> >> +	list_add(&sensor->list, &isp_dev->sensors);
> > 
> > In general, maintaining the information on the external subdevs on your own
> > adds complexity to the driver. You can get the information when you need it
> > from the data structures maintained by MC (see e.g. the omap3isp driver for
> > examples).
> > 
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
> >> +					  struct v4l2_subdev *sd)
> >> +{
> >> +	struct rkisp1_sensor *sensor;
> >> +
> >> +	list_for_each_entry(sensor, &dev->sensors, list)
> >> +		if (sensor->sd == sd)
> >> +			return sensor;
> >> +
> >> +	return NULL;
> >> +}
> >> +
> >> +static void subdev_notifier_unbind(struct v4l2_async_notifier *notifier,
> >> +				   struct v4l2_subdev *sd,
> >> +				   struct v4l2_async_subdev *asd)
> >> +{
> >> +	struct rkisp1_device *isp_dev = container_of(notifier,
> >> +						     struct rkisp1_device,
> >> +						     notifier);
> >> +	struct rkisp1_sensor *sensor = sd_to_sensor(isp_dev, sd);
> >> +
> >> +	/* TODO: check if a lock is required here */
> >> +	list_del(&sensor->list);
> >> +
> >> +	phy_exit(sensor->dphy);
> >> +}
> >> +
> >> +static int subdev_notifier_complete(struct v4l2_async_notifier *notifier)
> >> +{
> >> +	struct rkisp1_device *dev = container_of(notifier, struct rkisp1_device,
> >> +						 notifier);
> >> +	int ret;
> >> +
> >> +	mutex_lock(&dev->media_dev.graph_mutex);
> >> +	ret = rkisp1_create_links(dev);
> >> +	if (ret < 0)
> >> +		goto unlock;
> >> +	ret = v4l2_device_register_subdev_nodes(&dev->v4l2_dev);
> >> +	if (ret < 0)
> >> +		goto unlock;
> >> +
> >> +	v4l2_info(&dev->v4l2_dev, "Async subdev notifier completed\n");
> >> +
> >> +unlock:
> >> +	mutex_unlock(&dev->media_dev.graph_mutex);
> >> +	return ret;
> >> +}
> >> +
> >> +static int rkisp1_fwnode_parse(struct device *dev,
> >> +			       struct v4l2_fwnode_endpoint *vep,
> >> +			       struct v4l2_async_subdev *asd)
> >> +{
> >> +	struct sensor_async_subdev *s_asd =
> >> +			container_of(asd, struct sensor_async_subdev, asd);
> >> +
> >> +	if (vep->bus_type != V4L2_MBUS_CSI2_DPHY) {
> >> +		dev_err(dev, "Only CSI2 bus type is currently supported\n");
> >> +		return -EINVAL;
> >> +	}
> >> +
> >> +	if (vep->base.port != 0) {
> >> +		dev_err(dev, "The ISP has only port 0\n");
> >> +		return -EINVAL;
> >> +	}
> >> +
> >> +	s_asd->mbus.type = vep->bus_type;
> >> +	s_asd->mbus.flags = vep->bus.mipi_csi2.flags;
> >> +	s_asd->lanes = vep->bus.mipi_csi2.num_data_lanes;
> >> +
> >> +	switch (vep->bus.mipi_csi2.num_data_lanes) {
> >> +	case 1:
> >> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_1_LANE;
> >> +		break;
> >> +	case 2:
> >> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_2_LANE;
> >> +		break;
> >> +	case 3:
> >> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_3_LANE;
> >> +		break;
> >> +	case 4:
> >> +		s_asd->mbus.flags |= V4L2_MBUS_CSI2_4_LANE;
> >> +		break;
> >> +	default:
> >> +		return -EINVAL;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct v4l2_async_notifier_operations subdev_notifier_ops = {
> >> +	.bound = subdev_notifier_bound,
> >> +	.unbind = subdev_notifier_unbind,
> >> +	.complete = subdev_notifier_complete,
> >> +};
> >> +
> >> +static int isp_subdev_notifier(struct rkisp1_device *isp_dev)
> >> +{
> >> +	struct v4l2_async_notifier *ntf = &isp_dev->notifier;
> >> +	struct device *dev = isp_dev->dev;
> >> +	int ret;
> >> +
> >> +	v4l2_async_notifier_init(ntf);
> >> +
> >> +	ret = v4l2_async_notifier_parse_fwnode_endpoints_by_port(
> >> +		dev, ntf, sizeof(struct sensor_async_subdev), 0,
> >> +		rkisp1_fwnode_parse);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	if (list_empty(&ntf->asd_list))
> >> +		return -ENODEV;	/* no endpoint */
> >> +
> >> +	ntf->ops = &subdev_notifier_ops;
> >> +
> >> +	return v4l2_async_notifier_register(&isp_dev->v4l2_dev, ntf);
> >> +}
> >> +
> >> +/***************************** platform device *******************************/
> >> +
> >> +static int rkisp1_register_platform_subdevs(struct rkisp1_device *dev)
> >> +{
> >> +	int ret;
> >> +
> >> +	ret = rkisp1_register_isp_subdev(dev, &dev->v4l2_dev);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	ret = rkisp1_register_stream_vdevs(dev);
> >> +	if (ret < 0)
> >> +		goto err_unreg_isp_subdev;
> >> +
> >> +	ret = rkisp1_register_stats_vdev(&dev->stats_vdev, &dev->v4l2_dev, dev);
> >> +	if (ret < 0)
> >> +		goto err_unreg_stream_vdev;
> >> +
> >> +	ret = rkisp1_register_params_vdev(&dev->params_vdev, &dev->v4l2_dev,
> >> +					  dev);
> >> +	if (ret < 0)
> >> +		goto err_unreg_stats_vdev;
> >> +
> >> +	ret = isp_subdev_notifier(dev);
> >> +	if (ret < 0) {
> >> +		v4l2_err(&dev->v4l2_dev,
> >> +			 "Failed to register subdev notifier(%d)\n", ret);
> >> +		goto err_unreg_params_vdev;
> >> +	}
> >> +
> >> +	return 0;
> >> +err_unreg_params_vdev:
> >> +	rkisp1_unregister_params_vdev(&dev->params_vdev);
> >> +err_unreg_stats_vdev:
> >> +	rkisp1_unregister_stats_vdev(&dev->stats_vdev);
> >> +err_unreg_stream_vdev:
> >> +	rkisp1_unregister_stream_vdevs(dev);
> >> +err_unreg_isp_subdev:
> >> +	rkisp1_unregister_isp_subdev(dev);
> >> +	return ret;
> >> +}
> >> +
> >> +static const char * const rk3399_isp_clks[] = {
> >> +	"clk_isp",
> >> +	"aclk_isp",
> >> +	"hclk_isp",
> >> +	"aclk_isp_wrap",
> >> +	"hclk_isp_wrap",
> >> +};
> >> +
> >> +static const char * const rk3288_isp_clks[] = {
> >> +	"clk_isp",
> >> +	"aclk_isp",
> >> +	"hclk_isp",
> >> +	"pclk_isp_in",
> >> +	"sclk_isp_jpe",
> >> +};
> >> +
> >> +static const struct isp_match_data rk3288_isp_clk_data = {
> >> +	.clks = rk3288_isp_clks,
> >> +	.size = ARRAY_SIZE(rk3288_isp_clks),
> >> +};
> >> +
> >> +static const struct isp_match_data rk3399_isp_clk_data = {
> >> +	.clks = rk3399_isp_clks,
> >> +	.size = ARRAY_SIZE(rk3399_isp_clks),
> >> +};
> >> +
> >> +static const struct of_device_id rkisp1_plat_of_match[] = {
> >> +	{
> >> +		.compatible = "rockchip,rk3288-cif-isp",
> >> +		.data = &rk3288_isp_clk_data,
> >> +	}, {
> >> +		.compatible = "rockchip,rk3399-cif-isp",
> >> +		.data = &rk3399_isp_clk_data,
> >> +	},
> >> +	{},
> >> +};
> >> +MODULE_DEVICE_TABLE(of, rkisp1_plat_of_match);
> >> +
> >> +static irqreturn_t rkisp1_irq_handler(int irq, void *ctx)
> >> +{
> >> +	struct device *dev = ctx;
> >> +	struct rkisp1_device *rkisp1_dev = dev_get_drvdata(dev);
> >> +	unsigned int mis_val;
> >> +
> >> +	mis_val = readl(rkisp1_dev->base_addr + CIF_ISP_MIS);
> >> +	if (mis_val)
> >> +		rkisp1_isp_isr(mis_val, rkisp1_dev);
> >> +
> >> +	mis_val = readl(rkisp1_dev->base_addr + CIF_MIPI_MIS);
> >> +	if (mis_val)
> >> +		rkisp1_mipi_isr(mis_val, rkisp1_dev);
> >> +
> >> +	mis_val = readl(rkisp1_dev->base_addr + CIF_MI_MIS);
> >> +	if (mis_val)
> >> +		rkisp1_mi_isr(mis_val, rkisp1_dev);
> >> +
> >> +	return IRQ_HANDLED;
> >> +}
> >> +
> >> +static int rkisp1_plat_probe(struct platform_device *pdev)
> >> +{
> >> +	struct device_node *node = pdev->dev.of_node;
> >> +	const struct isp_match_data *clk_data;
> >> +	const struct of_device_id *match;
> >> +	struct device *dev = &pdev->dev;
> >> +	struct rkisp1_device *isp_dev;
> >> +	struct v4l2_device *v4l2_dev;
> >> +	unsigned int i;
> >> +	int ret, irq;
> >> +
> >> +	match = of_match_node(rkisp1_plat_of_match, node);
> >> +	isp_dev = devm_kzalloc(dev, sizeof(*isp_dev), GFP_KERNEL);
> >> +	if (!isp_dev)
> >> +		return -ENOMEM;
> >> +
> >> +	INIT_LIST_HEAD(&isp_dev->sensors);
> >> +
> >> +	dev_set_drvdata(dev, isp_dev);
> >> +	isp_dev->dev = dev;
> >> +
> >> +	isp_dev->base_addr = devm_platform_ioremap_resource(pdev, 0);
> >> +	if (IS_ERR(isp_dev->base_addr))
> >> +		return PTR_ERR(isp_dev->base_addr);
> >> +
> >> +	irq = platform_get_irq(pdev, 0);
> >> +	if (irq < 0)
> >> +		return irq;
> >> +
> >> +	ret = devm_request_irq(dev, irq, rkisp1_irq_handler, IRQF_SHARED,
> >> +			       dev_driver_string(dev), dev);
> >> +	if (ret < 0) {
> >> +		dev_err(dev, "request irq failed: %d\n", ret);
> >> +		return ret;
> >> +	}
> >> +
> >> +	isp_dev->irq = irq;
> >> +	clk_data = match->data;
> >> +
> >> +	for (i = 0; i < clk_data->size; i++)
> >> +		isp_dev->clks[i].id = clk_data->clks[i];
> >> +	ret = devm_clk_bulk_get(dev, clk_data->size, isp_dev->clks);
> >> +	if (ret)
> >> +		return ret;
> >> +	isp_dev->clk_size = clk_data->size;
> >> +
> >> +	atomic_set(&isp_dev->pipe.power_cnt, 0);
> >> +	atomic_set(&isp_dev->pipe.stream_cnt, 0);
> >> +	isp_dev->pipe.open = rkisp1_pipeline_open;
> >> +	isp_dev->pipe.close = rkisp1_pipeline_close;
> >> +	isp_dev->pipe.set_stream = rkisp1_pipeline_set_stream;
> >> +
> >> +	rkisp1_stream_init(isp_dev, RKISP1_STREAM_SP);
> >> +	rkisp1_stream_init(isp_dev, RKISP1_STREAM_MP);
> >> +
> >> +	strscpy(isp_dev->media_dev.model, "rkisp1",
> >> +		sizeof(isp_dev->media_dev.model));
> >> +	isp_dev->media_dev.dev = &pdev->dev;
> >> +	strscpy(isp_dev->media_dev.bus_info,
> >> +		"platform: " DRIVER_NAME, sizeof(isp_dev->media_dev.bus_info));
> >> +	media_device_init(&isp_dev->media_dev);
> >> +
> >> +	v4l2_dev = &isp_dev->v4l2_dev;
> >> +	v4l2_dev->mdev = &isp_dev->media_dev;
> >> +	strscpy(v4l2_dev->name, "rkisp1", sizeof(v4l2_dev->name));
> >> +	v4l2_ctrl_handler_init(&isp_dev->ctrl_handler, 5);
> >> +	v4l2_dev->ctrl_handler = &isp_dev->ctrl_handler;
> >> +
> >> +	ret = v4l2_device_register(isp_dev->dev, &isp_dev->v4l2_dev);
> >> +	if (ret < 0)
> > 
> > Once you've initialised the control handler, you'll need to free it in case
> > of an error. I.e. add one more label for that purpose near the end.
> 
> control handler is not required, I'll remove it for the next version.
> 
> > 
> >> +		return ret;
> >> +
> >> +	ret = media_device_register(&isp_dev->media_dev);
> >> +	if (ret < 0) {
> >> +		v4l2_err(v4l2_dev, "Failed to register media device: %d\n",
> >> +			 ret);
> >> +		goto err_unreg_v4l2_dev;
> >> +	}
> >> +
> >> +	/* create & register platefom subdev (from of_node) */
> >> +	ret = rkisp1_register_platform_subdevs(isp_dev);
> >> +	if (ret < 0)
> >> +		goto err_unreg_media_dev;
> >> +
> >> +	pm_runtime_enable(&pdev->dev);
> >> +
> >> +	return 0;
> >> +
> >> +err_unreg_media_dev:
> >> +	media_device_unregister(&isp_dev->media_dev);
> >> +err_unreg_v4l2_dev:
> >> +	v4l2_device_unregister(&isp_dev->v4l2_dev);
> >> +	return ret;
> >> +}
> >> +
> >> +static int rkisp1_plat_remove(struct platform_device *pdev)
> >> +{
> >> +	struct rkisp1_device *isp_dev = platform_get_drvdata(pdev);
> >> +
> >> +	pm_runtime_disable(&pdev->dev);
> >> +	media_device_unregister(&isp_dev->media_dev);
> >> +	v4l2_async_notifier_unregister(&isp_dev->notifier);
> >> +	v4l2_async_notifier_cleanup(&isp_dev->notifier);
> >> +	v4l2_device_unregister(&isp_dev->v4l2_dev);
> >> +	rkisp1_unregister_params_vdev(&isp_dev->params_vdev);
> >> +	rkisp1_unregister_stats_vdev(&isp_dev->stats_vdev);
> >> +	rkisp1_unregister_stream_vdevs(isp_dev);
> >> +	rkisp1_unregister_isp_subdev(isp_dev);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int __maybe_unused rkisp1_runtime_suspend(struct device *dev)
> >> +{
> >> +	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
> >> +
> >> +	clk_bulk_disable_unprepare(isp_dev->clk_size, isp_dev->clks);
> >> +	return pinctrl_pm_select_sleep_state(dev);
> >> +}
> >> +
> >> +static int __maybe_unused rkisp1_runtime_resume(struct device *dev)
> >> +{
> >> +	struct rkisp1_device *isp_dev = dev_get_drvdata(dev);
> >> +	int ret;
> >> +
> >> +	ret = pinctrl_pm_select_default_state(dev);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +	ret = clk_bulk_prepare_enable(isp_dev->clk_size, isp_dev->clks);
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct dev_pm_ops rkisp1_plat_pm_ops = {
> >> +	SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
> >> +				pm_runtime_force_resume)
> >> +	SET_RUNTIME_PM_OPS(rkisp1_runtime_suspend, rkisp1_runtime_resume, NULL)
> >> +};
> >> +
> >> +static struct platform_driver rkisp1_plat_drv = {
> >> +	.driver = {
> >> +		.name = DRIVER_NAME,
> >> +		.of_match_table = of_match_ptr(rkisp1_plat_of_match),
> >> +		.pm = &rkisp1_plat_pm_ops,
> >> +	},
> >> +	.probe = rkisp1_plat_probe,
> >> +	.remove = rkisp1_plat_remove,
> >> +};
> >> +
> >> +module_platform_driver(rkisp1_plat_drv);
> >> +MODULE_AUTHOR("Rockchip Camera/ISP team");
> >> +MODULE_DESCRIPTION("Rockchip ISP1 platform driver");
> >> +MODULE_LICENSE("Dual BSD/GPL");
> > 
> > BSD or MIT?
> 
> Thanks for spotting this, I'll verify.

I think this was the only place where the BSD license was present. The file
header has MIT.

> 
> > 
> >> diff --git a/drivers/media/platform/rockchip/isp1/dev.h b/drivers/media/platform/rockchip/isp1/dev.h
> >> new file mode 100644
> >> index 000000000000..f7cbee316523
> >> --- /dev/null
> >> +++ b/drivers/media/platform/rockchip/isp1/dev.h
> >> @@ -0,0 +1,97 @@
> >> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> >> +/*
> >> + * Rockchip isp1 driver
> >> + *
> >> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> >> + */
> >> +
> >> +#ifndef _RKISP1_DEV_H
> >> +#define _RKISP1_DEV_H
> >> +
> >> +#include <linux/clk.h>
> >> +
> >> +#include "capture.h"
> >> +#include "rkisp1.h"
> >> +#include "isp_params.h"
> >> +#include "isp_stats.h"
> >> +
> >> +#define DRIVER_NAME "rkisp1"
> >> +#define ISP_VDEV_NAME DRIVER_NAME  "_ispdev"
> >> +#define SP_VDEV_NAME DRIVER_NAME   "_selfpath"
> >> +#define MP_VDEV_NAME DRIVER_NAME   "_mainpath"
> >> +#define DMA_VDEV_NAME DRIVER_NAME  "_dmapath"
> >> +
> >> +#define GRP_ID_SENSOR			BIT(0)
> >> +#define GRP_ID_MIPIPHY			BIT(1)
> >> +#define GRP_ID_ISP			BIT(2)
> >> +#define GRP_ID_ISP_MP			BIT(3)
> >> +#define GRP_ID_ISP_SP			BIT(4)
> >> +
> >> +#define RKISP1_MAX_BUS_CLK	8
> >> +#define RKISP1_MAX_SENSOR	2
> >> +#define RKISP1_MAX_PIPELINE	4
> >> +
> >> +/*
> >> + * struct rkisp1_pipeline - An ISP hardware pipeline
> >> + *
> >> + * Capture device call other devices via pipeline
> >> + *
> >> + * @num_subdevs: number of linked subdevs
> >> + * @power_cnt: pipeline power count
> >> + * @stream_cnt: stream power count
> >> + */
> >> +struct rkisp1_pipeline {
> >> +	struct media_pipeline pipe;
> >> +	int num_subdevs;
> >> +	atomic_t power_cnt;
> >> +	atomic_t stream_cnt;
> >> +	struct v4l2_subdev *subdevs[RKISP1_MAX_PIPELINE];
> >> +	int (*open)(struct rkisp1_pipeline *p,
> >> +		    struct media_entity *me, bool prepare);
> >> +	int (*close)(struct rkisp1_pipeline *p);
> >> +	int (*set_stream)(struct rkisp1_pipeline *p, bool on);
> >> +};
> >> +
> >> +/*
> >> + * struct rkisp1_sensor - Sensor information
> >> + * @mbus: media bus configuration
> >> + */
> >> +struct rkisp1_sensor {
> >> +	struct v4l2_subdev *sd;
> >> +	struct v4l2_mbus_config mbus;
> >> +	unsigned int lanes;
> >> +	struct phy *dphy;
> >> +	struct list_head list;
> >> +};
> > 
> > You seem to also have struct sensor_async_subdev that appears to contain
> > similar information. Would it be possible to unify the two?
> > 
> > This would appear to allow you getting rid of functions such as
> > sd_to_sensor, for instance.
> 
> ack, I managed to get rid of this, and I don't even need to keep them
> on a list, I'll submit in the next version.

Nice!

> 
> Thanks a lot for your review

You're welcome!

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 00/14] Rockchip ISP1 Driver
  2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
                   ` (14 preceding siblings ...)
  2019-08-07 15:37 ` Sakari Ailus
@ 2019-08-09 18:45 ` Manivannan Sadhasivam
  15 siblings, 0 replies; 76+ messages in thread
From: Manivannan Sadhasivam @ 2019-08-09 18:45 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, laurent.pinchart, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Fri, Aug 09, 2019 at 03:40:02PM -0300, Helen Koike wrote:
> Hello,
> 
> I'm re-sending a new version of ISP(Camera) v4l2 driver for rockchip
> rk3399 SoC.
> 
> I didn't change much from the last version, just applying the
> suggestions made in the previous one.
> 
> This patchset is also available at:
> https://gitlab.collabora.com/koike/linux/tree/rockchip/isp/v8
> 
> Libcamera patched to work with this version:
> https://gitlab.collabora.com/koike/libcamera
> (also sent to the mailing list)
> 
> I tested on the rockpi 4 with a rpi v1.3 sensor and also with the
> Scarlet Chromebook.
> 

I just tested this patchset on Rock960 but getting below error while
configuring media link:

root@linaro-alip:~# media-ctl -p /dev/media0 -v
Opening media device /dev/media0
Enumerating entities
looking up device: 81:4
looking up device: 81:0
looking up device: 81:1
looking up device: 81:2
looking up device: 81:3
looking up device: 81:5
Found 6 entities
Enumerating pads and links
*** Error in `media-ctl': munmap_chunk(): invalid pointer: 0x01ce44d0 ***
Aborted

Here is the change I did for Rock960:
https://pastebin.ubuntu.com/p/CmdcqJ7bsJ/

Did I miss anything?

Thanks,
Mani

> Known issues (same as in v7):
> -------------
> - Reloading the module doesn't work (there is some missing cleanup when
> unloading)
> - When capturing in bayer format, changing the size doesn't seem to
> affect the image.
> - crop needs more tests
> - v4l2-compliance error:
>         fail: v4l2-test-controls.cpp(824): subscribe event for control 'Image Processing Controls' failed
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL
> It seems that if controls are supported, v4l2-compliance says that
> controls of type 'Image Processing Controls' are mandatory, is this
> correct?
> - It seems there are still some issues with interrupts, but I couldn't
> isolate them yet.
> 
> Previous changelog:
> -------------------
> 
> changes in V6:
>   - add mipi txrx phy support
>   - remove bool and enum from uapi header
>   - add buf_prepare op
>   - correct some spelling problems
>   - return all queued buffers when starting stream failed
> 
> changes in V5: Sync with local changes,
>   - fix the SP height limit
>   - speed up the second stream capture
>   - the second stream can't force sync for rsz when start/stop streaming
>   - add frame id to param vb2 buf
>   - enable luminance maximum threshold
> 
> changes in V4:
>   - fix some bugs during development
>   - move quantization settings to rkisp1 subdev
>   - correct some spelling problems
>   - describe ports in dt-binding documents
> 
> changes in V3:
>   - add some comments
>   - fix wrong use of v4l2_async_subdev_notifier_register
>   - optimize two paths capture at a time
>   - remove compose
>   - re-struct headers
>   - add a tmp wiki page: http://opensource.rock-chips.com/wiki_Rockchip-isp1
> 
> changes in V2:
>   mipi-phy:
>     - use async probing
>     - make it be a child device of the GRF
>   isp:
>     - add dummy buffer
>     - change the way to get bus configuration, which make it possible to
>             add parallel sensor support in the future(without mipi-phy driver).
> 
> ------------------
> 
> Changes in v8:
> - Add SPDX in the header
> - Remove emacs configs
> - Fix doc style
> - Remove boiler plate license text
> 
> Changes in v7:
> - s/IPU3/RK_ISP1
> - s/correspond/corresponding
> - s/use/uses
> - s/docuemnt/document
> - Fix checkpatch errors (lines over 80 and SPDX)
> - Add TODO to improve docs
> - Migrate dphy specific code from
> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
> to drivers/phy/rockchip/phy-rockchip-dphy.c
> - Drop support for rk3288
> - Drop support for dphy txrx
> - code styling and checkpatch fixes
> - fixed warning because of unknown entity type
> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> and default values
> - fix typo riksp1/rkisp1
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now. As a consequence, remove the
> hack in mipidphy_g_mbus_config() where information from the sensor was
> being propagated through the topology.
> - From the old dphy:
>         * cache get_remote_sensor() in s_stream
>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> - Replace stream state with a boolean
> - code styling and checkpatch fixes
> - fix stop_stream (return after calling stop, do not reenable the stream)
> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> - s/intput/input
> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> reused by the capture
> - s/strlcpy/strscpy
> - sort out the locks in isp stats
> - code styling and checkpatch fixes
> - s/strlcpy/strscpy
> - s/strcpy/strscpy
> - fix config lsc error
> LSC data table size is 17x17, but when configuring data to ISP,
> should be aligned to 18x17. That means every last data of last
> line should be filled with 0, and not filled with the data of
> next line.
> - Update new ISP parameters immediately
> For those sub modules that have shadow registers in core isp, the
> new programing parameters would not be active if both
> CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT and CFG_UPD are not set. Now
> we configure CFG_UPD to force update the shadow registers when new
> ISP parameters are configured.
> - fix some ISP parameters config error
> Some ISP parameter config functions may override the old enable
> bit value, because the enable bits of these modules are in the
> same registers with parameters. So we should save the old enable
> bits firstly.
> - code styling and checkpatch fixes
> - s/strlcpy/strscpy
> - Fix v4l2-compliance issues:
>         * remove input ioctls
> media api can be used to define the topology, this input api is not
> required. Besides it, if an input is enumerated, v4l2-compliance is not
> happy with G_FMT returning the default colorspace instead of something
> more specific.
>         * return the pixelformat to the userspace
> G_/S_/TRY_ FORMAT should return a valid pixelformat to the user, even if
> the user gave an invalid one
>         * add missing default colorspace and ycbcr
>         * fix wrong pixformat in mp_fmts[] table
>         * add buf type check in s_/g_selection
>         * queue_setup - check sizes
>         * normalize bus_info name
>         * fix field any v4l2-compliance -s complain - set field none
>         when streaming
> - Fix compiling error: s/vidioc_enum_fmt_vid_cap_mplane/vidioc_enum_fmt_vid_cap
> - Replace stream state with a boolean
> The rkisp1_state enum consists only of 3 entries, where 1 is completely
> unused and the other two respectively mean not streaming or streaming.
> Replace it with a boolean called "streaming".
> - Simplify MI interrupt handling
> Rather than adding unnecessary indirection, just use stream index to
> handle MI interrupt enable/disable/clear, since the stream index matches
> the order of bits now, thanks to previous patch. While at it, remove
> some dead code.
> - code styling and checkpatch fixes
> - add link_validate: don't allow a link with bayer/non-bayer mismatch
> - VIDEO_ROCKCHIP_ISP1 selects VIDEOBUF2_VMALLOC
> - add PHY_ROCKCHIP_DPHY as a dependency for VIDEO_ROCKCHIP_ISP1
> - Fix compilation and runtime errors due to bitrotting
> The code has bit-rotten since March 2018, fix compilation errors.
> The new V4L2 async notifier API requires notifiers to be initialized by
> a call to v4l2_async_notifier_init() before being used, do so.
> - Add missing module device table
> - use clk_bulk framework
> - add missing notifiers cleanups
> - s/strlcpy/strscpy
> - normalize bus_info name
> - fix s_stream error path, stream_cnt wans't being decremented properly
> - use devm_platform_ioremap_resource() helper
> - s/deice/device
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now.
> - remove "saved_state" member from rkisp1_stream struct
> - Reverse the order of MIs
> - Simplify MI interrupt handling
> Rather than adding unnecessary indirection, just use stream index to
> handle MI interrupt enable/disable/clear, since the stream index matches
> the order of bits now, thanks to previous patch. While at it, remove
> some dead code.
> - code styling and checkpatch fixes
> - update document with new design and tested example
> - updated doc with new design and tested example
> - add phy properties
> - add ports
> - add phy-cells
> 
> Helen Koike (1):
>   MAINTAINERS: add entry for Rockchip ISP1 driver
> 
> Jacob Chen (9):
>   media: doc: add document for rkisp1 meta buffer format
>   media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
>   media: rkisp1: add Rockchip ISP1 subdev driver
>   media: rkisp1: add ISP1 statistics driver
>   media: rkisp1: add ISP1 params driver
>   media: rkisp1: add capture device driver
>   media: rkisp1: add rockchip isp1 core driver
>   dt-bindings: Document the Rockchip ISP1 bindings
>   dt-bindings: Document the Rockchip MIPI RX D-PHY bindings
> 
> Jeffy Chen (1):
>   media: rkisp1: Add user space ABI definitions
> 
> Shunqian Zheng (3):
>   media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
>   arm64: dts: rockchip: add isp0 node for rk3399
>   arm64: dts: rockchip: add rx0 mipi-phy for rk3399
> 
>  .../bindings/media/rockchip-isp1.txt          |   71 +
>  .../bindings/media/rockchip-mipi-dphy.txt     |   38 +
>  Documentation/media/uapi/v4l/meta-formats.rst |    2 +
>  .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    |   23 +
>  .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      |   22 +
>  MAINTAINERS                                   |    8 +
>  arch/arm64/boot/dts/rockchip/rk3399.dtsi      |   36 +
>  drivers/media/platform/Kconfig                |   12 +
>  drivers/media/platform/Makefile               |    1 +
>  drivers/media/platform/rockchip/isp1/Makefile |    7 +
>  .../media/platform/rockchip/isp1/capture.c    | 1754 +++++++++++++++++
>  .../media/platform/rockchip/isp1/capture.h    |  164 ++
>  drivers/media/platform/rockchip/isp1/common.h |  101 +
>  drivers/media/platform/rockchip/isp1/dev.c    |  675 +++++++
>  drivers/media/platform/rockchip/isp1/dev.h    |   97 +
>  .../media/platform/rockchip/isp1/isp_params.c | 1604 +++++++++++++++
>  .../media/platform/rockchip/isp1/isp_params.h |   50 +
>  .../media/platform/rockchip/isp1/isp_stats.c  |  508 +++++
>  .../media/platform/rockchip/isp1/isp_stats.h  |   60 +
>  drivers/media/platform/rockchip/isp1/regs.c   |  223 +++
>  drivers/media/platform/rockchip/isp1/regs.h   | 1525 ++++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 ++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |    2 +
>  drivers/phy/rockchip/Kconfig                  |    8 +
>  drivers/phy/rockchip/Makefile                 |    1 +
>  drivers/phy/rockchip/phy-rockchip-dphy.c      |  408 ++++
>  include/uapi/linux/rkisp1-config.h            |  816 ++++++++
>  include/uapi/linux/videodev2.h                |    4 +
>  29 files changed, 9617 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt
>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
>  create mode 100644 drivers/media/platform/rockchip/isp1/Makefile
>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/capture.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/common.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/dev.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_params.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/isp_stats.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/regs.h
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
>  create mode 100644 include/uapi/linux/rkisp1-config.h
> 
> -- 
> 2.22.0
> 
> 
> -- 
> To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-08  9:14   ` Sakari Ailus
@ 2019-08-15  0:58     ` Helen Koike
  2019-08-15  8:24       ` Sakari Ailus
  0 siblings, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-08-15  0:58 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Sakari,

Thanks for your review. I just have some comments/questions below.

On 8/8/19 6:14 AM, Sakari Ailus wrote:
> Hi Helen,
> 
> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>
>> Add the subdev driver for rockchip isp1.
>>
>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
>> Signed-off-by: Jacob Chen <cc@rock-chips.com>
>> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>> [fixed unknown entity type / switched to PIXEL_RATE]
>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>> [update for upstream]
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>
>> ---
>>
>> Changes in v8: None
>> Changes in v7:
>> - fixed warning because of unknown entity type
>> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
>> and default values
>> - fix typo riksp1/rkisp1
>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>> isp subdevice in the media topology now. As a consequence, remove the
>> hack in mipidphy_g_mbus_config() where information from the sensor was
>> being propagated through the topology.
>> - From the old dphy:
>>         * cache get_remote_sensor() in s_stream
>>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
>> - Replace stream state with a boolean
>> - code styling and checkpatch fixes
>> - fix stop_stream (return after calling stop, do not reenable the stream)
>> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
>> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
>> - s/intput/input
>> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
>> reused by the capture
>>
>>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>>  2 files changed, 1397 insertions(+)
>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>>
>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
>> new file mode 100644
>> index 000000000000..6d0c0ffb5e03
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
>> @@ -0,0 +1,1286 @@
>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>> +/*
>> + * Rockchip isp1 driver
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#include <linux/iopoll.h>
>> +#include <linux/phy/phy.h>
>> +#include <linux/phy/phy-mipi-dphy.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/videodev2.h>
>> +#include <linux/vmalloc.h>
>> +#include <media/v4l2-event.h>
>> +
>> +#include "common.h"
>> +#include "regs.h"
>> +
>> +#define CIF_ISP_INPUT_W_MAX		4032
>> +#define CIF_ISP_INPUT_H_MAX		3024
>> +#define CIF_ISP_INPUT_W_MIN		32
>> +#define CIF_ISP_INPUT_H_MIN		32
>> +#define CIF_ISP_OUTPUT_W_MAX		CIF_ISP_INPUT_W_MAX
>> +#define CIF_ISP_OUTPUT_H_MAX		CIF_ISP_INPUT_H_MAX
>> +#define CIF_ISP_OUTPUT_W_MIN		CIF_ISP_INPUT_W_MIN
>> +#define CIF_ISP_OUTPUT_H_MIN		CIF_ISP_INPUT_H_MIN
>> +
>> +/*
>> + * NOTE: MIPI controller and input MUX are also configured in this file,
>> + * because ISP Subdev is not only describe ISP submodule(input size,format,
>> + * output size, format), but also a virtual route device.
>> + */
>> +
>> +/*
>> + * There are many variables named with format/frame in below code,
>> + * please see here for their meaning.
>> + *
>> + * Cropping regions of ISP
>> + *
>> + * +---------------------------------------------------------+
>> + * | Sensor image                                            |
>> + * | +---------------------------------------------------+   |
>> + * | | ISP_ACQ (for black level)                         |   |
>> + * | | in_frm                                            |   |
>> + * | | +--------------------------------------------+    |   |
>> + * | | |    ISP_OUT                                 |    |   |
>> + * | | |    in_crop                                 |    |   |
>> + * | | |    +---------------------------------+     |    |   |
>> + * | | |    |   ISP_IS                        |     |    |   |
>> + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
>> + * | | |    +---------------------------------+     |    |   |
>> + * | | +--------------------------------------------+    |   |
>> + * | +---------------------------------------------------+   |
>> + * +---------------------------------------------------------+
>> + */
>> +
>> +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
>> +{
>> +	return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
>> +}
>> +
>> +/* Get sensor by enabled media link */
>> +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
>> +{
>> +	struct media_pad *local, *remote;
>> +	struct media_entity *sensor_me;
>> +
>> +	local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
>> +	remote = media_entity_remote_pad(local);
>> +	if (!remote) {
>> +		v4l2_warn(sd, "No link between isp and sensor\n");
>> +		return NULL;
>> +	}
>> +
>> +	sensor_me = media_entity_remote_pad(local)->entity;
>> +	return media_entity_to_v4l2_subdev(sensor_me);
>> +}
>> +
>> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
>> +					       struct v4l2_subdev *sd)
> 
> Indentation.
> 
>> +{
>> +	struct rkisp1_sensor *sensor;
>> +
>> +	list_for_each_entry(sensor, &dev->sensors, list)
>> +		if (sensor->sd == sd)
>> +			return sensor;
>> +
>> +	return NULL;
>> +}
>> +
>> +/****************  register operations ****************/
>> +
>> +/*
>> + * Image Stabilization.
>> + * This should only be called when configuring CIF
>> + * or at the frame end interrupt
>> + */
>> +static void rkisp1_config_ism(struct rkisp1_device *dev)
>> +{
>> +	void __iomem *base = dev->base_addr;
>> +	struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
>> +	u32 val;
>> +
>> +	writel(0, base + CIF_ISP_IS_RECENTER);
>> +	writel(0, base + CIF_ISP_IS_MAX_DX);
>> +	writel(0, base + CIF_ISP_IS_MAX_DY);
>> +	writel(0, base + CIF_ISP_IS_DISPLACE);
>> +	writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
>> +	writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
>> +	writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
>> +	writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
>> +
>> +	/* IS(Image Stabilization) is always on, working as output crop */
>> +	writel(1, base + CIF_ISP_IS_CTRL);
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD;
>> +	writel(val, base + CIF_ISP_CTRL);
>> +}
>> +
>> +/*
>> + * configure isp blocks with input format, size......
>> + */
>> +static int rkisp1_config_isp(struct rkisp1_device *dev)
>> +{
>> +	u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
>> +	struct v4l2_rect *out_crop, *in_crop;
>> +	void __iomem *base = dev->base_addr;
>> +	struct v4l2_mbus_framefmt *in_frm;
>> +	struct ispsd_out_fmt *out_fmt;
>> +	struct rkisp1_sensor *sensor;
>> +	struct ispsd_in_fmt *in_fmt;
>> +
>> +	sensor = dev->active_sensor;
>> +	in_frm = &dev->isp_sdev.in_frm;
>> +	in_fmt = &dev->isp_sdev.in_fmt;
>> +	out_fmt = &dev->isp_sdev.out_fmt;
>> +	out_crop = &dev->isp_sdev.out_crop;
>> +	in_crop = &dev->isp_sdev.in_crop;
>> +
>> +	if (in_fmt->fmt_type == FMT_BAYER) {
>> +		acq_mult = 1;
>> +		if (out_fmt->fmt_type == FMT_BAYER) {
>> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
>> +				isp_ctrl =
>> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
>> +			else
>> +				isp_ctrl =
>> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
>> +		} else {
>> +			writel(CIF_ISP_DEMOSAIC_TH(0xc),
>> +			       base + CIF_ISP_DEMOSAIC);
>> +
>> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
>> +			else
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
>> +		}
>> +	} else if (in_fmt->fmt_type == FMT_YUV) {
>> +		acq_mult = 2;
>> +		if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>> +			isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
>> +		} else {
>> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
>> +			else
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
>> +
>> +		}
>> +
>> +		irq_mask |= CIF_ISP_DATA_LOSS;
>> +	}
>> +
>> +	/* Set up input acquisition properties */
>> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
>> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>> +		if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
>> +			signal = CIF_ISP_ACQ_PROP_POS_EDGE;
>> +	}
>> +
>> +	if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>> +		if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
>> +			signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
>> +
>> +		if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
>> +			signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
>> +	}
>> +
>> +	writel(isp_ctrl, base + CIF_ISP_CTRL);
>> +	writel(signal | in_fmt->yuv_seq |
>> +	       CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
>> +	       CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
>> +	writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
>> +
>> +	/* Acquisition Size */
>> +	writel(0, base + CIF_ISP_ACQ_H_OFFS);
>> +	writel(0, base + CIF_ISP_ACQ_V_OFFS);
>> +	writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
>> +	writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
>> +
>> +	/* ISP Out Area */
>> +	writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
>> +	writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
>> +	writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
>> +	writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
>> +
>> +	/* interrupt mask */
>> +	irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
>> +		    CIF_ISP_FRAME_IN;
>> +	writel(irq_mask, base + CIF_ISP_IMSC);
>> +
>> +	if (out_fmt->fmt_type == FMT_BAYER)
>> +		rkisp1_params_disable_isp(&dev->params_vdev);
>> +	else
>> +		rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
>> +					    dev->isp_sdev.quantization);
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_config_dvp(struct rkisp1_device *dev)
>> +{
>> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val, input_sel;
>> +
>> +	switch (in_fmt->bus_width) {
>> +	case 8:
>> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
>> +		break;
>> +	case 10:
>> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
>> +		break;
>> +	case 12:
>> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
>> +		break;
>> +	default:
>> +		v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	val = readl(base + CIF_ISP_ACQ_PROP);
>> +	writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_config_mipi(struct rkisp1_device *dev)
>> +{
>> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
>> +	struct rkisp1_sensor *sensor = dev->active_sensor;
>> +	void __iomem *base = dev->base_addr;
>> +	unsigned int lanes;
>> +	u32 mipi_ctrl;
>> +
>> +	/*
>> +	 * sensor->mbus is set in isp or d-phy notifier_bound function
>> +	 */
>> +	switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
>> +	case V4L2_MBUS_CSI2_4_LANE:
>> +		lanes = 4;
>> +		break;
>> +	case V4L2_MBUS_CSI2_3_LANE:
>> +		lanes = 3;
>> +		break;
>> +	case V4L2_MBUS_CSI2_2_LANE:
>> +		lanes = 2;
>> +		break;
>> +	case V4L2_MBUS_CSI2_1_LANE:
>> +		lanes = 1;
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
>> +		    CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
>> +		    CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
>> +		    CIF_MIPI_CTRL_CLOCKLANE_ENA;
>> +
>> +	writel(mipi_ctrl, base + CIF_MIPI_CTRL);
>> +
>> +	/* Configure Data Type and Virtual Channel */
>> +	writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
>> +	       base + CIF_MIPI_IMG_DATA_SEL);
>> +
>> +	/* Clear MIPI interrupts */
>> +	writel(~0, base + CIF_MIPI_ICR);
>> +	/*
>> +	 * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
>> +	 * isp bus may be dead when switch isp.
>> +	 */
>> +	writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
>> +	       CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
>> +	       base + CIF_MIPI_IMSC);
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
>> +		 "  MIPI_IMG_DATA_SEL 0x%08x\n"
>> +		 "  MIPI_STATUS 0x%08x\n"
>> +		 "  MIPI_IMSC 0x%08x\n",
>> +		 readl(base + CIF_MIPI_CTRL),
>> +		 readl(base + CIF_MIPI_IMG_DATA_SEL),
>> +		 readl(base + CIF_MIPI_STATUS),
>> +		 readl(base + CIF_MIPI_IMSC));
>> +
>> +	return 0;
>> +}
>> +
>> +/* Configure MUX */
>> +static int rkisp1_config_path(struct rkisp1_device *dev)
>> +{
>> +	struct rkisp1_sensor *sensor = dev->active_sensor;
>> +	u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
>> +	int ret = 0;
>> +
>> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
>> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>> +		ret = rkisp1_config_dvp(dev);
>> +		dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
>> +	} else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>> +		ret = rkisp1_config_mipi(dev);
>> +		dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
>> +	}
>> +
>> +	writel(dpcl, dev->base_addr + CIF_VI_DPCL);
>> +
>> +	return ret;
>> +}
>> +
>> +/* Hareware configure Entry */
>> +static int rkisp1_config_cif(struct rkisp1_device *dev)
>> +{
>> +	u32 cif_id;
>> +	int ret;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming);
>> +
>> +	cif_id = readl(dev->base_addr + CIF_VI_ID);
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
>> +
>> +	ret = rkisp1_config_isp(dev);
>> +	if (ret < 0)
>> +		return ret;
>> +	ret = rkisp1_config_path(dev);
>> +	if (ret < 0)
>> +		return ret;
>> +	rkisp1_config_ism(dev);
>> +
>> +	return 0;
>> +}
>> +
>> +/* Mess register operations to stop isp */
>> +static int rkisp1_isp_stop(struct rkisp1_device *dev)
>> +{
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming);
>> +
>> +	/*
>> +	 * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
>> +	 * Stop ISP(isp) ->wait for ISP isp off
>> +	 */
>> +	/* stop and clear MI, MIPI, and ISP interrupts */
>> +	writel(0, base + CIF_MIPI_IMSC);
>> +	writel(~0, base + CIF_MIPI_ICR);
>> +
>> +	writel(0, base + CIF_ISP_IMSC);
>> +	writel(~0, base + CIF_ISP_ICR);
>> +
>> +	writel(0, base + CIF_MI_IMSC);
>> +	writel(~0, base + CIF_MI_ICR);
>> +	val = readl(base + CIF_MIPI_CTRL);
>> +	writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
>> +	/* stop ISP */
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
>> +	writel(val, base + CIF_ISP_CTRL);
>> +
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
>> +
>> +	readx_poll_timeout(readl, base + CIF_ISP_RIS,
>> +			   val, val & CIF_ISP_OFF, 20, 100);
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		"streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming,
>> +		 readl(base + CIF_MI_CTRL),
>> +		 readl(base + CIF_ISP_CTRL),
>> +		 readl(base + CIF_MIPI_CTRL));
>> +
>> +	writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
>> +	writel(0x0, base + CIF_IRCL);
>> +
>> +	return 0;
>> +}
>> +
>> +/* Mess register operations to start isp */
>> +static int rkisp1_isp_start(struct rkisp1_device *dev)
>> +{
>> +	struct rkisp1_sensor *sensor = dev->active_sensor;
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming);
>> +
>> +	/* Activate MIPI */
>> +	if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>> +		val = readl(base + CIF_MIPI_CTRL);
>> +		writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
>> +	}
>> +	/* Activate ISP */
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
>> +	       CIF_ISP_CTRL_ISP_INFORM_ENABLE;
>> +	writel(val, base + CIF_ISP_CTRL);
>> +
>> +	/* XXX: Is the 1000us too long?
>> +	 * CIF spec says to wait for sufficient time after enabling
>> +	 * the MIPI interface and before starting the sensor output.
>> +	 */
>> +	usleep_range(1000, 1200);
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
>> +		 "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming,
>> +		 readl(base + CIF_MI_CTRL),
>> +		 readl(base + CIF_ISP_CTRL),
>> +		 readl(base + CIF_MIPI_CTRL));
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_config_clk(struct rkisp1_device *dev)
>> +{
>> +	u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
>> +		  CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
>> +		  CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
>> +
>> +	writel(val, dev->base_addr + CIF_ICCL);
>> +}
>> +
>> +/***************************** isp sub-devs *******************************/
>> +
>> +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
>> +	{
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_BGGR,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_RGGB,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_GBRG,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_GRBG,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_RGGB,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_BGGR,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_GBRG,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_GRBG,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_RGGB,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_BGGR,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_GBRG,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_GRBG,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCBYCR,
>> +		.bus_width	= 16,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_YVYU8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCRYCB,
>> +		.bus_width	= 16,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_UYVY8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CBYCRY,
>> +		.bus_width	= 16,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_VYUY8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CRYCBY,
>> +		.bus_width	= 16,
>> +	},
>> +};
>> +
>> +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
>> +	{
>> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_2X8,
>> +		.fmt_type	= FMT_YUV,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	},
>> +};
>> +
>> +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
>> +{
>> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
> 
> I think it'd be nicer to just use ARRAY_SIZE(...) in the condition. Same
> below.
> 
>> +	const struct ispsd_in_fmt *fmt;
>> +
>> +	for (i = 0; i < array_size; i++) {
>> +		fmt = &rkisp1_isp_input_formats[i];
>> +		if (fmt->mbus_code == mbus_code)
>> +			return fmt;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
>> +{
>> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
>> +	const struct ispsd_out_fmt *fmt;
>> +
>> +	for (i = 0; i < array_size; i++) {
>> +		fmt = &rkisp1_isp_output_formats[i];
>> +		if (fmt->mbus_code == mbus_code)
>> +			return fmt;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_pad_config *cfg,
>> +					struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	unsigned int i = code->index;
>> +
>> +	if ((code->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>> +		if (i > 0)
>> +			return -EINVAL;
>> +		code->code = MEDIA_BUS_FMT_FIXED;
>> +		return 0;
>> +	}
>> +
>> +	if (code->pad == RKISP1_ISP_PAD_SINK) {
>> +		if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
>> +			return -EINVAL;
>> +		code->code = rkisp1_isp_input_formats[i].mbus_code;
>> +	} else {
>> +		if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
>> +			return -EINVAL;
>> +		code->code = rkisp1_isp_output_formats[i].mbus_code;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
>> +				     struct v4l2_subdev_pad_config *cfg)
>> +{
>> +	struct v4l2_rect *mf_in_crop, *mf_out_crop;
>> +	struct v4l2_mbus_framefmt *mf_in, *mf_out;
>> +
>> +	mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
>> +	mf_in->width = RKISP1_DEFAULT_WIDTH;
>> +	mf_in->height = RKISP1_DEFAULT_HEIGHT;
>> +	mf_in->field = V4L2_FIELD_NONE;
>> +	mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
>> +
>> +	mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
>> +	mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
>> +	mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
>> +	mf_in_crop->left = 0;
>> +	mf_in_crop->top = 0;
>> +
>> +	mf_out = v4l2_subdev_get_try_format(sd, cfg,
>> +					    RKISP1_ISP_PAD_SOURCE_PATH);
>> +	*mf_out = *mf_in;
>> +	mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
>> +	mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +
>> +	mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
>> +					       RKISP1_ISP_PAD_SOURCE_PATH);
>> +	*mf_out_crop = *mf_in_crop;
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
>> +				 struct v4l2_subdev_pad_config *cfg,
>> +				 struct v4l2_subdev_format *fmt)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>> +
>> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
>> +		/*
>> +		 * NOTE: setting a format here doesn't make much sense
>> +		 * but v4l2-compliance complains
>> +		 */
>> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
>> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
>> +		fmt->format.field = V4L2_FIELD_NONE;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>> +		fmt->format = *mf;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>> +		*mf = isp_sd->in_frm;
>> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		/* format of source pad */
>> +		*mf = isp_sd->in_frm;
>> +		mf->code = isp_sd->out_fmt.mbus_code;
>> +		/* window size of source pad */
>> +		mf->width = isp_sd->out_crop.width;
>> +		mf->height = isp_sd->out_crop.height;
>> +		mf->quantization = isp_sd->quantization;
>> +	}
>> +	mf->field = V4L2_FIELD_NONE;
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
>> +				  unsigned int pad,
>> +				  struct v4l2_mbus_framefmt *fmt)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>> +	const struct ispsd_out_fmt *out_fmt;
>> +	const struct ispsd_in_fmt *in_fmt;
>> +
>> +	switch (pad) {
>> +	case RKISP1_ISP_PAD_SINK:
>> +		in_fmt = find_in_fmt(fmt->code);
>> +		if (in_fmt)
>> +			fmt->code = in_fmt->mbus_code;
>> +		else
>> +			fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
>> +		fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
>> +				      CIF_ISP_INPUT_W_MAX);
>> +		fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
>> +				      CIF_ISP_INPUT_H_MAX);
>> +		break;
>> +	case RKISP1_ISP_PAD_SOURCE_PATH:
>> +		out_fmt = find_out_fmt(fmt->code);
>> +		if (out_fmt)
>> +			fmt->code = out_fmt->mbus_code;
>> +		else
>> +			fmt->code = rkisp1_isp_output_formats[0].mbus_code;
>> +		/* window size is set in s_selection */
>> +		fmt->width  = isp_sd->out_crop.width;
>> +		fmt->height = isp_sd->out_crop.height;
>> +		/* full range by default */
>> +		if (!fmt->quantization)
>> +			fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +		break;
>> +	}
>> +
>> +	fmt->field = V4L2_FIELD_NONE;
>> +}
>> +
>> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
>> +				 struct v4l2_subdev_pad_config *cfg,
>> +				 struct v4l2_subdev_format *fmt)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>> +
> 
> Note that for sub-device nodes, the driver is itself responsible for
> serialising the access to its data structures.

But looking at subdev_do_ioctl_lock(), it seems that it serializes the
ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
most probably) ?

> 
>> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
>> +		return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
>> +
>> +	rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
>> +
>> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		struct v4l2_mbus_framefmt *try_mf;
>> +
>> +		try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>> +		*try_mf = *mf;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>> +		const struct ispsd_in_fmt *in_fmt;
>> +
>> +		in_fmt = find_in_fmt(mf->code);
>> +		isp_sd->in_fmt = *in_fmt;
>> +		isp_sd->in_frm = *mf;
>> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		const struct ispsd_out_fmt *out_fmt;
>> +
>> +		/* Ignore width/height */
>> +		out_fmt = find_out_fmt(mf->code);
>> +		isp_sd->out_fmt = *out_fmt;
>> +		/*
>> +		 * It is quantization for output,
>> +		 * isp use bt601 limit-range in internal
>> +		 */
>> +		isp_sd->quantization = mf->quantization;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
>> +				  struct v4l2_subdev_pad_config *cfg,
>> +				  struct v4l2_subdev_selection *sel)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
>> +	struct v4l2_rect in_crop = isp_sd->in_crop;
>> +	struct v4l2_rect *input = &sel->r;
>> +
>> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		in_frm = *v4l2_subdev_get_try_format(sd, cfg,
>> +						     RKISP1_ISP_PAD_SINK);
>> +		in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
>> +						    RKISP1_ISP_PAD_SINK);
>> +	}
>> +
>> +	input->left = ALIGN(input->left, 2);
>> +	input->width = ALIGN(input->width, 2);
>> +
>> +	if (sel->pad == RKISP1_ISP_PAD_SINK) {
>> +		input->left = clamp_t(u32, input->left, 0, in_frm.width);
>> +		input->top = clamp_t(u32, input->top, 0, in_frm.height);
>> +		input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
>> +				       in_frm.width - input->left);
>> +		input->height = clamp_t(u32, input->height,
>> +					CIF_ISP_INPUT_H_MIN,
>> +					in_frm.height - input->top);
>> +	} else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		input->left = clamp_t(u32, input->left, 0, in_crop.width);
>> +		input->top = clamp_t(u32, input->top, 0, in_crop.height);
>> +		input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
>> +				       in_crop.width - input->left);
>> +		input->height = clamp_t(u32, input->height,
>> +					CIF_ISP_OUTPUT_H_MIN,
>> +					in_crop.height - input->top);
>> +	}
>> +}
>> +
>> +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_pad_config *cfg,
>> +				       struct v4l2_subdev_selection *sel)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt *frm;
>> +	struct v4l2_rect *rect;
>> +
>> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
>> +	    sel->pad != RKISP1_ISP_PAD_SINK)
>> +		return -EINVAL;
>> +
>> +	switch (sel->target) {
>> +	case V4L2_SEL_TGT_CROP_BOUNDS:
>> +		if (sel->pad == RKISP1_ISP_PAD_SINK) {
>> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>> +				frm = v4l2_subdev_get_try_format(sd, cfg,
>> +								 sel->pad);
>> +			else
>> +				frm = &isp_sd->in_frm;
>> +
>> +			sel->r.height = frm->height;
>> +			sel->r.width = frm->width;
>> +			sel->r.left = 0;
>> +			sel->r.top = 0;
>> +		} else {
>> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>> +				rect = v4l2_subdev_get_try_crop(sd, cfg,
>> +							RKISP1_ISP_PAD_SINK);
>> +			else
>> +				rect = &isp_sd->in_crop;
>> +			sel->r = *rect;
>> +		}
>> +		break;
>> +	case V4L2_SEL_TGT_CROP:
>> +		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>> +			rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
>> +		else if (sel->pad == RKISP1_ISP_PAD_SINK)
>> +			rect = &isp_sd->in_crop;
>> +		else
>> +			rect = &isp_sd->out_crop;
>> +		sel->r = *rect;
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_pad_config *cfg,
>> +				       struct v4l2_subdev_selection *sel)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct rkisp1_device *dev = sd_to_isp_dev(sd);
>> +
>> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
>> +	    sel->pad != RKISP1_ISP_PAD_SINK)
>> +		return -EINVAL;
>> +	if (sel->target != V4L2_SEL_TGT_CROP)
>> +		return -EINVAL;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
>> +		 sel->r.left, sel->r.top, sel->r.width, sel->r.height);
>> +	rkisp1_isp_sd_try_crop(sd, cfg, sel);
>> +
>> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		struct v4l2_rect *try_sel =
>> +			v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
>> +
>> +		*try_sel = sel->r;
>> +		return 0;
>> +	}
>> +
>> +	if (sel->pad == RKISP1_ISP_PAD_SINK)
>> +		isp_sd->in_crop = sel->r;
>> +	else
>> +		isp_sd->out_crop = sel->r;
>> +
>> +	return 0;
>> +}
>> +
>> +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
>> +				    struct rkisp1_sensor *sensor)
>> +{
>> +	union phy_configure_opts opts = { 0 };
>> +	struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
>> +	struct v4l2_ctrl *pixel_rate;
>> +	s64 pixel_clock;
>> +
>> +	pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
>> +				    V4L2_CID_PIXEL_RATE);
>> +	if (!pixel_rate) {
>> +		v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
>> +		return -EPIPE;
>> +	}
>> +
>> +	pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
>> +	if (!pixel_clock) {
>> +		v4l2_err(sensor->sd, "Invalid pixel rate value\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
>> +					 sensor->lanes, cfg);
>> +	phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
>> +	phy_configure(sensor->dphy, &opts);
>> +	phy_power_on(sensor->dphy);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
>> +{
>> +	phy_power_off(sensor->dphy);
>> +}
>> +
>> +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	struct v4l2_subdev *sensor_sd;
>> +	int ret = 0;
>> +
>> +	if (!on) {
>> +		ret = rkisp1_isp_stop(isp_dev);
>> +		if (ret < 0)
>> +			return ret;
>> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
>> +		return 0;
>> +	}
>> +
>> +	sensor_sd = get_remote_sensor(sd);
>> +	if (!sensor_sd)
>> +		return -ENODEV;
>> +
>> +	isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
>> +	if (!isp_dev->active_sensor)
>> +		return -ENODEV;
>> +
>> +	atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
>> +	ret = rkisp1_config_cif(isp_dev);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* TODO: support other interfaces */
>> +	if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
>> +		return -EINVAL;
>> +
>> +	ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
>> +				       isp_dev->active_sensor);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	ret = rkisp1_isp_start(isp_dev);
>> +	if (ret)
>> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
>> +
>> +	return ret;
>> +}
>> +
>> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> 
> If you support runtime PM, you shouldn't implement the s_power op.

Is is ok to completly remove the usage of runtime PM then?
Like this http://ix.io/1RJb ?

tbh I'm not that familar with runtime PM and I'm not sure what is the
difference of it and using s_power op (and Documentation/power/runtime_pm.rst
is not being that helpful tbh).

> 
> You'll still need to call s_power on external subdevs though.
> 
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	int ret;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
>> +
>> +	if (on) {
>> +		ret = pm_runtime_get_sync(isp_dev->dev);

If this is not ok to remove suport for runtime PM, then where should I put
the call to pm_runtime_get_sync() if not in this s_power op ?

>> +		if (ret < 0)
>> +			return ret;
>> +
>> +		rkisp1_config_clk(isp_dev);
>> +	} else {
>> +		ret = pm_runtime_put(isp_dev->dev);
>> +		if (ret < 0)
>> +			return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_subdev_link_validate(struct media_link *link)
>> +{
>> +	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
> 
> Is this test correct? The source is the source end of the link, i.e. the
> video node.

Ah yes, it should be link->sink->index (and not source), thanks for spotting this.

> 
> How about the links that end in a video node?

I thought that the only possibilities were sensor->isp1 and params->isp1 (where params
is an output video node that should be catched by the corrected version of the if
statement above.

Or do you mean another thing?

> 
>> +		return 0;
>> +
>> +	return v4l2_subdev_link_validate(link);
>> +}
>> +
>> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
>> +					struct media_link *link,
>> +					struct v4l2_subdev_format *source_fmt,
>> +					struct v4l2_subdev_format *sink_fmt)
>> +{
>> +	if (source_fmt->format.code != sink_fmt->format.code)
>> +		return -EINVAL;

ops, should be -EPIPE

>> +
>> +	/* Crop is available */
>> +	if (source_fmt->format.width < sink_fmt->format.width ||
>> +	    source_fmt->format.height < sink_fmt->format.height)
>> +		return -EINVAL;

-EPIPE

>> +
> 
> Could you use v4l2_subdev_link_validate_default()?

v4l2_subdev_link_validate_default() only allows for an exact width/height match,
but here we allow the sink to be smaller then the source for cropping, no?

Thanks again for your review!
Helen

> 
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>> +{
>> +	struct v4l2_event event = {
>> +		.type = V4L2_EVENT_FRAME_SYNC,
>> +		.u.frame_sync.frame_sequence =
>> +			atomic_inc_return(&isp->frm_sync_seq) - 1,
>> +	};
>> +	v4l2_event_queue(isp->sd.devnode, &event);
>> +}
>> +
>> +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>> +				  struct v4l2_event_subscription *sub)
>> +{
>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
>> +		return -EINVAL;
>> +
>> +	/* Line number. For now only zero accepted. */
>> +	if (sub->id != 0)
>> +		return -EINVAL;
>> +
>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
>> +	.enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
>> +	.get_selection = rkisp1_isp_sd_get_selection,
>> +	.set_selection = rkisp1_isp_sd_set_selection,
>> +	.init_cfg = rkisp1_isp_sd_init_config,
>> +	.get_fmt = rkisp1_isp_sd_get_fmt,
>> +	.set_fmt = rkisp1_isp_sd_set_fmt,
>> +	.link_validate = rkisp1_subdev_fmt_link_validate,
>> +};
>> +
>> +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
>> +	.link_validate = rkisp1_subdev_link_validate,
>> +};
>> +
>> +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
>> +	.s_stream = rkisp1_isp_sd_s_stream,
>> +};
>> +
>> +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
>> +	.subscribe_event = rkisp1_isp_sd_subs_evt,
>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
>> +	.s_power = rkisp1_isp_sd_s_power,
>> +};
>> +
>> +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {
> 
> const
> 
>> +	.core = &rkisp1_isp_core_ops,
>> +	.video = &rkisp1_isp_sd_video_ops,
>> +	.pad = &rkisp1_isp_sd_pad_ops,
>> +};
>> +
>> +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
>> +{
>> +	struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
>> +	struct v4l2_rect *in_crop = &isp_sd->in_crop;
>> +	struct v4l2_rect *out_crop = &isp_sd->out_crop;
>> +	struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
>> +	struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
>> +
>> +	*in_fmt = rkisp1_isp_input_formats[0];
>> +	in_frm->width = RKISP1_DEFAULT_WIDTH;
>> +	in_frm->height = RKISP1_DEFAULT_HEIGHT;
>> +	in_frm->code = in_fmt->mbus_code;
>> +
>> +	in_crop->width = in_frm->width;
>> +	in_crop->height = in_frm->height;
>> +	in_crop->left = 0;
>> +	in_crop->top = 0;
>> +
>> +	/* propagate to source */
>> +	*out_crop = *in_crop;
>> +	*out_fmt = rkisp1_isp_output_formats[0];
>> +	isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +}
>> +
>> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
>> +			       struct v4l2_device *v4l2_dev)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
>> +	struct v4l2_subdev *sd = &isp_sdev->sd;
>> +	int ret;
>> +
>> +	v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>> +	sd->entity.ops = &rkisp1_isp_sd_media_ops;
>> +	snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
> 
> strscpy()
> 
>> +
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
>> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
>> +	ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
>> +				     isp_sdev->pads);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	sd->owner = THIS_MODULE;
>> +	v4l2_set_subdevdata(sd, isp_dev);
>> +
>> +	sd->grp_id = GRP_ID_ISP;
>> +	ret = v4l2_device_register_subdev(v4l2_dev, sd);
>> +	if (ret < 0) {
>> +		v4l2_err(sd, "Failed to register isp subdev\n");
>> +		goto err_cleanup_media_entity;
>> +	}
>> +
>> +	rkisp1_isp_sd_init_default_fmt(isp_sdev);
>> +
>> +	return 0;
> 
> A newline would be nice here.
> 
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&sd->entity);
> 
> And here.
> 
>> +	return ret;
>> +}
>> +
>> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
>> +{
>> +	struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
>> +
>> +	v4l2_device_unregister_subdev(sd);
>> +	media_entity_cleanup(&sd->entity);
>> +}
>> +
>> +/****************  Interrupter Handler ****************/
>> +
>> +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
>> +{
>> +	struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val;
>> +
>> +	writel(~0, base + CIF_MIPI_ICR);
>> +
>> +	/*
>> +	 * Disable DPHY errctrl interrupt, because this dphy
>> +	 * erctrl signal is asserted until the next changes
>> +	 * of line state. This time is may be too long and cpu
>> +	 * is hold in this interrupt.
>> +	 */
>> +	if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
>> +		val = readl(base + CIF_MIPI_IMSC);
>> +		writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
>> +		dev->isp_sdev.dphy_errctrl_disabled = true;
>> +	}
>> +
>> +	/*
>> +	 * Enable DPHY errctrl interrupt again, if mipi have receive
>> +	 * the whole frame without any error.
>> +	 */
>> +	if (mis == CIF_MIPI_FRAME_END) {
>> +		/*
>> +		 * Enable DPHY errctrl interrupt again, if mipi have receive
>> +		 * the whole frame without any error.
>> +		 */
>> +		if (dev->isp_sdev.dphy_errctrl_disabled) {
>> +			val = readl(base + CIF_MIPI_IMSC);
>> +			val |= CIF_MIPI_ERR_CTRL(0x0f);
>> +			writel(val, base + CIF_MIPI_IMSC);
>> +			dev->isp_sdev.dphy_errctrl_disabled = false;
>> +		}
>> +	} else {
>> +		v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
>> +	}
>> +}
>> +
>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>> +{
>> +	void __iomem *base = dev->base_addr;
>> +	unsigned int isp_mis_tmp = 0;
>> +	unsigned int isp_err = 0;
>> +
>> +	/* start edge of v_sync */
>> +	if (isp_mis & CIF_ISP_V_START) {
>> +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>> +
>> +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>> +		if (isp_mis_tmp & CIF_ISP_V_START)
>> +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>> +				 isp_mis_tmp);
>> +	}
>> +
>> +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> 
> Extra parentheses.
> 
>> +		/* Clear pic_size_error */
>> +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>> +		isp_err = readl(base + CIF_ISP_ERR);
>> +		v4l2_err(&dev->v4l2_dev,
>> +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>> +		writel(isp_err, base + CIF_ISP_ERR_CLR);
>> +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>> +		/* Clear data_loss */
>> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>> +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>> +	}
>> +
>> +	/* sampled input frame is complete */
>> +	if (isp_mis & CIF_ISP_FRAME_IN) {
>> +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>> +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>> +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>> +				 isp_mis_tmp);
>> +	}
>> +
>> +	/* frame was completely put out */
>> +	if (isp_mis & CIF_ISP_FRAME) {
>> +		u32 isp_ris = 0;
>> +		/* Clear Frame In (ISP) */
>> +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>> +		if (isp_mis_tmp & CIF_ISP_FRAME)
>> +			v4l2_err(&dev->v4l2_dev,
>> +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>> +
>> +		isp_ris = readl(base + CIF_ISP_RIS);
>> +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>> +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>> +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>> +	}
>> +
>> +	/*
>> +	 * Then update changed configs. Some of them involve
>> +	 * lot of register writes. Do those only one per frame.
>> +	 * Do the updates in the order of the processing flow.
>> +	 */
>> +	rkisp1_params_isr(&dev->params_vdev, isp_mis);
>> +}
>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
>> new file mode 100644
>> index 000000000000..b0366e354ec2
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
>> @@ -0,0 +1,111 @@
>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>> +/*
>> + * Rockchip isp1 driver
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#ifndef _RKISP1_H
>> +#define _RKISP1_H
>> +
>> +#include <linux/platform_device.h>
>> +#include <media/v4l2-fwnode.h>
>> +
>> +#include "common.h"
>> +
>> +struct rkisp1_stream;
>> +
>> +/*
>> + * struct ispsd_in_fmt - ISP intput-pad format
>> + *
>> + * Translate mbus_code to hardware format values
>> + *
>> + * @bus_width: used for parallel
>> + */
>> +struct ispsd_in_fmt {
>> +	u32 mbus_code;
>> +	u8 fmt_type;
>> +	u32 mipi_dt;
>> +	u32 yuv_seq;
>> +	enum rkisp1_fmt_raw_pat_type bayer_pat;
>> +	u8 bus_width;
>> +};
>> +
>> +struct ispsd_out_fmt {
>> +	u32 mbus_code;
>> +	u8 fmt_type;
>> +};
>> +
>> +struct rkisp1_ie_config {
>> +	unsigned int effect;
>> +};
>> +
>> +enum rkisp1_isp_pad {
>> +	RKISP1_ISP_PAD_SINK,
>> +	RKISP1_ISP_PAD_SINK_PARAMS,
>> +	RKISP1_ISP_PAD_SOURCE_PATH,
>> +	RKISP1_ISP_PAD_SOURCE_STATS,
>> +	RKISP1_ISP_PAD_MAX
>> +};
>> +
>> +/*
>> + * struct rkisp1_isp_subdev - ISP sub-device
>> + *
>> + * See Cropping regions of ISP in rkisp1.c for details
>> + * @in_frm: input size, don't have to be equal to sensor size
>> + * @in_fmt: input format
>> + * @in_crop: crop for sink pad
>> + * @out_fmt: output format
>> + * @out_crop: output size
>> + *
>> + * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)
>> + * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
>> + * @quantization: output quantization
>> + */
>> +struct rkisp1_isp_subdev {
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pads[RKISP1_ISP_PAD_MAX];
>> +	struct v4l2_ctrl_handler ctrl_handler;
>> +	struct v4l2_mbus_framefmt in_frm;
>> +	struct ispsd_in_fmt in_fmt;
>> +	struct v4l2_rect in_crop;
>> +	struct ispsd_out_fmt out_fmt;
>> +	struct v4l2_rect out_crop;
>> +	bool dphy_errctrl_disabled;
>> +	atomic_t frm_sync_seq;
>> +	enum v4l2_quantization quantization;
>> +};
>> +
>> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
>> +			       struct v4l2_device *v4l2_dev);
>> +
>> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
>> +
>> +void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
>> +
>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
>> +
>> +static inline
>> +struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
>> +{
>> +	return &isp_sdev->out_fmt;
>> +}
>> +
>> +static inline
>> +struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
>> +{
>> +	return &isp_sdev->in_fmt;
>> +}
>> +
>> +static inline
>> +struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
>> +{
>> +	return &isp_sdev->out_crop;
>> +}
> 
> I'd just use the struct fields directly in the code as it's easier to
> figure out which field in the struct is being accessed.
> 
>> +
>> +static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
>> +{
>> +	return container_of(sd, struct rkisp1_isp_subdev, sd);
>> +}
>> +
>> +#endif /* _RKISP1_H */
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-15  0:58     ` Helen Koike
@ 2019-08-15  8:24       ` Sakari Ailus
  2019-08-15 10:29         ` Tomasz Figa
  2019-08-15 13:17         ` Sakari Ailus
  0 siblings, 2 replies; 76+ messages in thread
From: Sakari Ailus @ 2019-08-15  8:24 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:

...

> >> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> >> +				 struct v4l2_subdev_pad_config *cfg,
> >> +				 struct v4l2_subdev_format *fmt)
> >> +{
> >> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> >> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> >> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> >> +
> > 
> > Note that for sub-device nodes, the driver is itself responsible for
> > serialising the access to its data structures.
> 
> But looking at subdev_do_ioctl_lock(), it seems that it serializes the
> ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
> most probably) ?

Good question. I had missed this change --- subdev_do_ioctl_lock() is
relatively new. But setting that lock is still not possible as the struct
is allocated in the framework and the device is registered before the
driver gets hold of it. It's a good idea to provide the same serialisation
for subdevs as well.

I'll get back to this later.

...

> >> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> > 
> > If you support runtime PM, you shouldn't implement the s_power op.
> 
> Is is ok to completly remove the usage of runtime PM then?
> Like this http://ix.io/1RJb ?

Please use runtime PM instead. In the long run we should get rid of the
s_power op. Drivers themselves know better when the hardware they control
should be powered on or off.

> 
> tbh I'm not that familar with runtime PM and I'm not sure what is the
> difference of it and using s_power op (and Documentation/power/runtime_pm.rst
> is not being that helpful tbh).

You can find a simple example e.g. in
drivers/media/platform/atmel/atmel-isi.c .

> 
> > 
> > You'll still need to call s_power on external subdevs though.
> > 
> >> +{
> >> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> >> +	int ret;
> >> +
> >> +	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> >> +
> >> +	if (on) {
> >> +		ret = pm_runtime_get_sync(isp_dev->dev);
> 
> If this is not ok to remove suport for runtime PM, then where should I put
> the call to pm_runtime_get_sync() if not in this s_power op ?

Basically the runtime_resume and runtime_suspend callbacks are where the
device power state changes are implemented, and pm_runtime_get_sync and
pm_runtime_put are how the driver controls the power state.

So you no longer need the s_power() op at all. The op needs to be called on
the pipeline however, as there are drivers that still use it.

> 
> >> +		if (ret < 0)
> >> +			return ret;
> >> +
> >> +		rkisp1_config_clk(isp_dev);
> >> +	} else {
> >> +		ret = pm_runtime_put(isp_dev->dev);
> >> +		if (ret < 0)
> >> +			return ret;
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int rkisp1_subdev_link_validate(struct media_link *link)
> >> +{
> >> +	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
> > 
> > Is this test correct? The source is the source end of the link, i.e. the
> > video node.
> 
> Ah yes, it should be link->sink->index (and not source), thanks for spotting this.
> 
> > 
> > How about the links that end in a video node?
> 
> I thought that the only possibilities were sensor->isp1 and params->isp1 (where params
> is an output video node that should be catched by the corrected version of the if
> statement above.
> 
> Or do you mean another thing?

The link_validate of the sink entity will be called only, for the knowledge
what is possible is generally in that end.

So you'll need this for all the sink pads this driver is in control of.
I suppose this means the sub-devices as well as capture video nodes in
practice.

> 
> > 
> >> +		return 0;
> >> +
> >> +	return v4l2_subdev_link_validate(link);
> >> +}
> >> +
> >> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
> >> +					struct media_link *link,
> >> +					struct v4l2_subdev_format *source_fmt,
> >> +					struct v4l2_subdev_format *sink_fmt)
> >> +{
> >> +	if (source_fmt->format.code != sink_fmt->format.code)
> >> +		return -EINVAL;
> 
> ops, should be -EPIPE
> 
> >> +
> >> +	/* Crop is available */
> >> +	if (source_fmt->format.width < sink_fmt->format.width ||
> >> +	    source_fmt->format.height < sink_fmt->format.height)
> >> +		return -EINVAL;
> 
> -EPIPE
> 
> >> +
> > 
> > Could you use v4l2_subdev_link_validate_default()?
> 
> v4l2_subdev_link_validate_default() only allows for an exact width/height match,
> but here we allow the sink to be smaller then the source for cropping, no?

The width and height generally must match over a link. But cropping takes
place inside a sub-device, it is not a concern in link validation as such.

> 
> Thanks again for your review!

You're welcome!

-- 
Kind regards,

Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-15  8:24       ` Sakari Ailus
@ 2019-08-15 10:29         ` Tomasz Figa
  2019-08-15 10:45           ` Sakari Ailus
  2019-08-15 13:17         ` Sakari Ailus
  1 sibling, 1 reply; 76+ messages in thread
From: Tomasz Figa @ 2019-08-15 10:29 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: devicetree, Eddie Cai, kernel, Heiko Stübner, Chen Jacob,
	Jeffy, 钟以崇,
	Linux Kernel Mailing List, Allon Huang,
	open list:ARM/Rockchip SoC...,
	Helen Koike, Jacob Chen, Hans Verkuil, Laurent Pinchart,
	Shunqian Zheng, Mauro Carvalho Chehab, Ezequiel Garcia,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

On Thu, Aug 15, 2019 at 5:25 PM Sakari Ailus
<sakari.ailus@linux.intel.com> wrote:
>
> Hi Helen,
>
> On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:
>
> ...
>
> > >> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> > >> +                           struct v4l2_subdev_pad_config *cfg,
> > >> +                           struct v4l2_subdev_format *fmt)
> > >> +{
> > >> +  struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > >> +  struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> > >> +  struct v4l2_mbus_framefmt *mf = &fmt->format;
> > >> +
> > >
> > > Note that for sub-device nodes, the driver is itself responsible for
> > > serialising the access to its data structures.
> >
> > But looking at subdev_do_ioctl_lock(), it seems that it serializes the
> > ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
> > most probably) ?
>
> Good question. I had missed this change --- subdev_do_ioctl_lock() is
> relatively new. But setting that lock is still not possible as the struct
> is allocated in the framework and the device is registered before the
> driver gets hold of it. It's a good idea to provide the same serialisation
> for subdevs as well.
>
> I'll get back to this later.
>
> ...
>
> > >> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> > >
> > > If you support runtime PM, you shouldn't implement the s_power op.
> >
> > Is is ok to completly remove the usage of runtime PM then?
> > Like this http://ix.io/1RJb ?
>
> Please use runtime PM instead. In the long run we should get rid of the
> s_power op. Drivers themselves know better when the hardware they control
> should be powered on or off.
>

One also needs to use runtime PM to handle power domains and power
dependencies on auxiliary devices, e.g. IOMMU.

> >
> > tbh I'm not that familar with runtime PM and I'm not sure what is the
> > difference of it and using s_power op (and Documentation/power/runtime_pm.rst
> > is not being that helpful tbh).
>
> You can find a simple example e.g. in
> drivers/media/platform/atmel/atmel-isi.c .
>
> >
> > >
> > > You'll still need to call s_power on external subdevs though.
> > >
> > >> +{
> > >> +  struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > >> +  int ret;
> > >> +
> > >> +  v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> > >> +
> > >> +  if (on) {
> > >> +          ret = pm_runtime_get_sync(isp_dev->dev);
> >
> > If this is not ok to remove suport for runtime PM, then where should I put
> > the call to pm_runtime_get_sync() if not in this s_power op ?
>
> Basically the runtime_resume and runtime_suspend callbacks are where the
> device power state changes are implemented, and pm_runtime_get_sync and
> pm_runtime_put are how the driver controls the power state.
>
> So you no longer need the s_power() op at all. The op needs to be called on
> the pipeline however, as there are drivers that still use it.
>

For this driver, I suppose we would _get_sync() when we start
streaming (in the hardware, i.e. we want the ISP to start capturing
frames) and _put() when we stop and the driver shouldn't perform any
access to the hardware when the streaming is not active.

Best regards,
Tomasz

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-15 10:29         ` Tomasz Figa
@ 2019-08-15 10:45           ` Sakari Ailus
  0 siblings, 0 replies; 76+ messages in thread
From: Sakari Ailus @ 2019-08-15 10:45 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, Eddie Cai, kernel, Heiko Stübner, Chen Jacob,
	Jeffy, 钟以崇,
	Linux Kernel Mailing List, Allon Huang,
	open list:ARM/Rockchip SoC...,
	Helen Koike, Jacob Chen, Hans Verkuil, Laurent Pinchart,
	Shunqian Zheng, Mauro Carvalho Chehab, Ezequiel Garcia,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

On Thu, Aug 15, 2019 at 07:29:59PM +0900, Tomasz Figa wrote:
> On Thu, Aug 15, 2019 at 5:25 PM Sakari Ailus
> <sakari.ailus@linux.intel.com> wrote:
> >
> > Hi Helen,
> >
> > On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:
> >
> > ...
> >
> > > >> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> > > >> +                           struct v4l2_subdev_pad_config *cfg,
> > > >> +                           struct v4l2_subdev_format *fmt)
> > > >> +{
> > > >> +  struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > >> +  struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> > > >> +  struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > >> +
> > > >
> > > > Note that for sub-device nodes, the driver is itself responsible for
> > > > serialising the access to its data structures.
> > >
> > > But looking at subdev_do_ioctl_lock(), it seems that it serializes the
> > > ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
> > > most probably) ?
> >
> > Good question. I had missed this change --- subdev_do_ioctl_lock() is
> > relatively new. But setting that lock is still not possible as the struct
> > is allocated in the framework and the device is registered before the
> > driver gets hold of it. It's a good idea to provide the same serialisation
> > for subdevs as well.
> >
> > I'll get back to this later.
> >
> > ...
> >
> > > >> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> > > >
> > > > If you support runtime PM, you shouldn't implement the s_power op.
> > >
> > > Is is ok to completly remove the usage of runtime PM then?
> > > Like this http://ix.io/1RJb ?
> >
> > Please use runtime PM instead. In the long run we should get rid of the
> > s_power op. Drivers themselves know better when the hardware they control
> > should be powered on or off.
> >
> 
> One also needs to use runtime PM to handle power domains and power
> dependencies on auxiliary devices, e.g. IOMMU.
> 
> > >
> > > tbh I'm not that familar with runtime PM and I'm not sure what is the
> > > difference of it and using s_power op (and Documentation/power/runtime_pm.rst
> > > is not being that helpful tbh).
> >
> > You can find a simple example e.g. in
> > drivers/media/platform/atmel/atmel-isi.c .
> >
> > >
> > > >
> > > > You'll still need to call s_power on external subdevs though.
> > > >
> > > >> +{
> > > >> +  struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > >> +  int ret;
> > > >> +
> > > >> +  v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> > > >> +
> > > >> +  if (on) {
> > > >> +          ret = pm_runtime_get_sync(isp_dev->dev);
> > >
> > > If this is not ok to remove suport for runtime PM, then where should I put
> > > the call to pm_runtime_get_sync() if not in this s_power op ?
> >
> > Basically the runtime_resume and runtime_suspend callbacks are where the
> > device power state changes are implemented, and pm_runtime_get_sync and
> > pm_runtime_put are how the driver controls the power state.
> >
> > So you no longer need the s_power() op at all. The op needs to be called on
> > the pipeline however, as there are drivers that still use it.
> >
> 
> For this driver, I suppose we would _get_sync() when we start
> streaming (in the hardware, i.e. we want the ISP to start capturing
> frames) and _put() when we stop and the driver shouldn't perform any
> access to the hardware when the streaming is not active.

Agreed.

-- 
Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-15  8:24       ` Sakari Ailus
  2019-08-15 10:29         ` Tomasz Figa
@ 2019-08-15 13:17         ` Sakari Ailus
  2020-01-31 19:38           ` Dafna Hirschfeld
  1 sibling, 1 reply; 76+ messages in thread
From: Sakari Ailus @ 2019-08-15 13:17 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, laurent.pinchart, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

On Thu, Aug 15, 2019 at 11:24:22AM +0300, Sakari Ailus wrote:
> Hi Helen,
> 
> On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:
> 
> ...
> 
> > >> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> > >> +				 struct v4l2_subdev_pad_config *cfg,
> > >> +				 struct v4l2_subdev_format *fmt)
> > >> +{
> > >> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > >> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> > >> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> > >> +
> > > 
> > > Note that for sub-device nodes, the driver is itself responsible for
> > > serialising the access to its data structures.
> > 
> > But looking at subdev_do_ioctl_lock(), it seems that it serializes the
> > ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
> > most probably) ?
> 
> Good question. I had missed this change --- subdev_do_ioctl_lock() is
> relatively new. But setting that lock is still not possible as the struct
> is allocated in the framework and the device is registered before the
> driver gets hold of it. It's a good idea to provide the same serialisation
> for subdevs as well.
> 
> I'll get back to this later.

The main reason is actually that these ops are also called through the
sub-device kAPI, not only through the uAPI, and the locks are only taken
through the calls via uAPI.

So adding the locks to uAPI calls alone would not address the issue.

-- 
Sakari Ailus
sakari.ailus@linux.intel.com

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format
  2019-07-30 18:42 ` [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format Helen Koike
@ 2019-08-15 13:30   ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 13:30 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

Hi Helen,

Thank you for the patch.

On Tue, Jul 30, 2019 at 03:42:43PM -0300, Helen Koike wrote:
> From: Shunqian Zheng <zhengsq@rock-chips.com>
> 
> Add the Rockchip ISP1 specific processing parameter format
> V4L2_META_FMT_RK_ISP1_PARAMS and metadata format
> V4L2_META_FMT_RK_ISP1_STAT_3A for 3A.

I don't like V4L2_META_FMT_RK_ISP1_PARAMS much, as the parameters are
not passed to the hardware through a buffer, but instead written to
registers by the kernel driver. This is an area where V4L2 controls are
traditionally used. I do however understand that it's easier for driver
authors to pass a large set of controls this way when they have to be
applied atomically, but it's clearly a hack in my opinion. I won't
oppose to it though, as I know there's no way we can support this kind
of feature in V4L2 at the moment without an extensive amount of work, so

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - s/IPU3/RK_ISP1
> 
>  drivers/media/v4l2-core/v4l2-ioctl.c | 2 ++
>  include/uapi/linux/videodev2.h       | 4 ++++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index bb5b4926538a..2df20801a299 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1330,6 +1330,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_META_FMT_VSP1_HGT:	descr = "R-Car VSP1 2-D Histogram"; break;
>  	case V4L2_META_FMT_UVC:		descr = "UVC Payload Header Metadata"; break;
>  	case V4L2_META_FMT_D4XX:	descr = "Intel D4xx UVC Metadata"; break;
> +	case V4L2_META_FMT_RK_ISP1_PARAMS:	descr = "Rockchip ISP1 3A params"; break;
> +	case V4L2_META_FMT_RK_ISP1_STAT_3A:	descr = "Rockchip ISP1 3A statistics"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 2427bc4d8eba..ae1e375a10d7 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -756,6 +756,10 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_UVC         v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
>  #define V4L2_META_FMT_D4XX        v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
>  
> +/* Vendor specific - used for RK_ISP1 camera sub-system */
> +#define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 params */
> +#define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A statistics */
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 02/14] media: doc: add document for rkisp1 meta buffer format
  2019-07-30 18:42 ` [PATCH v8 02/14] media: doc: add document for " Helen Koike
  2019-08-07 13:09   ` Sakari Ailus
@ 2019-08-15 13:51   ` Laurent Pinchart
  1 sibling, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 13:51 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Jacob Chen,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga,
	linux-rockchip, hans.verkuil, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

Thank you for the patch.

On Tue, Jul 30, 2019 at 03:42:44PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> This commit add document for rkisp1 meta buffer format
> 
> Signed-off-by: Jacob Chen <jacob-chen@rock-chips.com>
> Acked-by: Hans Verkuil <hans.verkuil@cisco.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8:
> - Add SPDX in the header
> - Remove emacs configs
> - Fix doc style
> 
> Changes in v7:
> - s/correspond/corresponding
> - s/use/uses
> - s/docuemnt/document
> 
>  Documentation/media/uapi/v4l/meta-formats.rst |  2 ++
>  .../uapi/v4l/pixfmt-meta-rkisp1-params.rst    | 23 +++++++++++++++++++
>  .../uapi/v4l/pixfmt-meta-rkisp1-stat.rst      | 22 ++++++++++++++++++
>  3 files changed, 47 insertions(+)
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
> 
> diff --git a/Documentation/media/uapi/v4l/meta-formats.rst b/Documentation/media/uapi/v4l/meta-formats.rst
> index b10ca9ee3968..5de621fea3cc 100644
> --- a/Documentation/media/uapi/v4l/meta-formats.rst
> +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> @@ -24,3 +24,5 @@ These formats are used for the :ref:`metadata` interface only.
>      pixfmt-meta-uvc
>      pixfmt-meta-vsp1-hgo
>      pixfmt-meta-vsp1-hgt
> +    pixfmt-meta-rkisp1-params
> +    pixfmt-meta-rkisp1-stat
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
> new file mode 100644
> index 000000000000..103b5cb79b7c
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-params.rst
> @@ -0,0 +1,23 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +
> +.. _v4l2-meta-fmt-rkisp1-params:
> +
> +============================
> +V4L2_META_FMT_RK_ISP1_PARAMS
> +============================
> +
> +Rockchip ISP1 Parameters Data
> +
> +Description
> +===========
> +
> +This format describes input parameters for the Rockchip ISP1.
> +
> +It uses c-struct :c:type:`rkisp1_isp_params_cfg`, which is defined in
> +the ``linux/rkisp1-config.h`` header file.

I would say that

"The buffer contains a single instance of the C structure
:c:type:`rkisp1_isp_params_cfg`, defined in the
``linux/rkisp1-config.h`` header file."

And add a sentence to explain what the alignment requirements are.

> +
> +The parameters consist of multiple modules.
> +The module won't be updated if the corresponding bit was not set in module_*_update.

Doesn't this belong to rkisp1-config.h ? I would group all the
information related to fields of the structure in the header file, and
only have here the information related to the buffer layout (this mainly
referencing the structure and talking about alignment/padding).

> +
> +.. kernel-doc:: include/uapi/linux/rkisp1-config.h
> +   :functions: rkisp1_isp_params_cfg
> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
> new file mode 100644
> index 000000000000..4ad303f96421
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-rkisp1-stat.rst
> @@ -0,0 +1,22 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +
> +.. _v4l2-meta-fmt-rkisp1-stat:
> +
> +=============================
> +V4L2_META_FMT_RK_ISP1_STAT_3A
> +=============================
> +
> +
> +Rockchip ISP1 Statistics Data
> +
> +Description
> +===========
> +
> +This format describes image color statistics information generated by the Rockchip
> +ISP1.
> +
> +It uses c-struct :c:type:`rkisp1_stat_buffer`, which is defined in
> +the ``linux/rkisp1-config.h`` header file.

Same comment here, I think we need to document alignment/padding
constraints.

> +
> +.. kernel-doc:: include/uapi/linux/rkisp1-config.h
> +   :functions: rkisp1_stat_buffer

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 14/14] MAINTAINERS: add entry for Rockchip ISP1 driver
  2019-07-30 18:42 ` [PATCH v8 14/14] MAINTAINERS: add entry for Rockchip ISP1 driver Helen Koike
@ 2019-08-15 13:56   ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 13:56 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

Hi Helen,

Thank you for the patch.

On Tue, Jul 30, 2019 at 03:42:56PM -0300, Helen Koike wrote:
> Add MAINTAINERS entry for the rockchip isp1 driver.
> 
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> 
> Changes in v8: None
> Changes in v7: None
> 
>  MAINTAINERS | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 6426db5198f0..7f38abcb4114 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -13743,6 +13743,14 @@ F:	drivers/hid/hid-roccat*
>  F:	include/linux/hid-roccat*
>  F:	Documentation/ABI/*/sysfs-driver-hid-roccat*
>  
> +ROCKCHIP ISP V1 DRIVER
> +M:	Helen Koike <helen.koike@collabora.com>
> +L:	linux-media@vger.kernel.org
> +S:	Maintained
> +F:	drivers/media/platform/rockchip/isp1/
> +F:	Documentation/devicetree/bindings/media/rockchip-isp1.txt
> +F:	Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt

This is missing the include/ files and the custom format documentation.
Apart from that, I'm happy to see that you will maintain this driver :-)

> +
>  ROCKCHIP RASTER 2D GRAPHIC ACCELERATION UNIT DRIVER
>  M:	Jacob chen <jacob2.chen@rock-chips.com>
>  L:	linux-media@vger.kernel.org

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-08-07 13:37     ` Helen Koike
@ 2019-08-15 17:54       ` Laurent Pinchart
  2019-08-15 18:26         ` Heiko Stübner
  2019-08-21 21:46         ` Helen Koike
  0 siblings, 2 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 17:54 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Sakari Ailus, sakari.ailus, zhengsq, hans.verkuil, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

Thank you for the patch.

On Wed, Aug 07, 2019 at 10:37:55AM -0300, Helen Koike wrote:
> On 8/7/19 10:05 AM, Sakari Ailus wrote:
> > On Tue, Jul 30, 2019 at 03:42:46PM -0300, Helen Koike wrote:
> >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> >>
> >> This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver
> >>
> >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> >> [migrate to phy framework]
> >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> >> [update for upstream]
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>
> >> ---
> >>
> >> Changes in v8:
> >> - Remove boiler plate license text
> >>
> >> Changes in v7:
> >> - Migrate dphy specific code from
> >> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
> >> to drivers/phy/rockchip/phy-rockchip-dphy.c
> >> - Drop support for rk3288
> >> - Drop support for dphy txrx
> >> - code styling and checkpatch fixes
> >>
> >>  drivers/phy/rockchip/Kconfig             |   8 +
> >>  drivers/phy/rockchip/Makefile            |   1 +
> >>  drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
> >>  3 files changed, 417 insertions(+)
> >>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
> >>
> >> diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
> >> index c454c90cd99e..afd072f135e6 100644
> >> --- a/drivers/phy/rockchip/Kconfig
> >> +++ b/drivers/phy/rockchip/Kconfig
> >> @@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
> >>  	help
> >>  	  Enable this to support the Rockchip Display Port PHY.
> >>  
> >> +config PHY_ROCKCHIP_DPHY
> >> +	tristate "Rockchip MIPI Synopsys DPHY driver"

How much of this PHY is Rockchip-specific ? Would it make sense to turn
it into a Synopsys DPHY driver, with some Rockchip glue ? I suppose this
could always be done later, if needed (and I also suppose there's no
existing driver in drivers/phy/ that support the same Synopsys IP).

> >> +	depends on ARCH_ROCKCHIP && OF
> > 
> > How about (...) || COMPILE_TEST ?
> > 
> >> +	select GENERIC_PHY_MIPI_DPHY
> >> +	select GENERIC_PHY
> >> +	help
> >> +	  Enable this to support the Rockchip MIPI Synopsys DPHY.
> >> +
> >>  config PHY_ROCKCHIP_EMMC
> >>  	tristate "Rockchip EMMC PHY Driver"
> >>  	depends on ARCH_ROCKCHIP && OF
> >> diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
> >> index fd21cbaf40dd..f62e9010bcaf 100644
> >> --- a/drivers/phy/rockchip/Makefile
> >> +++ b/drivers/phy/rockchip/Makefile
> >> @@ -1,5 +1,6 @@
> >>  # SPDX-License-Identifier: GPL-2.0
> >>  obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
> >> +obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
> >>  obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
> >>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
> >>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
> >> diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
> >> new file mode 100644
> >> index 000000000000..3a29976c2dff
> >> --- /dev/null
> >> +++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
> >> @@ -0,0 +1,408 @@
> >> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> >> +/*
> >> + * Rockchip MIPI Synopsys DPHY driver
> >> + *
> >> + * Based on:
> >> + *
> >> + * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
> >> + * Author: Yakir Yang <ykk@@rock-chips.com>
> >> + */
> >> +
> >> +#include <linux/clk.h>
> >> +#include <linux/io.h>
> >> +#include <linux/mfd/syscon.h>
> >> +#include <linux/module.h>
> >> +#include <linux/of.h>
> >> +#include <linux/of_device.h>
> >> +#include <linux/phy/phy.h>
> >> +#include <linux/phy/phy-mipi-dphy.h>
> >> +#include <linux/platform_device.h>
> >> +#include <linux/regmap.h>
> >> +
> >> +#define RK3399_GRF_SOC_CON9	0x6224
> >> +#define RK3399_GRF_SOC_CON21	0x6254
> >> +#define RK3399_GRF_SOC_CON22	0x6258
> >> +#define RK3399_GRF_SOC_CON23	0x625c
> >> +#define RK3399_GRF_SOC_CON24	0x6260
> >> +#define RK3399_GRF_SOC_CON25	0x6264
> >> +#define RK3399_GRF_SOC_STATUS1	0xe2a4
> >> +
> >> +#define CLOCK_LANE_HS_RX_CONTROL		0x34
> >> +#define LANE0_HS_RX_CONTROL			0x44
> >> +#define LANE1_HS_RX_CONTROL			0x54
> >> +#define LANE2_HS_RX_CONTROL			0x84
> >> +#define LANE3_HS_RX_CONTROL			0x94
> >> +#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
> >> +
> >> +#define MAX_DPHY_CLK 8
> >> +
> >> +#define PHY_TESTEN_ADDR			(0x1 << 16)
> >> +#define PHY_TESTEN_DATA			(0x0 << 16)
> >> +#define PHY_TESTCLK			(0x1 << 1)
> >> +#define PHY_TESTCLR			(0x1 << 0)

Maybe s/0x// for the previous four lines ?

> >> +#define THS_SETTLE_COUNTER_THRESHOLD	0x04
> >> +
> >> +#define HIWORD_UPDATE(val, mask, shift) \
> >> +	((val) << (shift) | (mask) << ((shift) + 16))

As you use this in a single place, I would inline it, possibly with a
small comment that explains what's happening.

> >> +
> >> +#define GRF_SOC_CON12                           0x0274
> >> +
> >> +#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
> >> +#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
> >> +
> >> +#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
> >> +#define GRF_EDP_PHY_SIDDQ_ON                    0
> >> +#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)

I would recommend aligning the value of of all macros in the same way.

> >> +
> >> +struct hsfreq_range {
> >> +	u32 range_h;

The structure would be more compact if you turned this into a u16.

> >> +	u8 cfg_bit;
> >> +};
> >> +
> >> +static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
> >> +	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
> >> +	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
> >> +	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
> >> +	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
> >> +	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
> >> +	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
> >> +	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
> >> +	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
> >> +	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
> >> +	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}

Maybe s/{/{ / and s/}/ }/ to give it a bit more air ? :-)

> >> +};
> >> +
> >> +static const char * const rk3399_mipidphy_clks[] = {
> >> +	"dphy-ref",
> >> +	"dphy-cfg",
> >> +	"grf",
> >> +};
> >> +
> >> +enum dphy_reg_id {
> >> +	GRF_DPHY_RX0_TURNDISABLE = 0,
> >> +	GRF_DPHY_RX0_FORCERXMODE,
> >> +	GRF_DPHY_RX0_FORCETXSTOPMODE,
> >> +	GRF_DPHY_RX0_ENABLE,
> >> +	GRF_DPHY_RX0_TESTCLR,
> >> +	GRF_DPHY_RX0_TESTCLK,
> >> +	GRF_DPHY_RX0_TESTEN,
> >> +	GRF_DPHY_RX0_TESTDIN,
> >> +	GRF_DPHY_RX0_TURNREQUEST,
> >> +	GRF_DPHY_RX0_TESTDOUT,
> >> +	GRF_DPHY_TX0_TURNDISABLE,
> >> +	GRF_DPHY_TX0_FORCERXMODE,
> >> +	GRF_DPHY_TX0_FORCETXSTOPMODE,
> >> +	GRF_DPHY_TX0_TURNREQUEST,
> >> +	GRF_DPHY_TX1RX1_TURNDISABLE,
> >> +	GRF_DPHY_TX1RX1_FORCERXMODE,
> >> +	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
> >> +	GRF_DPHY_TX1RX1_ENABLE,
> >> +	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
> >> +	GRF_DPHY_TX1RX1_BASEDIR,
> >> +	GRF_DPHY_TX1RX1_ENABLECLK,
> >> +	GRF_DPHY_TX1RX1_TURNREQUEST,
> >> +	GRF_DPHY_RX1_SRC_SEL,
> >> +	/* rk3288 only */
> >> +	GRF_CON_DISABLE_ISP,
> >> +	GRF_CON_ISP_DPHY_SEL,
> >> +	GRF_DSI_CSI_TESTBUS_SEL,
> >> +	GRF_DVP_V18SEL,
> >> +	/* below is for rk3399 only */
> >> +	GRF_DPHY_RX0_CLK_INV_SEL,
> >> +	GRF_DPHY_RX1_CLK_INV_SEL,
> >> +};
> >> +
> >> +struct dphy_reg {
> >> +	u32 offset;
> >> +	u32 mask;
> >> +	u32 shift;

The offset should hold in 16 bits and the mask and shift in 8 bits. That
would save space in the table below.

> >> +};
> >> +
> >> +#define PHY_REG(_offset, _width, _shift) \
> >> +	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
> >> +
> >> +static const struct dphy_reg rk3399_grf_dphy_regs[] = {
> >> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
> >> +	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
> >> +	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
> >> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
> >> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
> >> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
> >> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
> >> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
> >> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
> >> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
> >> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
> >> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
> >> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
> >> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
> >> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
> >> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
> >> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
> >> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
> >> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
> >> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
> >> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
> >> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
> >> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
> >> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
> >> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),

The annoying part with such an indirection is that you can't really
write multiple fields in a single register with a single operation. Is
the register mapping completely different between the rk3288 and the
rk3399, or are the fields grouped in registers in a similar way ? In the
latter case we could possibly optimise it.

> >> +};
> >> +
> >> +struct dphy_drv_data {
> >> +	const char * const *clks;
> >> +	int num_clks;

This is never negative, you can make it an unsigned int.

> >> +	const struct hsfreq_range *hsfreq_ranges;
> >> +	int num_hsfreq_ranges;

Same here.

> >> +	const struct dphy_reg *regs;
> >> +};
> >> +
> >> +struct rockchip_dphy {
> >> +	struct device *dev;
> >> +	struct regmap *grf;
> >> +	const struct dphy_reg *grf_regs;
> >> +	struct clk_bulk_data clks[MAX_DPHY_CLK];
> >> +
> >> +	const struct dphy_drv_data *drv_data;
> >> +	struct phy_configure_opts_mipi_dphy config;
> >> +};
> >> +
> >> +static inline void write_grf_reg(struct rockchip_dphy *priv,
> >> +				 int index, u8 value)

Maybe unsigned int index ?

> >> +{
> >> +	const struct dphy_reg *reg = &priv->grf_regs[index];
> >> +	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
> >> +
> >> +	WARN_ON(!reg->offset);
> >> +	regmap_write(priv->grf, reg->offset, val);
> >> +}
> >> +
> >> +static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
> >> +			     u8 test_code, u8 test_data)

Function (and structure) names have different prefixes, would it make
sense to standardise them ? Maybe rockchip_dphy_ ? Or rk_dphy_ for a
shorter version ? This could become rk_dphy_write_dphy(), and the
previous function rk_dphy_write_grf().

> >> +{
> >> +	/*
> >> +	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
> >> +	 * is latched internally as the current test code. Test data is
> >> +	 * programmed internally by rising edge on TESTCLK.
> >> +	 */

I've never understood why PHYs tend to have a register named TEST that
contains way more than test data :-)

> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> >> +}
> >> +
> >> +/* should be move to power_on */

s/move/moved/

Do you mean merging the two functions together ? What prevents from
doing so ? 

> >> +static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
> >> +{
> >> +	const struct dphy_drv_data *drv_data = priv->drv_data;
> >> +	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
> >> +	struct phy_configure_opts_mipi_dphy *config = &priv->config;
> >> +	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
> >> +	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
> >> +
> >> +	do_div(data_rate_mbps, 1000 * 1000);
> >> +
> >> +	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
> >> +		__func__, config->lanes, data_rate_mbps);
> >> +
> >> +	for (i = 0; i < num_hsfreq_ranges; i++) {
> >> +		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
> >> +			hsfreq = hsfreq_ranges[i].cfg_bit;
> >> +			break;
> >> +		}
> >> +	}

As num_hsfreq_ranges and hsfreq_ranges are only used in this loop, I
would remove the local variables.

> >> +
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
> >> +
> >> +	/* Disable lan turn around, which is ignored in receive mode */

Is it "lan turn around", or "lane turn around" ?

> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
> >> +
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
> >> +
> >> +	/* dphy start */
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
> >> +	usleep_range(100, 150);
> >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
> >> +	usleep_range(100, 150);
> >> +
> >> +	/* set clock lane */
> >> +	/* HS hsfreq_range & lane 0  settle bypass */
> >> +	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
> >> +	/* HS RX Control of lane0 */
> >> +	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
> >> +	/* HS RX Control of lane1 */
> >> +	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
> >> +	/* HS RX Control of lane2 */
> >> +	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
> >> +	/* HS RX Control of lane3 */
> >> +	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);

Does this hardcode usage of a single lane ?

> >> +	/* HS RX Data Lanes Settle State Time Control */
> >> +	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
> >> +			 THS_SETTLE_COUNTER_THRESHOLD);
> >> +
> >> +	/* Normal operation */
> >> +	mipidphy0_wr_reg(priv, 0x0, 0);
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
> >> +{
> >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >> +	int ret;
> >> +
> >> +	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */

I'm not sure to understand what this means.

> >> +	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	memcpy(&priv->config, opts, sizeof(priv->config));
> > 
> > You could to:
> > 
> > 	priv->config = *opts;
> > 
> > Up to you. Some people like memcpy(). :-)
> 
> your way is better thanks!
> 
> >> +
> >> +	return 0;
> >> +}
> >> +
> >> +static int rockchip_dphy_power_on(struct phy *phy)
> >> +{
> >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >> +	int ret;
> >> +
> >> +	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	return mipidphy_rx_stream_on(priv);

Should you call clk_bulk_disable() if mipidphy_rx_stream_on() fails ?
Actually that function never fails, so I'd make it a void function, and
return 0 here.

What happens if the clock rate is higher than the maximum supported by
the PHY ? Shouldn't rockchip_dphy_configure() fail in that case ?

> >> +}
> >> +
> >> +static int rockchip_dphy_power_off(struct phy *phy)
> >> +{
> >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >> +

No need to write any register ? That's scary, what will happen on the
next power on, when the clocks gets enabled ?

> >> +	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
> >> +	return 0;
> >> +}
> >> +
> >> +static int rockchip_dphy_init(struct phy *phy)
> >> +{
> >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >> +	int ret;
> >> +
> >> +	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);
> > 
> > return ...;
> > 
> >> +	if (ret)
> >> +		return ret;
> >> +	return 0;
> >> +}
> >> +
> >> +static int rockchip_dphy_exit(struct phy *phy)
> >> +{
> >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >> +
> >> +	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
> >> +	return 0;
> >> +}
> >> +
> >> +static const struct phy_ops rockchip_dphy_ops = {
> >> +	.power_on	= rockchip_dphy_power_on,
> >> +	.power_off	= rockchip_dphy_power_off,
> >> +	.init		= rockchip_dphy_init,
> >> +	.exit		= rockchip_dphy_exit,
> >> +	.configure	= rockchip_dphy_configure,
> >> +	.owner		= THIS_MODULE,
> >> +};
> >> +
> >> +static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
> >> +	.clks = rk3399_mipidphy_clks,
> >> +	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
> >> +	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
> >> +	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
> >> +	.regs = rk3399_grf_dphy_regs,
> > 
> > Do you expect to support more of the similar PHY(s) --- are there such? If
> > not, you could put these in the code that uses them.
> 
> Yes, for rk3288 in the future.
> 
> >> +};
> >> +
> >> +static const struct of_device_id rockchip_dphy_dt_ids[] = {
> >> +	{
> >> +		.compatible = "rockchip,rk3399-mipi-dphy",
> >> +		.data = &rk3399_mipidphy_drv_data,
> >> +	},
> >> +	{}
> >> +};
> >> +MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
> >> +
> >> +static int rockchip_dphy_probe(struct platform_device *pdev)
> >> +{
> >> +	struct device *dev = &pdev->dev;
> >> +	struct device_node *np = dev->of_node;
> >> +	const struct dphy_drv_data *drv_data;
> >> +	struct phy_provider *phy_provider;
> >> +	const struct of_device_id *of_id;
> >> +	struct rockchip_dphy *priv;
> >> +	struct regmap *grf;
> >> +	struct phy *phy;
> >> +	unsigned int i;
> >> +	int ret;
> >> +
> >> +	if (!dev->parent || !dev->parent->of_node)
> >> +		return -ENODEV;
> >> +
> >> +	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
> >> +		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");

You can replace pdev->dev with dev here and below.

s/rx/RX mode/ ?

> >> +		return -EINVAL;
> >> +	}
> >> +
> >> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> >> +	if (!priv)
> >> +		return -ENOMEM;
> >> +	priv->dev = dev;
> >> +
> >> +	grf = syscon_node_to_regmap(dev->parent->of_node);
> >> +	if (IS_ERR(grf)) {
> >> +		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
> >> +						      "rockchip,grf");
> >> +		if (IS_ERR(grf)) {
> >> +			dev_err(dev, "Can't find GRF syscon\n");
> >> +			return -ENODEV;
> >> +		}
> >> +	}
> >> +	priv->grf = grf;
> >> +
> >> +	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
> >> +	if (!of_id)
> >> +		return -EINVAL;
> >> +
> >> +	drv_data = of_id->data;
> >> +	priv->grf_regs = drv_data->regs;

Do you have to store grf_regs in priv, or could it be accessed through
priv->drv_data->regs ?

> >> +	priv->drv_data = drv_data;
> >> +	for (i = 0; i < drv_data->num_clks; i++)
> >> +		priv->clks[i].id = drv_data->clks[i];
> >> +	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
> >> +	if (IS_ERR(phy)) {
> >> +		dev_err(dev, "failed to create phy\n");
> >> +		return PTR_ERR(phy);
> >> +	}
> >> +	phy_set_drvdata(phy, priv);
> >> +
> >> +	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
> >> +
> >> +	return PTR_ERR_OR_ZERO(phy_provider);
> >> +}
> >> +
> >> +static struct platform_driver rockchip_dphy_driver = {
> >> +	.probe = rockchip_dphy_probe,
> >> +	.driver = {
> >> +		.name	= "rockchip-mipi-dphy",
> >> +		.of_match_table = rockchip_dphy_dt_ids,
> >> +	},
> >> +};
> >> +module_platform_driver(rockchip_dphy_driver);
> >> +
> >> +MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
> >> +MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
> >> +MODULE_LICENSE("Dual MIT/GPL");

Overall this is quite good, there are only small issues.

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 11/14] dt-bindings: Document the Rockchip MIPI RX D-PHY bindings
  2019-07-30 18:42 ` [PATCH v8 11/14] dt-bindings: Document the Rockchip MIPI RX D-PHY bindings Helen Koike
@ 2019-08-15 18:14   ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 18:14 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Rob Herring,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga,
	linux-rockchip, hans.verkuil, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

Thank you for the patch.

On Tue, Jul 30, 2019 at 03:42:53PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add DT bindings documentation for Rockchip MIPI D-PHY RX
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - updated doc with new design and tested example
> 
>  .../bindings/media/rockchip-mipi-dphy.txt     | 38 +++++++++++++++++++

Shouldn't this go to bindings/phy/ ?

>  1 file changed, 38 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
> 
> diff --git a/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt b/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
> new file mode 100644
> index 000000000000..2305d44d92db
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt
> @@ -0,0 +1,38 @@
> +Rockchip SoC MIPI RX D-PHY
> +-------------------------------------------------------------

How about already converting the bindings to yaml ? There's one example
in bindings/phy/ that looks quite similar to what you need here. Make
sure to have a look at Documentation/devicetree/writing-schema.md, and
in particular to run make dt_binding_check.

> +
> +Required properties:
> +- compatible: value should be one of the following
> +	"rockchip,rk3288-mipi-dphy"
> +	"rockchip,rk3399-mipi-dphy"
> +- clocks : list of clock specifiers, corresponding to entries in
> +	clock-names property;
> +- clock-names: required clock name.
> +- #phy-cells: Number of cells in a PHY specifier; Should be 0.
> +
> +MIPI RX D-PHY use registers in "general register files", it
> +should be a child of the GRF.
> +
> +Optional properties:
> +- reg: offset and length of the register set for the device.
> +- rockchip,grf: MIPI TX1RX1 D-PHY not only has its own register but also
> +		the GRF, so it is only necessary for MIPI TX1RX1 D-PHY.
> +
> +Device node example
> +-------------------
> +
> +grf: syscon@ff770000 {
> +	compatible = "rockchip,rk3399-grf", "syscon", "simple-mfd";
> +
> +...
> +
> +	dphy: mipi-dphy {
> +		compatible = "rockchip,rk3399-mipi-dphy";
> +		clocks = <&cru SCLK_MIPIDPHY_REF>,
> +			<&cru SCLK_DPHY_RX0_CFG>,
> +			<&cru PCLK_VIO_GRF>;
> +		clock-names = "dphy-ref", "dphy-cfg", "grf";
> +		power-domains = <&power RK3399_PD_VIO>;
> +		#phy-cells = <0>;
> +	};
> +};

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-08-15 17:54       ` Laurent Pinchart
@ 2019-08-15 18:26         ` Heiko Stübner
  2019-08-21 21:46         ` Helen Koike
  1 sibling, 0 replies; 76+ messages in thread
From: Heiko Stübner @ 2019-08-15 18:26 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: devicetree, eddie.cai.linux, kernel, jacob2.chen, jeffy.chen,
	zyc, linux-kernel, tfiga, linux-rockchip, Helen Koike,
	Sakari Ailus, sakari.ailus, zhengsq, hans.verkuil, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Laurent,

Am Donnerstag, 15. August 2019, 19:54:40 CEST schrieb Laurent Pinchart:
> On Wed, Aug 07, 2019 at 10:37:55AM -0300, Helen Koike wrote:
> > On 8/7/19 10:05 AM, Sakari Ailus wrote:
> > > On Tue, Jul 30, 2019 at 03:42:46PM -0300, Helen Koike wrote:
> > >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> > >>
> > >> This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver
> > >>
> > >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> > >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> > >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > >> [migrate to phy framework]
> > >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > >> [update for upstream]
> > >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > >>
> > >> ---
> > >>
> > >> Changes in v8:
> > >> - Remove boiler plate license text
> > >>
> > >> Changes in v7:
> > >> - Migrate dphy specific code from
> > >> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
> > >> to drivers/phy/rockchip/phy-rockchip-dphy.c
> > >> - Drop support for rk3288
> > >> - Drop support for dphy txrx
> > >> - code styling and checkpatch fixes
> > >>
> > >>  drivers/phy/rockchip/Kconfig             |   8 +
> > >>  drivers/phy/rockchip/Makefile            |   1 +
> > >>  drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
> > >>  3 files changed, 417 insertions(+)
> > >>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
> > >>
> > >> diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
> > >> index c454c90cd99e..afd072f135e6 100644
> > >> --- a/drivers/phy/rockchip/Kconfig
> > >> +++ b/drivers/phy/rockchip/Kconfig
> > >> @@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
> > >>  	help
> > >>  	  Enable this to support the Rockchip Display Port PHY.
> > >>  
> > >> +config PHY_ROCKCHIP_DPHY
> > >> +	tristate "Rockchip MIPI Synopsys DPHY driver"
> 
> How much of this PHY is Rockchip-specific ? Would it make sense to turn
> it into a Synopsys DPHY driver, with some Rockchip glue ? I suppose this
> could always be done later, if needed (and I also suppose there's no
> existing driver in drivers/phy/ that support the same Synopsys IP).

I'm not so versed in the video-stuff itself but I should be able to shed
some light on where the this thingy sits on the soc :-) .

SoCs like the rk3399 have 3 mipi-dphy instances:
- TX0 - output-only for dsi0
  with its access living _inside_ the mipi-dsi0 controller
  the registers to control it are part of the dsi-registers]

- TX1RX1 - output for dsi1 / input for isp1 (shared between them)
  its access lives inside the dsi1 controller's registers as well
  [not part of Helen's initial series, but I do have a wip-patch
   that exposes a phy-device from the dsi-driver for a later addon]

- RX1 - input for isp0 - this driver
  it's access is exposed through the GRF (General Register Files) of the soc
  The GRF is "chip designer's playground", registers change between each
  soc ... and the "randomness" can be seen by the RK3399_GRF_SOC_CONx
  register-naming, which is what they're called in all docs as well.

So I do think this is highly Rockchip-specific and it doesn't really make
a lot of sense trying to squeeze commonality out of it _before_ a second
actual instance of this rather strange system appears from some other
vendor :-) ... and even then I'd think keeping them separate would 
actually make things a lot easier long term.


Heiko


> 
> > >> +	depends on ARCH_ROCKCHIP && OF
> > > 
> > > How about (...) || COMPILE_TEST ?
> > > 
> > >> +	select GENERIC_PHY_MIPI_DPHY
> > >> +	select GENERIC_PHY
> > >> +	help
> > >> +	  Enable this to support the Rockchip MIPI Synopsys DPHY.
> > >> +
> > >>  config PHY_ROCKCHIP_EMMC
> > >>  	tristate "Rockchip EMMC PHY Driver"
> > >>  	depends on ARCH_ROCKCHIP && OF
> > >> diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
> > >> index fd21cbaf40dd..f62e9010bcaf 100644
> > >> --- a/drivers/phy/rockchip/Makefile
> > >> +++ b/drivers/phy/rockchip/Makefile
> > >> @@ -1,5 +1,6 @@
> > >>  # SPDX-License-Identifier: GPL-2.0
> > >>  obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
> > >> +obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
> > >>  obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
> > >>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
> > >>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
> > >> diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
> > >> new file mode 100644
> > >> index 000000000000..3a29976c2dff
> > >> --- /dev/null
> > >> +++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
> > >> @@ -0,0 +1,408 @@
> > >> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> > >> +/*
> > >> + * Rockchip MIPI Synopsys DPHY driver
> > >> + *
> > >> + * Based on:
> > >> + *
> > >> + * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
> > >> + * Author: Yakir Yang <ykk@@rock-chips.com>
> > >> + */
> > >> +
> > >> +#include <linux/clk.h>
> > >> +#include <linux/io.h>
> > >> +#include <linux/mfd/syscon.h>
> > >> +#include <linux/module.h>
> > >> +#include <linux/of.h>
> > >> +#include <linux/of_device.h>
> > >> +#include <linux/phy/phy.h>
> > >> +#include <linux/phy/phy-mipi-dphy.h>
> > >> +#include <linux/platform_device.h>
> > >> +#include <linux/regmap.h>
> > >> +
> > >> +#define RK3399_GRF_SOC_CON9	0x6224
> > >> +#define RK3399_GRF_SOC_CON21	0x6254
> > >> +#define RK3399_GRF_SOC_CON22	0x6258
> > >> +#define RK3399_GRF_SOC_CON23	0x625c
> > >> +#define RK3399_GRF_SOC_CON24	0x6260
> > >> +#define RK3399_GRF_SOC_CON25	0x6264
> > >> +#define RK3399_GRF_SOC_STATUS1	0xe2a4
> > >> +
> > >> +#define CLOCK_LANE_HS_RX_CONTROL		0x34
> > >> +#define LANE0_HS_RX_CONTROL			0x44
> > >> +#define LANE1_HS_RX_CONTROL			0x54
> > >> +#define LANE2_HS_RX_CONTROL			0x84
> > >> +#define LANE3_HS_RX_CONTROL			0x94
> > >> +#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
> > >> +
> > >> +#define MAX_DPHY_CLK 8
> > >> +
> > >> +#define PHY_TESTEN_ADDR			(0x1 << 16)
> > >> +#define PHY_TESTEN_DATA			(0x0 << 16)
> > >> +#define PHY_TESTCLK			(0x1 << 1)
> > >> +#define PHY_TESTCLR			(0x1 << 0)
> 
> Maybe s/0x// for the previous four lines ?
> 
> > >> +#define THS_SETTLE_COUNTER_THRESHOLD	0x04
> > >> +
> > >> +#define HIWORD_UPDATE(val, mask, shift) \
> > >> +	((val) << (shift) | (mask) << ((shift) + 16))
> 
> As you use this in a single place, I would inline it, possibly with a
> small comment that explains what's happening.
> 
> > >> +
> > >> +#define GRF_SOC_CON12                           0x0274
> > >> +
> > >> +#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
> > >> +#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
> > >> +
> > >> +#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
> > >> +#define GRF_EDP_PHY_SIDDQ_ON                    0
> > >> +#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)
> 
> I would recommend aligning the value of of all macros in the same way.
> 
> > >> +
> > >> +struct hsfreq_range {
> > >> +	u32 range_h;
> 
> The structure would be more compact if you turned this into a u16.
> 
> > >> +	u8 cfg_bit;
> > >> +};
> > >> +
> > >> +static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
> > >> +	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
> > >> +	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
> > >> +	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
> > >> +	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
> > >> +	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
> > >> +	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
> > >> +	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
> > >> +	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
> > >> +	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
> > >> +	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}
> 
> Maybe s/{/{ / and s/}/ }/ to give it a bit more air ? :-)
> 
> > >> +};
> > >> +
> > >> +static const char * const rk3399_mipidphy_clks[] = {
> > >> +	"dphy-ref",
> > >> +	"dphy-cfg",
> > >> +	"grf",
> > >> +};
> > >> +
> > >> +enum dphy_reg_id {
> > >> +	GRF_DPHY_RX0_TURNDISABLE = 0,
> > >> +	GRF_DPHY_RX0_FORCERXMODE,
> > >> +	GRF_DPHY_RX0_FORCETXSTOPMODE,
> > >> +	GRF_DPHY_RX0_ENABLE,
> > >> +	GRF_DPHY_RX0_TESTCLR,
> > >> +	GRF_DPHY_RX0_TESTCLK,
> > >> +	GRF_DPHY_RX0_TESTEN,
> > >> +	GRF_DPHY_RX0_TESTDIN,
> > >> +	GRF_DPHY_RX0_TURNREQUEST,
> > >> +	GRF_DPHY_RX0_TESTDOUT,
> > >> +	GRF_DPHY_TX0_TURNDISABLE,
> > >> +	GRF_DPHY_TX0_FORCERXMODE,
> > >> +	GRF_DPHY_TX0_FORCETXSTOPMODE,
> > >> +	GRF_DPHY_TX0_TURNREQUEST,
> > >> +	GRF_DPHY_TX1RX1_TURNDISABLE,
> > >> +	GRF_DPHY_TX1RX1_FORCERXMODE,
> > >> +	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
> > >> +	GRF_DPHY_TX1RX1_ENABLE,
> > >> +	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
> > >> +	GRF_DPHY_TX1RX1_BASEDIR,
> > >> +	GRF_DPHY_TX1RX1_ENABLECLK,
> > >> +	GRF_DPHY_TX1RX1_TURNREQUEST,
> > >> +	GRF_DPHY_RX1_SRC_SEL,
> > >> +	/* rk3288 only */
> > >> +	GRF_CON_DISABLE_ISP,
> > >> +	GRF_CON_ISP_DPHY_SEL,
> > >> +	GRF_DSI_CSI_TESTBUS_SEL,
> > >> +	GRF_DVP_V18SEL,
> > >> +	/* below is for rk3399 only */
> > >> +	GRF_DPHY_RX0_CLK_INV_SEL,
> > >> +	GRF_DPHY_RX1_CLK_INV_SEL,
> > >> +};
> > >> +
> > >> +struct dphy_reg {
> > >> +	u32 offset;
> > >> +	u32 mask;
> > >> +	u32 shift;
> 
> The offset should hold in 16 bits and the mask and shift in 8 bits. That
> would save space in the table below.
> 
> > >> +};
> > >> +
> > >> +#define PHY_REG(_offset, _width, _shift) \
> > >> +	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
> > >> +
> > >> +static const struct dphy_reg rk3399_grf_dphy_regs[] = {
> > >> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
> > >> +	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
> > >> +	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
> > >> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
> > >> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
> > >> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
> > >> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
> > >> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
> > >> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
> > >> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
> > >> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
> > >> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
> > >> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
> > >> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
> > >> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
> > >> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
> > >> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
> > >> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
> > >> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
> > >> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
> > >> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
> > >> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
> > >> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
> > >> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
> > >> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),
> 
> The annoying part with such an indirection is that you can't really
> write multiple fields in a single register with a single operation. Is
> the register mapping completely different between the rk3288 and the
> rk3399, or are the fields grouped in registers in a similar way ? In the
> latter case we could possibly optimise it.
> 
> > >> +};
> > >> +
> > >> +struct dphy_drv_data {
> > >> +	const char * const *clks;
> > >> +	int num_clks;
> 
> This is never negative, you can make it an unsigned int.
> 
> > >> +	const struct hsfreq_range *hsfreq_ranges;
> > >> +	int num_hsfreq_ranges;
> 
> Same here.
> 
> > >> +	const struct dphy_reg *regs;
> > >> +};
> > >> +
> > >> +struct rockchip_dphy {
> > >> +	struct device *dev;
> > >> +	struct regmap *grf;
> > >> +	const struct dphy_reg *grf_regs;
> > >> +	struct clk_bulk_data clks[MAX_DPHY_CLK];
> > >> +
> > >> +	const struct dphy_drv_data *drv_data;
> > >> +	struct phy_configure_opts_mipi_dphy config;
> > >> +};
> > >> +
> > >> +static inline void write_grf_reg(struct rockchip_dphy *priv,
> > >> +				 int index, u8 value)
> 
> Maybe unsigned int index ?
> 
> > >> +{
> > >> +	const struct dphy_reg *reg = &priv->grf_regs[index];
> > >> +	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
> > >> +
> > >> +	WARN_ON(!reg->offset);
> > >> +	regmap_write(priv->grf, reg->offset, val);
> > >> +}
> > >> +
> > >> +static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
> > >> +			     u8 test_code, u8 test_data)
> 
> Function (and structure) names have different prefixes, would it make
> sense to standardise them ? Maybe rockchip_dphy_ ? Or rk_dphy_ for a
> shorter version ? This could become rk_dphy_write_dphy(), and the
> previous function rk_dphy_write_grf().
> 
> > >> +{
> > >> +	/*
> > >> +	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
> > >> +	 * is latched internally as the current test code. Test data is
> > >> +	 * programmed internally by rising edge on TESTCLK.
> > >> +	 */
> 
> I've never understood why PHYs tend to have a register named TEST that
> contains way more than test data :-)
> 
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> > >> +}
> > >> +
> > >> +/* should be move to power_on */
> 
> s/move/moved/
> 
> Do you mean merging the two functions together ? What prevents from
> doing so ? 
> 
> > >> +static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
> > >> +{
> > >> +	const struct dphy_drv_data *drv_data = priv->drv_data;
> > >> +	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
> > >> +	struct phy_configure_opts_mipi_dphy *config = &priv->config;
> > >> +	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
> > >> +	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
> > >> +
> > >> +	do_div(data_rate_mbps, 1000 * 1000);
> > >> +
> > >> +	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
> > >> +		__func__, config->lanes, data_rate_mbps);
> > >> +
> > >> +	for (i = 0; i < num_hsfreq_ranges; i++) {
> > >> +		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
> > >> +			hsfreq = hsfreq_ranges[i].cfg_bit;
> > >> +			break;
> > >> +		}
> > >> +	}
> 
> As num_hsfreq_ranges and hsfreq_ranges are only used in this loop, I
> would remove the local variables.
> 
> > >> +
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
> > >> +
> > >> +	/* Disable lan turn around, which is ignored in receive mode */
> 
> Is it "lan turn around", or "lane turn around" ?
> 
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
> > >> +
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
> > >> +
> > >> +	/* dphy start */
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
> > >> +	usleep_range(100, 150);
> > >> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
> > >> +	usleep_range(100, 150);
> > >> +
> > >> +	/* set clock lane */
> > >> +	/* HS hsfreq_range & lane 0  settle bypass */
> > >> +	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
> > >> +	/* HS RX Control of lane0 */
> > >> +	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
> > >> +	/* HS RX Control of lane1 */
> > >> +	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
> > >> +	/* HS RX Control of lane2 */
> > >> +	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
> > >> +	/* HS RX Control of lane3 */
> > >> +	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);
> 
> Does this hardcode usage of a single lane ?
> 
> > >> +	/* HS RX Data Lanes Settle State Time Control */
> > >> +	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
> > >> +			 THS_SETTLE_COUNTER_THRESHOLD);
> > >> +
> > >> +	/* Normal operation */
> > >> +	mipidphy0_wr_reg(priv, 0x0, 0);
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
> > >> +{
> > >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> > >> +	int ret;
> > >> +
> > >> +	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */
> 
> I'm not sure to understand what this means.
> 
> > >> +	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	memcpy(&priv->config, opts, sizeof(priv->config));
> > > 
> > > You could to:
> > > 
> > > 	priv->config = *opts;
> > > 
> > > Up to you. Some people like memcpy(). :-)
> > 
> > your way is better thanks!
> > 
> > >> +
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int rockchip_dphy_power_on(struct phy *phy)
> > >> +{
> > >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> > >> +	int ret;
> > >> +
> > >> +	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	return mipidphy_rx_stream_on(priv);
> 
> Should you call clk_bulk_disable() if mipidphy_rx_stream_on() fails ?
> Actually that function never fails, so I'd make it a void function, and
> return 0 here.
> 
> What happens if the clock rate is higher than the maximum supported by
> the PHY ? Shouldn't rockchip_dphy_configure() fail in that case ?
> 
> > >> +}
> > >> +
> > >> +static int rockchip_dphy_power_off(struct phy *phy)
> > >> +{
> > >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> > >> +
> 
> No need to write any register ? That's scary, what will happen on the
> next power on, when the clocks gets enabled ?
> 
> > >> +	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int rockchip_dphy_init(struct phy *phy)
> > >> +{
> > >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> > >> +	int ret;
> > >> +
> > >> +	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);
> > > 
> > > return ...;
> > > 
> > >> +	if (ret)
> > >> +		return ret;
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static int rockchip_dphy_exit(struct phy *phy)
> > >> +{
> > >> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> > >> +
> > >> +	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
> > >> +	return 0;
> > >> +}
> > >> +
> > >> +static const struct phy_ops rockchip_dphy_ops = {
> > >> +	.power_on	= rockchip_dphy_power_on,
> > >> +	.power_off	= rockchip_dphy_power_off,
> > >> +	.init		= rockchip_dphy_init,
> > >> +	.exit		= rockchip_dphy_exit,
> > >> +	.configure	= rockchip_dphy_configure,
> > >> +	.owner		= THIS_MODULE,
> > >> +};
> > >> +
> > >> +static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
> > >> +	.clks = rk3399_mipidphy_clks,
> > >> +	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
> > >> +	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
> > >> +	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
> > >> +	.regs = rk3399_grf_dphy_regs,
> > > 
> > > Do you expect to support more of the similar PHY(s) --- are there such? If
> > > not, you could put these in the code that uses them.
> > 
> > Yes, for rk3288 in the future.
> > 
> > >> +};
> > >> +
> > >> +static const struct of_device_id rockchip_dphy_dt_ids[] = {
> > >> +	{
> > >> +		.compatible = "rockchip,rk3399-mipi-dphy",
> > >> +		.data = &rk3399_mipidphy_drv_data,
> > >> +	},
> > >> +	{}
> > >> +};
> > >> +MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
> > >> +
> > >> +static int rockchip_dphy_probe(struct platform_device *pdev)
> > >> +{
> > >> +	struct device *dev = &pdev->dev;
> > >> +	struct device_node *np = dev->of_node;
> > >> +	const struct dphy_drv_data *drv_data;
> > >> +	struct phy_provider *phy_provider;
> > >> +	const struct of_device_id *of_id;
> > >> +	struct rockchip_dphy *priv;
> > >> +	struct regmap *grf;
> > >> +	struct phy *phy;
> > >> +	unsigned int i;
> > >> +	int ret;
> > >> +
> > >> +	if (!dev->parent || !dev->parent->of_node)
> > >> +		return -ENODEV;
> > >> +
> > >> +	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
> > >> +		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");
> 
> You can replace pdev->dev with dev here and below.
> 
> s/rx/RX mode/ ?
> 
> > >> +		return -EINVAL;
> > >> +	}
> > >> +
> > >> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > >> +	if (!priv)
> > >> +		return -ENOMEM;
> > >> +	priv->dev = dev;
> > >> +
> > >> +	grf = syscon_node_to_regmap(dev->parent->of_node);
> > >> +	if (IS_ERR(grf)) {
> > >> +		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
> > >> +						      "rockchip,grf");
> > >> +		if (IS_ERR(grf)) {
> > >> +			dev_err(dev, "Can't find GRF syscon\n");
> > >> +			return -ENODEV;
> > >> +		}
> > >> +	}
> > >> +	priv->grf = grf;
> > >> +
> > >> +	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
> > >> +	if (!of_id)
> > >> +		return -EINVAL;
> > >> +
> > >> +	drv_data = of_id->data;
> > >> +	priv->grf_regs = drv_data->regs;
> 
> Do you have to store grf_regs in priv, or could it be accessed through
> priv->drv_data->regs ?
> 
> > >> +	priv->drv_data = drv_data;
> > >> +	for (i = 0; i < drv_data->num_clks; i++)
> > >> +		priv->clks[i].id = drv_data->clks[i];
> > >> +	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
> > >> +	if (ret)
> > >> +		return ret;
> > >> +
> > >> +	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
> > >> +	if (IS_ERR(phy)) {
> > >> +		dev_err(dev, "failed to create phy\n");
> > >> +		return PTR_ERR(phy);
> > >> +	}
> > >> +	phy_set_drvdata(phy, priv);
> > >> +
> > >> +	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
> > >> +
> > >> +	return PTR_ERR_OR_ZERO(phy_provider);
> > >> +}
> > >> +
> > >> +static struct platform_driver rockchip_dphy_driver = {
> > >> +	.probe = rockchip_dphy_probe,
> > >> +	.driver = {
> > >> +		.name	= "rockchip-mipi-dphy",
> > >> +		.of_match_table = rockchip_dphy_dt_ids,
> > >> +	},
> > >> +};
> > >> +module_platform_driver(rockchip_dphy_driver);
> > >> +
> > >> +MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
> > >> +MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
> > >> +MODULE_LICENSE("Dual MIT/GPL");
> 
> Overall this is quite good, there are only small issues.
> 
> 





_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions
  2019-07-30 18:42 ` [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions Helen Koike
@ 2019-08-15 18:46   ` Laurent Pinchart
  2020-07-10 12:59     ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 18:46 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

Hi Helen,

Thank you ofor the patch.

On Tue, Jul 30, 2019 at 03:42:45PM -0300, Helen Koike wrote:
> From: Jeffy Chen <jeffy.chen@rock-chips.com>
> 
> Add the header for userspace
> 
> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> Hi,
> 
> I don't have the "REF_01 - ISP_user_manual, Rev 2.57" document that was
> mentioned in previous version, so I'm adding a TODO for now to improve
> the docs once we collect the information.

I would keep the document name in the TODO entry, for reference.

> If Rockchip people could help here it would be great.

As you don't have access to the documentation I won't focus on the
issues that you can't address, but in general I think we need more
documentation than what is provided here. In particular we need a
description of the pipeline, with the order of the processing blocks.

> Changes in v8: None
> Changes in v7:
> - Fix checkpatch errors (lines over 80 and SPDX)
> - Add TODO to improve docs
> 
>  include/uapi/linux/rkisp1-config.h | 816 +++++++++++++++++++++++++++++
>  1 file changed, 816 insertions(+)
>  create mode 100644 include/uapi/linux/rkisp1-config.h
> 
> diff --git a/include/uapi/linux/rkisp1-config.h b/include/uapi/linux/rkisp1-config.h
> new file mode 100644
> index 000000000000..9ab979bb4adb
> --- /dev/null
> +++ b/include/uapi/linux/rkisp1-config.h
> @@ -0,0 +1,816 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver

Maybe "Rockchip ISP1 userspace API" ?

> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +/*
> + * TODO: Improve documentation, mostly regarding abbreviation and hardware
> + * specificities.
> + */
> +
> +#ifndef _UAPI_RKISP1_CONFIG_H
> +#define _UAPI_RKISP1_CONFIG_H
> +
> +#include <linux/types.h>
> +#include <linux/v4l2-controls.h>

Is v4l2-controls.h needed ?

> +
> +#define CIFISP_MODULE_DPCC              (1 << 0)
> +#define CIFISP_MODULE_BLS               (1 << 1)
> +#define CIFISP_MODULE_SDG               (1 << 2)
> +#define CIFISP_MODULE_HST               (1 << 3)
> +#define CIFISP_MODULE_LSC               (1 << 4)
> +#define CIFISP_MODULE_AWB_GAIN          (1 << 5)
> +#define CIFISP_MODULE_FLT               (1 << 6)
> +#define CIFISP_MODULE_BDM               (1 << 7)
> +#define CIFISP_MODULE_CTK               (1 << 8)
> +#define CIFISP_MODULE_GOC               (1 << 9)
> +#define CIFISP_MODULE_CPROC             (1 << 10)
> +#define CIFISP_MODULE_AFC               (1 << 11)
> +#define CIFISP_MODULE_AWB               (1 << 12)
> +#define CIFISP_MODULE_IE                (1 << 13)
> +#define CIFISP_MODULE_AEC               (1 << 14)
> +#define CIFISP_MODULE_WDR               (1 << 15)
> +#define CIFISP_MODULE_DPF               (1 << 16)
> +#define CIFISP_MODULE_DPF_STRENGTH      (1 << 17)

Should we use BIT() here ?

> +
> +#define CIFISP_CTK_COEFF_MAX            0x100
> +#define CIFISP_CTK_OFFSET_MAX           0x800
> +
> +#define CIFISP_AE_MEAN_MAX              25
> +#define CIFISP_HIST_BIN_N_MAX           16
> +#define CIFISP_AFM_MAX_WINDOWS          3
> +#define CIFISP_DEGAMMA_CURVE_SIZE       17
> +
> +#define CIFISP_BDM_MAX_TH               0xFF

Hex constants are usually expressed lower-case in the kernel.

> +
> +/*
> + * Black level compensation
> + */
> +/* maximum value for horizontal start address */
> +#define CIFISP_BLS_START_H_MAX             0x00000FFF
> +/* maximum value for horizontal stop address */
> +#define CIFISP_BLS_STOP_H_MAX              0x00000FFF
> +/* maximum value for vertical start address */
> +#define CIFISP_BLS_START_V_MAX             0x00000FFF
> +/* maximum value for vertical stop address */
> +#define CIFISP_BLS_STOP_V_MAX              0x00000FFF
> +/* maximum is 2^18 = 262144*/
> +#define CIFISP_BLS_SAMPLES_MAX             0x00000012
> +/* maximum value for fixed black level */
> +#define CIFISP_BLS_FIX_SUB_MAX             0x00000FFF
> +/* minimum value for fixed black level */
> +#define CIFISP_BLS_FIX_SUB_MIN             0xFFFFF000
> +/* 13 bit range (signed)*/
> +#define CIFISP_BLS_FIX_MASK                0x00001FFF
> +
> +/*
> + * Automatic white balance measurments
> + */
> +#define CIFISP_AWB_MAX_GRID                1
> +#define CIFISP_AWB_MAX_FRAMES              7
> +
> +/*
> + * Gamma out
> + */
> +/* Maximum number of color samples supported */
> +#define CIFISP_GAMMA_OUT_MAX_SAMPLES       17
> +
> +/*
> + * Lens shade correction
> + */
> +#define CIFISP_LSC_GRAD_TBL_SIZE           8
> +#define CIFISP_LSC_SIZE_TBL_SIZE           8
> +/*
> + * The following matches the tuning process,
> + * not the max capabilities of the chip.
> + * Last value unused.
> + */
> +#define	CIFISP_LSC_DATA_TBL_SIZE           290
> +
> +/*
> + * Histogram calculation
> + */
> +/* Last 3 values unused. */
> +#define CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE 28
> +
> +/*
> + * Defect Pixel Cluster Correction
> + */
> +#define CIFISP_DPCC_METHODS_MAX       3
> +
> +/*
> + * Denoising pre filter
> + */
> +#define CIFISP_DPF_MAX_NLF_COEFFS      17
> +#define CIFISP_DPF_MAX_SPATIAL_COEFFS  6
> +
> +/*
> + * Measurement types
> + */
> +#define CIFISP_STAT_AWB           (1 << 0)
> +#define CIFISP_STAT_AUTOEXP       (1 << 1)
> +#define CIFISP_STAT_AFM_FIN       (1 << 2)
> +#define CIFISP_STAT_HIST          (1 << 3)
> +
> +enum cifisp_histogram_mode {
> +	CIFISP_HISTOGRAM_MODE_DISABLE,
> +	CIFISP_HISTOGRAM_MODE_RGB_COMBINED,
> +	CIFISP_HISTOGRAM_MODE_R_HISTOGRAM,
> +	CIFISP_HISTOGRAM_MODE_G_HISTOGRAM,
> +	CIFISP_HISTOGRAM_MODE_B_HISTOGRAM,
> +	CIFISP_HISTOGRAM_MODE_Y_HISTOGRAM
> +};
> +
> +enum cifisp_awb_mode_type {
> +	CIFISP_AWB_MODE_MANUAL,
> +	CIFISP_AWB_MODE_RGB,
> +	CIFISP_AWB_MODE_YCBCR
> +};
> +
> +enum cifisp_flt_mode {
> +	CIFISP_FLT_STATIC_MODE,
> +	CIFISP_FLT_DYNAMIC_MODE
> +};
> +
> +/**
> + * enum cifisp_exp_ctrl_autostop - stop modes
> + * @CIFISP_EXP_CTRL_AUTOSTOP_0: continuous measurement
> + * @CIFISP_EXP_CTRL_AUTOSTOP_1: stop measuring after a complete frame
> + */
> +enum cifisp_exp_ctrl_autostop {
> +	CIFISP_EXP_CTRL_AUTOSTOP_0 = 0,
> +	CIFISP_EXP_CTRL_AUTOSTOP_1 = 1,
> +};
> +
> +/**
> + * enum cifisp_exp_meas_mode - Exposure measure mode
> + * @CIFISP_EXP_MEASURING_MODE_0: Y = 16 + 0.25R + 0.5G + 0.1094B
> + * @CIFISP_EXP_MEASURING_MODE_1: Y = (R + G + B) x (85/256)
> + */
> +enum cifisp_exp_meas_mode {
> +	CIFISP_EXP_MEASURING_MODE_0,
> +	CIFISP_EXP_MEASURING_MODE_1,
> +};
> +
> +/*---------- PART1: Input Parameters ------------*/
> +
> +struct cifisp_window {
> +	__u16 h_offs;
> +	__u16 v_offs;
> +	__u16 h_size;
> +	__u16 v_size;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_bls_fixed_val - BLS fixed subtraction values
> + *
> + * The values will be subtracted from the sensor
> + * values. Therefore a negative value means addition instead of subtraction!
> + *
> + * @r: Fixed (signed!) subtraction value for Bayer pattern R
> + * @gr: Fixed (signed!) subtraction value for Bayer pattern Gr
> + * @gb: Fixed (signed!) subtraction value for Bayer pattern Gb
> + * @b: Fixed (signed!) subtraction value for Bayer pattern B
> + */
> +struct cifisp_bls_fixed_val {
> +	__s16 r;
> +	__s16 gr;
> +	__s16 gb;
> +	__s16 b;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_bls_config - Configuration used by black level subtraction
> + *
> + * @enable_auto: Automatic mode activated means that the measured values
> + *		 are subtracted. Otherwise the fixed subtraction
> + *		 values will be subtracted.
> + * @en_windows: enabled window
> + * @bls_window1: Measurement window 1 size
> + * @bls_window2: Measurement window 2 size
> + * @bls_samples: Set amount of measured pixels for each Bayer position
> + *		 (A, B,C and D) to 2^bls_samples.
> + * @cifisp_bls_fixed_val: Fixed subtraction values
> + */
> +struct cifisp_bls_config {
> +	__u8 enable_auto;
> +	__u8 en_windows;
> +	struct cifisp_window bls_window1;
> +	struct cifisp_window bls_window2;
> +	__u8 bls_samples;
> +	struct cifisp_bls_fixed_val fixed_val;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpcc_methods_config - Methods Configuration used by DPCC
> + *
> + * Methods Configuration used by Defect Pixel Cluster Correction
> + *
> + * @method: Method enable bits
> + * @line_thresh: Line threshold
> + * @line_mad_fac: Line MAD factor
> + * @pg_fac: Peak gradient factor
> + * @rnd_thresh: Rank Neighbor Difference threshold
> + * @rg_fac: Rank gradient factor
> + */
> +struct cifisp_dpcc_methods_config {
> +	__u32 method;
> +	__u32 line_thresh;
> +	__u32 line_mad_fac;
> +	__u32 pg_fac;
> +	__u32 rnd_thresh;
> +	__u32 rg_fac;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpcc_methods_config - Configuration used by DPCC

Should be cifisp_dpcc_config.

> + *
> + * Configuration used by Defect Pixel Cluster Correction
> + *
> + * @mode: dpcc output mode
> + * @output_mode: whether use hard coded methods
> + * @set_use: stage1 methods set
> + * @methods: methods config
> + * @ro_limits: rank order limits
> + * @rnd_offs: differential rank offsets for rank neighbor difference
> + */
> +struct cifisp_dpcc_config {
> +	__u32 mode;
> +	__u32 output_mode;
> +	__u32 set_use;
> +	struct cifisp_dpcc_methods_config methods[CIFISP_DPCC_METHODS_MAX];
> +	__u32 ro_limits;
> +	__u32 rnd_offs;
> +} __attribute__ ((packed));
> +
> +struct cifisp_gamma_corr_curve {
> +	__u16 gamma_y[CIFISP_DEGAMMA_CURVE_SIZE];
> +} __attribute__ ((packed));
> +
> +struct cifisp_gamma_curve_x_axis_pnts {
> +	__u32 gamma_dx0;
> +	__u32 gamma_dx1;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_gamma_corr_curve - Configuration used by sensor degamma

Should be cifisp_sdg_config

> + *
> + * @curve_x: gamma curve point definition axis for x
> + * @xa_pnts: x increments
> + */
> +struct cifisp_sdg_config {
> +	struct cifisp_gamma_corr_curve curve_r;
> +	struct cifisp_gamma_corr_curve curve_g;
> +	struct cifisp_gamma_corr_curve curve_b;
> +	struct cifisp_gamma_curve_x_axis_pnts xa_pnts;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_lsc_config - Configuration used by Lens shading correction
> + *
> + * refer to REF_01 for details
> + */
> +struct cifisp_lsc_config {
> +	__u32 r_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> +	__u32 gr_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> +	__u32 gb_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> +	__u32 b_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> +
> +	__u32 x_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
> +	__u32 y_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
> +
> +	__u32 x_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
> +	__u32 y_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
> +	__u16 config_width;
> +	__u16 config_height;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_ie_config - Configuration used by image effects
> + *
> + * @eff_mat_1: 3x3 Matrix Coefficients for Emboss Effect 1
> + * @eff_mat_2: 3x3 Matrix Coefficients for Emboss Effect 2
> + * @eff_mat_3: 3x3 Matrix Coefficients for Emboss 3/Sketch 1
> + * @eff_mat_4: 3x3 Matrix Coefficients for Sketch Effect 2
> + * @eff_mat_5: 3x3 Matrix Coefficients for Sketch Effect 3
> + * @eff_tint: Chrominance increment values of tint (used for sepia effect)
> + */
> +struct cifisp_ie_config {
> +	__u16 effect;
> +	__u16 color_sel;
> +	__u16 eff_mat_1;
> +	__u16 eff_mat_2;
> +	__u16 eff_mat_3;
> +	__u16 eff_mat_4;
> +	__u16 eff_mat_5;
> +	__u16 eff_tint;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_cproc_config - Configuration used by Color Processing
> + *
> + * @c_out_range: Chrominance pixel clipping range at output.
> + *		 (0 for limit, 1 for full)
> + * @y_in_range: Luminance pixel clipping range at output.
> + * @y_out_range: Luminance pixel clipping range at output.
> + * @contrast: 00~ff, 0.0~1.992
> + * @brightness: 80~7F, -128~+127
> + * @sat: saturation, 00~FF, 0.0~1.992
> + * @hue: 80~7F, -90~+87.188
> + */
> +struct cifisp_cproc_config {
> +	__u8 c_out_range;
> +	__u8 y_in_range;
> +	__u8 y_out_range;
> +	__u8 contrast;
> +	__u8 brightness;
> +	__u8 sat;
> +	__u8 hue;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_awb_meas_config - Configuration used by auto white balance
> + *
> + * @awb_wnd: white balance measurement window (in pixels)
> + *	     (from enum cifisp_awb_mode_type)
> + * @max_y: only pixels values < max_y contribute to awb measurement, set to 0
> + *	   to disable this feature
> + * @min_y: only pixels values > min_y contribute to awb measurement
> + * @max_csum: Chrominance sum maximum value, only consider pixels with Cb+Cr,
> + *	      smaller than threshold for awb measurements
> + * @min_c: Chrominance minimum value, only consider pixels with Cb/Cr
> + *	   each greater than threshold value for awb measurements
> + * @frames: number of frames - 1 used for mean value calculation
> + *	    (ucFrames=0 means 1 Frame)
> + * @awb_ref_cr: reference Cr value for AWB regulation, target for AWB
> + * @awb_ref_cb: reference Cb value for AWB regulation, target for AWB
> + */
> +struct cifisp_awb_meas_config {
> +	/*
> +	 * Note: currently the h and v offsets are mapped to grid offsets
> +	 */
> +	struct cifisp_window awb_wnd;
> +	__u32 awb_mode;
> +	__u8 max_y;
> +	__u8 min_y;
> +	__u8 max_csum;
> +	__u8 min_c;
> +	__u8 frames;
> +	__u8 awb_ref_cr;
> +	__u8 awb_ref_cb;
> +	__u8 enable_ymax_cmp;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_awb_gain_config - Configuration used by auto white balance gain
> + *
> + * out_data_x = ( AWB_GEAIN_X * in_data + 128) >> 8
> + */
> +struct cifisp_awb_gain_config {
> +	__u16 gain_red;
> +	__u16 gain_green_r;
> +	__u16 gain_blue;
> +	__u16 gain_green_b;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_flt_config - Configuration used by ISP filtering
> + *
> + * @mode: ISP_FILT_MODE register fields (from enum cifisp_flt_mode)
> + * @grn_stage1: ISP_FILT_MODE register fields
> + * @chr_h_mode: ISP_FILT_MODE register fields
> + * @chr_v_mode: ISP_FILT_MODE register fields
> + *
> + * refer to REF_01 for details.
> + */
> +
> +struct cifisp_flt_config {
> +	__u32 mode;
> +	__u8 grn_stage1;
> +	__u8 chr_h_mode;
> +	__u8 chr_v_mode;
> +	__u32 thresh_bl0;
> +	__u32 thresh_bl1;
> +	__u32 thresh_sh0;
> +	__u32 thresh_sh1;
> +	__u32 lum_weight;
> +	__u32 fac_sh1;
> +	__u32 fac_sh0;
> +	__u32 fac_mid;
> +	__u32 fac_bl0;
> +	__u32 fac_bl1;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_bdm_config - Configuration used by Bayer DeMosaic
> + *
> + * @demosaic_th: threshod for bayer demosaicing texture detection
> + */
> +struct cifisp_bdm_config {
> +	__u8 demosaic_th;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_ctk_config - Configuration used by Cross Talk correction
> + *
> + * @coeff: color correction matrix
> + * @ct_offset_b: offset for the crosstalk correction matrix
> + */
> +struct cifisp_ctk_config {
> +	__u16 coeff0;
> +	__u16 coeff1;
> +	__u16 coeff2;
> +	__u16 coeff3;
> +	__u16 coeff4;
> +	__u16 coeff5;
> +	__u16 coeff6;
> +	__u16 coeff7;
> +	__u16 coeff8;
> +	__u16 ct_offset_r;
> +	__u16 ct_offset_g;
> +	__u16 ct_offset_b;
> +} __attribute__ ((packed));
> +
> +enum cifisp_goc_mode {
> +	CIFISP_GOC_MODE_LOGARITHMIC,
> +	CIFISP_GOC_MODE_EQUIDISTANT
> +};
> +
> +/**
> + * struct cifisp_goc_config - Configuration used by Gamma Out correction
> + *
> + * @mode: goc mode (from enum cifisp_goc_mode)
> + * @gamma_y: gamma out curve y-axis for all color components
> + */
> +struct cifisp_goc_config {
> +	__u32 mode;
> +	__u16 gamma_y[CIFISP_GAMMA_OUT_MAX_SAMPLES];
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_hst_config - Configuration used by Histogram
> + *
> + * @mode: histogram mode (from enum cifisp_histogram_mode)
> + * @histogram_predivider: process every stepsize pixel, all other pixels are
> + *			  skipped
> + * @meas_window: coordinates of the measure window
> + * @hist_weight: weighting factor for sub-windows
> + */
> +struct cifisp_hst_config {
> +	__u32 mode;
> +	__u8 histogram_predivider;
> +	struct cifisp_window meas_window;
> +	__u8 hist_weight[CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE];
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_aec_config - Configuration used by Auto Exposure Control
> + *
> + * @mode: Exposure measure mode (from enum cifisp_exp_meas_mode)
> + * @autostop: stop mode (from enum cifisp_exp_ctrl_autostop)
> + * @meas_window: coordinates of the measure window
> + */
> +struct cifisp_aec_config {
> +	__u32 mode;
> +	__u32 autostop;
> +	struct cifisp_window meas_window;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_afc_config - Configuration used by Auto Focus Control
> + *
> + * @num_afm_win: max CIFISP_AFM_MAX_WINDOWS
> + * @afm_win: coordinates of the meas window
> + * @thres: threshold used for minimizing the influence of noise
> + * @var_shift: the number of bits for the shift operation at the end of the
> + *	       calculation chain.
> + */
> +struct cifisp_afc_config {
> +	__u8 num_afm_win;
> +	struct cifisp_window afm_win[CIFISP_AFM_MAX_WINDOWS];
> +	__u32 thres;
> +	__u32 var_shift;
> +} __attribute__ ((packed));
> +
> +/**
> + * enum cifisp_dpf_gain_usage - dpf gain usage
> + * @CIFISP_DPF_GAIN_USAGE_DISABLED: don't use any gains in preprocessing stage
> + * @CIFISP_DPF_GAIN_USAGE_NF_GAINS: use only the noise function gains from
> + *				    registers DPF_NF_GAIN_R, ...
> + * @CIFISP_DPF_GAIN_USAGE_LSC_GAINS:  use only the gains from LSC module
> + * @CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS: use the noise function gains and the
> + *					gains from LSC module
> + * @CIFISP_DPF_GAIN_USAGE_AWB_GAINS: use only the gains from AWB module
> + * @CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS: use the gains from AWB and LSC module
> + * @CIFISP_DPF_GAIN_USAGE_MAX: upper border (only for an internal evaluation)
> + */
> +enum cifisp_dpf_gain_usage {
> +	CIFISP_DPF_GAIN_USAGE_DISABLED,
> +	CIFISP_DPF_GAIN_USAGE_NF_GAINS,
> +	CIFISP_DPF_GAIN_USAGE_LSC_GAINS,
> +	CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS,
> +	CIFISP_DPF_GAIN_USAGE_AWB_GAINS,
> +	CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS,
> +	CIFISP_DPF_GAIN_USAGE_MAX
> +};
> +
> +/**
> + * enum cifisp_dpf_gain_usage - dpf gain usage

This should be cifisp_dpf_rb_filtersize and the description should be
updated.

> + * @CIFISP_DPF_RB_FILTERSIZE_13x9: red and blue filter kernel size 13x9
> + *				   (means 7x5 active pixel)
> + * @CIFISP_DPF_RB_FILTERSIZE_9x9: red and blue filter kernel size 9x9
> + *				   (means 5x5 active pixel)
> + */
> +enum cifisp_dpf_rb_filtersize {
> +	CIFISP_DPF_RB_FILTERSIZE_13x9,
> +	CIFISP_DPF_RB_FILTERSIZE_9x9,
> +};
> +
> +/**
> + * enum cifisp_dpf_nll_scale_mode - dpf noise level scale mode
> + * @CIFISP_NLL_SCALE_LINEAR: use a linear scaling
> + * @CIFISP_NLL_SCALE_LOGARITHMIC: use a logarithmic scaling
> + */
> +enum cifisp_dpf_nll_scale_mode {
> +	CIFISP_NLL_SCALE_LINEAR,
> +	CIFISP_NLL_SCALE_LOGARITHMIC,
> +};
> +
> +/**
> + * struct cifisp_dpf_nll - Noise level lookup
> + *
> + * @coeff: Noise level Lookup coefficient
> + * @scale_mode: dpf noise level scale mode (from enum cifisp_dpf_nll_scale_mode)
> + */
> +struct cifisp_dpf_nll {
> +	__u16 coeff[CIFISP_DPF_MAX_NLF_COEFFS];
> +	__u32 scale_mode;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpf_rb_flt - Red blue filter config
> + *
> + * @fltsize: The filter size for the red and blue pixels
> + *	     (from enum cifisp_dpf_rb_filtersize)
> + * @spatial_coeff: Spatial weights
> + * @r_enable: enable filter processing for red pixels
> + * @b_enable: enable filter processing for blue pixels
> + */
> +struct cifisp_dpf_rb_flt {
> +	__u32 fltsize;
> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
> +	__u8 r_enable;
> +	__u8 b_enable;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpf_g_flt - Green filter Configuration
> + *
> + * @spatial_coeff: Spatial weights
> + * @gr_enable: enable filter processing for green pixels in green/red lines
> + * @gb_enable: enable filter processing for green pixels in green/blue lines
> + */
> +struct cifisp_dpf_g_flt {
> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
> +	__u8 gr_enable;
> +	__u8 gb_enable;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpf_gain - Noise function Configuration
> + *
> + * @mode: dpf gain usage  (from enum cifisp_dpf_gain_usage)
> + * @nf_r_gain: Noise function Gain that replaces the AWB gain for red pixels
> + * @nf_b_gain: Noise function Gain that replaces the AWB gain for blue pixels
> + * @nf_gr_gain: Noise function Gain that replaces the AWB gain
> + *		for green pixels in a red line
> + * @nf_gb_gain: Noise function Gain that replaces the AWB gain
> + *		for green pixels in a blue line
> + */
> +struct cifisp_dpf_gain {
> +	__u32 mode;
> +	__u16 nf_r_gain;
> +	__u16 nf_b_gain;
> +	__u16 nf_gr_gain;
> +	__u16 nf_gb_gain;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpf_config - Configuration used by De-noising pre-filter
> + *
> + * @gain: noise function gain
> + * @g_flt: green filter config
> + * @rb_flt: red blue filter config
> + * @nll: noise level lookup
> + */
> +struct cifisp_dpf_config {
> +	struct cifisp_dpf_gain gain;
> +	struct cifisp_dpf_g_flt g_flt;
> +	struct cifisp_dpf_rb_flt rb_flt;
> +	struct cifisp_dpf_nll nll;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_dpf_strength_config - strength of the filter
> + *
> + * @r: filter strength of the RED filter
> + * @g: filter strength of the GREEN filter
> + * @b: filter strength of the BLUE filter
> + */
> +struct cifisp_dpf_strength_config {
> +	__u8 r;
> +	__u8 g;
> +	__u8 b;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_isp_other_cfg - Parameters for some blocks in rockchip isp1
> + *
> + * @dpcc_config: Defect Pixel Cluster Correction config
> + * @bls_config: Black Level Subtraction config
> + * @sdg_config: sensor degamma config
> + * @lsc_config: Lens Shade config
> + * @awb_gain_config: Auto White balance gain config
> + * @flt_config: filter config
> + * @bdm_config: demosaic config
> + * @ctk_config: cross talk config
> + * @goc_config: gamma out config
> + * @bls_config: black level subtraction config
> + * @dpf_config: De-noising pre-filter config
> + * @dpf_strength_config: dpf strength config
> + * @cproc_config: color process config
> + * @ie_config: image effects config
> + */
> +struct cifisp_isp_other_cfg {
> +	struct cifisp_dpcc_config dpcc_config;
> +	struct cifisp_bls_config bls_config;
> +	struct cifisp_sdg_config sdg_config;
> +	struct cifisp_lsc_config lsc_config;
> +	struct cifisp_awb_gain_config awb_gain_config;
> +	struct cifisp_flt_config flt_config;
> +	struct cifisp_bdm_config bdm_config;
> +	struct cifisp_ctk_config ctk_config;
> +	struct cifisp_goc_config goc_config;
> +	struct cifisp_dpf_config dpf_config;
> +	struct cifisp_dpf_strength_config dpf_strength_config;
> +	struct cifisp_cproc_config cproc_config;
> +	struct cifisp_ie_config ie_config;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_isp_meas_cfg - Rockchip ISP1 Measure Parameters
> + *
> + * @awb_meas_config: auto white balance config
> + * @hst_config: histogram config
> + * @aec_config: auto exposure config
> + * @afc_config: auto focus config
> + */
> +struct cifisp_isp_meas_cfg {
> +	struct cifisp_awb_meas_config awb_meas_config;
> +	struct cifisp_hst_config hst_config;
> +	struct cifisp_aec_config aec_config;
> +	struct cifisp_afc_config afc_config;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct rkisp1_isp_params_cfg - Rockchip ISP1 Input Parameters Meta Data
> + *
> + * @module_en_update: mask the enable bits of which module should be updated
> + * @module_ens: mask the enable value of each module, only update the module
> + *		which correspond bit was set in module_en_update
> + * @module_cfg_update: mask the config bits of which module should be updated
> + * @meas: measurement config
> + * @others: other config
> + */
> +struct rkisp1_isp_params_cfg {
> +	__u32 module_en_update;
> +	__u32 module_ens;
> +	__u32 module_cfg_update;
> +
> +	struct cifisp_isp_meas_cfg meas;
> +	struct cifisp_isp_other_cfg others;
> +} __attribute__ ((packed));

As far as I understand most (all ?) of the structures defined above are
not used directly by the hardware, is that right ? In that case, do they
need to be packed ?

Some of the structures have a _cfg suffix, while others have a _config
suffix. Should we standardise on one of the two ?

> +
> +/*---------- PART2: Measurement Statistics ------------*/
> +
> +/**
> + * struct cifisp_bls_meas_val - AWB measured values

This should be cifisp_awb_meas

> + *
> + * @cnt: White pixel count, number of "white pixels" found during laster

s/laster/last/ ?

> + *	 measurement
> + * @mean_y_or_g: Mean value of Y within window and frames,
> + *		 Green if RGB is selected.
> + * @mean_cb_or_b: Mean value of Cb within window and frames,
> + *		  Blue if RGB is selected.
> + * @mean_cr_or_r: Mean value of Cr within window and frames,
> + *		  Red if RGB is selected.
> + */
> +struct cifisp_awb_meas {
> +	__u32 cnt;
> +	__u8 mean_y_or_g;
> +	__u8 mean_cb_or_b;
> +	__u8 mean_cr_or_r;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_awb_stat - statistics automatic white balance data
> + *
> + * @awb_mean: Mean measured data
> + */
> +struct cifisp_awb_stat {
> +	struct cifisp_awb_meas awb_mean[CIFISP_AWB_MAX_GRID];
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_bls_meas_val - BLS measured values
> + *
> + * @meas_r: Mean measured value for Bayer pattern R
> + * @meas_gr: Mean measured value for Bayer pattern Gr
> + * @meas_gb: Mean measured value for Bayer pattern Gb
> + * @meas_b: Mean measured value for Bayer pattern B
> + */
> +struct cifisp_bls_meas_val {
> +	__u16 meas_r;
> +	__u16 meas_gr;
> +	__u16 meas_gb;
> +	__u16 meas_b;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_ae_stat - statistics auto exposure data
> + *
> + * @exp_mean: Mean luminance value of block xx
> + * @bls_val:  BLS measured values
> + *
> + * Image is divided into 5x5 blocks.
> + */
> +struct cifisp_ae_stat {
> +	__u8 exp_mean[CIFISP_AE_MEAN_MAX];
> +	struct cifisp_bls_meas_val bls_val;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_af_meas_val - AF measured values
> + *
> + * @sum: sharpness, refer to REF_01 for definition
> + * @lum: luminance, refer to REF_01 for definition

That will be lovely to use without documentation...

> + */
> +struct cifisp_af_meas_val {
> +	__u32 sum;
> +	__u32 lum;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_af_stat - statistics auto focus data
> + *
> + * @window: AF measured value of window x
> + *
> + * The module measures the sharpness in 3 windows of selectable size via
> + * register settings(ISP_AFM_*_A/B/C)
> + */
> +struct cifisp_af_stat {
> +	struct cifisp_af_meas_val window[CIFISP_AFM_MAX_WINDOWS];
> +} __attribute__ ((packed));
> +
> +/**
> + * struct cifisp_hist_stat - statistics histogram data
> + *
> + * @hist_bins: measured bin counters
> + *
> + * Measurement window divided into 25 sub-windows, set
> + * with ISP_HIST_XXX

What is ISP_HIST_XXX ?

> + */
> +struct cifisp_hist_stat {
> +	__u16 hist_bins[CIFISP_HIST_BIN_N_MAX];
> +} __attribute__ ((packed));
> +
> +/**
> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Data
> + *
> + * @cifisp_awb_stat: statistics data for automatic white balance
> + * @cifisp_ae_stat: statistics data for auto exposure
> + * @cifisp_af_stat: statistics data for auto focus
> + * @cifisp_hist_stat: statistics histogram data
> + */
> +struct cifisp_stat {
> +	struct cifisp_awb_stat awb;
> +	struct cifisp_ae_stat ae;
> +	struct cifisp_af_stat af;
> +	struct cifisp_hist_stat hist;
> +} __attribute__ ((packed));
> +
> +/**
> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Meta Data
> + *
> + * @meas_type: measurement types (CIFISP_STAT_ definitions)
> + * @frame_id: frame ID for sync
> + * @params: statistics data
> + */
> +struct rkisp1_stat_buffer {
> +	__u32 meas_type;
> +	__u32 frame_id;
> +	struct cifisp_stat params;
> +} __attribute__ ((packed));
> +
> +#endif /* _UAPI_RKISP1_CONFIG_H */

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-07 10:39     ` Hans Verkuil
@ 2019-08-15 19:35       ` Laurent Pinchart
  2020-03-25  6:34         ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-15 19:35 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, Allon Huang,
	linux-rockchip, Helen Koike, Jacob Chen, hans.verkuil,
	sakari.ailus, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media

Hi Hans,

On Wed, Aug 07, 2019 at 12:39:17PM +0200, Hans Verkuil wrote:
> On 8/6/19 8:51 PM, Helen Koike wrote:
> > On 7/30/19 3:42 PM, Helen Koike wrote:
> >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> >>
> >> Add the subdev driver for rockchip isp1.
> >>
> >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> >> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> >> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> >> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> >> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> >> [fixed unknown entity type / switched to PIXEL_RATE]
> >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> >> [update for upstream]
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>
> >> ---
> >>
> >> Changes in v8: None
> >> Changes in v7:
> >> - fixed warning because of unknown entity type
> >> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> >> and default values
> >> - fix typo riksp1/rkisp1
> >> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> >> isp subdevice in the media topology now. As a consequence, remove the
> >> hack in mipidphy_g_mbus_config() where information from the sensor was
> >> being propagated through the topology.
> >> - From the old dphy:
> >>         * cache get_remote_sensor() in s_stream
> >>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> >> - Replace stream state with a boolean
> >> - code styling and checkpatch fixes
> >> - fix stop_stream (return after calling stop, do not reenable the stream)
> >> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> >> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> >> - s/intput/input
> >> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> >> reused by the capture
> >>
> >>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
> >>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
> >>  2 files changed, 1397 insertions(+)
> >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
> >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> >>
> >> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> >> new file mode 100644
> >> index 000000000000..6d0c0ffb5e03
> >> --- /dev/null
> >> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> >> @@ -0,0 +1,1286 @@
> 
> <snip>
> 
> >> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> >> +				 struct v4l2_subdev_pad_config *cfg,
> >> +				 struct v4l2_subdev_format *fmt)
> >> +{
> >> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> >> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> >> +
> >> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> >> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> >> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
> >> +		/*
> >> +		 * NOTE: setting a format here doesn't make much sense
> >> +		 * but v4l2-compliance complains
> >> +		 */
> >> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
> >> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> > 
> > As I had mentioned to you, this is called for the isp pads connected to the
> > DMA engines for statistics and parameters (meta data).
> > 
> > If I remove those, I get the following errors:
> > 
> > Sub-Device ioctls (Sink Pad 1):
> >         test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> >         test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
> >         test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> >         test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> >         test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
> >         test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> >         test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
> > 
> > Here is the full log: http://ix.io/1QNt
> > 
> > Is this a bug in v4l2-compliance?
> 
> Yes and no :-)
> 
> Currently v4l2-compliance assumes that only video is transferred over a media bus.
> But that's not the case here, and testing the code field doesn't help v4l2-compliance
> since MEDIA_BUS_FMT_FIXED is also still used by some older subdev drivers for video.
> 
> I think we need a new bus format: MEDIA_BUS_FMT_FIXED_METADATA. Then v4l2-compliance
> can tell it apart from the regular fixed video bus format.

Wouldn't a pad flag that identifies the type of data transmitted by a
pad be a better, backward-compatible option ? This could be useful for
audio as well.

> If I do a 'git grep MEDIA_BUS_FMT_FIXED' then I see that it is also in use by vsp1
> for histogram information, so that should also be converted to use the new FIXED_METADATA
> format, although that might be too late (there might be userspace complications).

Yes, probably not a good idea.

> >> +		fmt->format.field = V4L2_FIELD_NONE;
> >> +		return 0;
> >> +	}
> >> +
> >> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> >> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> >> +		fmt->format = *mf;
> >> +		return 0;
> >> +	}
> >> +
> >> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> >> +		*mf = isp_sd->in_frm;
> >> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> >> +		/* format of source pad */
> >> +		*mf = isp_sd->in_frm;
> >> +		mf->code = isp_sd->out_fmt.mbus_code;
> >> +		/* window size of source pad */
> >> +		mf->width = isp_sd->out_crop.width;
> >> +		mf->height = isp_sd->out_crop.height;
> >> +		mf->quantization = isp_sd->quantization;
> >> +	}
> >> +	mf->field = V4L2_FIELD_NONE;
> >> +
> >> +	return 0;
> >> +}

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-07-30 18:42 ` [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver Helen Koike
  2019-08-06 18:51   ` Helen Koike
  2019-08-08  9:14   ` Sakari Ailus
@ 2019-08-16  0:13   ` Laurent Pinchart
  2020-07-11 11:04     ` Dafna Hirschfeld
  2 siblings, 1 reply; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-16  0:13 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, sakari.ailus, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hello Helen,

Thank you for the patch.

On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add the subdev driver for rockchip isp1.
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> [fixed unknown entity type / switched to PIXEL_RATE]
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - fixed warning because of unknown entity type
> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> and default values
> - fix typo riksp1/rkisp1
> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> isp subdevice in the media topology now. As a consequence, remove the
> hack in mipidphy_g_mbus_config() where information from the sensor was
> being propagated through the topology.
> - From the old dphy:
>         * cache get_remote_sensor() in s_stream
>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> - Replace stream state with a boolean
> - code styling and checkpatch fixes
> - fix stop_stream (return after calling stop, do not reenable the stream)
> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> - s/intput/input
> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> reused by the capture
> 
>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>  2 files changed, 1397 insertions(+)
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> 
> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> new file mode 100644
> index 000000000000..6d0c0ffb5e03
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> @@ -0,0 +1,1286 @@
> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +/*
> + * Rockchip isp1 driver

Shouldn't each file describe what it contains ? Maybe

 * Rockchip ISP1 Driver - ISP Core

for this one ? Same for other .c or .h files.

> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#include <linux/iopoll.h>
> +#include <linux/phy/phy.h>
> +#include <linux/phy/phy-mipi-dphy.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/videodev2.h>
> +#include <linux/vmalloc.h>
> +#include <media/v4l2-event.h>
> +
> +#include "common.h"
> +#include "regs.h"

common.h and regs.h aren't available yet. This won't break bisection as
this file isn't referenced from the makefile yet, but it makes it a bit
annoying when reviewing patches in order :-S

> +
> +#define CIF_ISP_INPUT_W_MAX		4032
> +#define CIF_ISP_INPUT_H_MAX		3024
> +#define CIF_ISP_INPUT_W_MIN		32
> +#define CIF_ISP_INPUT_H_MIN		32
> +#define CIF_ISP_OUTPUT_W_MAX		CIF_ISP_INPUT_W_MAX
> +#define CIF_ISP_OUTPUT_H_MAX		CIF_ISP_INPUT_H_MAX
> +#define CIF_ISP_OUTPUT_W_MIN		CIF_ISP_INPUT_W_MIN
> +#define CIF_ISP_OUTPUT_H_MIN		CIF_ISP_INPUT_H_MIN
> +
> +/*
> + * NOTE: MIPI controller and input MUX are also configured in this file,
> + * because ISP Subdev is not only describe ISP submodule(input size,format,
> + * output size, format), but also a virtual route device.
> + */
> +
> +/*
> + * There are many variables named with format/frame in below code,
> + * please see here for their meaning.
> + *
> + * Cropping regions of ISP
> + *
> + * +---------------------------------------------------------+
> + * | Sensor image                                            |
> + * | +---------------------------------------------------+   |
> + * | | ISP_ACQ (for black level)                         |   |
> + * | | in_frm                                            |   |
> + * | | +--------------------------------------------+    |   |
> + * | | |    ISP_OUT                                 |    |   |
> + * | | |    in_crop                                 |    |   |

in_crop at the ISP output ? That seems a bit weird. I'm guessing that
this is really the ISP output, while ISP_IS is related to the resizer ?

> + * | | |    +---------------------------------+     |    |   |
> + * | | |    |   ISP_IS                        |     |    |   |
> + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
> + * | | |    +---------------------------------+     |    |   |
> + * | | +--------------------------------------------+    |   |
> + * | +---------------------------------------------------+   |
> + * +---------------------------------------------------------+
> + */
> +
> +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
> +}
> +
> +/* Get sensor by enabled media link */
> +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
> +{
> +	struct media_pad *local, *remote;
> +	struct media_entity *sensor_me;
> +
> +	local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
> +	remote = media_entity_remote_pad(local);
> +	if (!remote) {
> +		v4l2_warn(sd, "No link between isp and sensor\n");
> +		return NULL;
> +	}
> +
> +	sensor_me = media_entity_remote_pad(local)->entity;
> +	return media_entity_to_v4l2_subdev(sensor_me);
> +}
> +
> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
> +					       struct v4l2_subdev *sd)
> +{
> +	struct rkisp1_sensor *sensor;
> +
> +	list_for_each_entry(sensor, &dev->sensors, list)
> +		if (sensor->sd == sd)
> +			return sensor;
> +
> +	return NULL;
> +}
> +
> +/****************  register operations ****************/
> +
> +/*
> + * Image Stabilization.
> + * This should only be called when configuring CIF
> + * or at the frame end interrupt
> + */
> +static void rkisp1_config_ism(struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
> +	u32 val;
> +
> +	writel(0, base + CIF_ISP_IS_RECENTER);

How about read/write wrappers that take a rkisp1_device pointer, a
register offset and a value (for the write wrapper) and compute
dev->base_addr + offset internally ? That would make the code easier to
read.

> +	writel(0, base + CIF_ISP_IS_MAX_DX);
> +	writel(0, base + CIF_ISP_IS_MAX_DY);
> +	writel(0, base + CIF_ISP_IS_DISPLACE);
> +	writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
> +	writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
> +	writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
> +	writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
> +
> +	/* IS(Image Stabilization) is always on, working as output crop */
> +	writel(1, base + CIF_ISP_IS_CTRL);
> +	val = readl(base + CIF_ISP_CTRL);
> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD;
> +	writel(val, base + CIF_ISP_CTRL);
> +}
> +
> +/*
> + * configure isp blocks with input format, size......
> + */
> +static int rkisp1_config_isp(struct rkisp1_device *dev)
> +{
> +	u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
> +	struct v4l2_rect *out_crop, *in_crop;
> +	void __iomem *base = dev->base_addr;
> +	struct v4l2_mbus_framefmt *in_frm;
> +	struct ispsd_out_fmt *out_fmt;
> +	struct rkisp1_sensor *sensor;
> +	struct ispsd_in_fmt *in_fmt;
> +
> +	sensor = dev->active_sensor;
> +	in_frm = &dev->isp_sdev.in_frm;
> +	in_fmt = &dev->isp_sdev.in_fmt;
> +	out_fmt = &dev->isp_sdev.out_fmt;
> +	out_crop = &dev->isp_sdev.out_crop;
> +	in_crop = &dev->isp_sdev.in_crop;
> +
> +	if (in_fmt->fmt_type == FMT_BAYER) {
> +		acq_mult = 1;
> +		if (out_fmt->fmt_type == FMT_BAYER) {
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl =
> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
> +			else
> +				isp_ctrl =
> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
> +		} else {
> +			writel(CIF_ISP_DEMOSAIC_TH(0xc),
> +			       base + CIF_ISP_DEMOSAIC);
> +
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
> +			else
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
> +		}
> +	} else if (in_fmt->fmt_type == FMT_YUV) {
> +		acq_mult = 2;
> +		if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +			isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> +		} else {
> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
> +			else
> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> +
> +		}
> +
> +		irq_mask |= CIF_ISP_DATA_LOSS;
> +	}
> +
> +	/* Set up input acquisition properties */
> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
> +			signal = CIF_ISP_ACQ_PROP_POS_EDGE;
> +	}
> +
> +	if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
> +			signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
> +
> +		if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
> +			signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
> +	}
> +
> +	writel(isp_ctrl, base + CIF_ISP_CTRL);
> +	writel(signal | in_fmt->yuv_seq |
> +	       CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
> +	       CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
> +	writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
> +
> +	/* Acquisition Size */
> +	writel(0, base + CIF_ISP_ACQ_H_OFFS);
> +	writel(0, base + CIF_ISP_ACQ_V_OFFS);
> +	writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
> +	writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
> +
> +	/* ISP Out Area */
> +	writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
> +	writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
> +	writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
> +	writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
> +
> +	/* interrupt mask */
> +	irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
> +		    CIF_ISP_FRAME_IN;
> +	writel(irq_mask, base + CIF_ISP_IMSC);
> +
> +	if (out_fmt->fmt_type == FMT_BAYER)
> +		rkisp1_params_disable_isp(&dev->params_vdev);
> +	else
> +		rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
> +					    dev->isp_sdev.quantization);
> +
> +	return 0;
> +}
> +
> +static int rkisp1_config_dvp(struct rkisp1_device *dev)
> +{
> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> +	void __iomem *base = dev->base_addr;
> +	u32 val, input_sel;
> +
> +	switch (in_fmt->bus_width) {
> +	case 8:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
> +		break;
> +	case 10:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
> +		break;
> +	case 12:
> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
> +		break;
> +	default:
> +		v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
> +		return -EINVAL;
> +	}
> +
> +	val = readl(base + CIF_ISP_ACQ_PROP);
> +	writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
> +
> +	return 0;
> +}
> +
> +static int rkisp1_config_mipi(struct rkisp1_device *dev)
> +{
> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	void __iomem *base = dev->base_addr;
> +	unsigned int lanes;
> +	u32 mipi_ctrl;
> +
> +	/*
> +	 * sensor->mbus is set in isp or d-phy notifier_bound function
> +	 */
> +	switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
> +	case V4L2_MBUS_CSI2_4_LANE:
> +		lanes = 4;
> +		break;
> +	case V4L2_MBUS_CSI2_3_LANE:
> +		lanes = 3;
> +		break;
> +	case V4L2_MBUS_CSI2_2_LANE:
> +		lanes = 2;
> +		break;
> +	case V4L2_MBUS_CSI2_1_LANE:
> +		lanes = 1;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
> +		    CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
> +		    CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
> +		    CIF_MIPI_CTRL_CLOCKLANE_ENA;
> +
> +	writel(mipi_ctrl, base + CIF_MIPI_CTRL);
> +
> +	/* Configure Data Type and Virtual Channel */
> +	writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
> +	       base + CIF_MIPI_IMG_DATA_SEL);
> +
> +	/* Clear MIPI interrupts */
> +	writel(~0, base + CIF_MIPI_ICR);
> +	/*
> +	 * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
> +	 * isp bus may be dead when switch isp.
> +	 */
> +	writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
> +	       CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
> +	       base + CIF_MIPI_IMSC);
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
> +		 "  MIPI_IMG_DATA_SEL 0x%08x\n"
> +		 "  MIPI_STATUS 0x%08x\n"
> +		 "  MIPI_IMSC 0x%08x\n",
> +		 readl(base + CIF_MIPI_CTRL),
> +		 readl(base + CIF_MIPI_IMG_DATA_SEL),
> +		 readl(base + CIF_MIPI_STATUS),
> +		 readl(base + CIF_MIPI_IMSC));
> +
> +	return 0;
> +}
> +
> +/* Configure MUX */
> +static int rkisp1_config_path(struct rkisp1_device *dev)
> +{
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
> +	int ret = 0;
> +
> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> +		ret = rkisp1_config_dvp(dev);
> +		dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
> +	} else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +		ret = rkisp1_config_mipi(dev);
> +		dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
> +	}
> +
> +	writel(dpcl, dev->base_addr + CIF_VI_DPCL);
> +
> +	return ret;
> +}
> +
> +/* Hareware configure Entry */

s/Hareware/Hardware/

> +static int rkisp1_config_cif(struct rkisp1_device *dev)
> +{
> +	u32 cif_id;
> +	int ret;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	cif_id = readl(dev->base_addr + CIF_VI_ID);
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
> +
> +	ret = rkisp1_config_isp(dev);
> +	if (ret < 0)
> +		return ret;
> +	ret = rkisp1_config_path(dev);
> +	if (ret < 0)
> +		return ret;
> +	rkisp1_config_ism(dev);
> +
> +	return 0;
> +}
> +
> +/* Mess register operations to stop isp */

Is it such a mess ? :-)

I would capitalise ISP in all comments.

> +static int rkisp1_isp_stop(struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	/*
> +	 * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
> +	 * Stop ISP(isp) ->wait for ISP isp off
> +	 */
> +	/* stop and clear MI, MIPI, and ISP interrupts */
> +	writel(0, base + CIF_MIPI_IMSC);
> +	writel(~0, base + CIF_MIPI_ICR);
> +
> +	writel(0, base + CIF_ISP_IMSC);
> +	writel(~0, base + CIF_ISP_ICR);
> +
> +	writel(0, base + CIF_MI_IMSC);
> +	writel(~0, base + CIF_MI_ICR);
> +	val = readl(base + CIF_MIPI_CTRL);
> +	writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
> +	/* stop ISP */
> +	val = readl(base + CIF_ISP_CTRL);
> +	val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
> +	writel(val, base + CIF_ISP_CTRL);
> +
> +	val = readl(base + CIF_ISP_CTRL);
> +	writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
> +
> +	readx_poll_timeout(readl, base + CIF_ISP_RIS,
> +			   val, val & CIF_ISP_OFF, 20, 100);
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		"streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming,
> +		 readl(base + CIF_MI_CTRL),
> +		 readl(base + CIF_ISP_CTRL),
> +		 readl(base + CIF_MIPI_CTRL));
> +
> +	writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
> +	writel(0x0, base + CIF_IRCL);
> +
> +	return 0;
> +}
> +
> +/* Mess register operations to start isp */
> +static int rkisp1_isp_start(struct rkisp1_device *dev)
> +{
> +	struct rkisp1_sensor *sensor = dev->active_sensor;
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming);
> +
> +	/* Activate MIPI */
> +	if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> +		val = readl(base + CIF_MIPI_CTRL);
> +		writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
> +	}
> +	/* Activate ISP */
> +	val = readl(base + CIF_ISP_CTRL);
> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
> +	       CIF_ISP_CTRL_ISP_INFORM_ENABLE;
> +	writel(val, base + CIF_ISP_CTRL);
> +
> +	/* XXX: Is the 1000us too long?
> +	 * CIF spec says to wait for sufficient time after enabling
> +	 * the MIPI interface and before starting the sensor output.
> +	 */
> +	usleep_range(1000, 1200);
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
> +		 "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
> +		 dev->stream[RKISP1_STREAM_SP].streaming,
> +		 dev->stream[RKISP1_STREAM_MP].streaming,
> +		 readl(base + CIF_MI_CTRL),
> +		 readl(base + CIF_ISP_CTRL),
> +		 readl(base + CIF_MIPI_CTRL));
> +
> +	return 0;
> +}
> +
> +static void rkisp1_config_clk(struct rkisp1_device *dev)
> +{
> +	u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
> +		  CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
> +		  CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
> +
> +	writel(val, dev->base_addr + CIF_ICCL);
> +}
> +
> +/***************************** isp sub-devs *******************************/
> +
> +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
> +	{
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 10,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,

Is there a reason why the 10-bit Bayer format start with BGGR while the
12-bit formats start with RGGB ? Not a big deal, just OCD kicking in :-)

> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 12,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_RGGB,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_BGGR,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_GBRG,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
> +		.bayer_pat	= RAW_GRBG,
> +		.bus_width	= 8,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCBYCR,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_YVYU8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCRYCB,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_UYVY8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CBYCRY,
> +		.bus_width	= 16,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_VYUY8_1X16,
> +		.fmt_type	= FMT_YUV,
> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CRYCBY,
> +		.bus_width	= 16,
> +	},
> +};
> +
> +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
> +	{
> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_2X8,
> +		.fmt_type	= FMT_YUV,
> +	}, {

This is the only entry not present in the previous table, so I'm
wondering if it would make sense to merge the two tables and rename
ispsd_in_fmt to rkisp1_format_info. You would need to add a field that
tells, for each format, if it's valid as an input format, and output
format, or both. Hmmmm and also make the enum logic a bit more complex.
Maybe it's not worth it after all, but it bothers me a bit to have two
tables :-) I'll let you decide what's best.

> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	}, {
> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
> +		.fmt_type	= FMT_BAYER,
> +	},
> +};
> +
> +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
> +{
> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
> +	const struct ispsd_in_fmt *fmt;

You can move this variable inside the loop.

> +
> +	for (i = 0; i < array_size; i++) {

I would remove the array_size local variable, it doesn't improve
readability. Same for the next function.

> +		fmt = &rkisp1_isp_input_formats[i];
> +		if (fmt->mbus_code == mbus_code)
> +			return fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
> +{
> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
> +	const struct ispsd_out_fmt *fmt;
> +
> +	for (i = 0; i < array_size; i++) {
> +		fmt = &rkisp1_isp_output_formats[i];
> +		if (fmt->mbus_code == mbus_code)
> +			return fmt;
> +	}
> +
> +	return NULL;
> +}
> +
> +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
> +					struct v4l2_subdev_pad_config *cfg,
> +					struct v4l2_subdev_mbus_code_enum *code)
> +{
> +	unsigned int i = code->index;
> +
> +	if ((code->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> +		if (i > 0)
> +			return -EINVAL;
> +		code->code = MEDIA_BUS_FMT_FIXED;
> +		return 0;
> +	}
> +
> +	if (code->pad == RKISP1_ISP_PAD_SINK) {
> +		if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
> +			return -EINVAL;
> +		code->code = rkisp1_isp_input_formats[i].mbus_code;
> +	} else {
> +		if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
> +			return -EINVAL;
> +		code->code = rkisp1_isp_output_formats[i].mbus_code;
> +	}

On the other hand, merging the two tables above into one, you could
merge the two branches here and only consider formats indicated as valid
for to the pad you want. Maybe the code would be cleaner in the end.

> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
> +				     struct v4l2_subdev_pad_config *cfg)
> +{
> +	struct v4l2_rect *mf_in_crop, *mf_out_crop;
> +	struct v4l2_mbus_framefmt *mf_in, *mf_out;
> +
> +	mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
> +	mf_in->width = RKISP1_DEFAULT_WIDTH;
> +	mf_in->height = RKISP1_DEFAULT_HEIGHT;
> +	mf_in->field = V4L2_FIELD_NONE;
> +	mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
> +
> +	mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
> +	mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
> +	mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
> +	mf_in_crop->left = 0;
> +	mf_in_crop->top = 0;
> +
> +	mf_out = v4l2_subdev_get_try_format(sd, cfg,
> +					    RKISP1_ISP_PAD_SOURCE_PATH);
> +	*mf_out = *mf_in;
> +	mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
> +	mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +
> +	mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
> +					       RKISP1_ISP_PAD_SOURCE_PATH);
> +	*mf_out_crop = *mf_in_crop;
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_format *fmt)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> +
> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
> +		/*
> +		 * NOTE: setting a format here doesn't make much sense
> +		 * but v4l2-compliance complains
> +		 */

For the params pad I agreed it makes no sense, and I think
v4l2-compliance is at fault, so I'd set width and height to 0. For the
stats pad we *could* use the size of the image from which stats are
computed, but because v4l2_meta_format has no width/height, I think 0
would also be appropriate.

> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> +		fmt->format.field = V4L2_FIELD_NONE;
> +		return 0;
> +	}
> +
> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> +		fmt->format = *mf;
> +		return 0;
> +	}
> +
> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> +		*mf = isp_sd->in_frm;
> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		/* format of source pad */
> +		*mf = isp_sd->in_frm;
> +		mf->code = isp_sd->out_fmt.mbus_code;
> +		/* window size of source pad */
> +		mf->width = isp_sd->out_crop.width;
> +		mf->height = isp_sd->out_crop.height;
> +		mf->quantization = isp_sd->quantization;
> +	}
> +	mf->field = V4L2_FIELD_NONE;

This can be simplified, please read through, or jump to the review of
rkisp1_isp_subdev at the end.

> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
> +				  unsigned int pad,
> +				  struct v4l2_mbus_framefmt *fmt)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> +	const struct ispsd_out_fmt *out_fmt;
> +	const struct ispsd_in_fmt *in_fmt;
> +
> +	switch (pad) {
> +	case RKISP1_ISP_PAD_SINK:
> +		in_fmt = find_in_fmt(fmt->code);
> +		if (in_fmt)
> +			fmt->code = in_fmt->mbus_code;
> +		else
> +			fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;

You write MEDIA_BUS_FMT_SRGGB10_1X10 explicitly here, while you use
rkisp1_isp_output_formats[0].mbus_code below (and in other places). I
would standardise on one of the two (explicit formats or array[0]), with
a preference for the first as that would allow merging the input and
output arrays more easily. I would then create two #define,
RKISP1_DEF_INPUT_FORMAT and RKISP2_DEF_OUTPUT_FORMAT (or similar).
Similar macros for the default width and height could also be useful, to
make it easier to change them.

> +		fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
> +				      CIF_ISP_INPUT_W_MAX);
> +		fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
> +				      CIF_ISP_INPUT_H_MAX);
> +		break;
> +	case RKISP1_ISP_PAD_SOURCE_PATH:
> +		out_fmt = find_out_fmt(fmt->code);
> +		if (out_fmt)
> +			fmt->code = out_fmt->mbus_code;
> +		else
> +			fmt->code = rkisp1_isp_output_formats[0].mbus_code;
> +		/* window size is set in s_selection */
> +		fmt->width  = isp_sd->out_crop.width;
> +		fmt->height = isp_sd->out_crop.height;

This function operates on the TRY configuration too, in which case you
should use the TRY crop rectangle here, not the ACTIVE one. If you've
already jumped to the review of rkisp1_isp_subdev you know my proposal
to simplify this. Otherwise now may be a good time to do so :-)

> +		/* full range by default */
> +		if (!fmt->quantization)
> +			fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +		break;
> +	}
> +
> +	fmt->field = V4L2_FIELD_NONE;
> +}
> +
> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_pad_config *cfg,
> +				 struct v4l2_subdev_format *fmt)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> +
> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
> +		return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
> +
> +	rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
> +
> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		struct v4l2_mbus_framefmt *try_mf;
> +
> +		try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> +		*try_mf = *mf;
> +		return 0;

When setting the format on the sink pad the crop rectangles need to be
reset (here, and for the ACTIVE format below too).

> +	}
> +
> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> +		const struct ispsd_in_fmt *in_fmt;
> +
> +		in_fmt = find_in_fmt(mf->code);
> +		isp_sd->in_fmt = *in_fmt;
> +		isp_sd->in_frm = *mf;
> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		const struct ispsd_out_fmt *out_fmt;
> +
> +		/* Ignore width/height */
> +		out_fmt = find_out_fmt(mf->code);
> +		isp_sd->out_fmt = *out_fmt;

I would return the in_fmt and out_fmt from rkisp1_isp_sd_try_fmt() as it
already looks them up. If you merge the input and output tables, you'll
have a single format info structure type, and rkisp1_isp_sd_try_fmt()
could return the entry for the pad it operates on.

> +		/*
> +		 * It is quantization for output,
> +		 * isp use bt601 limit-range in internal
> +		 */
> +		isp_sd->quantization = mf->quantization;
> +	}
> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_pad_config *cfg,
> +				  struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
> +	struct v4l2_rect in_crop = isp_sd->in_crop;
> +	struct v4l2_rect *input = &sel->r;
> +
> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		in_frm = *v4l2_subdev_get_try_format(sd, cfg,
> +						     RKISP1_ISP_PAD_SINK);
> +		in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
> +						    RKISP1_ISP_PAD_SINK);
> +	}
> +
> +	input->left = ALIGN(input->left, 2);
> +	input->width = ALIGN(input->width, 2);
> +
> +	if (sel->pad == RKISP1_ISP_PAD_SINK) {
> +		input->left = clamp_t(u32, input->left, 0, in_frm.width);
> +		input->top = clamp_t(u32, input->top, 0, in_frm.height);
> +		input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
> +				       in_frm.width - input->left);
> +		input->height = clamp_t(u32, input->height,
> +					CIF_ISP_INPUT_H_MIN,
> +					in_frm.height - input->top);
> +	} else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> +		input->left = clamp_t(u32, input->left, 0, in_crop.width);
> +		input->top = clamp_t(u32, input->top, 0, in_crop.height);
> +		input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
> +				       in_crop.width - input->left);
> +		input->height = clamp_t(u32, input->height,
> +					CIF_ISP_OUTPUT_H_MIN,
> +					in_crop.height - input->top);
> +	}
> +}
> +
> +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_pad_config *cfg,
> +				       struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct v4l2_mbus_framefmt *frm;
> +	struct v4l2_rect *rect;
> +
> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> +	    sel->pad != RKISP1_ISP_PAD_SINK)
> +		return -EINVAL;
> +
> +	switch (sel->target) {
> +	case V4L2_SEL_TGT_CROP_BOUNDS:
> +		if (sel->pad == RKISP1_ISP_PAD_SINK) {
> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +				frm = v4l2_subdev_get_try_format(sd, cfg,
> +								 sel->pad);
> +			else
> +				frm = &isp_sd->in_frm;
> +
> +			sel->r.height = frm->height;
> +			sel->r.width = frm->width;
> +			sel->r.left = 0;
> +			sel->r.top = 0;
> +		} else {
> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +				rect = v4l2_subdev_get_try_crop(sd, cfg,
> +							RKISP1_ISP_PAD_SINK);
> +			else
> +				rect = &isp_sd->in_crop;
> +			sel->r = *rect;
> +		}
> +		break;
> +	case V4L2_SEL_TGT_CROP:
> +		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> +			rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> +		else if (sel->pad == RKISP1_ISP_PAD_SINK)
> +			rect = &isp_sd->in_crop;
> +		else
> +			rect = &isp_sd->out_crop;
> +		sel->r = *rect;
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
> +				       struct v4l2_subdev_pad_config *cfg,
> +				       struct v4l2_subdev_selection *sel)
> +{
> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> +	struct rkisp1_device *dev = sd_to_isp_dev(sd);
> +
> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> +	    sel->pad != RKISP1_ISP_PAD_SINK)
> +		return -EINVAL;
> +	if (sel->target != V4L2_SEL_TGT_CROP)
> +		return -EINVAL;
> +
> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> +		 "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
> +		 sel->r.left, sel->r.top, sel->r.width, sel->r.height);
> +	rkisp1_isp_sd_try_crop(sd, cfg, sel);
> +
> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> +		struct v4l2_rect *try_sel =
> +			v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> +
> +		*try_sel = sel->r;
> +		return 0;

When setting the crop rectangle on the sink pad the rectangle on the
source pad should be reset (and below for the ACTIVE format too).

The resizer logic, through selection rectangles, seems to be missing. I
assume it's configured through the video nodes, but that's not correct
I'm afraid, scaling should be configured on subdevs, see [1]. I'm afraid
this means that we'll need separate subdevs for the resizers :-S

[1] https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html#order-of-configuration-and-format-propagation

> +	}
> +
> +	if (sel->pad == RKISP1_ISP_PAD_SINK)
> +		isp_sd->in_crop = sel->r;
> +	else
> +		isp_sd->out_crop = sel->r;
> +
> +	return 0;
> +}
> +

I'll skip the part related to power on/off below as Sakari requested it
to be handled through runtime PM.

> +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
> +				    struct rkisp1_sensor *sensor)
> +{
> +	union phy_configure_opts opts = { 0 };
> +	struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
> +	struct v4l2_ctrl *pixel_rate;
> +	s64 pixel_clock;
> +
> +	pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
> +				    V4L2_CID_PIXEL_RATE);
> +	if (!pixel_rate) {
> +		v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
> +		return -EPIPE;
> +	}

Would it make sense to retrieve (and cache) the control pointer (not its
value of course) at probe time already ?

> +
> +	pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
> +	if (!pixel_clock) {
> +		v4l2_err(sensor->sd, "Invalid pixel rate value\n");
> +		return -EINVAL;
> +	}
> +
> +	phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
> +					 sensor->lanes, cfg);
> +	phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
> +	phy_configure(sensor->dphy, &opts);
> +	phy_power_on(sensor->dphy);
> +
> +	return 0;
> +}
> +
> +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
> +{
> +	phy_power_off(sensor->dphy);
> +}
> +
> +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	struct v4l2_subdev *sensor_sd;
> +	int ret = 0;
> +
> +	if (!on) {
> +		ret = rkisp1_isp_stop(isp_dev);
> +		if (ret < 0)
> +			return ret;
> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> +		return 0;
> +	}
> +
> +	sensor_sd = get_remote_sensor(sd);
> +	if (!sensor_sd)
> +		return -ENODEV;
> +
> +	isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
> +	if (!isp_dev->active_sensor)
> +		return -ENODEV;
> +
> +	atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
> +	ret = rkisp1_config_cif(isp_dev);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* TODO: support other interfaces */
> +	if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
> +		return -EINVAL;
> +
> +	ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
> +				       isp_dev->active_sensor);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = rkisp1_isp_start(isp_dev);
> +	if (ret)
> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> +
> +	return ret;
> +}
> +
> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> +{
> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> +	int ret;
> +
> +	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> +
> +	if (on) {
> +		ret = pm_runtime_get_sync(isp_dev->dev);
> +		if (ret < 0)
> +			return ret;
> +
> +		rkisp1_config_clk(isp_dev);
> +	} else {
> +		ret = pm_runtime_put(isp_dev->dev);
> +		if (ret < 0)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int rkisp1_subdev_link_validate(struct media_link *link)
> +{
> +	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
> +		return 0;
> +
> +	return v4l2_subdev_link_validate(link);
> +}
> +
> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
> +					struct media_link *link,
> +					struct v4l2_subdev_format *source_fmt,
> +					struct v4l2_subdev_format *sink_fmt)
> +{
> +	if (source_fmt->format.code != sink_fmt->format.code)
> +		return -EINVAL;
> +
> +	/* Crop is available */
> +	if (source_fmt->format.width < sink_fmt->format.width ||
> +	    source_fmt->format.height < sink_fmt->format.height)
> +		return -EINVAL;

Crop should be handled on pads, not on links. The size at the ends of
the link should match. This will likely require a bit of a redesign of
the format and selection rectangles handling, but I think it's also an
opportunity to simplify the code.

> +
> +	return 0;
> +}
> +
> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> +{
> +	struct v4l2_event event = {
> +		.type = V4L2_EVENT_FRAME_SYNC,
> +		.u.frame_sync.frame_sequence =
> +			atomic_inc_return(&isp->frm_sync_seq) - 1,

I would move the increment to the caller, hiding it in this function is
error-prone (and if you look at the caller I'm pointing out one possible
error :-)).

In general usage of frm_sync_seq through the driver seems to be very
race-prone. It's read in various IRQ handling functions, all coming from
the same IRQ, so that part is fine (and wouldn't require an atomic
variable), but when read from the buffer queue handlers I really get a
red light flashing in my head. I'll try to investigate more when
reviewing the next patches.

> +	};
> +	v4l2_event_queue(isp->sd.devnode, &event);
> +}
> +
> +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> +				  struct v4l2_event_subscription *sub)
> +{
> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
> +		return -EINVAL;
> +
> +	/* Line number. For now only zero accepted. */

Is the id a line number for V4L2_EVENT_FRAME_SYNC ? It's not mentioned
for V4L2_EVENT_FRAME_SYNC in the V4L2 spec, so I think this check is
correct, but the comment doesn't seem to be.

> +	if (sub->id != 0)
> +		return -EINVAL;
> +
> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
> +}
> +
> +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
> +	.enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
> +	.get_selection = rkisp1_isp_sd_get_selection,
> +	.set_selection = rkisp1_isp_sd_set_selection,
> +	.init_cfg = rkisp1_isp_sd_init_config,
> +	.get_fmt = rkisp1_isp_sd_get_fmt,
> +	.set_fmt = rkisp1_isp_sd_set_fmt,
> +	.link_validate = rkisp1_subdev_fmt_link_validate,
> +};
> +
> +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
> +	.link_validate = rkisp1_subdev_link_validate,
> +};
> +
> +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
> +	.s_stream = rkisp1_isp_sd_s_stream,
> +};
> +
> +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
> +	.subscribe_event = rkisp1_isp_sd_subs_evt,
> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
> +	.s_power = rkisp1_isp_sd_s_power,
> +};
> +
> +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {

static const

> +	.core = &rkisp1_isp_core_ops,
> +	.video = &rkisp1_isp_sd_video_ops,
> +	.pad = &rkisp1_isp_sd_pad_ops,
> +};
> +
> +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
> +{
> +	struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
> +	struct v4l2_rect *in_crop = &isp_sd->in_crop;
> +	struct v4l2_rect *out_crop = &isp_sd->out_crop;
> +	struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
> +	struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
> +
> +	*in_fmt = rkisp1_isp_input_formats[0];
> +	in_frm->width = RKISP1_DEFAULT_WIDTH;
> +	in_frm->height = RKISP1_DEFAULT_HEIGHT;
> +	in_frm->code = in_fmt->mbus_code;
> +
> +	in_crop->width = in_frm->width;
> +	in_crop->height = in_frm->height;
> +	in_crop->left = 0;
> +	in_crop->top = 0;
> +
> +	/* propagate to source */
> +	*out_crop = *in_crop;
> +	*out_fmt = rkisp1_isp_output_formats[0];
> +	isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;

I wonder, could this be implemented by sharing code with .init_cfg() if
you store the active configuration in v4l2_subdev_pad_config instances
(please see the comments about the rkisp1_isp_subdev structure below) ?

> +}
> +
> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> +			       struct v4l2_device *v4l2_dev)
> +{
> +	struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
> +	struct v4l2_subdev *sd = &isp_sdev->sd;
> +	int ret;
> +
> +	v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> +	sd->entity.ops = &rkisp1_isp_sd_media_ops;
> +	snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
> +
> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> +	ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
> +				     isp_sdev->pads);
> +	if (ret < 0)
> +		return ret;
> +
> +	sd->owner = THIS_MODULE;
> +	v4l2_set_subdevdata(sd, isp_dev);
> +
> +	sd->grp_id = GRP_ID_ISP;

I think you can drop this, as well as all the GRP_ID_* macros, they're
not used.

> +	ret = v4l2_device_register_subdev(v4l2_dev, sd);
> +	if (ret < 0) {
> +		v4l2_err(sd, "Failed to register isp subdev\n");
> +		goto err_cleanup_media_entity;
> +	}
> +
> +	rkisp1_isp_sd_init_default_fmt(isp_sdev);
> +
> +	return 0;
> +err_cleanup_media_entity:
> +	media_entity_cleanup(&sd->entity);
> +	return ret;
> +}
> +
> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
> +{
> +	struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
> +
> +	v4l2_device_unregister_subdev(sd);
> +	media_entity_cleanup(&sd->entity);
> +}
> +
> +/****************  Interrupter Handler ****************/

s/Handler/Handlers/

> +
> +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)

I would make the mis field an u32 as it contains register values. Should
it also be renamed to status ? Same for rkisp1_isp_isr() below.

> +{
> +	struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
> +	void __iomem *base = dev->base_addr;
> +	u32 val;
> +

How about moving the CIF_MIPI_MIS read here and removing the mis
argument to the function ? It would be more logical to group all
register access related to interrupts in a single place. Same for the
other interrupt handler functions.

> +	writel(~0, base + CIF_MIPI_ICR);
> +
> +	/*
> +	 * Disable DPHY errctrl interrupt, because this dphy
> +	 * erctrl signal is asserted until the next changes
> +	 * of line state. This time is may be too long and cpu
> +	 * is hold in this interrupt.
> +	 */
> +	if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
> +		val = readl(base + CIF_MIPI_IMSC);
> +		writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
> +		dev->isp_sdev.dphy_errctrl_disabled = true;
> +	}
> +
> +	/*
> +	 * Enable DPHY errctrl interrupt again, if mipi have receive
> +	 * the whole frame without any error.
> +	 */
> +	if (mis == CIF_MIPI_FRAME_END) {
> +		/*
> +		 * Enable DPHY errctrl interrupt again, if mipi have receive
> +		 * the whole frame without any error.
> +		 */
> +		if (dev->isp_sdev.dphy_errctrl_disabled) {
> +			val = readl(base + CIF_MIPI_IMSC);
> +			val |= CIF_MIPI_ERR_CTRL(0x0f);
> +			writel(val, base + CIF_MIPI_IMSC);
> +			dev->isp_sdev.dphy_errctrl_disabled = false;
> +		}
> +	} else {
> +		v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
> +	}
> +}
> +
> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> +{
> +	void __iomem *base = dev->base_addr;
> +	unsigned int isp_mis_tmp = 0;

_tmp are never good names :-S

> +	unsigned int isp_err = 0;

Neither of these variable need to be initialised to 0.

> +
> +	/* start edge of v_sync */
> +	if (isp_mis & CIF_ISP_V_START) {
> +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);

This will increment the frame sequence number. What if the interrupt is
slightly delayed and the next frame starts before we get a change to
copy the sequence number to the buffers (before they will complete
below) ?

> +
> +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);

Do you need to clear all interrupt bits individually, can't you write
isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
all in one go ?

> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_V_START)
> +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> +				 isp_mis_tmp);

This require some explanation. It looks like a naive way to protect
against something, but I think it could trigger under normal
circumstances if IRQ handling is delayed, and wouldn't do much anyway.
Same for the similar constructs below.

> +	}
> +
> +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> +		/* Clear pic_size_error */
> +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> +		isp_err = readl(base + CIF_ISP_ERR);
> +		v4l2_err(&dev->v4l2_dev,
> +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);

What does this mean ?

> +		writel(isp_err, base + CIF_ISP_ERR_CLR);
> +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {

Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?

> +		/* Clear data_loss */
> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> +	}
> +
> +	/* sampled input frame is complete */
> +	if (isp_mis & CIF_ISP_FRAME_IN) {
> +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> +				 isp_mis_tmp);
> +	}
> +
> +	/* frame was completely put out */

"put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
? The two comments could do with a bit of brush up, and I think the
ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
action.

> +	if (isp_mis & CIF_ISP_FRAME) {
> +		u32 isp_ris = 0;

No need to initialise this to 0.

> +		/* Clear Frame In (ISP) */
> +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> +		if (isp_mis_tmp & CIF_ISP_FRAME)
> +			v4l2_err(&dev->v4l2_dev,
> +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> +
> +		isp_ris = readl(base + CIF_ISP_RIS);
> +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);

Is there a guarantee that the statistics will be fully written out
before the video frame itself ? And doesn't this test if any of the
statistics is complete, not all of them ? I think the logic is wrong, it
seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
own. It's hard to tell for sure without extra information though (for
instance why are the stats-related bits read from CIF_ISP_RIS, when
they seem to be documented as valid in CIF_ISP_ISR), but this should be
validated, and most probably fixed. Care should be taken to keep
synchronisation of sequence number between the different queues.

> +	}
> +
> +	/*
> +	 * Then update changed configs. Some of them involve
> +	 * lot of register writes. Do those only one per frame.
> +	 * Do the updates in the order of the processing flow.
> +	 */
> +	rkisp1_params_isr(&dev->params_vdev, isp_mis);

You should pass dev to this function, and use it internally for the I/O
accessors instead of going through the vdev.

> +}
> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
> new file mode 100644
> index 000000000000..b0366e354ec2
> --- /dev/null
> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
> @@ -0,0 +1,111 @@
> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> +/*
> + * Rockchip isp1 driver
> + *
> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> + */
> +
> +#ifndef _RKISP1_H
> +#define _RKISP1_H
> +
> +#include <linux/platform_device.h>
> +#include <media/v4l2-fwnode.h>

I don't think you need these headers. You however need

#include <media/v4l2-ctrls.h>
#include <media/v4l2-subdev.h>

to make this header self-contained. Other headers could be added too
(videodev2.h, v4l2-mediabus.h and media-entity.h), they're included
through the above two headers at the moment.

> +
> +#include "common.h"
> +
> +struct rkisp1_stream;
> +
> +/*
> + * struct ispsd_in_fmt - ISP intput-pad format
> + *
> + * Translate mbus_code to hardware format values
> + *
> + * @bus_width: used for parallel
> + */
> +struct ispsd_in_fmt {
> +	u32 mbus_code;
> +	u8 fmt_type;
> +	u32 mipi_dt;
> +	u32 yuv_seq;
> +	enum rkisp1_fmt_raw_pat_type bayer_pat;
> +	u8 bus_width;
> +};

Some of the structures are prefixed with ispsd_, some with rkisp1_. I
think we should standardise on a single prefix.

> +
> +struct ispsd_out_fmt {
> +	u32 mbus_code;
> +	u8 fmt_type;
> +};
> +
> +struct rkisp1_ie_config {
> +	unsigned int effect;
> +};

This structure isn't used.

> +
> +enum rkisp1_isp_pad {
> +	RKISP1_ISP_PAD_SINK,

Should this be named SINK_VIDEO to match SINK_PARAMS ?

> +	RKISP1_ISP_PAD_SINK_PARAMS,
> +	RKISP1_ISP_PAD_SOURCE_PATH,

And SOURCE_VIDEO ?

> +	RKISP1_ISP_PAD_SOURCE_STATS,
> +	RKISP1_ISP_PAD_MAX
> +};
> +
> +/*
> + * struct rkisp1_isp_subdev - ISP sub-device
> + *
> + * See Cropping regions of ISP in rkisp1.c for details

Could you please document all fields from the structure ?

> + * @in_frm: input size, don't have to be equal to sensor size
> + * @in_fmt: input format
> + * @in_crop: crop for sink pad
> + * @out_fmt: output format
> + * @out_crop: output size
> + *
> + * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)

s/disabled/disabled /

> + * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
> + * @quantization: output quantization
> + */
> +struct rkisp1_isp_subdev {
> +	struct v4l2_subdev sd;
> +	struct media_pad pads[RKISP1_ISP_PAD_MAX];
> +	struct v4l2_ctrl_handler ctrl_handler;
> +	struct v4l2_mbus_framefmt in_frm;
> +	struct ispsd_in_fmt in_fmt;

I think this (and out_fmt) could be stored as pointers.

> +	struct v4l2_rect in_crop;
> +	struct ispsd_out_fmt out_fmt;
> +	struct v4l2_rect out_crop;

I think this could be greatly simplified if you stored all the data as
v4l2_subdev_pad_config instances for both the sink and source video pads
instead of using ad-hoc structures. In particular your get and set
format handlers would become much simpler. I recommend looking at the
mt9p031 sensor driver for an example. In a nutshell, the get handler
would retrieve the format from the v4l2_subdev_pad_config instance
either from rkisp1_isp_subdev (for V4L2_SUBDEV_FORMAT_ACTIVE) or from
the passed cfg argument (for V4L2_SUBDEV_FORMAT_TRY). The set handler
would operate similarly, with just a test one ACTIVE/TRY at the
beginning, followed by common code that only operates on the cfg, and
finally storing the format in rkisp1_isp_subdev for
V4L2_SUBDEV_FORMAT_ACTIVE.

> +	bool dphy_errctrl_disabled;
> +	atomic_t frm_sync_seq;
> +	enum v4l2_quantization quantization;

The quantization would then be stored in the v4l2_subdev_pad_config too.

> +};
> +
> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> +			       struct v4l2_device *v4l2_dev);
> +
> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
> +
> +void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
> +
> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
> +
> +static inline
> +struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->out_fmt;
> +}
> +
> +static inline
> +struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->in_fmt;
> +}
> +
> +static inline
> +struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
> +{
> +	return &isp_sdev->out_crop;
> +}

I agree with Sakari that accessing fields directly would be more
readable.

> +
> +static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
> +{
> +	return container_of(sd, struct rkisp1_isp_subdev, sd);
> +}
> +
> +#endif /* _RKISP1_H */

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 10/14] dt-bindings: Document the Rockchip ISP1 bindings
  2019-07-30 18:42 ` [PATCH v8 10/14] dt-bindings: Document the Rockchip ISP1 bindings Helen Koike
@ 2019-08-16  0:21   ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-16  0:21 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, Rob Herring,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, tfiga,
	linux-rockchip, hans.verkuil, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

Thank you for the patch.

On Tue, Jul 30, 2019 at 03:42:52PM -0300, Helen Koike wrote:
> From: Jacob Chen <jacob2.chen@rock-chips.com>
> 
> Add DT bindings documentation for Rockchip ISP1
> 
> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> [update for upstream]
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> 
> ---
> 
> Changes in v8: None
> Changes in v7:
> - update document with new design and tested example
> 
>  .../bindings/media/rockchip-isp1.txt          | 71 +++++++++++++++++++
>  1 file changed, 71 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/media/rockchip-isp1.txt
> 
> diff --git a/Documentation/devicetree/bindings/media/rockchip-isp1.txt b/Documentation/devicetree/bindings/media/rockchip-isp1.txt
> new file mode 100644
> index 000000000000..a97fef0f189f
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/rockchip-isp1.txt

If there wasn't enough work on your plate already I'd propose converting
this to yaml, but I'll refrain from doing so :-)

> @@ -0,0 +1,71 @@
> +Rockchip SoC Image Signal Processing unit v1
> +----------------------------------------------
> +
> +Rockchip ISP1 is the Camera interface for the Rockchip series of SoCs
> +which contains image processing, scaling, and compression funcitons.

s/funcitons/functions/

> +
> +Required properties:
> +- compatible: value should be one of the following
> +	"rockchip,rk3288-cif-isp";
> +	"rockchip,rk3399-cif-isp";
> +- reg : offset and length of the register set for the device.
> +- interrupts: should contain ISP interrupt.
> +- clocks: phandle to the required clocks.
> +- clock-names: required clock name.
> +- iommus: required a iommu node.
> +- phys: the phandle for the PHY port
> +- phy-names: must contain "dphy"
> +
> +port node
> +-------------------
> +
> +The device node should contain one 'ports' child node, with children 'port'
> +with child 'endpoint'.

Extra . and line break ?

> +nodes, according to the bindings defined in Documentation/devicetree/bindings/
> +media/video-interfaces.txt.
> +
> +- endpoint(mipi):
> +	- remote-endpoint: Connecting to Rockchip MIPI-DPHY,
> +		which is defined in rockchip-mipi-dphy.txt.
> +
> +The port node must contain at least one endpoint, either parallel or mipi.

If I understand things correctly, each ISP has a single CSI-2 receiver
and a single parallel output, and can select one of them at runtime.
This should be modelled as two separate ports. In addition to this,
multiple CSI-2 sensors can be connected to the same CSI-2 receiver as
long as all but one of them is held in reset (this is a poor man's CSI-2
switch, which exists in device out in the market, so we have to support
that). This should be modelled by multiple endpoints in the CSI-2 port.

> +It could have multiple endpoints, but please note the hardware don't support
> +two sensors work at a time, they are supposed to work asynchronously.

I assume you meant "are supposed to be mutually exclusive" or something
similar ?

> +
> +Device node example
> +-------------------
> +
> +	isp0: isp0@ff910000 {
> +		compatible = "rockchip,rk3399-cif-isp";
> +		reg = <0x0 0xff910000 0x0 0x4000>;
> +		interrupts = <GIC_SPI 43 IRQ_TYPE_LEVEL_HIGH 0>;
> +		clocks = <&cru SCLK_ISP0>,
> +			 <&cru ACLK_ISP0>, <&cru ACLK_ISP0_WRAPPER>,
> +			 <&cru HCLK_ISP0>, <&cru HCLK_ISP0_WRAPPER>;
> +		clock-names = "clk_isp",
> +			      "aclk_isp", "aclk_isp_wrap",
> +			      "hclk_isp", "hclk_isp_wrap";
> +		power-domains = <&power RK3399_PD_ISP0>;
> +		iommus = <&isp0_mmu>;
> +		phys = <&dphy>;
> +		phy-names = "dphy";
> +
> +		ports {
> +			#address-cells = <1>;
> +			#size-cells = <0>;
> +
> +			port@0 {
> +				mipi_in_wcam: endpoint@0 {
> +					reg = <0>;
> +					remote-endpoint = <&wcam_out>;
> +					data-lanes = <1 2>;
> +				};
> +
> +				mipi_in_ucam: endpoint@1 {
> +					reg = <1>;
> +					remote-endpoint = <&ucam_out>;
> +					data-lanes = <1>;
> +				};

What are wcam and ucam ? It would help if you showed the sensor nodes in
the example.

> +			};
> +		};
> +	};

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-08-15 17:54       ` Laurent Pinchart
  2019-08-15 18:26         ` Heiko Stübner
@ 2019-08-21 21:46         ` Helen Koike
  2019-08-22  2:32           ` Laurent Pinchart
  1 sibling, 1 reply; 76+ messages in thread
From: Helen Koike @ 2019-08-21 21:46 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Sakari Ailus, sakari.ailus, zhengsq, hans.verkuil, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Laurent,

Thanks for your review, I just have some comments/questions below.

On 8/15/19 2:54 PM, Laurent Pinchart wrote:
> Hi Helen,
> 
> Thank you for the patch.
> 
> On Wed, Aug 07, 2019 at 10:37:55AM -0300, Helen Koike wrote:
>> On 8/7/19 10:05 AM, Sakari Ailus wrote:
>>> On Tue, Jul 30, 2019 at 03:42:46PM -0300, Helen Koike wrote:
>>>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>>>
>>>> This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver
>>>>
>>>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>>>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>>>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>>>> [migrate to phy framework]
>>>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>>>> [update for upstream]
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>
>>>> ---
>>>>
>>>> Changes in v8:
>>>> - Remove boiler plate license text
>>>>
>>>> Changes in v7:
>>>> - Migrate dphy specific code from
>>>> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
>>>> to drivers/phy/rockchip/phy-rockchip-dphy.c
>>>> - Drop support for rk3288
>>>> - Drop support for dphy txrx
>>>> - code styling and checkpatch fixes
>>>>
>>>>  drivers/phy/rockchip/Kconfig             |   8 +
>>>>  drivers/phy/rockchip/Makefile            |   1 +
>>>>  drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
>>>>  3 files changed, 417 insertions(+)
>>>>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
>>>>
>>>> diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
>>>> index c454c90cd99e..afd072f135e6 100644
>>>> --- a/drivers/phy/rockchip/Kconfig
>>>> +++ b/drivers/phy/rockchip/Kconfig
>>>> @@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
>>>>  	help
>>>>  	  Enable this to support the Rockchip Display Port PHY.
>>>>  
>>>> +config PHY_ROCKCHIP_DPHY
>>>> +	tristate "Rockchip MIPI Synopsys DPHY driver"
> 
> How much of this PHY is Rockchip-specific ? Would it make sense to turn
> it into a Synopsys DPHY driver, with some Rockchip glue ? I suppose this
> could always be done later, if needed (and I also suppose there's no
> existing driver in drivers/phy/ that support the same Synopsys IP).
> 
>>>> +	depends on ARCH_ROCKCHIP && OF
>>>
>>> How about (...) || COMPILE_TEST ?
>>>
>>>> +	select GENERIC_PHY_MIPI_DPHY
>>>> +	select GENERIC_PHY
>>>> +	help
>>>> +	  Enable this to support the Rockchip MIPI Synopsys DPHY.
>>>> +
>>>>  config PHY_ROCKCHIP_EMMC
>>>>  	tristate "Rockchip EMMC PHY Driver"
>>>>  	depends on ARCH_ROCKCHIP && OF
>>>> diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
>>>> index fd21cbaf40dd..f62e9010bcaf 100644
>>>> --- a/drivers/phy/rockchip/Makefile
>>>> +++ b/drivers/phy/rockchip/Makefile
>>>> @@ -1,5 +1,6 @@
>>>>  # SPDX-License-Identifier: GPL-2.0
>>>>  obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
>>>> +obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
>>>>  obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
>>>>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
>>>>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
>>>> diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
>>>> new file mode 100644
>>>> index 000000000000..3a29976c2dff
>>>> --- /dev/null
>>>> +++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
>>>> @@ -0,0 +1,408 @@
>>>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>>>> +/*
>>>> + * Rockchip MIPI Synopsys DPHY driver
>>>> + *
>>>> + * Based on:
>>>> + *
>>>> + * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
>>>> + * Author: Yakir Yang <ykk@@rock-chips.com>
>>>> + */
>>>> +
>>>> +#include <linux/clk.h>
>>>> +#include <linux/io.h>
>>>> +#include <linux/mfd/syscon.h>
>>>> +#include <linux/module.h>
>>>> +#include <linux/of.h>
>>>> +#include <linux/of_device.h>
>>>> +#include <linux/phy/phy.h>
>>>> +#include <linux/phy/phy-mipi-dphy.h>
>>>> +#include <linux/platform_device.h>
>>>> +#include <linux/regmap.h>
>>>> +
>>>> +#define RK3399_GRF_SOC_CON9	0x6224
>>>> +#define RK3399_GRF_SOC_CON21	0x6254
>>>> +#define RK3399_GRF_SOC_CON22	0x6258
>>>> +#define RK3399_GRF_SOC_CON23	0x625c
>>>> +#define RK3399_GRF_SOC_CON24	0x6260
>>>> +#define RK3399_GRF_SOC_CON25	0x6264
>>>> +#define RK3399_GRF_SOC_STATUS1	0xe2a4
>>>> +
>>>> +#define CLOCK_LANE_HS_RX_CONTROL		0x34
>>>> +#define LANE0_HS_RX_CONTROL			0x44
>>>> +#define LANE1_HS_RX_CONTROL			0x54
>>>> +#define LANE2_HS_RX_CONTROL			0x84
>>>> +#define LANE3_HS_RX_CONTROL			0x94
>>>> +#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
>>>> +
>>>> +#define MAX_DPHY_CLK 8
>>>> +
>>>> +#define PHY_TESTEN_ADDR			(0x1 << 16)
>>>> +#define PHY_TESTEN_DATA			(0x0 << 16)
>>>> +#define PHY_TESTCLK			(0x1 << 1)
>>>> +#define PHY_TESTCLR			(0x1 << 0)
> 
> Maybe s/0x// for the previous four lines ?
> 
>>>> +#define THS_SETTLE_COUNTER_THRESHOLD	0x04
>>>> +
>>>> +#define HIWORD_UPDATE(val, mask, shift) \
>>>> +	((val) << (shift) | (mask) << ((shift) + 16))
> 
> As you use this in a single place, I would inline it, possibly with a
> small comment that explains what's happening.
> 
>>>> +
>>>> +#define GRF_SOC_CON12                           0x0274
>>>> +
>>>> +#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
>>>> +#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
>>>> +
>>>> +#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
>>>> +#define GRF_EDP_PHY_SIDDQ_ON                    0
>>>> +#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)
> 
> I would recommend aligning the value of of all macros in the same way.
> 
>>>> +
>>>> +struct hsfreq_range {
>>>> +	u32 range_h;
> 
> The structure would be more compact if you turned this into a u16.
> 
>>>> +	u8 cfg_bit;
>>>> +};
>>>> +
>>>> +static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
>>>> +	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
>>>> +	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
>>>> +	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
>>>> +	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
>>>> +	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
>>>> +	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
>>>> +	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
>>>> +	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
>>>> +	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
>>>> +	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}
> 
> Maybe s/{/{ / and s/}/ }/ to give it a bit more air ? :-)
> 
>>>> +};
>>>> +
>>>> +static const char * const rk3399_mipidphy_clks[] = {
>>>> +	"dphy-ref",
>>>> +	"dphy-cfg",
>>>> +	"grf",
>>>> +};
>>>> +
>>>> +enum dphy_reg_id {
>>>> +	GRF_DPHY_RX0_TURNDISABLE = 0,
>>>> +	GRF_DPHY_RX0_FORCERXMODE,
>>>> +	GRF_DPHY_RX0_FORCETXSTOPMODE,
>>>> +	GRF_DPHY_RX0_ENABLE,
>>>> +	GRF_DPHY_RX0_TESTCLR,
>>>> +	GRF_DPHY_RX0_TESTCLK,
>>>> +	GRF_DPHY_RX0_TESTEN,
>>>> +	GRF_DPHY_RX0_TESTDIN,
>>>> +	GRF_DPHY_RX0_TURNREQUEST,
>>>> +	GRF_DPHY_RX0_TESTDOUT,
>>>> +	GRF_DPHY_TX0_TURNDISABLE,
>>>> +	GRF_DPHY_TX0_FORCERXMODE,
>>>> +	GRF_DPHY_TX0_FORCETXSTOPMODE,
>>>> +	GRF_DPHY_TX0_TURNREQUEST,
>>>> +	GRF_DPHY_TX1RX1_TURNDISABLE,
>>>> +	GRF_DPHY_TX1RX1_FORCERXMODE,
>>>> +	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
>>>> +	GRF_DPHY_TX1RX1_ENABLE,
>>>> +	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
>>>> +	GRF_DPHY_TX1RX1_BASEDIR,
>>>> +	GRF_DPHY_TX1RX1_ENABLECLK,
>>>> +	GRF_DPHY_TX1RX1_TURNREQUEST,
>>>> +	GRF_DPHY_RX1_SRC_SEL,
>>>> +	/* rk3288 only */
>>>> +	GRF_CON_DISABLE_ISP,
>>>> +	GRF_CON_ISP_DPHY_SEL,
>>>> +	GRF_DSI_CSI_TESTBUS_SEL,
>>>> +	GRF_DVP_V18SEL,
>>>> +	/* below is for rk3399 only */
>>>> +	GRF_DPHY_RX0_CLK_INV_SEL,
>>>> +	GRF_DPHY_RX1_CLK_INV_SEL,
>>>> +};
>>>> +
>>>> +struct dphy_reg {
>>>> +	u32 offset;
>>>> +	u32 mask;
>>>> +	u32 shift;
> 
> The offset should hold in 16 bits and the mask and shift in 8 bits. That
> would save space in the table below.
> 
>>>> +};
>>>> +
>>>> +#define PHY_REG(_offset, _width, _shift) \
>>>> +	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
>>>> +
>>>> +static const struct dphy_reg rk3399_grf_dphy_regs[] = {
>>>> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
>>>> +	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
>>>> +	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
>>>> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
>>>> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
>>>> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
>>>> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
>>>> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
>>>> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
>>>> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
>>>> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
>>>> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
>>>> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
>>>> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
>>>> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
>>>> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
>>>> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
>>>> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
>>>> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
>>>> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
>>>> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
>>>> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
>>>> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
>>>> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
>>>> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),
> 
> The annoying part with such an indirection is that you can't really
> write multiple fields in a single register with a single operation.
> Is
> the register mapping completely different between the rk3288 and the
> rk3399, or are the fields grouped in registers in a similar way ? In the
> latter case we could possibly optimise it.

This would be the rk3288 version:

+static const struct dphy_reg rk3288_grf_dphy_regs[] = {
+	[GRF_CON_DISABLE_ISP] = PHY_REG(RK3288_GRF_SOC_CON6, 1, 0),
+	[GRF_CON_ISP_DPHY_SEL] = PHY_REG(RK3288_GRF_SOC_CON6, 1, 1),
+	[GRF_DSI_CSI_TESTBUS_SEL] = PHY_REG(RK3288_GRF_SOC_CON6, 1, 14),
+	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3288_GRF_SOC_CON8, 4, 0),
+	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3288_GRF_SOC_CON8, 4, 4),
+	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3288_GRF_SOC_CON8, 4, 8),
+	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 0),
+	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 4),
+	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 8),
+	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 12),
+	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 0),
+	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 4),
+	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 8),
+	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 12),
+	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 0),
+	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 1),
+	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 2),
+	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3288_GRF_SOC_CON14, 8, 3),
+	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 12),
+	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 13),
+	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 14),
+	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 15),
+	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3288_GRF_SOC_CON15, 4, 0),
+	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3288_GRF_SOC_CON15, 4, 4),
+	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3288_GRF_SOC_CON15, 3, 8),
+	[GRF_DVP_V18SEL] = PHY_REG(RK3288_GRF_IO_VSEL, 1, 1),
+	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3288_GRF_SOC_STATUS21, 8, 0),
+};

Which seems different mask and shifts from rk3399. If you have any ideas in
how to optimize this I would appreciate it.

> 
>>>> +};
>>>> +
>>>> +struct dphy_drv_data {
>>>> +	const char * const *clks;
>>>> +	int num_clks;
> 
> This is never negative, you can make it an unsigned int.
> 
>>>> +	const struct hsfreq_range *hsfreq_ranges;
>>>> +	int num_hsfreq_ranges;
> 
> Same here.
> 
>>>> +	const struct dphy_reg *regs;
>>>> +};
>>>> +
>>>> +struct rockchip_dphy {
>>>> +	struct device *dev;
>>>> +	struct regmap *grf;
>>>> +	const struct dphy_reg *grf_regs;
>>>> +	struct clk_bulk_data clks[MAX_DPHY_CLK];
>>>> +
>>>> +	const struct dphy_drv_data *drv_data;
>>>> +	struct phy_configure_opts_mipi_dphy config;
>>>> +};
>>>> +
>>>> +static inline void write_grf_reg(struct rockchip_dphy *priv,
>>>> +				 int index, u8 value)
> 
> Maybe unsigned int index ?
> 
>>>> +{
>>>> +	const struct dphy_reg *reg = &priv->grf_regs[index];
>>>> +	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
>>>> +
>>>> +	WARN_ON(!reg->offset);
>>>> +	regmap_write(priv->grf, reg->offset, val);
>>>> +}
>>>> +
>>>> +static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
>>>> +			     u8 test_code, u8 test_data)
> 
> Function (and structure) names have different prefixes, would it make
> sense to standardise them ? Maybe rockchip_dphy_ ? Or rk_dphy_ for a
> shorter version ? This could become rk_dphy_write_dphy(), and the
> previous function rk_dphy_write_grf().
> 
>>>> +{
>>>> +	/*
>>>> +	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
>>>> +	 * is latched internally as the current test code. Test data is
>>>> +	 * programmed internally by rising edge on TESTCLK.
>>>> +	 */
> 
> I've never understood why PHYs tend to have a register named TEST that
> contains way more than test data :-)
> 
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
>>>> +}
>>>> +
>>>> +/* should be move to power_on */
> 
> s/move/moved/
> 
> Do you mean merging the two functions together ? What prevents from
> doing so ? 

Nothing really, this is a left over command as mipidphy_rx_stream_on() is already
being called from power_on, and I don't think we should merge it because
in the future we'll probably going to have mipidphy_txrx_stream_on() for dphy1.

> 
>>>> +static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
>>>> +{
>>>> +	const struct dphy_drv_data *drv_data = priv->drv_data;
>>>> +	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
>>>> +	struct phy_configure_opts_mipi_dphy *config = &priv->config;
>>>> +	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
>>>> +	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
>>>> +
>>>> +	do_div(data_rate_mbps, 1000 * 1000);
>>>> +
>>>> +	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
>>>> +		__func__, config->lanes, data_rate_mbps);
>>>> +
>>>> +	for (i = 0; i < num_hsfreq_ranges; i++) {
>>>> +		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
>>>> +			hsfreq = hsfreq_ranges[i].cfg_bit;
>>>> +			break;
>>>> +		}
>>>> +	}
> 
> As num_hsfreq_ranges and hsfreq_ranges are only used in this loop, I
> would remove the local variables.
> 
>>>> +
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
>>>> +
>>>> +	/* Disable lan turn around, which is ignored in receive mode */
> 
> Is it "lan turn around", or "lane turn around" ?
> 
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
>>>> +
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
>>>> +
>>>> +	/* dphy start */
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
>>>> +	usleep_range(100, 150);
>>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
>>>> +	usleep_range(100, 150);
>>>> +
>>>> +	/* set clock lane */
>>>> +	/* HS hsfreq_range & lane 0  settle bypass */
>>>> +	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
>>>> +	/* HS RX Control of lane0 */
>>>> +	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
>>>> +	/* HS RX Control of lane1 */
>>>> +	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
>>>> +	/* HS RX Control of lane2 */
>>>> +	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
>>>> +	/* HS RX Control of lane3 */
>>>> +	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);
> 
> Does this hardcode usage of a single lane ?

Rockchip seems to uses TEST* registers to set the hsfreqrange.
It mentions the test code 0x44 (which is LANE0_HS_RX_CONTROL)
but it doesn't mention the others lanes.

Replacing those call by
mipidphy0_wr_reg(priv, LANEx_HS_RX_CONTROL, hsfreq << 1);
seems to be working.

I can check if this changes the datarate (I just need to figure a proper
way to test this or get some docs).

Thanks for spotting this.

> 
>>>> +	/* HS RX Data Lanes Settle State Time Control */
>>>> +	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
>>>> +			 THS_SETTLE_COUNTER_THRESHOLD);
>>>> +
>>>> +	/* Normal operation */
>>>> +	mipidphy0_wr_reg(priv, 0x0, 0);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
>>>> +{
>>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>>>> +	int ret;
>>>> +
>>>> +	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */
> 
> I'm not sure to understand what this means.
> 

iirc, the question is if we should fail when phy_mipi_dphy_config_validate() fails,
or if we should use a default config.

Looking at other examples, is seems that only two drivers call
phy_mipi_dphy_get_default_config() in a totally diferent context, not in mipi path.
So I guess I would just remove this comment if this is ok with you.


>>>> +	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	memcpy(&priv->config, opts, sizeof(priv->config));
>>>
>>> You could to:
>>>
>>> 	priv->config = *opts;
>>>
>>> Up to you. Some people like memcpy(). :-)
>>
>> your way is better thanks!
>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int rockchip_dphy_power_on(struct phy *phy)
>>>> +{
>>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>>>> +	int ret;
>>>> +
>>>> +	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	return mipidphy_rx_stream_on(priv);
> 
> Should you call clk_bulk_disable() if mipidphy_rx_stream_on() fails ?
> Actually that function never fails, so I'd make it a void function, and
> return 0 here.

Ack, I made it void, I'll send it in the next version.

> 
> What happens if the clock rate is higher than the maximum supported by
> the PHY ? Shouldn't rockchip_dphy_configure() fail in that case ?

This is checked in function mipidphy_rx_stream_on(), if it is higher we just
configure the maximum supported rate. Is this ok?

> 
>>>> +}
>>>> +
>>>> +static int rockchip_dphy_power_off(struct phy *phy)
>>>> +{
>>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>>>> +
> 
> No need to write any register ? That's scary, what will happen on the
> next power on, when the clocks gets enabled ?

Just for testing, I hacked the code to only call mipidphy_rx_stream_on() once,
when streaming for the first time, then I don't call it anymore and starting/stopping
streaming always works, so I guess it keeps the previous configuration when clocks
get enabled.
I wonder if this can be a problem when switching from dphy rx to txrx, but for now
we just support rx.

Maybe just calling rk_dphy_write_grf(priv, GRF_DPHY_RX0_ENABLE, 0) is enough.

> 
>>>> +	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int rockchip_dphy_init(struct phy *phy)
>>>> +{
>>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>>>> +	int ret;
>>>> +
>>>> +	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);
>>>
>>> return ...;
>>>
>>>> +	if (ret)
>>>> +		return ret;
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int rockchip_dphy_exit(struct phy *phy)
>>>> +{
>>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
>>>> +
>>>> +	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static const struct phy_ops rockchip_dphy_ops = {
>>>> +	.power_on	= rockchip_dphy_power_on,
>>>> +	.power_off	= rockchip_dphy_power_off,
>>>> +	.init		= rockchip_dphy_init,
>>>> +	.exit		= rockchip_dphy_exit,
>>>> +	.configure	= rockchip_dphy_configure,
>>>> +	.owner		= THIS_MODULE,
>>>> +};
>>>> +
>>>> +static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
>>>> +	.clks = rk3399_mipidphy_clks,
>>>> +	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
>>>> +	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
>>>> +	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
>>>> +	.regs = rk3399_grf_dphy_regs,
>>>
>>> Do you expect to support more of the similar PHY(s) --- are there such? If
>>> not, you could put these in the code that uses them.
>>
>> Yes, for rk3288 in the future.
>>
>>>> +};
>>>> +
>>>> +static const struct of_device_id rockchip_dphy_dt_ids[] = {
>>>> +	{
>>>> +		.compatible = "rockchip,rk3399-mipi-dphy",
>>>> +		.data = &rk3399_mipidphy_drv_data,
>>>> +	},
>>>> +	{}
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
>>>> +
>>>> +static int rockchip_dphy_probe(struct platform_device *pdev)
>>>> +{
>>>> +	struct device *dev = &pdev->dev;
>>>> +	struct device_node *np = dev->of_node;
>>>> +	const struct dphy_drv_data *drv_data;
>>>> +	struct phy_provider *phy_provider;
>>>> +	const struct of_device_id *of_id;
>>>> +	struct rockchip_dphy *priv;
>>>> +	struct regmap *grf;
>>>> +	struct phy *phy;
>>>> +	unsigned int i;
>>>> +	int ret;
>>>> +
>>>> +	if (!dev->parent || !dev->parent->of_node)
>>>> +		return -ENODEV;
>>>> +
>>>> +	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
>>>> +		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");
> 
> You can replace pdev->dev with dev here and below.
> 
> s/rx/RX mode/ ?
> 
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
>>>> +	if (!priv)
>>>> +		return -ENOMEM;
>>>> +	priv->dev = dev;
>>>> +
>>>> +	grf = syscon_node_to_regmap(dev->parent->of_node);
>>>> +	if (IS_ERR(grf)) {
>>>> +		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
>>>> +						      "rockchip,grf");
>>>> +		if (IS_ERR(grf)) {
>>>> +			dev_err(dev, "Can't find GRF syscon\n");
>>>> +			return -ENODEV;
>>>> +		}
>>>> +	}
>>>> +	priv->grf = grf;
>>>> +
>>>> +	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
>>>> +	if (!of_id)
>>>> +		return -EINVAL;
>>>> +
>>>> +	drv_data = of_id->data;
>>>> +	priv->grf_regs = drv_data->regs;
> 
> Do you have to store grf_regs in priv, or could it be accessed through
> priv->drv_data->regs ?
> 
>>>> +	priv->drv_data = drv_data;
>>>> +	for (i = 0; i < drv_data->num_clks; i++)
>>>> +		priv->clks[i].id = drv_data->clks[i];
>>>> +	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
>>>> +	if (IS_ERR(phy)) {
>>>> +		dev_err(dev, "failed to create phy\n");
>>>> +		return PTR_ERR(phy);
>>>> +	}
>>>> +	phy_set_drvdata(phy, priv);
>>>> +
>>>> +	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
>>>> +
>>>> +	return PTR_ERR_OR_ZERO(phy_provider);
>>>> +}
>>>> +
>>>> +static struct platform_driver rockchip_dphy_driver = {
>>>> +	.probe = rockchip_dphy_probe,
>>>> +	.driver = {
>>>> +		.name	= "rockchip-mipi-dphy",
>>>> +		.of_match_table = rockchip_dphy_dt_ids,
>>>> +	},
>>>> +};
>>>> +module_platform_driver(rockchip_dphy_driver);
>>>> +
>>>> +MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
>>>> +MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
>>>> +MODULE_LICENSE("Dual MIT/GPL");
> 
> Overall this is quite good, there are only small issues.
> 

Thank you a lot for your review
Helen

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver
  2019-08-21 21:46         ` Helen Koike
@ 2019-08-22  2:32           ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2019-08-22  2:32 UTC (permalink / raw)
  To: Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Sakari Ailus, sakari.ailus, zhengsq, hans.verkuil, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Helen,

On Wed, Aug 21, 2019 at 06:46:15PM -0300, Helen Koike wrote:
> On 8/15/19 2:54 PM, Laurent Pinchart wrote:
> > On Wed, Aug 07, 2019 at 10:37:55AM -0300, Helen Koike wrote:
> >> On 8/7/19 10:05 AM, Sakari Ailus wrote:
> >>> On Tue, Jul 30, 2019 at 03:42:46PM -0300, Helen Koike wrote:
> >>>> From: Jacob Chen <jacob2.chen@rock-chips.com>
> >>>>
> >>>> This commit adds a subdev driver for Rockchip MIPI Synopsys DPHY driver
> >>>>
> >>>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> >>>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> >>>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> >>>> [migrate to phy framework]
> >>>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> >>>> [update for upstream]
> >>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>
> >>>> ---
> >>>>
> >>>> Changes in v8:
> >>>> - Remove boiler plate license text
> >>>>
> >>>> Changes in v7:
> >>>> - Migrate dphy specific code from
> >>>> drivers/media/platform/rockchip/isp1/mipi_dphy_sy.c
> >>>> to drivers/phy/rockchip/phy-rockchip-dphy.c
> >>>> - Drop support for rk3288
> >>>> - Drop support for dphy txrx
> >>>> - code styling and checkpatch fixes
> >>>>
> >>>>  drivers/phy/rockchip/Kconfig             |   8 +
> >>>>  drivers/phy/rockchip/Makefile            |   1 +
> >>>>  drivers/phy/rockchip/phy-rockchip-dphy.c | 408 +++++++++++++++++++++++
> >>>>  3 files changed, 417 insertions(+)
> >>>>  create mode 100644 drivers/phy/rockchip/phy-rockchip-dphy.c
> >>>>
> >>>> diff --git a/drivers/phy/rockchip/Kconfig b/drivers/phy/rockchip/Kconfig
> >>>> index c454c90cd99e..afd072f135e6 100644
> >>>> --- a/drivers/phy/rockchip/Kconfig
> >>>> +++ b/drivers/phy/rockchip/Kconfig
> >>>> @@ -9,6 +9,14 @@ config PHY_ROCKCHIP_DP
> >>>>  	help
> >>>>  	  Enable this to support the Rockchip Display Port PHY.
> >>>>  
> >>>> +config PHY_ROCKCHIP_DPHY
> >>>> +	tristate "Rockchip MIPI Synopsys DPHY driver"
> > 
> > How much of this PHY is Rockchip-specific ? Would it make sense to turn
> > it into a Synopsys DPHY driver, with some Rockchip glue ? I suppose this
> > could always be done later, if needed (and I also suppose there's no
> > existing driver in drivers/phy/ that support the same Synopsys IP).
> > 
> >>>> +	depends on ARCH_ROCKCHIP && OF
> >>>
> >>> How about (...) || COMPILE_TEST ?
> >>>
> >>>> +	select GENERIC_PHY_MIPI_DPHY
> >>>> +	select GENERIC_PHY
> >>>> +	help
> >>>> +	  Enable this to support the Rockchip MIPI Synopsys DPHY.
> >>>> +
> >>>>  config PHY_ROCKCHIP_EMMC
> >>>>  	tristate "Rockchip EMMC PHY Driver"
> >>>>  	depends on ARCH_ROCKCHIP && OF
> >>>> diff --git a/drivers/phy/rockchip/Makefile b/drivers/phy/rockchip/Makefile
> >>>> index fd21cbaf40dd..f62e9010bcaf 100644
> >>>> --- a/drivers/phy/rockchip/Makefile
> >>>> +++ b/drivers/phy/rockchip/Makefile
> >>>> @@ -1,5 +1,6 @@
> >>>>  # SPDX-License-Identifier: GPL-2.0
> >>>>  obj-$(CONFIG_PHY_ROCKCHIP_DP)		+= phy-rockchip-dp.o
> >>>> +obj-$(CONFIG_PHY_ROCKCHIP_DPHY)		+= phy-rockchip-dphy.o
> >>>>  obj-$(CONFIG_PHY_ROCKCHIP_EMMC)		+= phy-rockchip-emmc.o
> >>>>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_HDMI)	+= phy-rockchip-inno-hdmi.o
> >>>>  obj-$(CONFIG_PHY_ROCKCHIP_INNO_USB2)	+= phy-rockchip-inno-usb2.o
> >>>> diff --git a/drivers/phy/rockchip/phy-rockchip-dphy.c b/drivers/phy/rockchip/phy-rockchip-dphy.c
> >>>> new file mode 100644
> >>>> index 000000000000..3a29976c2dff
> >>>> --- /dev/null
> >>>> +++ b/drivers/phy/rockchip/phy-rockchip-dphy.c
> >>>> @@ -0,0 +1,408 @@
> >>>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> >>>> +/*
> >>>> + * Rockchip MIPI Synopsys DPHY driver
> >>>> + *
> >>>> + * Based on:
> >>>> + *
> >>>> + * Copyright (C) 2016 FuZhou Rockchip Co., Ltd.
> >>>> + * Author: Yakir Yang <ykk@@rock-chips.com>
> >>>> + */
> >>>> +
> >>>> +#include <linux/clk.h>
> >>>> +#include <linux/io.h>
> >>>> +#include <linux/mfd/syscon.h>
> >>>> +#include <linux/module.h>
> >>>> +#include <linux/of.h>
> >>>> +#include <linux/of_device.h>
> >>>> +#include <linux/phy/phy.h>
> >>>> +#include <linux/phy/phy-mipi-dphy.h>
> >>>> +#include <linux/platform_device.h>
> >>>> +#include <linux/regmap.h>
> >>>> +
> >>>> +#define RK3399_GRF_SOC_CON9	0x6224
> >>>> +#define RK3399_GRF_SOC_CON21	0x6254
> >>>> +#define RK3399_GRF_SOC_CON22	0x6258
> >>>> +#define RK3399_GRF_SOC_CON23	0x625c
> >>>> +#define RK3399_GRF_SOC_CON24	0x6260
> >>>> +#define RK3399_GRF_SOC_CON25	0x6264
> >>>> +#define RK3399_GRF_SOC_STATUS1	0xe2a4
> >>>> +
> >>>> +#define CLOCK_LANE_HS_RX_CONTROL		0x34
> >>>> +#define LANE0_HS_RX_CONTROL			0x44
> >>>> +#define LANE1_HS_RX_CONTROL			0x54
> >>>> +#define LANE2_HS_RX_CONTROL			0x84
> >>>> +#define LANE3_HS_RX_CONTROL			0x94
> >>>> +#define HS_RX_DATA_LANES_THS_SETTLE_CONTROL	0x75
> >>>> +
> >>>> +#define MAX_DPHY_CLK 8
> >>>> +
> >>>> +#define PHY_TESTEN_ADDR			(0x1 << 16)
> >>>> +#define PHY_TESTEN_DATA			(0x0 << 16)
> >>>> +#define PHY_TESTCLK			(0x1 << 1)
> >>>> +#define PHY_TESTCLR			(0x1 << 0)
> > 
> > Maybe s/0x// for the previous four lines ?
> > 
> >>>> +#define THS_SETTLE_COUNTER_THRESHOLD	0x04
> >>>> +
> >>>> +#define HIWORD_UPDATE(val, mask, shift) \
> >>>> +	((val) << (shift) | (mask) << ((shift) + 16))
> > 
> > As you use this in a single place, I would inline it, possibly with a
> > small comment that explains what's happening.
> > 
> >>>> +
> >>>> +#define GRF_SOC_CON12                           0x0274
> >>>> +
> >>>> +#define GRF_EDP_REF_CLK_SEL_INTER_HIWORD_MASK   BIT(20)
> >>>> +#define GRF_EDP_REF_CLK_SEL_INTER               BIT(4)
> >>>> +
> >>>> +#define GRF_EDP_PHY_SIDDQ_HIWORD_MASK           BIT(21)
> >>>> +#define GRF_EDP_PHY_SIDDQ_ON                    0
> >>>> +#define GRF_EDP_PHY_SIDDQ_OFF                   BIT(5)
> > 
> > I would recommend aligning the value of of all macros in the same way.
> > 
> >>>> +
> >>>> +struct hsfreq_range {
> >>>> +	u32 range_h;
> > 
> > The structure would be more compact if you turned this into a u16.
> > 
> >>>> +	u8 cfg_bit;
> >>>> +};
> >>>> +
> >>>> +static const struct hsfreq_range rk3399_mipidphy_hsfreq_ranges[] = {
> >>>> +	{  89, 0x00}, {  99, 0x10}, { 109, 0x20}, { 129, 0x01},
> >>>> +	{ 139, 0x11}, { 149, 0x21}, { 169, 0x02}, { 179, 0x12},
> >>>> +	{ 199, 0x22}, { 219, 0x03}, { 239, 0x13}, { 249, 0x23},
> >>>> +	{ 269, 0x04}, { 299, 0x14}, { 329, 0x05}, { 359, 0x15},
> >>>> +	{ 399, 0x25}, { 449, 0x06}, { 499, 0x16}, { 549, 0x07},
> >>>> +	{ 599, 0x17}, { 649, 0x08}, { 699, 0x18}, { 749, 0x09},
> >>>> +	{ 799, 0x19}, { 849, 0x29}, { 899, 0x39}, { 949, 0x0a},
> >>>> +	{ 999, 0x1a}, {1049, 0x2a}, {1099, 0x3a}, {1149, 0x0b},
> >>>> +	{1199, 0x1b}, {1249, 0x2b}, {1299, 0x3b}, {1349, 0x0c},
> >>>> +	{1399, 0x1c}, {1449, 0x2c}, {1500, 0x3c}
> > 
> > Maybe s/{/{ / and s/}/ }/ to give it a bit more air ? :-)
> > 
> >>>> +};
> >>>> +
> >>>> +static const char * const rk3399_mipidphy_clks[] = {
> >>>> +	"dphy-ref",
> >>>> +	"dphy-cfg",
> >>>> +	"grf",
> >>>> +};
> >>>> +
> >>>> +enum dphy_reg_id {
> >>>> +	GRF_DPHY_RX0_TURNDISABLE = 0,
> >>>> +	GRF_DPHY_RX0_FORCERXMODE,
> >>>> +	GRF_DPHY_RX0_FORCETXSTOPMODE,
> >>>> +	GRF_DPHY_RX0_ENABLE,
> >>>> +	GRF_DPHY_RX0_TESTCLR,
> >>>> +	GRF_DPHY_RX0_TESTCLK,
> >>>> +	GRF_DPHY_RX0_TESTEN,
> >>>> +	GRF_DPHY_RX0_TESTDIN,
> >>>> +	GRF_DPHY_RX0_TURNREQUEST,
> >>>> +	GRF_DPHY_RX0_TESTDOUT,
> >>>> +	GRF_DPHY_TX0_TURNDISABLE,
> >>>> +	GRF_DPHY_TX0_FORCERXMODE,
> >>>> +	GRF_DPHY_TX0_FORCETXSTOPMODE,
> >>>> +	GRF_DPHY_TX0_TURNREQUEST,
> >>>> +	GRF_DPHY_TX1RX1_TURNDISABLE,
> >>>> +	GRF_DPHY_TX1RX1_FORCERXMODE,
> >>>> +	GRF_DPHY_TX1RX1_FORCETXSTOPMODE,
> >>>> +	GRF_DPHY_TX1RX1_ENABLE,
> >>>> +	GRF_DPHY_TX1RX1_MASTERSLAVEZ,
> >>>> +	GRF_DPHY_TX1RX1_BASEDIR,
> >>>> +	GRF_DPHY_TX1RX1_ENABLECLK,
> >>>> +	GRF_DPHY_TX1RX1_TURNREQUEST,
> >>>> +	GRF_DPHY_RX1_SRC_SEL,
> >>>> +	/* rk3288 only */
> >>>> +	GRF_CON_DISABLE_ISP,
> >>>> +	GRF_CON_ISP_DPHY_SEL,
> >>>> +	GRF_DSI_CSI_TESTBUS_SEL,
> >>>> +	GRF_DVP_V18SEL,
> >>>> +	/* below is for rk3399 only */
> >>>> +	GRF_DPHY_RX0_CLK_INV_SEL,
> >>>> +	GRF_DPHY_RX1_CLK_INV_SEL,
> >>>> +};
> >>>> +
> >>>> +struct dphy_reg {
> >>>> +	u32 offset;
> >>>> +	u32 mask;
> >>>> +	u32 shift;
> > 
> > The offset should hold in 16 bits and the mask and shift in 8 bits. That
> > would save space in the table below.
> > 
> >>>> +};
> >>>> +
> >>>> +#define PHY_REG(_offset, _width, _shift) \
> >>>> +	{ .offset = _offset, .mask = BIT(_width) - 1, .shift = _shift, }
> >>>> +
> >>>> +static const struct dphy_reg rk3399_grf_dphy_regs[] = {
> >>>> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON9, 4, 0),
> >>>> +	[GRF_DPHY_RX0_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 10),
> >>>> +	[GRF_DPHY_RX1_CLK_INV_SEL] = PHY_REG(RK3399_GRF_SOC_CON9, 1, 11),
> >>>> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 0),
> >>>> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 4),
> >>>> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 8),
> >>>> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON21, 4, 12),
> >>>> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 0),
> >>>> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 4),
> >>>> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 8),
> >>>> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON22, 4, 12),
> >>>> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 0),
> >>>> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 4),
> >>>> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 8),
> >>>> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3399_GRF_SOC_CON23, 4, 12),
> >>>> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3399_GRF_SOC_CON24, 4, 0),
> >>>> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 4),
> >>>> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 5),
> >>>> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 6),
> >>>> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3399_GRF_SOC_CON24, 1, 7),
> >>>> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3399_GRF_SOC_CON25, 8, 0),
> >>>> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 8),
> >>>> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 9),
> >>>> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3399_GRF_SOC_CON25, 1, 10),
> >>>> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3399_GRF_SOC_STATUS1, 8, 0),
> > 
> > The annoying part with such an indirection is that you can't really
> > write multiple fields in a single register with a single operation.
> > Is the register mapping completely different between the rk3288 and the
> > rk3399, or are the fields grouped in registers in a similar way ? In the
> > latter case we could possibly optimise it.
> 
> This would be the rk3288 version:
> 
> +static const struct dphy_reg rk3288_grf_dphy_regs[] = {
> +	[GRF_CON_DISABLE_ISP] = PHY_REG(RK3288_GRF_SOC_CON6, 1, 0),
> +	[GRF_CON_ISP_DPHY_SEL] = PHY_REG(RK3288_GRF_SOC_CON6, 1, 1),
> +	[GRF_DSI_CSI_TESTBUS_SEL] = PHY_REG(RK3288_GRF_SOC_CON6, 1, 14),
> +	[GRF_DPHY_TX0_TURNDISABLE] = PHY_REG(RK3288_GRF_SOC_CON8, 4, 0),
> +	[GRF_DPHY_TX0_FORCERXMODE] = PHY_REG(RK3288_GRF_SOC_CON8, 4, 4),
> +	[GRF_DPHY_TX0_FORCETXSTOPMODE] = PHY_REG(RK3288_GRF_SOC_CON8, 4, 8),
> +	[GRF_DPHY_TX1RX1_TURNDISABLE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 0),
> +	[GRF_DPHY_TX1RX1_FORCERXMODE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 4),
> +	[GRF_DPHY_TX1RX1_FORCETXSTOPMODE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 8),
> +	[GRF_DPHY_TX1RX1_ENABLE] = PHY_REG(RK3288_GRF_SOC_CON9, 4, 12),
> +	[GRF_DPHY_RX0_TURNDISABLE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 0),
> +	[GRF_DPHY_RX0_FORCERXMODE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 4),
> +	[GRF_DPHY_RX0_FORCETXSTOPMODE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 8),
> +	[GRF_DPHY_RX0_ENABLE] = PHY_REG(RK3288_GRF_SOC_CON10, 4, 12),
> +	[GRF_DPHY_RX0_TESTCLR] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 0),
> +	[GRF_DPHY_RX0_TESTCLK] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 1),
> +	[GRF_DPHY_RX0_TESTEN] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 2),
> +	[GRF_DPHY_RX0_TESTDIN] = PHY_REG(RK3288_GRF_SOC_CON14, 8, 3),
> +	[GRF_DPHY_TX1RX1_ENABLECLK] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 12),
> +	[GRF_DPHY_RX1_SRC_SEL] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 13),
> +	[GRF_DPHY_TX1RX1_MASTERSLAVEZ] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 14),
> +	[GRF_DPHY_TX1RX1_BASEDIR] = PHY_REG(RK3288_GRF_SOC_CON14, 1, 15),
> +	[GRF_DPHY_RX0_TURNREQUEST] = PHY_REG(RK3288_GRF_SOC_CON15, 4, 0),
> +	[GRF_DPHY_TX1RX1_TURNREQUEST] = PHY_REG(RK3288_GRF_SOC_CON15, 4, 4),
> +	[GRF_DPHY_TX0_TURNREQUEST] = PHY_REG(RK3288_GRF_SOC_CON15, 3, 8),
> +	[GRF_DVP_V18SEL] = PHY_REG(RK3288_GRF_IO_VSEL, 1, 1),
> +	[GRF_DPHY_RX0_TESTDOUT] = PHY_REG(RK3288_GRF_SOC_STATUS21, 8, 0),
> +};
> 
> Which seems different mask and shifts from rk3399. If you have any ideas in
> how to optimize this I would appreciate it.

It would be tricky indeed :-( Nevermind for now.

> >>>> +};
> >>>> +
> >>>> +struct dphy_drv_data {
> >>>> +	const char * const *clks;
> >>>> +	int num_clks;
> > 
> > This is never negative, you can make it an unsigned int.
> > 
> >>>> +	const struct hsfreq_range *hsfreq_ranges;
> >>>> +	int num_hsfreq_ranges;
> > 
> > Same here.
> > 
> >>>> +	const struct dphy_reg *regs;
> >>>> +};
> >>>> +
> >>>> +struct rockchip_dphy {
> >>>> +	struct device *dev;
> >>>> +	struct regmap *grf;
> >>>> +	const struct dphy_reg *grf_regs;
> >>>> +	struct clk_bulk_data clks[MAX_DPHY_CLK];
> >>>> +
> >>>> +	const struct dphy_drv_data *drv_data;
> >>>> +	struct phy_configure_opts_mipi_dphy config;
> >>>> +};
> >>>> +
> >>>> +static inline void write_grf_reg(struct rockchip_dphy *priv,
> >>>> +				 int index, u8 value)
> > 
> > Maybe unsigned int index ?
> > 
> >>>> +{
> >>>> +	const struct dphy_reg *reg = &priv->grf_regs[index];
> >>>> +	unsigned int val = HIWORD_UPDATE(value, reg->mask, reg->shift);
> >>>> +
> >>>> +	WARN_ON(!reg->offset);
> >>>> +	regmap_write(priv->grf, reg->offset, val);
> >>>> +}
> >>>> +
> >>>> +static void mipidphy0_wr_reg(struct rockchip_dphy *priv,
> >>>> +			     u8 test_code, u8 test_data)
> > 
> > Function (and structure) names have different prefixes, would it make
> > sense to standardise them ? Maybe rockchip_dphy_ ? Or rk_dphy_ for a
> > shorter version ? This could become rk_dphy_write_dphy(), and the
> > previous function rk_dphy_write_grf().
> > 
> >>>> +{
> >>>> +	/*
> >>>> +	 * With the falling edge on TESTCLK, the TESTDIN[7:0] signal content
> >>>> +	 * is latched internally as the current test code. Test data is
> >>>> +	 * programmed internally by rising edge on TESTCLK.
> >>>> +	 */
> > 
> > I've never understood why PHYs tend to have a register named TEST that
> > contains way more than test data :-)
> > 
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_code);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 1);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 0);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTEN, 0);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTDIN, test_data);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> >>>> +}
> >>>> +
> >>>> +/* should be move to power_on */
> > 
> > s/move/moved/
> > 
> > Do you mean merging the two functions together ? What prevents from
> > doing so ? 
> 
> Nothing really, this is a left over command as mipidphy_rx_stream_on() is already
> being called from power_on, and I don't think we should merge it because
> in the future we'll probably going to have mipidphy_txrx_stream_on() for dphy1.

Fine with me, let's just remove the comment then.

> >>>> +static int mipidphy_rx_stream_on(struct rockchip_dphy *priv)
> >>>> +{
> >>>> +	const struct dphy_drv_data *drv_data = priv->drv_data;
> >>>> +	const struct hsfreq_range *hsfreq_ranges = drv_data->hsfreq_ranges;
> >>>> +	struct phy_configure_opts_mipi_dphy *config = &priv->config;
> >>>> +	unsigned int i, hsfreq = 0, data_rate_mbps = config->hs_clk_rate;
> >>>> +	int num_hsfreq_ranges = drv_data->num_hsfreq_ranges;
> >>>> +
> >>>> +	do_div(data_rate_mbps, 1000 * 1000);
> >>>> +
> >>>> +	dev_dbg(priv->dev, "%s: lanes %d - data_rate_mbps %u\n",
> >>>> +		__func__, config->lanes, data_rate_mbps);
> >>>> +
> >>>> +	for (i = 0; i < num_hsfreq_ranges; i++) {
> >>>> +		if (hsfreq_ranges[i].range_h >= data_rate_mbps) {
> >>>> +			hsfreq = hsfreq_ranges[i].cfg_bit;
> >>>> +			break;
> >>>> +		}
> >>>> +	}
> > 
> > As num_hsfreq_ranges and hsfreq_ranges are only used in this loop, I
> > would remove the local variables.
> > 
> >>>> +
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCERXMODE, 0);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_FORCETXSTOPMODE, 0);
> >>>> +
> >>>> +	/* Disable lan turn around, which is ignored in receive mode */
> > 
> > Is it "lan turn around", or "lane turn around" ?
> > 
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNREQUEST, 0);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TURNDISABLE, 0xf);
> >>>> +
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_ENABLE, GENMASK(config->lanes - 1, 0));
> >>>> +
> >>>> +	/* dphy start */
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLK, 1);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 1);
> >>>> +	usleep_range(100, 150);
> >>>> +	write_grf_reg(priv, GRF_DPHY_RX0_TESTCLR, 0);
> >>>> +	usleep_range(100, 150);
> >>>> +
> >>>> +	/* set clock lane */
> >>>> +	/* HS hsfreq_range & lane 0  settle bypass */
> >>>> +	mipidphy0_wr_reg(priv, CLOCK_LANE_HS_RX_CONTROL, 0);
> >>>> +	/* HS RX Control of lane0 */
> >>>> +	mipidphy0_wr_reg(priv, LANE0_HS_RX_CONTROL, hsfreq << 1);
> >>>> +	/* HS RX Control of lane1 */
> >>>> +	mipidphy0_wr_reg(priv, LANE1_HS_RX_CONTROL, 0);
> >>>> +	/* HS RX Control of lane2 */
> >>>> +	mipidphy0_wr_reg(priv, LANE2_HS_RX_CONTROL, 0);
> >>>> +	/* HS RX Control of lane3 */
> >>>> +	mipidphy0_wr_reg(priv, LANE3_HS_RX_CONTROL, 0);
> > 
> > Does this hardcode usage of a single lane ?
> 
> Rockchip seems to uses TEST* registers to set the hsfreqrange.
> It mentions the test code 0x44 (which is LANE0_HS_RX_CONTROL)
> but it doesn't mention the others lanes.
> 
> Replacing those call by
> mipidphy0_wr_reg(priv, LANEx_HS_RX_CONTROL, hsfreq << 1);
> seems to be working.
> 
> I can check if this changes the datarate (I just need to figure a proper
> way to test this or get some docs).
> 
> Thanks for spotting this.

We've discussed this on IRC, it's not clear if the above code is
incorrect or not. Let's add this to a list of open issues.

> >>>> +	/* HS RX Data Lanes Settle State Time Control */
> >>>> +	mipidphy0_wr_reg(priv, HS_RX_DATA_LANES_THS_SETTLE_CONTROL,
> >>>> +			 THS_SETTLE_COUNTER_THRESHOLD);
> >>>> +
> >>>> +	/* Normal operation */
> >>>> +	mipidphy0_wr_reg(priv, 0x0, 0);
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int rockchip_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
> >>>> +{
> >>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >>>> +	int ret;
> >>>> +
> >>>> +	/* pass with phy_mipi_dphy_get_default_config (with pixel rate?) */
> > 
> > I'm not sure to understand what this means.
> 
> iirc, the question is if we should fail when phy_mipi_dphy_config_validate() fails,
> or if we should use a default config.
> 
> Looking at other examples, is seems that only two drivers call
> phy_mipi_dphy_get_default_config() in a totally diferent context, not in mipi path.
> So I guess I would just remove this comment if this is ok with you.

OK.

> >>>> +	ret = phy_mipi_dphy_config_validate(&opts->mipi_dphy);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	memcpy(&priv->config, opts, sizeof(priv->config));
> >>>
> >>> You could to:
> >>>
> >>> 	priv->config = *opts;
> >>>
> >>> Up to you. Some people like memcpy(). :-)
> >>
> >> your way is better thanks!
> >>
> >>>> +
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int rockchip_dphy_power_on(struct phy *phy)
> >>>> +{
> >>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >>>> +	int ret;
> >>>> +
> >>>> +	ret = clk_bulk_enable(priv->drv_data->num_clks, priv->clks);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	return mipidphy_rx_stream_on(priv);
> > 
> > Should you call clk_bulk_disable() if mipidphy_rx_stream_on() fails ?
> > Actually that function never fails, so I'd make it a void function, and
> > return 0 here.
> 
> Ack, I made it void, I'll send it in the next version.
> 
> > What happens if the clock rate is higher than the maximum supported by
> > the PHY ? Shouldn't rockchip_dphy_configure() fail in that case ?
> 
> This is checked in function mipidphy_rx_stream_on(), if it is higher we just
> configure the maximum supported rate. Is this ok?

I think it would be better to reject that in rockchip_dphy_configure()
in order to let the user of the PHY handle the error as early as
possible.

> >>>> +}
> >>>> +
> >>>> +static int rockchip_dphy_power_off(struct phy *phy)
> >>>> +{
> >>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >>>> +
> > 
> > No need to write any register ? That's scary, what will happen on the
> > next power on, when the clocks gets enabled ?
> 
> Just for testing, I hacked the code to only call mipidphy_rx_stream_on() once,
> when streaming for the first time, then I don't call it anymore and starting/stopping
> streaming always works, so I guess it keeps the previous configuration when clocks
> get enabled.
> I wonder if this can be a problem when switching from dphy rx to txrx, but for now
> we just support rx.
> 
> Maybe just calling rk_dphy_write_grf(priv, GRF_DPHY_RX0_ENABLE, 0) is enough.

If that works with disable/enable sequences I think it would be good to
include it.

> >>>> +	clk_bulk_disable(priv->drv_data->num_clks, priv->clks);
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int rockchip_dphy_init(struct phy *phy)
> >>>> +{
> >>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >>>> +	int ret;
> >>>> +
> >>>> +	ret = clk_bulk_prepare(priv->drv_data->num_clks, priv->clks);
> >>>
> >>> return ...;
> >>>
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int rockchip_dphy_exit(struct phy *phy)
> >>>> +{
> >>>> +	struct rockchip_dphy *priv = phy_get_drvdata(phy);
> >>>> +
> >>>> +	clk_bulk_unprepare(priv->drv_data->num_clks, priv->clks);
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static const struct phy_ops rockchip_dphy_ops = {
> >>>> +	.power_on	= rockchip_dphy_power_on,
> >>>> +	.power_off	= rockchip_dphy_power_off,
> >>>> +	.init		= rockchip_dphy_init,
> >>>> +	.exit		= rockchip_dphy_exit,
> >>>> +	.configure	= rockchip_dphy_configure,
> >>>> +	.owner		= THIS_MODULE,
> >>>> +};
> >>>> +
> >>>> +static const struct dphy_drv_data rk3399_mipidphy_drv_data = {
> >>>> +	.clks = rk3399_mipidphy_clks,
> >>>> +	.num_clks = ARRAY_SIZE(rk3399_mipidphy_clks),
> >>>> +	.hsfreq_ranges = rk3399_mipidphy_hsfreq_ranges,
> >>>> +	.num_hsfreq_ranges = ARRAY_SIZE(rk3399_mipidphy_hsfreq_ranges),
> >>>> +	.regs = rk3399_grf_dphy_regs,
> >>>
> >>> Do you expect to support more of the similar PHY(s) --- are there such? If
> >>> not, you could put these in the code that uses them.
> >>
> >> Yes, for rk3288 in the future.
> >>
> >>>> +};
> >>>> +
> >>>> +static const struct of_device_id rockchip_dphy_dt_ids[] = {
> >>>> +	{
> >>>> +		.compatible = "rockchip,rk3399-mipi-dphy",
> >>>> +		.data = &rk3399_mipidphy_drv_data,
> >>>> +	},
> >>>> +	{}
> >>>> +};
> >>>> +MODULE_DEVICE_TABLE(of, rockchip_dphy_dt_ids);
> >>>> +
> >>>> +static int rockchip_dphy_probe(struct platform_device *pdev)
> >>>> +{
> >>>> +	struct device *dev = &pdev->dev;
> >>>> +	struct device_node *np = dev->of_node;
> >>>> +	const struct dphy_drv_data *drv_data;
> >>>> +	struct phy_provider *phy_provider;
> >>>> +	const struct of_device_id *of_id;
> >>>> +	struct rockchip_dphy *priv;
> >>>> +	struct regmap *grf;
> >>>> +	struct phy *phy;
> >>>> +	unsigned int i;
> >>>> +	int ret;
> >>>> +
> >>>> +	if (!dev->parent || !dev->parent->of_node)
> >>>> +		return -ENODEV;
> >>>> +
> >>>> +	if (platform_get_resource(pdev, IORESOURCE_MEM, 0)) {
> >>>> +		dev_err(&pdev->dev, "Rockchip DPHY driver only suports rx\n");
> > 
> > You can replace pdev->dev with dev here and below.
> > 
> > s/rx/RX mode/ ?
> > 
> >>>> +		return -EINVAL;
> >>>> +	}
> >>>> +
> >>>> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> >>>> +	if (!priv)
> >>>> +		return -ENOMEM;
> >>>> +	priv->dev = dev;
> >>>> +
> >>>> +	grf = syscon_node_to_regmap(dev->parent->of_node);
> >>>> +	if (IS_ERR(grf)) {
> >>>> +		grf = syscon_regmap_lookup_by_phandle(dev->of_node,
> >>>> +						      "rockchip,grf");
> >>>> +		if (IS_ERR(grf)) {
> >>>> +			dev_err(dev, "Can't find GRF syscon\n");
> >>>> +			return -ENODEV;
> >>>> +		}
> >>>> +	}
> >>>> +	priv->grf = grf;
> >>>> +
> >>>> +	of_id = of_match_device(rockchip_dphy_dt_ids, dev);
> >>>> +	if (!of_id)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	drv_data = of_id->data;
> >>>> +	priv->grf_regs = drv_data->regs;
> > 
> > Do you have to store grf_regs in priv, or could it be accessed through
> > priv->drv_data->regs ?
> > 
> >>>> +	priv->drv_data = drv_data;
> >>>> +	for (i = 0; i < drv_data->num_clks; i++)
> >>>> +		priv->clks[i].id = drv_data->clks[i];
> >>>> +	ret = devm_clk_bulk_get(&pdev->dev, drv_data->num_clks, priv->clks);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	phy = devm_phy_create(dev, np, &rockchip_dphy_ops);
> >>>> +	if (IS_ERR(phy)) {
> >>>> +		dev_err(dev, "failed to create phy\n");
> >>>> +		return PTR_ERR(phy);
> >>>> +	}
> >>>> +	phy_set_drvdata(phy, priv);
> >>>> +
> >>>> +	phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
> >>>> +
> >>>> +	return PTR_ERR_OR_ZERO(phy_provider);
> >>>> +}
> >>>> +
> >>>> +static struct platform_driver rockchip_dphy_driver = {
> >>>> +	.probe = rockchip_dphy_probe,
> >>>> +	.driver = {
> >>>> +		.name	= "rockchip-mipi-dphy",
> >>>> +		.of_match_table = rockchip_dphy_dt_ids,
> >>>> +	},
> >>>> +};
> >>>> +module_platform_driver(rockchip_dphy_driver);
> >>>> +
> >>>> +MODULE_AUTHOR("Ezequiel Garcia <ezequiel@collabora.com>");
> >>>> +MODULE_DESCRIPTION("Rockchip MIPI Synopsys DPHY driver");
> >>>> +MODULE_LICENSE("Dual MIT/GPL");
> > 
> > Overall this is quite good, there are only small issues.
> 
> Thank you a lot for your review

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-15 13:17         ` Sakari Ailus
@ 2020-01-31 19:38           ` Dafna Hirschfeld
  2020-02-12 21:13             ` Sakari Ailus
  0 siblings, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-01-31 19:38 UTC (permalink / raw)
  To: Sakari Ailus, Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	Dafna Hirschfeld, jeffy.chen, zyc, linux-kernel, tfiga,
	linux-rockchip, Allon Huang, Jacob Chen, hans.verkuil,
	laurent.pinchart, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media

Hi,
I (Dafna Hirschfeld) will work in following months with Helen Koike to fix the issues
in the TODO file of this driver: drivers/staging/media/rkisp1/TODO

On 15.08.19 15:17, Sakari Ailus wrote:
> On Thu, Aug 15, 2019 at 11:24:22AM +0300, Sakari Ailus wrote:
>> Hi Helen,
>>
>> On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:
>>
>> ...
>>
>>>>> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
>>>>> +				 struct v4l2_subdev_pad_config *cfg,
>>>>> +				 struct v4l2_subdev_format *fmt)
>>>>> +{
>>>>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>>>>> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>>>>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>>>>> +
>>>>
>>>> Note that for sub-device nodes, the driver is itself responsible for
>>>> serialising the access to its data structures.
>>>
>>> But looking at subdev_do_ioctl_lock(), it seems that it serializes the
>>> ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
>>> most probably) ?
>>
>> Good question. I had missed this change --- subdev_do_ioctl_lock() is
>> relatively new. But setting that lock is still not possible as the struct

'the struct' - do you mean the 'vdev' struct allocated in
'v4l2_device_register_subdev_nodes' ?

>> is allocated in the framework and the device is registered before the

>> driver gets hold of it. It's a good idea to provide the same serialisation
>> for subdevs as well.
>>
>> I'll get back to this later.
> 
> The main reason is actually that these ops are also called through the
> sub-device kAPI, not only through the uAPI, and the locks are only taken
> through the calls via uAPI.

actually it seems that although 'subdev_do_ioctl_lock' exit, I wonder if
any subdevice uses that vdev->lock in  subdev_do_ioctl_lock.
It is not initialized in v4l2_device_register_subdev_nodes where the vdev is allocated
and I wonder if any subdevice actually initialize it somewhere else. For example it is null in this
driver and in vimc.

> 
> So adding the locks to uAPI calls alone would not address the issue.

What I can do is add a mutex to every struct of a subdevice and lock it
at the beginning of each subdevice operation.
Is this an acceptable solution?

Thanks,
Dafna
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-01-31 19:38           ` Dafna Hirschfeld
@ 2020-02-12 21:13             ` Sakari Ailus
  2020-02-13 12:50               ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Sakari Ailus @ 2020-02-12 21:13 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	Dafna Hirschfeld, jeffy.chen, zyc, linux-kernel, tfiga,
	Allon Huang, linux-rockchip, Helen Koike, Jacob Chen,
	hans.verkuil, laurent.pinchart, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media

Hi Dafna,

Apologies for the late reply. I learned the mail had got lost due to mail
server issues.

On Fri, Jan 31, 2020 at 08:38:34PM +0100, Dafna Hirschfeld wrote:
> Hi,
> I (Dafna Hirschfeld) will work in following months with Helen Koike to fix the issues
> in the TODO file of this driver: drivers/staging/media/rkisp1/TODO
> 
> On 15.08.19 15:17, Sakari Ailus wrote:
> > On Thu, Aug 15, 2019 at 11:24:22AM +0300, Sakari Ailus wrote:
> > > Hi Helen,
> > > 
> > > On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:
> > > 
> > > ...
> > > 
> > > > > > +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> > > > > > +				 struct v4l2_subdev_pad_config *cfg,
> > > > > > +				 struct v4l2_subdev_format *fmt)
> > > > > > +{
> > > > > > +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > > > > +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> > > > > > +	struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > > > > +
> > > > > 
> > > > > Note that for sub-device nodes, the driver is itself responsible for
> > > > > serialising the access to its data structures.
> > > > 
> > > > But looking at subdev_do_ioctl_lock(), it seems that it serializes the
> > > > ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
> > > > most probably) ?
> > > 
> > > Good question. I had missed this change --- subdev_do_ioctl_lock() is
> > > relatively new. But setting that lock is still not possible as the struct
> 
> 'the struct' - do you mean the 'vdev' struct allocated in
> 'v4l2_device_register_subdev_nodes' ?

Yes.

> 
> > > is allocated in the framework and the device is registered before the
> 
> > > driver gets hold of it. It's a good idea to provide the same serialisation
> > > for subdevs as well.
> > > 
> > > I'll get back to this later.
> > 
> > The main reason is actually that these ops are also called through the
> > sub-device kAPI, not only through the uAPI, and the locks are only taken
> > through the calls via uAPI.
> 
> actually it seems that although 'subdev_do_ioctl_lock' exit, I wonder if
> any subdevice uses that vdev->lock in  subdev_do_ioctl_lock.
> It is not initialized in v4l2_device_register_subdev_nodes where the vdev is allocated
> and I wonder if any subdevice actually initialize it somewhere else. For example it is null in this
> driver and in vimc.

It needs to be set before the video device is registered, so indeed, it
seems no driver can make use it.

> 
> > 
> > So adding the locks to uAPI calls alone would not address the issue.
> 
> What I can do is add a mutex to every struct of a subdevice and lock it
> at the beginning of each subdevice operation.
> Is this an acceptable solution?

Please do. That's what other drivers do at the moment as well.

-- 
Kind regards,

Sakari Ailus

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-02-12 21:13             ` Sakari Ailus
@ 2020-02-13 12:50               ` Dafna Hirschfeld
  0 siblings, 0 replies; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-02-13 12:50 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	Dafna Hirschfeld, jeffy.chen, zyc, linux-kernel, tfiga,
	Allon Huang, linux-rockchip, Helen Koike, Jacob Chen,
	hans.verkuil, laurent.pinchart, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media



On 12.02.20 22:13, Sakari Ailus wrote:
> Hi Dafna,
> 
> Apologies for the late reply. I learned the mail had got lost due to mail
> server issues.
> 
np:)

> On Fri, Jan 31, 2020 at 08:38:34PM +0100, Dafna Hirschfeld wrote:
>> Hi,
>> I (Dafna Hirschfeld) will work in following months with Helen Koike to fix the issues
>> in the TODO file of this driver: drivers/staging/media/rkisp1/TODO
>>
>> On 15.08.19 15:17, Sakari Ailus wrote:
>>> On Thu, Aug 15, 2019 at 11:24:22AM +0300, Sakari Ailus wrote:
>>>> Hi Helen,
>>>>
>>>> On Wed, Aug 14, 2019 at 09:58:05PM -0300, Helen Koike wrote:
>>>>
>>>> ...
>>>>
>>>>>>> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
>>>>>>> +				 struct v4l2_subdev_pad_config *cfg,
>>>>>>> +				 struct v4l2_subdev_format *fmt)
>>>>>>> +{
>>>>>>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>>>>>>> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>>>>>>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>>>>>>> +
>>>>>>
>>>>>> Note that for sub-device nodes, the driver is itself responsible for
>>>>>> serialising the access to its data structures.
>>>>>
>>>>> But looking at subdev_do_ioctl_lock(), it seems that it serializes the
>>>>> ioctl calls for subdevs, no? Or I'm misunderstanding something (which is
>>>>> most probably) ?
>>>>
>>>> Good question. I had missed this change --- subdev_do_ioctl_lock() is
>>>> relatively new. But setting that lock is still not possible as the struct
>>
>> 'the struct' - do you mean the 'vdev' struct allocated in
>> 'v4l2_device_register_subdev_nodes' ?
> 
> Yes.
> 
>>
>>>> is allocated in the framework and the device is registered before the
>>
>>>> driver gets hold of it. It's a good idea to provide the same serialisation
>>>> for subdevs as well.
>>>>
>>>> I'll get back to this later.
>>>
>>> The main reason is actually that these ops are also called through the
>>> sub-device kAPI, not only through the uAPI, and the locks are only taken
>>> through the calls via uAPI.
>>
>> actually it seems that although 'subdev_do_ioctl_lock' exit, I wonder if
>> any subdevice uses that vdev->lock in  subdev_do_ioctl_lock.
>> It is not initialized in v4l2_device_register_subdev_nodes where the vdev is allocated
>> and I wonder if any subdevice actually initialize it somewhere else. For example it is null in this
>> driver and in vimc.
> 
> It needs to be set before the video device is registered, so indeed, it
> seems no driver can make use it.
> 
>>
>>>
>>> So adding the locks to uAPI calls alone would not address the issue.
>>
>> What I can do is add a mutex to every struct of a subdevice and lock it
>> at the beginning of each subdevice operation.
>> Is this an acceptable solution?
> 
> Please do. That's what other drivers do at the moment as well.
> 
I already sent it as the patchset "media: staging: rkisp1: add serialization to the isp and resizer ops"

Thanks,
Dafna

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-15 19:35       ` Laurent Pinchart
@ 2020-03-25  6:34         ` Dafna Hirschfeld
  2020-03-25  7:11           ` Laurent Pinchart
  0 siblings, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-03-25  6:34 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Tomasz Figa, Hans Verkuil,
	linux-rockchip, Helen Koike, Jacob Chen, hans.verkuil,
	sakari.ailus, zhengsq, Allon Huang, mchehab, Ezequiel Garcia,
	linux-arm-kernel, Linux Media Mailing List

On Thu, Aug 15, 2019 at 10:17 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Hans,
>
> On Wed, Aug 07, 2019 at 12:39:17PM +0200, Hans Verkuil wrote:
> > On 8/6/19 8:51 PM, Helen Koike wrote:
> > > On 7/30/19 3:42 PM, Helen Koike wrote:
> > >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> > >>
> > >> Add the subdev driver for rockchip isp1.
> > >>
> > >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> > >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> > >> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> > >> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> > >> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> > >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> > >> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> > >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > >> [fixed unknown entity type / switched to PIXEL_RATE]
> > >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > >> [update for upstream]
> > >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > >>
> > >> ---
> > >>
> > >> Changes in v8: None
> > >> Changes in v7:
> > >> - fixed warning because of unknown entity type
> > >> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> > >> and default values
> > >> - fix typo riksp1/rkisp1
> > >> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> > >> isp subdevice in the media topology now. As a consequence, remove the
> > >> hack in mipidphy_g_mbus_config() where information from the sensor was
> > >> being propagated through the topology.
> > >> - From the old dphy:
> > >>         * cache get_remote_sensor() in s_stream
> > >>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> > >> - Replace stream state with a boolean
> > >> - code styling and checkpatch fixes
> > >> - fix stop_stream (return after calling stop, do not reenable the stream)
> > >> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> > >> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> > >> - s/intput/input
> > >> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> > >> reused by the capture
> > >>
> > >>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
> > >>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
> > >>  2 files changed, 1397 insertions(+)
> > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
> > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> > >>
> > >> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > >> new file mode 100644
> > >> index 000000000000..6d0c0ffb5e03
> > >> --- /dev/null
> > >> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > >> @@ -0,0 +1,1286 @@
> >
> > <snip>
> >
> > >> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> > >> +                           struct v4l2_subdev_pad_config *cfg,
> > >> +                           struct v4l2_subdev_format *fmt)
> > >> +{
> > >> +  struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > >> +  struct v4l2_mbus_framefmt *mf = &fmt->format;
> > >> +
> > >> +  if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> > >> +      (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> > >> +          fmt->format.code = MEDIA_BUS_FMT_FIXED;
> > >> +          /*
> > >> +           * NOTE: setting a format here doesn't make much sense
> > >> +           * but v4l2-compliance complains
> > >> +           */
> > >> +          fmt->format.width = RKISP1_DEFAULT_WIDTH;
> > >> +          fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> > >
> > > As I had mentioned to you, this is called for the isp pads connected to the
> > > DMA engines for statistics and parameters (meta data).
> > >
> > > If I remove those, I get the following errors:
> > >
> > > Sub-Device ioctls (Sink Pad 1):
> > >         test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > >         test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
> > >         test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > >         test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > >         test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
> > >         test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > >         test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
> > >
> > > Here is the full log: http://ix.io/1QNt
> > >
> > > Is this a bug in v4l2-compliance?
> >
> > Yes and no :-)
> >
> > Currently v4l2-compliance assumes that only video is transferred over a media bus.
> > But that's not the case here, and testing the code field doesn't help v4l2-compliance
> > since MEDIA_BUS_FMT_FIXED is also still used by some older subdev drivers for video.
> >
> > I think we need a new bus format: MEDIA_BUS_FMT_FIXED_METADATA. Then v4l2-compliance
> > can tell it apart from the regular fixed video bus format.
>
> Wouldn't a pad flag that identifies the type of data transmitted by a
> pad be a better, backward-compatible option ? This could be useful for
> audio as well.

Hi,
Can you explain what pad flag do you mean?
Do you mean adding a flag in the 'MEDIA_LNK_FL_*' list ?
Also, some valid value should be set to  'fmt->format.code' so with
the flags solution
that you suggest it should stay  MEDIA_BUS_FMT_FIXED ?

thanks,
Dafna
>
> > If I do a 'git grep MEDIA_BUS_FMT_FIXED' then I see that it is also in use by vsp1
> > for histogram information, so that should also be converted to use the new FIXED_METADATA
> > format, although that might be too late (there might be userspace complications).
>
> Yes, probably not a good idea.
>
> > >> +          fmt->format.field = V4L2_FIELD_NONE;
> > >> +          return 0;
> > >> +  }
> > >> +
> > >> +  if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> > >> +          mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> > >> +          fmt->format = *mf;
> > >> +          return 0;
> > >> +  }
> > >> +
> > >> +  if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> > >> +          *mf = isp_sd->in_frm;
> > >> +  } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > >> +          /* format of source pad */
> > >> +          *mf = isp_sd->in_frm;
> > >> +          mf->code = isp_sd->out_fmt.mbus_code;
> > >> +          /* window size of source pad */
> > >> +          mf->width = isp_sd->out_crop.width;
> > >> +          mf->height = isp_sd->out_crop.height;
> > >> +          mf->quantization = isp_sd->quantization;
> > >> +  }
> > >> +  mf->field = V4L2_FIELD_NONE;
> > >> +
> > >> +  return 0;
> > >> +}
>
> --
> Regards,
>
> Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-03-25  6:34         ` Dafna Hirschfeld
@ 2020-03-25  7:11           ` Laurent Pinchart
  2020-03-25  8:51             ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Laurent Pinchart @ 2020-03-25  7:11 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Tomasz Figa, Hans Verkuil,
	linux-rockchip, Helen Koike, Jacob Chen, hans.verkuil,
	sakari.ailus, zhengsq, Allon Huang, mchehab, Ezequiel Garcia,
	linux-arm-kernel, Linux Media Mailing List

Hi Dafna,

On Wed, Mar 25, 2020 at 07:34:37AM +0100, Dafna Hirschfeld wrote:
> On Thu, Aug 15, 2019 at 10:17 PM Laurent Pinchart wrote:
> > On Wed, Aug 07, 2019 at 12:39:17PM +0200, Hans Verkuil wrote:
> > > On 8/6/19 8:51 PM, Helen Koike wrote:
> > > > On 7/30/19 3:42 PM, Helen Koike wrote:
> > > >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> > > >>
> > > >> Add the subdev driver for rockchip isp1.
> > > >>
> > > >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> > > >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> > > >> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> > > >> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> > > >> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> > > >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> > > >> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> > > >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > > >> [fixed unknown entity type / switched to PIXEL_RATE]
> > > >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > >> [update for upstream]
> > > >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > >>
> > > >> ---
> > > >>
> > > >> Changes in v8: None
> > > >> Changes in v7:
> > > >> - fixed warning because of unknown entity type
> > > >> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> > > >> and default values
> > > >> - fix typo riksp1/rkisp1
> > > >> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> > > >> isp subdevice in the media topology now. As a consequence, remove the
> > > >> hack in mipidphy_g_mbus_config() where information from the sensor was
> > > >> being propagated through the topology.
> > > >> - From the old dphy:
> > > >>         * cache get_remote_sensor() in s_stream
> > > >>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> > > >> - Replace stream state with a boolean
> > > >> - code styling and checkpatch fixes
> > > >> - fix stop_stream (return after calling stop, do not reenable the stream)
> > > >> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> > > >> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> > > >> - s/intput/input
> > > >> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> > > >> reused by the capture
> > > >>
> > > >>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
> > > >>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
> > > >>  2 files changed, 1397 insertions(+)
> > > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
> > > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> > > >>
> > > >> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > >> new file mode 100644
> > > >> index 000000000000..6d0c0ffb5e03
> > > >> --- /dev/null
> > > >> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > >> @@ -0,0 +1,1286 @@
> > >
> > > <snip>
> > >
> > > >> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> > > >> +                           struct v4l2_subdev_pad_config *cfg,
> > > >> +                           struct v4l2_subdev_format *fmt)
> > > >> +{
> > > >> +  struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > >> +  struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > >> +
> > > >> +  if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> > > >> +      (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> > > >> +          fmt->format.code = MEDIA_BUS_FMT_FIXED;
> > > >> +          /*
> > > >> +           * NOTE: setting a format here doesn't make much sense
> > > >> +           * but v4l2-compliance complains
> > > >> +           */
> > > >> +          fmt->format.width = RKISP1_DEFAULT_WIDTH;
> > > >> +          fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> > > >
> > > > As I had mentioned to you, this is called for the isp pads connected to the
> > > > DMA engines for statistics and parameters (meta data).
> > > >
> > > > If I remove those, I get the following errors:
> > > >
> > > > Sub-Device ioctls (Sink Pad 1):
> > > >         test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > > >         test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
> > > >         test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > > >         test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > > >         test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
> > > >         test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > > >         test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
> > > >
> > > > Here is the full log: http://ix.io/1QNt
> > > >
> > > > Is this a bug in v4l2-compliance?
> > >
> > > Yes and no :-)
> > >
> > > Currently v4l2-compliance assumes that only video is transferred over a media bus.
> > > But that's not the case here, and testing the code field doesn't help v4l2-compliance
> > > since MEDIA_BUS_FMT_FIXED is also still used by some older subdev drivers for video.
> > >
> > > I think we need a new bus format: MEDIA_BUS_FMT_FIXED_METADATA. Then v4l2-compliance
> > > can tell it apart from the regular fixed video bus format.
> >
> > Wouldn't a pad flag that identifies the type of data transmitted by a
> > pad be a better, backward-compatible option ? This could be useful for
> > audio as well.
> 
> Hi,
> Can you explain what pad flag do you mean?
> Do you mean adding a flag in the 'MEDIA_LNK_FL_*' list ?

I meant MEDIA_PAD_FL_*. We could reserve a few bits in
media_pad_desc.flags to tell what type of data is being transported.

> Also, some valid value should be set to  'fmt->format.code' so with
> the flags solution
> that you suggest it should stay  MEDIA_BUS_FMT_FIXED ?

Correct.

> > > If I do a 'git grep MEDIA_BUS_FMT_FIXED' then I see that it is also in use by vsp1
> > > for histogram information, so that should also be converted to use the new FIXED_METADATA
> > > format, although that might be too late (there might be userspace complications).
> >
> > Yes, probably not a good idea.
> >
> > > >> +          fmt->format.field = V4L2_FIELD_NONE;
> > > >> +          return 0;
> > > >> +  }
> > > >> +
> > > >> +  if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > >> +          mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> > > >> +          fmt->format = *mf;
> > > >> +          return 0;
> > > >> +  }
> > > >> +
> > > >> +  if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> > > >> +          *mf = isp_sd->in_frm;
> > > >> +  } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > > >> +          /* format of source pad */
> > > >> +          *mf = isp_sd->in_frm;
> > > >> +          mf->code = isp_sd->out_fmt.mbus_code;
> > > >> +          /* window size of source pad */
> > > >> +          mf->width = isp_sd->out_crop.width;
> > > >> +          mf->height = isp_sd->out_crop.height;
> > > >> +          mf->quantization = isp_sd->quantization;
> > > >> +  }
> > > >> +  mf->field = V4L2_FIELD_NONE;
> > > >> +
> > > >> +  return 0;
> > > >> +}

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-03-25  7:11           ` Laurent Pinchart
@ 2020-03-25  8:51             ` Dafna Hirschfeld
  2020-03-25  9:11               ` Laurent Pinchart
  0 siblings, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-03-25  8:51 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Tomasz Figa, Hans Verkuil,
	linux-rockchip, Helen Koike, Jacob Chen, hans.verkuil,
	sakari.ailus, zhengsq, Allon Huang, mchehab, Ezequiel Garcia,
	linux-arm-kernel, Linux Media Mailing List

On Wed, Mar 25, 2020 at 8:12 AM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dafna,
>
> On Wed, Mar 25, 2020 at 07:34:37AM +0100, Dafna Hirschfeld wrote:
> > On Thu, Aug 15, 2019 at 10:17 PM Laurent Pinchart wrote:
> > > On Wed, Aug 07, 2019 at 12:39:17PM +0200, Hans Verkuil wrote:
> > > > On 8/6/19 8:51 PM, Helen Koike wrote:
> > > > > On 7/30/19 3:42 PM, Helen Koike wrote:
> > > > >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> > > > >>
> > > > >> Add the subdev driver for rockchip isp1.
> > > > >>
> > > > >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> > > > >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> > > > >> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> > > > >> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> > > > >> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> > > > >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> > > > >> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> > > > >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > > > >> [fixed unknown entity type / switched to PIXEL_RATE]
> > > > >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > >> [update for upstream]
> > > > >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > > >>
> > > > >> ---
> > > > >>
> > > > >> Changes in v8: None
> > > > >> Changes in v7:
> > > > >> - fixed warning because of unknown entity type
> > > > >> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> > > > >> and default values
> > > > >> - fix typo riksp1/rkisp1
> > > > >> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> > > > >> isp subdevice in the media topology now. As a consequence, remove the
> > > > >> hack in mipidphy_g_mbus_config() where information from the sensor was
> > > > >> being propagated through the topology.
> > > > >> - From the old dphy:
> > > > >>         * cache get_remote_sensor() in s_stream
> > > > >>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> > > > >> - Replace stream state with a boolean
> > > > >> - code styling and checkpatch fixes
> > > > >> - fix stop_stream (return after calling stop, do not reenable the stream)
> > > > >> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> > > > >> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> > > > >> - s/intput/input
> > > > >> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> > > > >> reused by the capture
> > > > >>
> > > > >>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
> > > > >>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
> > > > >>  2 files changed, 1397 insertions(+)
> > > > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> > > > >>
> > > > >> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > >> new file mode 100644
> > > > >> index 000000000000..6d0c0ffb5e03
> > > > >> --- /dev/null
> > > > >> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > >> @@ -0,0 +1,1286 @@
> > > >
> > > > <snip>
> > > >
> > > > >> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> > > > >> +                           struct v4l2_subdev_pad_config *cfg,
> > > > >> +                           struct v4l2_subdev_format *fmt)
> > > > >> +{
> > > > >> +  struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > > >> +  struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > > >> +
> > > > >> +  if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> > > > >> +      (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> > > > >> +          fmt->format.code = MEDIA_BUS_FMT_FIXED;
> > > > >> +          /*
> > > > >> +           * NOTE: setting a format here doesn't make much sense
> > > > >> +           * but v4l2-compliance complains
> > > > >> +           */
> > > > >> +          fmt->format.width = RKISP1_DEFAULT_WIDTH;
> > > > >> +          fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> > > > >
> > > > > As I had mentioned to you, this is called for the isp pads connected to the
> > > > > DMA engines for statistics and parameters (meta data).
> > > > >
> > > > > If I remove those, I get the following errors:
> > > > >
> > > > > Sub-Device ioctls (Sink Pad 1):
> > > > >         test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > > > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > > > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > > > >         test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
> > > > >         test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > > > >         test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > > > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > > > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > > > >         test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
> > > > >         test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > > > >         test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
> > > > >
> > > > > Here is the full log: http://ix.io/1QNt
> > > > >
> > > > > Is this a bug in v4l2-compliance?
> > > >
> > > > Yes and no :-)
> > > >
> > > > Currently v4l2-compliance assumes that only video is transferred over a media bus.
> > > > But that's not the case here, and testing the code field doesn't help v4l2-compliance
> > > > since MEDIA_BUS_FMT_FIXED is also still used by some older subdev drivers for video.
> > > >
> > > > I think we need a new bus format: MEDIA_BUS_FMT_FIXED_METADATA. Then v4l2-compliance
> > > > can tell it apart from the regular fixed video bus format.
> > >
> > > Wouldn't a pad flag that identifies the type of data transmitted by a
> > > pad be a better, backward-compatible option ? This could be useful for
> > > audio as well.
> >
> > Hi,
> > Can you explain what pad flag do you mean?
> > Do you mean adding a flag in the 'MEDIA_LNK_FL_*' list ?
>
> I meant MEDIA_PAD_FL_*. We could reserve a few bits in
> media_pad_desc.flags to tell what type of data is being transported.
>
So the idea is that when the MBUS format is MEDIA_BUS_FMT_FIXED,
then userspace should look at the flags of the pad to see what format is it?
So if I add a flag  'MEDIA_PAD_FL_METADATA' it knows it is a metadata ?
What makes is more backward-comaptible ?

> > Also, some valid value should be set to  'fmt->format.code' so with
> > the flags solution
> > that you suggest it should stay  MEDIA_BUS_FMT_FIXED ?
>
> Correct.
>
> > > > If I do a 'git grep MEDIA_BUS_FMT_FIXED' then I see that it is also in use by vsp1
> > > > for histogram information, so that should also be converted to use the new FIXED_METADATA
> > > > format, although that might be too late (there might be userspace complications).
> > >
> > > Yes, probably not a good idea.
> > >
> > > > >> +          fmt->format.field = V4L2_FIELD_NONE;
> > > > >> +          return 0;
> > > > >> +  }
> > > > >> +
> > > > >> +  if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > > >> +          mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> > > > >> +          fmt->format = *mf;
> > > > >> +          return 0;
> > > > >> +  }
> > > > >> +
> > > > >> +  if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> > > > >> +          *mf = isp_sd->in_frm;
> > > > >> +  } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > > > >> +          /* format of source pad */
> > > > >> +          *mf = isp_sd->in_frm;
> > > > >> +          mf->code = isp_sd->out_fmt.mbus_code;
> > > > >> +          /* window size of source pad */
> > > > >> +          mf->width = isp_sd->out_crop.width;
> > > > >> +          mf->height = isp_sd->out_crop.height;
> > > > >> +          mf->quantization = isp_sd->quantization;
> > > > >> +  }
> > > > >> +  mf->field = V4L2_FIELD_NONE;
> > > > >> +
> > > > >> +  return 0;
> > > > >> +}
>
> --
> Regards,
>
> Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-03-25  8:51             ` Dafna Hirschfeld
@ 2020-03-25  9:11               ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2020-03-25  9:11 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Tomasz Figa, Hans Verkuil,
	linux-rockchip, Helen Koike, Jacob Chen, hans.verkuil,
	sakari.ailus, zhengsq, Allon Huang, mchehab, Ezequiel Garcia,
	linux-arm-kernel, Linux Media Mailing List

Hi Dafna,

On Wed, Mar 25, 2020 at 09:51:51AM +0100, Dafna Hirschfeld wrote:
> On Wed, Mar 25, 2020 at 8:12 AM Laurent Pinchart wrote:
> > On Wed, Mar 25, 2020 at 07:34:37AM +0100, Dafna Hirschfeld wrote:
> > > On Thu, Aug 15, 2019 at 10:17 PM Laurent Pinchart wrote:
> > > > On Wed, Aug 07, 2019 at 12:39:17PM +0200, Hans Verkuil wrote:
> > > > > On 8/6/19 8:51 PM, Helen Koike wrote:
> > > > > > On 7/30/19 3:42 PM, Helen Koike wrote:
> > > > > >> From: Jacob Chen <jacob2.chen@rock-chips.com>
> > > > > >>
> > > > > >> Add the subdev driver for rockchip isp1.
> > > > > >>
> > > > > >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> > > > > >> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> > > > > >> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> > > > > >> Signed-off-by: Jacob Chen <cc@rock-chips.com>
> > > > > >> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> > > > > >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> > > > > >> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> > > > > >> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > > > > >> [fixed unknown entity type / switched to PIXEL_RATE]
> > > > > >> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > > >> [update for upstream]
> > > > > >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > > > >>
> > > > > >> ---
> > > > > >>
> > > > > >> Changes in v8: None
> > > > > >> Changes in v7:
> > > > > >> - fixed warning because of unknown entity type
> > > > > >> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> > > > > >> and default values
> > > > > >> - fix typo riksp1/rkisp1
> > > > > >> - redesign: remove mipi/csi subdevice, sensors connect directly to the
> > > > > >> isp subdevice in the media topology now. As a consequence, remove the
> > > > > >> hack in mipidphy_g_mbus_config() where information from the sensor was
> > > > > >> being propagated through the topology.
> > > > > >> - From the old dphy:
> > > > > >>         * cache get_remote_sensor() in s_stream
> > > > > >>         * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> > > > > >> - Replace stream state with a boolean
> > > > > >> - code styling and checkpatch fixes
> > > > > >> - fix stop_stream (return after calling stop, do not reenable the stream)
> > > > > >> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> > > > > >> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> > > > > >> - s/intput/input
> > > > > >> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> > > > > >> reused by the capture
> > > > > >>
> > > > > >>  drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
> > > > > >>  drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
> > > > > >>  2 files changed, 1397 insertions(+)
> > > > > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > > >>  create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> > > > > >>
> > > > > >> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > > >> new file mode 100644
> > > > > >> index 000000000000..6d0c0ffb5e03
> > > > > >> --- /dev/null
> > > > > >> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > > >> @@ -0,0 +1,1286 @@
> > > > >
> > > > > <snip>
> > > > >
> > > > > >> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> > > > > >> +                           struct v4l2_subdev_pad_config *cfg,
> > > > > >> +                           struct v4l2_subdev_format *fmt)
> > > > > >> +{
> > > > > >> +  struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > > > >> +  struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > > > >> +
> > > > > >> +  if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> > > > > >> +      (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> > > > > >> +          fmt->format.code = MEDIA_BUS_FMT_FIXED;
> > > > > >> +          /*
> > > > > >> +           * NOTE: setting a format here doesn't make much sense
> > > > > >> +           * but v4l2-compliance complains
> > > > > >> +           */
> > > > > >> +          fmt->format.width = RKISP1_DEFAULT_WIDTH;
> > > > > >> +          fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> > > > > >
> > > > > > As I had mentioned to you, this is called for the isp pads connected to the
> > > > > > DMA engines for statistics and parameters (meta data).
> > > > > >
> > > > > > If I remove those, I get the following errors:
> > > > > >
> > > > > > Sub-Device ioctls (Sink Pad 1):
> > > > > >         test Try VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > > > > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > > > > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > > > > >         test Try VIDIOC_SUBDEV_G/S_FMT: FAIL
> > > > > >         test Try VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > > > > >         test Active VIDIOC_SUBDEV_ENUM_MBUS_CODE/FRAME_SIZE/FRAME_INTERVAL: OK
> > > > > >                 fail: v4l2-test-subdevs.cpp(311): fmt.width == 0 || fmt.width > 65536
> > > > > >                 fail: v4l2-test-subdevs.cpp(356): checkMBusFrameFmt(node, fmt.format)
> > > > > >         test Active VIDIOC_SUBDEV_G/S_FMT: FAIL
> > > > > >         test Active VIDIOC_SUBDEV_G/S_SELECTION/CROP: OK
> > > > > >         test VIDIOC_SUBDEV_G/S_FRAME_INTERVAL: OK (Not Supported)
> > > > > >
> > > > > > Here is the full log: http://ix.io/1QNt
> > > > > >
> > > > > > Is this a bug in v4l2-compliance?
> > > > >
> > > > > Yes and no :-)
> > > > >
> > > > > Currently v4l2-compliance assumes that only video is transferred over a media bus.
> > > > > But that's not the case here, and testing the code field doesn't help v4l2-compliance
> > > > > since MEDIA_BUS_FMT_FIXED is also still used by some older subdev drivers for video.
> > > > >
> > > > > I think we need a new bus format: MEDIA_BUS_FMT_FIXED_METADATA. Then v4l2-compliance
> > > > > can tell it apart from the regular fixed video bus format.
> > > >
> > > > Wouldn't a pad flag that identifies the type of data transmitted by a
> > > > pad be a better, backward-compatible option ? This could be useful for
> > > > audio as well.
> > >
> > > Hi,
> > > Can you explain what pad flag do you mean?
> > > Do you mean adding a flag in the 'MEDIA_LNK_FL_*' list ?
> >
> > I meant MEDIA_PAD_FL_*. We could reserve a few bits in
> > media_pad_desc.flags to tell what type of data is being transported.
> >
> So the idea is that when the MBUS format is MEDIA_BUS_FMT_FIXED,
> then userspace should look at the flags of the pad to see what format is it?
> So if I add a flag  'MEDIA_PAD_FL_METADATA' it knows it is a metadata ?

Not just when it's MEDIA_BUS_FMT_FIXED. Userspace can look at the pad
flags to determine the type of data carried by the pad. This can, for
instance, allow userspace to walk graphs to find video data paths
without going down metadata or audio links.

> What makes is more backward-comaptible ?

We won't need to change the current format used on those pads
(MEDIA_BUS_FMT_FIXED), which would break userspace that relies on that
format. Of course, if it's just for the rkisp1 driver, we don't care
much about breakages as the driver is in staging :-)

> > > Also, some valid value should be set to  'fmt->format.code' so with
> > > the flags solution
> > > that you suggest it should stay  MEDIA_BUS_FMT_FIXED ?
> >
> > Correct.
> >
> > > > > If I do a 'git grep MEDIA_BUS_FMT_FIXED' then I see that it is also in use by vsp1
> > > > > for histogram information, so that should also be converted to use the new FIXED_METADATA
> > > > > format, although that might be too late (there might be userspace complications).
> > > >
> > > > Yes, probably not a good idea.
> > > >
> > > > > >> +          fmt->format.field = V4L2_FIELD_NONE;
> > > > > >> +          return 0;
> > > > > >> +  }
> > > > > >> +
> > > > > >> +  if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > > > >> +          mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> > > > > >> +          fmt->format = *mf;
> > > > > >> +          return 0;
> > > > > >> +  }
> > > > > >> +
> > > > > >> +  if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> > > > > >> +          *mf = isp_sd->in_frm;
> > > > > >> +  } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > > > > >> +          /* format of source pad */
> > > > > >> +          *mf = isp_sd->in_frm;
> > > > > >> +          mf->code = isp_sd->out_fmt.mbus_code;
> > > > > >> +          /* window size of source pad */
> > > > > >> +          mf->width = isp_sd->out_crop.width;
> > > > > >> +          mf->height = isp_sd->out_crop.height;
> > > > > >> +          mf->quantization = isp_sd->quantization;
> > > > > >> +  }
> > > > > >> +  mf->field = V4L2_FIELD_NONE;
> > > > > >> +
> > > > > >> +  return 0;
> > > > > >> +}

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions
  2019-08-15 18:46   ` Laurent Pinchart
@ 2020-07-10 12:59     ` Dafna Hirschfeld
  2020-07-10 13:36       ` Laurent Pinchart
  0 siblings, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-07-10 12:59 UTC (permalink / raw)
  To: Laurent Pinchart, Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	hans.verkuil, sakari.ailus, zhengsq, mchehab, ezequiel,
	linux-arm-kernel, linux-media



On 15.08.19 20:46, Laurent Pinchart wrote:
> Hi Helen,
> 
> Thank you ofor the patch.
> 
> On Tue, Jul 30, 2019 at 03:42:45PM -0300, Helen Koike wrote:
>> From: Jeffy Chen <jeffy.chen@rock-chips.com>
>>
>> Add the header for userspace
>>
>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>> [update for upstream]
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>
>> ---
>> Hi,
>>
>> I don't have the "REF_01 - ISP_user_manual, Rev 2.57" document that was
>> mentioned in previous version, so I'm adding a TODO for now to improve
>> the docs once we collect the information.
> 
> I would keep the document name in the TODO entry, for reference.
> 
>> If Rockchip people could help here it would be great.
> 
> As you don't have access to the documentation I won't focus on the
> issues that you can't address, but in general I think we need more
> documentation than what is provided here. In particular we need a
> description of the pipeline, with the order of the processing blocks.
> 
>> Changes in v8: None
>> Changes in v7:
>> - Fix checkpatch errors (lines over 80 and SPDX)
>> - Add TODO to improve docs
>>
>>   include/uapi/linux/rkisp1-config.h | 816 +++++++++++++++++++++++++++++
>>   1 file changed, 816 insertions(+)
>>   create mode 100644 include/uapi/linux/rkisp1-config.h
>>
>> diff --git a/include/uapi/linux/rkisp1-config.h b/include/uapi/linux/rkisp1-config.h
>> new file mode 100644
>> index 000000000000..9ab979bb4adb
>> --- /dev/null
>> +++ b/include/uapi/linux/rkisp1-config.h
>> @@ -0,0 +1,816 @@
>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>> +/*
>> + * Rockchip isp1 driver
> 
> Maybe "Rockchip ISP1 userspace API" ?
> 
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +/*
>> + * TODO: Improve documentation, mostly regarding abbreviation and hardware
>> + * specificities.
>> + */
>> +
>> +#ifndef _UAPI_RKISP1_CONFIG_H
>> +#define _UAPI_RKISP1_CONFIG_H
>> +
>> +#include <linux/types.h>
>> +#include <linux/v4l2-controls.h>
> 
> Is v4l2-controls.h needed ?
> 
>> +
>> +#define CIFISP_MODULE_DPCC              (1 << 0)
>> +#define CIFISP_MODULE_BLS               (1 << 1)
>> +#define CIFISP_MODULE_SDG               (1 << 2)
>> +#define CIFISP_MODULE_HST               (1 << 3)
>> +#define CIFISP_MODULE_LSC               (1 << 4)
>> +#define CIFISP_MODULE_AWB_GAIN          (1 << 5)
>> +#define CIFISP_MODULE_FLT               (1 << 6)
>> +#define CIFISP_MODULE_BDM               (1 << 7)
>> +#define CIFISP_MODULE_CTK               (1 << 8)
>> +#define CIFISP_MODULE_GOC               (1 << 9)
>> +#define CIFISP_MODULE_CPROC             (1 << 10)
>> +#define CIFISP_MODULE_AFC               (1 << 11)
>> +#define CIFISP_MODULE_AWB               (1 << 12)
>> +#define CIFISP_MODULE_IE                (1 << 13)
>> +#define CIFISP_MODULE_AEC               (1 << 14)
>> +#define CIFISP_MODULE_WDR               (1 << 15)
>> +#define CIFISP_MODULE_DPF               (1 << 16)
>> +#define CIFISP_MODULE_DPF_STRENGTH      (1 << 17)
> 
> Should we use BIT() here ?
> 
>> +
>> +#define CIFISP_CTK_COEFF_MAX            0x100
>> +#define CIFISP_CTK_OFFSET_MAX           0x800
>> +
>> +#define CIFISP_AE_MEAN_MAX              25
>> +#define CIFISP_HIST_BIN_N_MAX           16
>> +#define CIFISP_AFM_MAX_WINDOWS          3
>> +#define CIFISP_DEGAMMA_CURVE_SIZE       17
>> +
>> +#define CIFISP_BDM_MAX_TH               0xFF
> 
> Hex constants are usually expressed lower-case in the kernel.
> 
>> +
>> +/*
>> + * Black level compensation
>> + */
>> +/* maximum value for horizontal start address */
>> +#define CIFISP_BLS_START_H_MAX             0x00000FFF
>> +/* maximum value for horizontal stop address */
>> +#define CIFISP_BLS_STOP_H_MAX              0x00000FFF
>> +/* maximum value for vertical start address */
>> +#define CIFISP_BLS_START_V_MAX             0x00000FFF
>> +/* maximum value for vertical stop address */
>> +#define CIFISP_BLS_STOP_V_MAX              0x00000FFF
>> +/* maximum is 2^18 = 262144*/
>> +#define CIFISP_BLS_SAMPLES_MAX             0x00000012
>> +/* maximum value for fixed black level */
>> +#define CIFISP_BLS_FIX_SUB_MAX             0x00000FFF
>> +/* minimum value for fixed black level */
>> +#define CIFISP_BLS_FIX_SUB_MIN             0xFFFFF000
>> +/* 13 bit range (signed)*/
>> +#define CIFISP_BLS_FIX_MASK                0x00001FFF
>> +
>> +/*
>> + * Automatic white balance measurments
>> + */
>> +#define CIFISP_AWB_MAX_GRID                1
>> +#define CIFISP_AWB_MAX_FRAMES              7
>> +
>> +/*
>> + * Gamma out
>> + */
>> +/* Maximum number of color samples supported */
>> +#define CIFISP_GAMMA_OUT_MAX_SAMPLES       17
>> +
>> +/*
>> + * Lens shade correction
>> + */
>> +#define CIFISP_LSC_GRAD_TBL_SIZE           8
>> +#define CIFISP_LSC_SIZE_TBL_SIZE           8
>> +/*
>> + * The following matches the tuning process,
>> + * not the max capabilities of the chip.
>> + * Last value unused.
>> + */
>> +#define	CIFISP_LSC_DATA_TBL_SIZE           290
>> +
>> +/*
>> + * Histogram calculation
>> + */
>> +/* Last 3 values unused. */
>> +#define CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE 28
>> +
>> +/*
>> + * Defect Pixel Cluster Correction
>> + */
>> +#define CIFISP_DPCC_METHODS_MAX       3
>> +
>> +/*
>> + * Denoising pre filter
>> + */
>> +#define CIFISP_DPF_MAX_NLF_COEFFS      17
>> +#define CIFISP_DPF_MAX_SPATIAL_COEFFS  6
>> +
>> +/*
>> + * Measurement types
>> + */
>> +#define CIFISP_STAT_AWB           (1 << 0)
>> +#define CIFISP_STAT_AUTOEXP       (1 << 1)
>> +#define CIFISP_STAT_AFM_FIN       (1 << 2)
>> +#define CIFISP_STAT_HIST          (1 << 3)
>> +
>> +enum cifisp_histogram_mode {
>> +	CIFISP_HISTOGRAM_MODE_DISABLE,
>> +	CIFISP_HISTOGRAM_MODE_RGB_COMBINED,
>> +	CIFISP_HISTOGRAM_MODE_R_HISTOGRAM,
>> +	CIFISP_HISTOGRAM_MODE_G_HISTOGRAM,
>> +	CIFISP_HISTOGRAM_MODE_B_HISTOGRAM,
>> +	CIFISP_HISTOGRAM_MODE_Y_HISTOGRAM
>> +};
>> +
>> +enum cifisp_awb_mode_type {
>> +	CIFISP_AWB_MODE_MANUAL,
>> +	CIFISP_AWB_MODE_RGB,
>> +	CIFISP_AWB_MODE_YCBCR
>> +};
>> +
>> +enum cifisp_flt_mode {
>> +	CIFISP_FLT_STATIC_MODE,
>> +	CIFISP_FLT_DYNAMIC_MODE
>> +};
>> +
>> +/**
>> + * enum cifisp_exp_ctrl_autostop - stop modes
>> + * @CIFISP_EXP_CTRL_AUTOSTOP_0: continuous measurement
>> + * @CIFISP_EXP_CTRL_AUTOSTOP_1: stop measuring after a complete frame
>> + */
>> +enum cifisp_exp_ctrl_autostop {
>> +	CIFISP_EXP_CTRL_AUTOSTOP_0 = 0,
>> +	CIFISP_EXP_CTRL_AUTOSTOP_1 = 1,
>> +};
>> +
>> +/**
>> + * enum cifisp_exp_meas_mode - Exposure measure mode
>> + * @CIFISP_EXP_MEASURING_MODE_0: Y = 16 + 0.25R + 0.5G + 0.1094B
>> + * @CIFISP_EXP_MEASURING_MODE_1: Y = (R + G + B) x (85/256)
>> + */
>> +enum cifisp_exp_meas_mode {
>> +	CIFISP_EXP_MEASURING_MODE_0,
>> +	CIFISP_EXP_MEASURING_MODE_1,
>> +};
>> +
>> +/*---------- PART1: Input Parameters ------------*/
>> +
>> +struct cifisp_window {
>> +	__u16 h_offs;
>> +	__u16 v_offs;
>> +	__u16 h_size;
>> +	__u16 v_size;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_bls_fixed_val - BLS fixed subtraction values
>> + *
>> + * The values will be subtracted from the sensor
>> + * values. Therefore a negative value means addition instead of subtraction!
>> + *
>> + * @r: Fixed (signed!) subtraction value for Bayer pattern R
>> + * @gr: Fixed (signed!) subtraction value for Bayer pattern Gr
>> + * @gb: Fixed (signed!) subtraction value for Bayer pattern Gb
>> + * @b: Fixed (signed!) subtraction value for Bayer pattern B
>> + */
>> +struct cifisp_bls_fixed_val {
>> +	__s16 r;
>> +	__s16 gr;
>> +	__s16 gb;
>> +	__s16 b;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_bls_config - Configuration used by black level subtraction
>> + *
>> + * @enable_auto: Automatic mode activated means that the measured values
>> + *		 are subtracted. Otherwise the fixed subtraction
>> + *		 values will be subtracted.
>> + * @en_windows: enabled window
>> + * @bls_window1: Measurement window 1 size
>> + * @bls_window2: Measurement window 2 size
>> + * @bls_samples: Set amount of measured pixels for each Bayer position
>> + *		 (A, B,C and D) to 2^bls_samples.
>> + * @cifisp_bls_fixed_val: Fixed subtraction values
>> + */
>> +struct cifisp_bls_config {
>> +	__u8 enable_auto;
>> +	__u8 en_windows;
>> +	struct cifisp_window bls_window1;
>> +	struct cifisp_window bls_window2;
>> +	__u8 bls_samples;
>> +	struct cifisp_bls_fixed_val fixed_val;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpcc_methods_config - Methods Configuration used by DPCC
>> + *
>> + * Methods Configuration used by Defect Pixel Cluster Correction
>> + *
>> + * @method: Method enable bits
>> + * @line_thresh: Line threshold
>> + * @line_mad_fac: Line MAD factor
>> + * @pg_fac: Peak gradient factor
>> + * @rnd_thresh: Rank Neighbor Difference threshold
>> + * @rg_fac: Rank gradient factor
>> + */
>> +struct cifisp_dpcc_methods_config {
>> +	__u32 method;
>> +	__u32 line_thresh;
>> +	__u32 line_mad_fac;
>> +	__u32 pg_fac;
>> +	__u32 rnd_thresh;
>> +	__u32 rg_fac;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpcc_methods_config - Configuration used by DPCC
> 
> Should be cifisp_dpcc_config.
> 
>> + *
>> + * Configuration used by Defect Pixel Cluster Correction
>> + *
>> + * @mode: dpcc output mode
>> + * @output_mode: whether use hard coded methods
>> + * @set_use: stage1 methods set
>> + * @methods: methods config
>> + * @ro_limits: rank order limits
>> + * @rnd_offs: differential rank offsets for rank neighbor difference
>> + */
>> +struct cifisp_dpcc_config {
>> +	__u32 mode;
>> +	__u32 output_mode;
>> +	__u32 set_use;
>> +	struct cifisp_dpcc_methods_config methods[CIFISP_DPCC_METHODS_MAX];
>> +	__u32 ro_limits;
>> +	__u32 rnd_offs;
>> +} __attribute__ ((packed));
>> +
>> +struct cifisp_gamma_corr_curve {
>> +	__u16 gamma_y[CIFISP_DEGAMMA_CURVE_SIZE];
>> +} __attribute__ ((packed));
>> +
>> +struct cifisp_gamma_curve_x_axis_pnts {
>> +	__u32 gamma_dx0;
>> +	__u32 gamma_dx1;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_gamma_corr_curve - Configuration used by sensor degamma
> 
> Should be cifisp_sdg_config
> 
>> + *
>> + * @curve_x: gamma curve point definition axis for x
>> + * @xa_pnts: x increments
>> + */
>> +struct cifisp_sdg_config {
>> +	struct cifisp_gamma_corr_curve curve_r;
>> +	struct cifisp_gamma_corr_curve curve_g;
>> +	struct cifisp_gamma_corr_curve curve_b;
>> +	struct cifisp_gamma_curve_x_axis_pnts xa_pnts;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_lsc_config - Configuration used by Lens shading correction
>> + *
>> + * refer to REF_01 for details
>> + */
>> +struct cifisp_lsc_config {
>> +	__u32 r_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>> +	__u32 gr_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>> +	__u32 gb_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>> +	__u32 b_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>> +
>> +	__u32 x_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
>> +	__u32 y_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
>> +
>> +	__u32 x_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
>> +	__u32 y_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
>> +	__u16 config_width;
>> +	__u16 config_height;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_ie_config - Configuration used by image effects
>> + *
>> + * @eff_mat_1: 3x3 Matrix Coefficients for Emboss Effect 1
>> + * @eff_mat_2: 3x3 Matrix Coefficients for Emboss Effect 2
>> + * @eff_mat_3: 3x3 Matrix Coefficients for Emboss 3/Sketch 1
>> + * @eff_mat_4: 3x3 Matrix Coefficients for Sketch Effect 2
>> + * @eff_mat_5: 3x3 Matrix Coefficients for Sketch Effect 3
>> + * @eff_tint: Chrominance increment values of tint (used for sepia effect)
>> + */
>> +struct cifisp_ie_config {
>> +	__u16 effect;
>> +	__u16 color_sel;
>> +	__u16 eff_mat_1;
>> +	__u16 eff_mat_2;
>> +	__u16 eff_mat_3;
>> +	__u16 eff_mat_4;
>> +	__u16 eff_mat_5;
>> +	__u16 eff_tint;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_cproc_config - Configuration used by Color Processing
>> + *
>> + * @c_out_range: Chrominance pixel clipping range at output.
>> + *		 (0 for limit, 1 for full)
>> + * @y_in_range: Luminance pixel clipping range at output.
>> + * @y_out_range: Luminance pixel clipping range at output.
>> + * @contrast: 00~ff, 0.0~1.992
>> + * @brightness: 80~7F, -128~+127
>> + * @sat: saturation, 00~FF, 0.0~1.992
>> + * @hue: 80~7F, -90~+87.188
>> + */
>> +struct cifisp_cproc_config {
>> +	__u8 c_out_range;
>> +	__u8 y_in_range;
>> +	__u8 y_out_range;
>> +	__u8 contrast;
>> +	__u8 brightness;
>> +	__u8 sat;
>> +	__u8 hue;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_awb_meas_config - Configuration used by auto white balance
>> + *
>> + * @awb_wnd: white balance measurement window (in pixels)
>> + *	     (from enum cifisp_awb_mode_type)
>> + * @max_y: only pixels values < max_y contribute to awb measurement, set to 0
>> + *	   to disable this feature
>> + * @min_y: only pixels values > min_y contribute to awb measurement
>> + * @max_csum: Chrominance sum maximum value, only consider pixels with Cb+Cr,
>> + *	      smaller than threshold for awb measurements
>> + * @min_c: Chrominance minimum value, only consider pixels with Cb/Cr
>> + *	   each greater than threshold value for awb measurements
>> + * @frames: number of frames - 1 used for mean value calculation
>> + *	    (ucFrames=0 means 1 Frame)
>> + * @awb_ref_cr: reference Cr value for AWB regulation, target for AWB
>> + * @awb_ref_cb: reference Cb value for AWB regulation, target for AWB
>> + */
>> +struct cifisp_awb_meas_config {
>> +	/*
>> +	 * Note: currently the h and v offsets are mapped to grid offsets
>> +	 */
>> +	struct cifisp_window awb_wnd;
>> +	__u32 awb_mode;
>> +	__u8 max_y;
>> +	__u8 min_y;
>> +	__u8 max_csum;
>> +	__u8 min_c;
>> +	__u8 frames;
>> +	__u8 awb_ref_cr;
>> +	__u8 awb_ref_cb;
>> +	__u8 enable_ymax_cmp;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_awb_gain_config - Configuration used by auto white balance gain
>> + *
>> + * out_data_x = ( AWB_GEAIN_X * in_data + 128) >> 8
>> + */
>> +struct cifisp_awb_gain_config {
>> +	__u16 gain_red;
>> +	__u16 gain_green_r;
>> +	__u16 gain_blue;
>> +	__u16 gain_green_b;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_flt_config - Configuration used by ISP filtering
>> + *
>> + * @mode: ISP_FILT_MODE register fields (from enum cifisp_flt_mode)
>> + * @grn_stage1: ISP_FILT_MODE register fields
>> + * @chr_h_mode: ISP_FILT_MODE register fields
>> + * @chr_v_mode: ISP_FILT_MODE register fields
>> + *
>> + * refer to REF_01 for details.
>> + */
>> +
>> +struct cifisp_flt_config {
>> +	__u32 mode;
>> +	__u8 grn_stage1;
>> +	__u8 chr_h_mode;
>> +	__u8 chr_v_mode;
>> +	__u32 thresh_bl0;
>> +	__u32 thresh_bl1;
>> +	__u32 thresh_sh0;
>> +	__u32 thresh_sh1;
>> +	__u32 lum_weight;
>> +	__u32 fac_sh1;
>> +	__u32 fac_sh0;
>> +	__u32 fac_mid;
>> +	__u32 fac_bl0;
>> +	__u32 fac_bl1;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_bdm_config - Configuration used by Bayer DeMosaic
>> + *
>> + * @demosaic_th: threshod for bayer demosaicing texture detection
>> + */
>> +struct cifisp_bdm_config {
>> +	__u8 demosaic_th;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_ctk_config - Configuration used by Cross Talk correction
>> + *
>> + * @coeff: color correction matrix
>> + * @ct_offset_b: offset for the crosstalk correction matrix
>> + */
>> +struct cifisp_ctk_config {
>> +	__u16 coeff0;
>> +	__u16 coeff1;
>> +	__u16 coeff2;
>> +	__u16 coeff3;
>> +	__u16 coeff4;
>> +	__u16 coeff5;
>> +	__u16 coeff6;
>> +	__u16 coeff7;
>> +	__u16 coeff8;
>> +	__u16 ct_offset_r;
>> +	__u16 ct_offset_g;
>> +	__u16 ct_offset_b;
>> +} __attribute__ ((packed));
>> +
>> +enum cifisp_goc_mode {
>> +	CIFISP_GOC_MODE_LOGARITHMIC,
>> +	CIFISP_GOC_MODE_EQUIDISTANT
>> +};
>> +
>> +/**
>> + * struct cifisp_goc_config - Configuration used by Gamma Out correction
>> + *
>> + * @mode: goc mode (from enum cifisp_goc_mode)
>> + * @gamma_y: gamma out curve y-axis for all color components
>> + */
>> +struct cifisp_goc_config {
>> +	__u32 mode;
>> +	__u16 gamma_y[CIFISP_GAMMA_OUT_MAX_SAMPLES];
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_hst_config - Configuration used by Histogram
>> + *
>> + * @mode: histogram mode (from enum cifisp_histogram_mode)
>> + * @histogram_predivider: process every stepsize pixel, all other pixels are
>> + *			  skipped
>> + * @meas_window: coordinates of the measure window
>> + * @hist_weight: weighting factor for sub-windows
>> + */
>> +struct cifisp_hst_config {
>> +	__u32 mode;
>> +	__u8 histogram_predivider;
>> +	struct cifisp_window meas_window;
>> +	__u8 hist_weight[CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE];
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_aec_config - Configuration used by Auto Exposure Control
>> + *
>> + * @mode: Exposure measure mode (from enum cifisp_exp_meas_mode)
>> + * @autostop: stop mode (from enum cifisp_exp_ctrl_autostop)
>> + * @meas_window: coordinates of the measure window
>> + */
>> +struct cifisp_aec_config {
>> +	__u32 mode;
>> +	__u32 autostop;
>> +	struct cifisp_window meas_window;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_afc_config - Configuration used by Auto Focus Control
>> + *
>> + * @num_afm_win: max CIFISP_AFM_MAX_WINDOWS
>> + * @afm_win: coordinates of the meas window
>> + * @thres: threshold used for minimizing the influence of noise
>> + * @var_shift: the number of bits for the shift operation at the end of the
>> + *	       calculation chain.
>> + */
>> +struct cifisp_afc_config {
>> +	__u8 num_afm_win;
>> +	struct cifisp_window afm_win[CIFISP_AFM_MAX_WINDOWS];
>> +	__u32 thres;
>> +	__u32 var_shift;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * enum cifisp_dpf_gain_usage - dpf gain usage
>> + * @CIFISP_DPF_GAIN_USAGE_DISABLED: don't use any gains in preprocessing stage
>> + * @CIFISP_DPF_GAIN_USAGE_NF_GAINS: use only the noise function gains from
>> + *				    registers DPF_NF_GAIN_R, ...
>> + * @CIFISP_DPF_GAIN_USAGE_LSC_GAINS:  use only the gains from LSC module
>> + * @CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS: use the noise function gains and the
>> + *					gains from LSC module
>> + * @CIFISP_DPF_GAIN_USAGE_AWB_GAINS: use only the gains from AWB module
>> + * @CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS: use the gains from AWB and LSC module
>> + * @CIFISP_DPF_GAIN_USAGE_MAX: upper border (only for an internal evaluation)
>> + */
>> +enum cifisp_dpf_gain_usage {
>> +	CIFISP_DPF_GAIN_USAGE_DISABLED,
>> +	CIFISP_DPF_GAIN_USAGE_NF_GAINS,
>> +	CIFISP_DPF_GAIN_USAGE_LSC_GAINS,
>> +	CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS,
>> +	CIFISP_DPF_GAIN_USAGE_AWB_GAINS,
>> +	CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS,
>> +	CIFISP_DPF_GAIN_USAGE_MAX
>> +};
>> +
>> +/**
>> + * enum cifisp_dpf_gain_usage - dpf gain usage
> 
> This should be cifisp_dpf_rb_filtersize and the description should be
> updated.
> 
>> + * @CIFISP_DPF_RB_FILTERSIZE_13x9: red and blue filter kernel size 13x9
>> + *				   (means 7x5 active pixel)
>> + * @CIFISP_DPF_RB_FILTERSIZE_9x9: red and blue filter kernel size 9x9
>> + *				   (means 5x5 active pixel)
>> + */
>> +enum cifisp_dpf_rb_filtersize {
>> +	CIFISP_DPF_RB_FILTERSIZE_13x9,
>> +	CIFISP_DPF_RB_FILTERSIZE_9x9,
>> +};
>> +
>> +/**
>> + * enum cifisp_dpf_nll_scale_mode - dpf noise level scale mode
>> + * @CIFISP_NLL_SCALE_LINEAR: use a linear scaling
>> + * @CIFISP_NLL_SCALE_LOGARITHMIC: use a logarithmic scaling
>> + */
>> +enum cifisp_dpf_nll_scale_mode {
>> +	CIFISP_NLL_SCALE_LINEAR,
>> +	CIFISP_NLL_SCALE_LOGARITHMIC,
>> +};
>> +
>> +/**
>> + * struct cifisp_dpf_nll - Noise level lookup
>> + *
>> + * @coeff: Noise level Lookup coefficient
>> + * @scale_mode: dpf noise level scale mode (from enum cifisp_dpf_nll_scale_mode)
>> + */
>> +struct cifisp_dpf_nll {
>> +	__u16 coeff[CIFISP_DPF_MAX_NLF_COEFFS];
>> +	__u32 scale_mode;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpf_rb_flt - Red blue filter config
>> + *
>> + * @fltsize: The filter size for the red and blue pixels
>> + *	     (from enum cifisp_dpf_rb_filtersize)
>> + * @spatial_coeff: Spatial weights
>> + * @r_enable: enable filter processing for red pixels
>> + * @b_enable: enable filter processing for blue pixels
>> + */
>> +struct cifisp_dpf_rb_flt {
>> +	__u32 fltsize;
>> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
>> +	__u8 r_enable;
>> +	__u8 b_enable;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpf_g_flt - Green filter Configuration
>> + *
>> + * @spatial_coeff: Spatial weights
>> + * @gr_enable: enable filter processing for green pixels in green/red lines
>> + * @gb_enable: enable filter processing for green pixels in green/blue lines
>> + */
>> +struct cifisp_dpf_g_flt {
>> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
>> +	__u8 gr_enable;
>> +	__u8 gb_enable;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpf_gain - Noise function Configuration
>> + *
>> + * @mode: dpf gain usage  (from enum cifisp_dpf_gain_usage)
>> + * @nf_r_gain: Noise function Gain that replaces the AWB gain for red pixels
>> + * @nf_b_gain: Noise function Gain that replaces the AWB gain for blue pixels
>> + * @nf_gr_gain: Noise function Gain that replaces the AWB gain
>> + *		for green pixels in a red line
>> + * @nf_gb_gain: Noise function Gain that replaces the AWB gain
>> + *		for green pixels in a blue line
>> + */
>> +struct cifisp_dpf_gain {
>> +	__u32 mode;
>> +	__u16 nf_r_gain;
>> +	__u16 nf_b_gain;
>> +	__u16 nf_gr_gain;
>> +	__u16 nf_gb_gain;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpf_config - Configuration used by De-noising pre-filter
>> + *
>> + * @gain: noise function gain
>> + * @g_flt: green filter config
>> + * @rb_flt: red blue filter config
>> + * @nll: noise level lookup
>> + */
>> +struct cifisp_dpf_config {
>> +	struct cifisp_dpf_gain gain;
>> +	struct cifisp_dpf_g_flt g_flt;
>> +	struct cifisp_dpf_rb_flt rb_flt;
>> +	struct cifisp_dpf_nll nll;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_dpf_strength_config - strength of the filter
>> + *
>> + * @r: filter strength of the RED filter
>> + * @g: filter strength of the GREEN filter
>> + * @b: filter strength of the BLUE filter
>> + */
>> +struct cifisp_dpf_strength_config {
>> +	__u8 r;
>> +	__u8 g;
>> +	__u8 b;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_isp_other_cfg - Parameters for some blocks in rockchip isp1
>> + *
>> + * @dpcc_config: Defect Pixel Cluster Correction config
>> + * @bls_config: Black Level Subtraction config
>> + * @sdg_config: sensor degamma config
>> + * @lsc_config: Lens Shade config
>> + * @awb_gain_config: Auto White balance gain config
>> + * @flt_config: filter config
>> + * @bdm_config: demosaic config
>> + * @ctk_config: cross talk config
>> + * @goc_config: gamma out config
>> + * @bls_config: black level subtraction config
>> + * @dpf_config: De-noising pre-filter config
>> + * @dpf_strength_config: dpf strength config
>> + * @cproc_config: color process config
>> + * @ie_config: image effects config
>> + */
>> +struct cifisp_isp_other_cfg {
>> +	struct cifisp_dpcc_config dpcc_config;
>> +	struct cifisp_bls_config bls_config;
>> +	struct cifisp_sdg_config sdg_config;
>> +	struct cifisp_lsc_config lsc_config;
>> +	struct cifisp_awb_gain_config awb_gain_config;
>> +	struct cifisp_flt_config flt_config;
>> +	struct cifisp_bdm_config bdm_config;
>> +	struct cifisp_ctk_config ctk_config;
>> +	struct cifisp_goc_config goc_config;
>> +	struct cifisp_dpf_config dpf_config;
>> +	struct cifisp_dpf_strength_config dpf_strength_config;
>> +	struct cifisp_cproc_config cproc_config;
>> +	struct cifisp_ie_config ie_config;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_isp_meas_cfg - Rockchip ISP1 Measure Parameters
>> + *
>> + * @awb_meas_config: auto white balance config
>> + * @hst_config: histogram config
>> + * @aec_config: auto exposure config
>> + * @afc_config: auto focus config
>> + */
>> +struct cifisp_isp_meas_cfg {
>> +	struct cifisp_awb_meas_config awb_meas_config;
>> +	struct cifisp_hst_config hst_config;
>> +	struct cifisp_aec_config aec_config;
>> +	struct cifisp_afc_config afc_config;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct rkisp1_isp_params_cfg - Rockchip ISP1 Input Parameters Meta Data
>> + *
>> + * @module_en_update: mask the enable bits of which module should be updated
>> + * @module_ens: mask the enable value of each module, only update the module
>> + *		which correspond bit was set in module_en_update
>> + * @module_cfg_update: mask the config bits of which module should be updated
>> + * @meas: measurement config
>> + * @others: other config
>> + */
>> +struct rkisp1_isp_params_cfg {
>> +	__u32 module_en_update;
>> +	__u32 module_ens;
>> +	__u32 module_cfg_update;
>> +
>> +	struct cifisp_isp_meas_cfg meas;
>> +	struct cifisp_isp_other_cfg others;
>> +} __attribute__ ((packed));
> 
> As far as I understand most (all ?) of the structures defined above are
> not used directly by the hardware, is that right ? In that case, do they
> need to be packed ?

Hi,
I guess the 'packed' is a mean to serialize the struct since it is sent to userspace as
a buffer. I see that the ipu3 also uses 'packed' for the
metadata formats. If we don't have the 'packed' it means we relay on that
the userspace uses the same padding as the kernel for the struct.
Is it ok to assume that?

Thanks,
Dafna


> 
> Some of the structures have a _cfg suffix, while others have a _config
> suffix. Should we standardise on one of the two ?
> 
>> +
>> +/*---------- PART2: Measurement Statistics ------------*/
>> +
>> +/**
>> + * struct cifisp_bls_meas_val - AWB measured values
> 
> This should be cifisp_awb_meas
> 
>> + *
>> + * @cnt: White pixel count, number of "white pixels" found during laster
> 
> s/laster/last/ ?
> 
>> + *	 measurement
>> + * @mean_y_or_g: Mean value of Y within window and frames,
>> + *		 Green if RGB is selected.
>> + * @mean_cb_or_b: Mean value of Cb within window and frames,
>> + *		  Blue if RGB is selected.
>> + * @mean_cr_or_r: Mean value of Cr within window and frames,
>> + *		  Red if RGB is selected.
>> + */
>> +struct cifisp_awb_meas {
>> +	__u32 cnt;
>> +	__u8 mean_y_or_g;
>> +	__u8 mean_cb_or_b;
>> +	__u8 mean_cr_or_r;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_awb_stat - statistics automatic white balance data
>> + *
>> + * @awb_mean: Mean measured data
>> + */
>> +struct cifisp_awb_stat {
>> +	struct cifisp_awb_meas awb_mean[CIFISP_AWB_MAX_GRID];
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_bls_meas_val - BLS measured values
>> + *
>> + * @meas_r: Mean measured value for Bayer pattern R
>> + * @meas_gr: Mean measured value for Bayer pattern Gr
>> + * @meas_gb: Mean measured value for Bayer pattern Gb
>> + * @meas_b: Mean measured value for Bayer pattern B
>> + */
>> +struct cifisp_bls_meas_val {
>> +	__u16 meas_r;
>> +	__u16 meas_gr;
>> +	__u16 meas_gb;
>> +	__u16 meas_b;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_ae_stat - statistics auto exposure data
>> + *
>> + * @exp_mean: Mean luminance value of block xx
>> + * @bls_val:  BLS measured values
>> + *
>> + * Image is divided into 5x5 blocks.
>> + */
>> +struct cifisp_ae_stat {
>> +	__u8 exp_mean[CIFISP_AE_MEAN_MAX];
>> +	struct cifisp_bls_meas_val bls_val;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_af_meas_val - AF measured values
>> + *
>> + * @sum: sharpness, refer to REF_01 for definition
>> + * @lum: luminance, refer to REF_01 for definition
> 
> That will be lovely to use without documentation...
> 
>> + */
>> +struct cifisp_af_meas_val {
>> +	__u32 sum;
>> +	__u32 lum;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_af_stat - statistics auto focus data
>> + *
>> + * @window: AF measured value of window x
>> + *
>> + * The module measures the sharpness in 3 windows of selectable size via
>> + * register settings(ISP_AFM_*_A/B/C)
>> + */
>> +struct cifisp_af_stat {
>> +	struct cifisp_af_meas_val window[CIFISP_AFM_MAX_WINDOWS];
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct cifisp_hist_stat - statistics histogram data
>> + *
>> + * @hist_bins: measured bin counters
>> + *
>> + * Measurement window divided into 25 sub-windows, set
>> + * with ISP_HIST_XXX
> 
> What is ISP_HIST_XXX ?
> 
>> + */
>> +struct cifisp_hist_stat {
>> +	__u16 hist_bins[CIFISP_HIST_BIN_N_MAX];
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Data
>> + *
>> + * @cifisp_awb_stat: statistics data for automatic white balance
>> + * @cifisp_ae_stat: statistics data for auto exposure
>> + * @cifisp_af_stat: statistics data for auto focus
>> + * @cifisp_hist_stat: statistics histogram data
>> + */
>> +struct cifisp_stat {
>> +	struct cifisp_awb_stat awb;
>> +	struct cifisp_ae_stat ae;
>> +	struct cifisp_af_stat af;
>> +	struct cifisp_hist_stat hist;
>> +} __attribute__ ((packed));
>> +
>> +/**
>> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Meta Data
>> + *
>> + * @meas_type: measurement types (CIFISP_STAT_ definitions)
>> + * @frame_id: frame ID for sync
>> + * @params: statistics data
>> + */
>> +struct rkisp1_stat_buffer {
>> +	__u32 meas_type;
>> +	__u32 frame_id;
>> +	struct cifisp_stat params;
>> +} __attribute__ ((packed));
>> +
>> +#endif /* _UAPI_RKISP1_CONFIG_H */
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions
  2020-07-10 12:59     ` Dafna Hirschfeld
@ 2020-07-10 13:36       ` Laurent Pinchart
  2020-07-10 14:30         ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Laurent Pinchart @ 2020-07-10 13:36 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Helen Koike, hans.verkuil, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media

Hi Dafna,

On Fri, Jul 10, 2020 at 02:59:36PM +0200, Dafna Hirschfeld wrote:
> On 15.08.19 20:46, Laurent Pinchart wrote:
> > On Tue, Jul 30, 2019 at 03:42:45PM -0300, Helen Koike wrote:
> >> From: Jeffy Chen <jeffy.chen@rock-chips.com>
> >>
> >> Add the header for userspace
> >>
> >> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> >> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> >> [update for upstream]
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>
> >> ---
> >> Hi,
> >>
> >> I don't have the "REF_01 - ISP_user_manual, Rev 2.57" document that was
> >> mentioned in previous version, so I'm adding a TODO for now to improve
> >> the docs once we collect the information.
> > 
> > I would keep the document name in the TODO entry, for reference.
> > 
> >> If Rockchip people could help here it would be great.
> > 
> > As you don't have access to the documentation I won't focus on the
> > issues that you can't address, but in general I think we need more
> > documentation than what is provided here. In particular we need a
> > description of the pipeline, with the order of the processing blocks.
> > 
> >> Changes in v8: None
> >> Changes in v7:
> >> - Fix checkpatch errors (lines over 80 and SPDX)
> >> - Add TODO to improve docs
> >>
> >>   include/uapi/linux/rkisp1-config.h | 816 +++++++++++++++++++++++++++++
> >>   1 file changed, 816 insertions(+)
> >>   create mode 100644 include/uapi/linux/rkisp1-config.h
> >>
> >> diff --git a/include/uapi/linux/rkisp1-config.h b/include/uapi/linux/rkisp1-config.h
> >> new file mode 100644
> >> index 000000000000..9ab979bb4adb
> >> --- /dev/null
> >> +++ b/include/uapi/linux/rkisp1-config.h
> >> @@ -0,0 +1,816 @@
> >> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
> >> +/*
> >> + * Rockchip isp1 driver
> > 
> > Maybe "Rockchip ISP1 userspace API" ?
> > 
> >> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> >> + */
> >> +
> >> +/*
> >> + * TODO: Improve documentation, mostly regarding abbreviation and hardware
> >> + * specificities.
> >> + */
> >> +
> >> +#ifndef _UAPI_RKISP1_CONFIG_H
> >> +#define _UAPI_RKISP1_CONFIG_H
> >> +
> >> +#include <linux/types.h>
> >> +#include <linux/v4l2-controls.h>
> > 
> > Is v4l2-controls.h needed ?
> > 
> >> +
> >> +#define CIFISP_MODULE_DPCC              (1 << 0)
> >> +#define CIFISP_MODULE_BLS               (1 << 1)
> >> +#define CIFISP_MODULE_SDG               (1 << 2)
> >> +#define CIFISP_MODULE_HST               (1 << 3)
> >> +#define CIFISP_MODULE_LSC               (1 << 4)
> >> +#define CIFISP_MODULE_AWB_GAIN          (1 << 5)
> >> +#define CIFISP_MODULE_FLT               (1 << 6)
> >> +#define CIFISP_MODULE_BDM               (1 << 7)
> >> +#define CIFISP_MODULE_CTK               (1 << 8)
> >> +#define CIFISP_MODULE_GOC               (1 << 9)
> >> +#define CIFISP_MODULE_CPROC             (1 << 10)
> >> +#define CIFISP_MODULE_AFC               (1 << 11)
> >> +#define CIFISP_MODULE_AWB               (1 << 12)
> >> +#define CIFISP_MODULE_IE                (1 << 13)
> >> +#define CIFISP_MODULE_AEC               (1 << 14)
> >> +#define CIFISP_MODULE_WDR               (1 << 15)
> >> +#define CIFISP_MODULE_DPF               (1 << 16)
> >> +#define CIFISP_MODULE_DPF_STRENGTH      (1 << 17)
> > 
> > Should we use BIT() here ?
> > 
> >> +
> >> +#define CIFISP_CTK_COEFF_MAX            0x100
> >> +#define CIFISP_CTK_OFFSET_MAX           0x800
> >> +
> >> +#define CIFISP_AE_MEAN_MAX              25
> >> +#define CIFISP_HIST_BIN_N_MAX           16
> >> +#define CIFISP_AFM_MAX_WINDOWS          3
> >> +#define CIFISP_DEGAMMA_CURVE_SIZE       17
> >> +
> >> +#define CIFISP_BDM_MAX_TH               0xFF
> > 
> > Hex constants are usually expressed lower-case in the kernel.
> > 
> >> +
> >> +/*
> >> + * Black level compensation
> >> + */
> >> +/* maximum value for horizontal start address */
> >> +#define CIFISP_BLS_START_H_MAX             0x00000FFF
> >> +/* maximum value for horizontal stop address */
> >> +#define CIFISP_BLS_STOP_H_MAX              0x00000FFF
> >> +/* maximum value for vertical start address */
> >> +#define CIFISP_BLS_START_V_MAX             0x00000FFF
> >> +/* maximum value for vertical stop address */
> >> +#define CIFISP_BLS_STOP_V_MAX              0x00000FFF
> >> +/* maximum is 2^18 = 262144*/
> >> +#define CIFISP_BLS_SAMPLES_MAX             0x00000012
> >> +/* maximum value for fixed black level */
> >> +#define CIFISP_BLS_FIX_SUB_MAX             0x00000FFF
> >> +/* minimum value for fixed black level */
> >> +#define CIFISP_BLS_FIX_SUB_MIN             0xFFFFF000
> >> +/* 13 bit range (signed)*/
> >> +#define CIFISP_BLS_FIX_MASK                0x00001FFF
> >> +
> >> +/*
> >> + * Automatic white balance measurments
> >> + */
> >> +#define CIFISP_AWB_MAX_GRID                1
> >> +#define CIFISP_AWB_MAX_FRAMES              7
> >> +
> >> +/*
> >> + * Gamma out
> >> + */
> >> +/* Maximum number of color samples supported */
> >> +#define CIFISP_GAMMA_OUT_MAX_SAMPLES       17
> >> +
> >> +/*
> >> + * Lens shade correction
> >> + */
> >> +#define CIFISP_LSC_GRAD_TBL_SIZE           8
> >> +#define CIFISP_LSC_SIZE_TBL_SIZE           8
> >> +/*
> >> + * The following matches the tuning process,
> >> + * not the max capabilities of the chip.
> >> + * Last value unused.
> >> + */
> >> +#define	CIFISP_LSC_DATA_TBL_SIZE           290
> >> +
> >> +/*
> >> + * Histogram calculation
> >> + */
> >> +/* Last 3 values unused. */
> >> +#define CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE 28
> >> +
> >> +/*
> >> + * Defect Pixel Cluster Correction
> >> + */
> >> +#define CIFISP_DPCC_METHODS_MAX       3
> >> +
> >> +/*
> >> + * Denoising pre filter
> >> + */
> >> +#define CIFISP_DPF_MAX_NLF_COEFFS      17
> >> +#define CIFISP_DPF_MAX_SPATIAL_COEFFS  6
> >> +
> >> +/*
> >> + * Measurement types
> >> + */
> >> +#define CIFISP_STAT_AWB           (1 << 0)
> >> +#define CIFISP_STAT_AUTOEXP       (1 << 1)
> >> +#define CIFISP_STAT_AFM_FIN       (1 << 2)
> >> +#define CIFISP_STAT_HIST          (1 << 3)
> >> +
> >> +enum cifisp_histogram_mode {
> >> +	CIFISP_HISTOGRAM_MODE_DISABLE,
> >> +	CIFISP_HISTOGRAM_MODE_RGB_COMBINED,
> >> +	CIFISP_HISTOGRAM_MODE_R_HISTOGRAM,
> >> +	CIFISP_HISTOGRAM_MODE_G_HISTOGRAM,
> >> +	CIFISP_HISTOGRAM_MODE_B_HISTOGRAM,
> >> +	CIFISP_HISTOGRAM_MODE_Y_HISTOGRAM
> >> +};
> >> +
> >> +enum cifisp_awb_mode_type {
> >> +	CIFISP_AWB_MODE_MANUAL,
> >> +	CIFISP_AWB_MODE_RGB,
> >> +	CIFISP_AWB_MODE_YCBCR
> >> +};
> >> +
> >> +enum cifisp_flt_mode {
> >> +	CIFISP_FLT_STATIC_MODE,
> >> +	CIFISP_FLT_DYNAMIC_MODE
> >> +};
> >> +
> >> +/**
> >> + * enum cifisp_exp_ctrl_autostop - stop modes
> >> + * @CIFISP_EXP_CTRL_AUTOSTOP_0: continuous measurement
> >> + * @CIFISP_EXP_CTRL_AUTOSTOP_1: stop measuring after a complete frame
> >> + */
> >> +enum cifisp_exp_ctrl_autostop {
> >> +	CIFISP_EXP_CTRL_AUTOSTOP_0 = 0,
> >> +	CIFISP_EXP_CTRL_AUTOSTOP_1 = 1,
> >> +};
> >> +
> >> +/**
> >> + * enum cifisp_exp_meas_mode - Exposure measure mode
> >> + * @CIFISP_EXP_MEASURING_MODE_0: Y = 16 + 0.25R + 0.5G + 0.1094B
> >> + * @CIFISP_EXP_MEASURING_MODE_1: Y = (R + G + B) x (85/256)
> >> + */
> >> +enum cifisp_exp_meas_mode {
> >> +	CIFISP_EXP_MEASURING_MODE_0,
> >> +	CIFISP_EXP_MEASURING_MODE_1,
> >> +};
> >> +
> >> +/*---------- PART1: Input Parameters ------------*/
> >> +
> >> +struct cifisp_window {
> >> +	__u16 h_offs;
> >> +	__u16 v_offs;
> >> +	__u16 h_size;
> >> +	__u16 v_size;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_bls_fixed_val - BLS fixed subtraction values
> >> + *
> >> + * The values will be subtracted from the sensor
> >> + * values. Therefore a negative value means addition instead of subtraction!
> >> + *
> >> + * @r: Fixed (signed!) subtraction value for Bayer pattern R
> >> + * @gr: Fixed (signed!) subtraction value for Bayer pattern Gr
> >> + * @gb: Fixed (signed!) subtraction value for Bayer pattern Gb
> >> + * @b: Fixed (signed!) subtraction value for Bayer pattern B
> >> + */
> >> +struct cifisp_bls_fixed_val {
> >> +	__s16 r;
> >> +	__s16 gr;
> >> +	__s16 gb;
> >> +	__s16 b;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_bls_config - Configuration used by black level subtraction
> >> + *
> >> + * @enable_auto: Automatic mode activated means that the measured values
> >> + *		 are subtracted. Otherwise the fixed subtraction
> >> + *		 values will be subtracted.
> >> + * @en_windows: enabled window
> >> + * @bls_window1: Measurement window 1 size
> >> + * @bls_window2: Measurement window 2 size
> >> + * @bls_samples: Set amount of measured pixels for each Bayer position
> >> + *		 (A, B,C and D) to 2^bls_samples.
> >> + * @cifisp_bls_fixed_val: Fixed subtraction values
> >> + */
> >> +struct cifisp_bls_config {
> >> +	__u8 enable_auto;
> >> +	__u8 en_windows;
> >> +	struct cifisp_window bls_window1;
> >> +	struct cifisp_window bls_window2;
> >> +	__u8 bls_samples;
> >> +	struct cifisp_bls_fixed_val fixed_val;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpcc_methods_config - Methods Configuration used by DPCC
> >> + *
> >> + * Methods Configuration used by Defect Pixel Cluster Correction
> >> + *
> >> + * @method: Method enable bits
> >> + * @line_thresh: Line threshold
> >> + * @line_mad_fac: Line MAD factor
> >> + * @pg_fac: Peak gradient factor
> >> + * @rnd_thresh: Rank Neighbor Difference threshold
> >> + * @rg_fac: Rank gradient factor
> >> + */
> >> +struct cifisp_dpcc_methods_config {
> >> +	__u32 method;
> >> +	__u32 line_thresh;
> >> +	__u32 line_mad_fac;
> >> +	__u32 pg_fac;
> >> +	__u32 rnd_thresh;
> >> +	__u32 rg_fac;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpcc_methods_config - Configuration used by DPCC
> > 
> > Should be cifisp_dpcc_config.
> > 
> >> + *
> >> + * Configuration used by Defect Pixel Cluster Correction
> >> + *
> >> + * @mode: dpcc output mode
> >> + * @output_mode: whether use hard coded methods
> >> + * @set_use: stage1 methods set
> >> + * @methods: methods config
> >> + * @ro_limits: rank order limits
> >> + * @rnd_offs: differential rank offsets for rank neighbor difference
> >> + */
> >> +struct cifisp_dpcc_config {
> >> +	__u32 mode;
> >> +	__u32 output_mode;
> >> +	__u32 set_use;
> >> +	struct cifisp_dpcc_methods_config methods[CIFISP_DPCC_METHODS_MAX];
> >> +	__u32 ro_limits;
> >> +	__u32 rnd_offs;
> >> +} __attribute__ ((packed));
> >> +
> >> +struct cifisp_gamma_corr_curve {
> >> +	__u16 gamma_y[CIFISP_DEGAMMA_CURVE_SIZE];
> >> +} __attribute__ ((packed));
> >> +
> >> +struct cifisp_gamma_curve_x_axis_pnts {
> >> +	__u32 gamma_dx0;
> >> +	__u32 gamma_dx1;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_gamma_corr_curve - Configuration used by sensor degamma
> > 
> > Should be cifisp_sdg_config
> > 
> >> + *
> >> + * @curve_x: gamma curve point definition axis for x
> >> + * @xa_pnts: x increments
> >> + */
> >> +struct cifisp_sdg_config {
> >> +	struct cifisp_gamma_corr_curve curve_r;
> >> +	struct cifisp_gamma_corr_curve curve_g;
> >> +	struct cifisp_gamma_corr_curve curve_b;
> >> +	struct cifisp_gamma_curve_x_axis_pnts xa_pnts;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_lsc_config - Configuration used by Lens shading correction
> >> + *
> >> + * refer to REF_01 for details
> >> + */
> >> +struct cifisp_lsc_config {
> >> +	__u32 r_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> >> +	__u32 gr_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> >> +	__u32 gb_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> >> +	__u32 b_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
> >> +
> >> +	__u32 x_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
> >> +	__u32 y_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
> >> +
> >> +	__u32 x_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
> >> +	__u32 y_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
> >> +	__u16 config_width;
> >> +	__u16 config_height;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_ie_config - Configuration used by image effects
> >> + *
> >> + * @eff_mat_1: 3x3 Matrix Coefficients for Emboss Effect 1
> >> + * @eff_mat_2: 3x3 Matrix Coefficients for Emboss Effect 2
> >> + * @eff_mat_3: 3x3 Matrix Coefficients for Emboss 3/Sketch 1
> >> + * @eff_mat_4: 3x3 Matrix Coefficients for Sketch Effect 2
> >> + * @eff_mat_5: 3x3 Matrix Coefficients for Sketch Effect 3
> >> + * @eff_tint: Chrominance increment values of tint (used for sepia effect)
> >> + */
> >> +struct cifisp_ie_config {
> >> +	__u16 effect;
> >> +	__u16 color_sel;
> >> +	__u16 eff_mat_1;
> >> +	__u16 eff_mat_2;
> >> +	__u16 eff_mat_3;
> >> +	__u16 eff_mat_4;
> >> +	__u16 eff_mat_5;
> >> +	__u16 eff_tint;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_cproc_config - Configuration used by Color Processing
> >> + *
> >> + * @c_out_range: Chrominance pixel clipping range at output.
> >> + *		 (0 for limit, 1 for full)
> >> + * @y_in_range: Luminance pixel clipping range at output.
> >> + * @y_out_range: Luminance pixel clipping range at output.
> >> + * @contrast: 00~ff, 0.0~1.992
> >> + * @brightness: 80~7F, -128~+127
> >> + * @sat: saturation, 00~FF, 0.0~1.992
> >> + * @hue: 80~7F, -90~+87.188
> >> + */
> >> +struct cifisp_cproc_config {
> >> +	__u8 c_out_range;
> >> +	__u8 y_in_range;
> >> +	__u8 y_out_range;
> >> +	__u8 contrast;
> >> +	__u8 brightness;
> >> +	__u8 sat;
> >> +	__u8 hue;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_awb_meas_config - Configuration used by auto white balance
> >> + *
> >> + * @awb_wnd: white balance measurement window (in pixels)
> >> + *	     (from enum cifisp_awb_mode_type)
> >> + * @max_y: only pixels values < max_y contribute to awb measurement, set to 0
> >> + *	   to disable this feature
> >> + * @min_y: only pixels values > min_y contribute to awb measurement
> >> + * @max_csum: Chrominance sum maximum value, only consider pixels with Cb+Cr,
> >> + *	      smaller than threshold for awb measurements
> >> + * @min_c: Chrominance minimum value, only consider pixels with Cb/Cr
> >> + *	   each greater than threshold value for awb measurements
> >> + * @frames: number of frames - 1 used for mean value calculation
> >> + *	    (ucFrames=0 means 1 Frame)
> >> + * @awb_ref_cr: reference Cr value for AWB regulation, target for AWB
> >> + * @awb_ref_cb: reference Cb value for AWB regulation, target for AWB
> >> + */
> >> +struct cifisp_awb_meas_config {
> >> +	/*
> >> +	 * Note: currently the h and v offsets are mapped to grid offsets
> >> +	 */
> >> +	struct cifisp_window awb_wnd;
> >> +	__u32 awb_mode;
> >> +	__u8 max_y;
> >> +	__u8 min_y;
> >> +	__u8 max_csum;
> >> +	__u8 min_c;
> >> +	__u8 frames;
> >> +	__u8 awb_ref_cr;
> >> +	__u8 awb_ref_cb;
> >> +	__u8 enable_ymax_cmp;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_awb_gain_config - Configuration used by auto white balance gain
> >> + *
> >> + * out_data_x = ( AWB_GEAIN_X * in_data + 128) >> 8
> >> + */
> >> +struct cifisp_awb_gain_config {
> >> +	__u16 gain_red;
> >> +	__u16 gain_green_r;
> >> +	__u16 gain_blue;
> >> +	__u16 gain_green_b;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_flt_config - Configuration used by ISP filtering
> >> + *
> >> + * @mode: ISP_FILT_MODE register fields (from enum cifisp_flt_mode)
> >> + * @grn_stage1: ISP_FILT_MODE register fields
> >> + * @chr_h_mode: ISP_FILT_MODE register fields
> >> + * @chr_v_mode: ISP_FILT_MODE register fields
> >> + *
> >> + * refer to REF_01 for details.
> >> + */
> >> +
> >> +struct cifisp_flt_config {
> >> +	__u32 mode;
> >> +	__u8 grn_stage1;
> >> +	__u8 chr_h_mode;
> >> +	__u8 chr_v_mode;
> >> +	__u32 thresh_bl0;
> >> +	__u32 thresh_bl1;
> >> +	__u32 thresh_sh0;
> >> +	__u32 thresh_sh1;
> >> +	__u32 lum_weight;
> >> +	__u32 fac_sh1;
> >> +	__u32 fac_sh0;
> >> +	__u32 fac_mid;
> >> +	__u32 fac_bl0;
> >> +	__u32 fac_bl1;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_bdm_config - Configuration used by Bayer DeMosaic
> >> + *
> >> + * @demosaic_th: threshod for bayer demosaicing texture detection
> >> + */
> >> +struct cifisp_bdm_config {
> >> +	__u8 demosaic_th;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_ctk_config - Configuration used by Cross Talk correction
> >> + *
> >> + * @coeff: color correction matrix
> >> + * @ct_offset_b: offset for the crosstalk correction matrix
> >> + */
> >> +struct cifisp_ctk_config {
> >> +	__u16 coeff0;
> >> +	__u16 coeff1;
> >> +	__u16 coeff2;
> >> +	__u16 coeff3;
> >> +	__u16 coeff4;
> >> +	__u16 coeff5;
> >> +	__u16 coeff6;
> >> +	__u16 coeff7;
> >> +	__u16 coeff8;
> >> +	__u16 ct_offset_r;
> >> +	__u16 ct_offset_g;
> >> +	__u16 ct_offset_b;
> >> +} __attribute__ ((packed));
> >> +
> >> +enum cifisp_goc_mode {
> >> +	CIFISP_GOC_MODE_LOGARITHMIC,
> >> +	CIFISP_GOC_MODE_EQUIDISTANT
> >> +};
> >> +
> >> +/**
> >> + * struct cifisp_goc_config - Configuration used by Gamma Out correction
> >> + *
> >> + * @mode: goc mode (from enum cifisp_goc_mode)
> >> + * @gamma_y: gamma out curve y-axis for all color components
> >> + */
> >> +struct cifisp_goc_config {
> >> +	__u32 mode;
> >> +	__u16 gamma_y[CIFISP_GAMMA_OUT_MAX_SAMPLES];
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_hst_config - Configuration used by Histogram
> >> + *
> >> + * @mode: histogram mode (from enum cifisp_histogram_mode)
> >> + * @histogram_predivider: process every stepsize pixel, all other pixels are
> >> + *			  skipped
> >> + * @meas_window: coordinates of the measure window
> >> + * @hist_weight: weighting factor for sub-windows
> >> + */
> >> +struct cifisp_hst_config {
> >> +	__u32 mode;
> >> +	__u8 histogram_predivider;
> >> +	struct cifisp_window meas_window;
> >> +	__u8 hist_weight[CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE];
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_aec_config - Configuration used by Auto Exposure Control
> >> + *
> >> + * @mode: Exposure measure mode (from enum cifisp_exp_meas_mode)
> >> + * @autostop: stop mode (from enum cifisp_exp_ctrl_autostop)
> >> + * @meas_window: coordinates of the measure window
> >> + */
> >> +struct cifisp_aec_config {
> >> +	__u32 mode;
> >> +	__u32 autostop;
> >> +	struct cifisp_window meas_window;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_afc_config - Configuration used by Auto Focus Control
> >> + *
> >> + * @num_afm_win: max CIFISP_AFM_MAX_WINDOWS
> >> + * @afm_win: coordinates of the meas window
> >> + * @thres: threshold used for minimizing the influence of noise
> >> + * @var_shift: the number of bits for the shift operation at the end of the
> >> + *	       calculation chain.
> >> + */
> >> +struct cifisp_afc_config {
> >> +	__u8 num_afm_win;
> >> +	struct cifisp_window afm_win[CIFISP_AFM_MAX_WINDOWS];
> >> +	__u32 thres;
> >> +	__u32 var_shift;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * enum cifisp_dpf_gain_usage - dpf gain usage
> >> + * @CIFISP_DPF_GAIN_USAGE_DISABLED: don't use any gains in preprocessing stage
> >> + * @CIFISP_DPF_GAIN_USAGE_NF_GAINS: use only the noise function gains from
> >> + *				    registers DPF_NF_GAIN_R, ...
> >> + * @CIFISP_DPF_GAIN_USAGE_LSC_GAINS:  use only the gains from LSC module
> >> + * @CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS: use the noise function gains and the
> >> + *					gains from LSC module
> >> + * @CIFISP_DPF_GAIN_USAGE_AWB_GAINS: use only the gains from AWB module
> >> + * @CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS: use the gains from AWB and LSC module
> >> + * @CIFISP_DPF_GAIN_USAGE_MAX: upper border (only for an internal evaluation)
> >> + */
> >> +enum cifisp_dpf_gain_usage {
> >> +	CIFISP_DPF_GAIN_USAGE_DISABLED,
> >> +	CIFISP_DPF_GAIN_USAGE_NF_GAINS,
> >> +	CIFISP_DPF_GAIN_USAGE_LSC_GAINS,
> >> +	CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS,
> >> +	CIFISP_DPF_GAIN_USAGE_AWB_GAINS,
> >> +	CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS,
> >> +	CIFISP_DPF_GAIN_USAGE_MAX
> >> +};
> >> +
> >> +/**
> >> + * enum cifisp_dpf_gain_usage - dpf gain usage
> > 
> > This should be cifisp_dpf_rb_filtersize and the description should be
> > updated.
> > 
> >> + * @CIFISP_DPF_RB_FILTERSIZE_13x9: red and blue filter kernel size 13x9
> >> + *				   (means 7x5 active pixel)
> >> + * @CIFISP_DPF_RB_FILTERSIZE_9x9: red and blue filter kernel size 9x9
> >> + *				   (means 5x5 active pixel)
> >> + */
> >> +enum cifisp_dpf_rb_filtersize {
> >> +	CIFISP_DPF_RB_FILTERSIZE_13x9,
> >> +	CIFISP_DPF_RB_FILTERSIZE_9x9,
> >> +};
> >> +
> >> +/**
> >> + * enum cifisp_dpf_nll_scale_mode - dpf noise level scale mode
> >> + * @CIFISP_NLL_SCALE_LINEAR: use a linear scaling
> >> + * @CIFISP_NLL_SCALE_LOGARITHMIC: use a logarithmic scaling
> >> + */
> >> +enum cifisp_dpf_nll_scale_mode {
> >> +	CIFISP_NLL_SCALE_LINEAR,
> >> +	CIFISP_NLL_SCALE_LOGARITHMIC,
> >> +};
> >> +
> >> +/**
> >> + * struct cifisp_dpf_nll - Noise level lookup
> >> + *
> >> + * @coeff: Noise level Lookup coefficient
> >> + * @scale_mode: dpf noise level scale mode (from enum cifisp_dpf_nll_scale_mode)
> >> + */
> >> +struct cifisp_dpf_nll {
> >> +	__u16 coeff[CIFISP_DPF_MAX_NLF_COEFFS];
> >> +	__u32 scale_mode;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpf_rb_flt - Red blue filter config
> >> + *
> >> + * @fltsize: The filter size for the red and blue pixels
> >> + *	     (from enum cifisp_dpf_rb_filtersize)
> >> + * @spatial_coeff: Spatial weights
> >> + * @r_enable: enable filter processing for red pixels
> >> + * @b_enable: enable filter processing for blue pixels
> >> + */
> >> +struct cifisp_dpf_rb_flt {
> >> +	__u32 fltsize;
> >> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
> >> +	__u8 r_enable;
> >> +	__u8 b_enable;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpf_g_flt - Green filter Configuration
> >> + *
> >> + * @spatial_coeff: Spatial weights
> >> + * @gr_enable: enable filter processing for green pixels in green/red lines
> >> + * @gb_enable: enable filter processing for green pixels in green/blue lines
> >> + */
> >> +struct cifisp_dpf_g_flt {
> >> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
> >> +	__u8 gr_enable;
> >> +	__u8 gb_enable;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpf_gain - Noise function Configuration
> >> + *
> >> + * @mode: dpf gain usage  (from enum cifisp_dpf_gain_usage)
> >> + * @nf_r_gain: Noise function Gain that replaces the AWB gain for red pixels
> >> + * @nf_b_gain: Noise function Gain that replaces the AWB gain for blue pixels
> >> + * @nf_gr_gain: Noise function Gain that replaces the AWB gain
> >> + *		for green pixels in a red line
> >> + * @nf_gb_gain: Noise function Gain that replaces the AWB gain
> >> + *		for green pixels in a blue line
> >> + */
> >> +struct cifisp_dpf_gain {
> >> +	__u32 mode;
> >> +	__u16 nf_r_gain;
> >> +	__u16 nf_b_gain;
> >> +	__u16 nf_gr_gain;
> >> +	__u16 nf_gb_gain;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpf_config - Configuration used by De-noising pre-filter
> >> + *
> >> + * @gain: noise function gain
> >> + * @g_flt: green filter config
> >> + * @rb_flt: red blue filter config
> >> + * @nll: noise level lookup
> >> + */
> >> +struct cifisp_dpf_config {
> >> +	struct cifisp_dpf_gain gain;
> >> +	struct cifisp_dpf_g_flt g_flt;
> >> +	struct cifisp_dpf_rb_flt rb_flt;
> >> +	struct cifisp_dpf_nll nll;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_dpf_strength_config - strength of the filter
> >> + *
> >> + * @r: filter strength of the RED filter
> >> + * @g: filter strength of the GREEN filter
> >> + * @b: filter strength of the BLUE filter
> >> + */
> >> +struct cifisp_dpf_strength_config {
> >> +	__u8 r;
> >> +	__u8 g;
> >> +	__u8 b;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_isp_other_cfg - Parameters for some blocks in rockchip isp1
> >> + *
> >> + * @dpcc_config: Defect Pixel Cluster Correction config
> >> + * @bls_config: Black Level Subtraction config
> >> + * @sdg_config: sensor degamma config
> >> + * @lsc_config: Lens Shade config
> >> + * @awb_gain_config: Auto White balance gain config
> >> + * @flt_config: filter config
> >> + * @bdm_config: demosaic config
> >> + * @ctk_config: cross talk config
> >> + * @goc_config: gamma out config
> >> + * @bls_config: black level subtraction config
> >> + * @dpf_config: De-noising pre-filter config
> >> + * @dpf_strength_config: dpf strength config
> >> + * @cproc_config: color process config
> >> + * @ie_config: image effects config
> >> + */
> >> +struct cifisp_isp_other_cfg {
> >> +	struct cifisp_dpcc_config dpcc_config;
> >> +	struct cifisp_bls_config bls_config;
> >> +	struct cifisp_sdg_config sdg_config;
> >> +	struct cifisp_lsc_config lsc_config;
> >> +	struct cifisp_awb_gain_config awb_gain_config;
> >> +	struct cifisp_flt_config flt_config;
> >> +	struct cifisp_bdm_config bdm_config;
> >> +	struct cifisp_ctk_config ctk_config;
> >> +	struct cifisp_goc_config goc_config;
> >> +	struct cifisp_dpf_config dpf_config;
> >> +	struct cifisp_dpf_strength_config dpf_strength_config;
> >> +	struct cifisp_cproc_config cproc_config;
> >> +	struct cifisp_ie_config ie_config;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_isp_meas_cfg - Rockchip ISP1 Measure Parameters
> >> + *
> >> + * @awb_meas_config: auto white balance config
> >> + * @hst_config: histogram config
> >> + * @aec_config: auto exposure config
> >> + * @afc_config: auto focus config
> >> + */
> >> +struct cifisp_isp_meas_cfg {
> >> +	struct cifisp_awb_meas_config awb_meas_config;
> >> +	struct cifisp_hst_config hst_config;
> >> +	struct cifisp_aec_config aec_config;
> >> +	struct cifisp_afc_config afc_config;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct rkisp1_isp_params_cfg - Rockchip ISP1 Input Parameters Meta Data
> >> + *
> >> + * @module_en_update: mask the enable bits of which module should be updated
> >> + * @module_ens: mask the enable value of each module, only update the module
> >> + *		which correspond bit was set in module_en_update
> >> + * @module_cfg_update: mask the config bits of which module should be updated
> >> + * @meas: measurement config
> >> + * @others: other config
> >> + */
> >> +struct rkisp1_isp_params_cfg {
> >> +	__u32 module_en_update;
> >> +	__u32 module_ens;
> >> +	__u32 module_cfg_update;
> >> +
> >> +	struct cifisp_isp_meas_cfg meas;
> >> +	struct cifisp_isp_other_cfg others;
> >> +} __attribute__ ((packed));
> > 
> > As far as I understand most (all ?) of the structures defined above are
> > not used directly by the hardware, is that right ? In that case, do they
> > need to be packed ?
> 
> Hi,
> I guess the 'packed' is a mean to serialize the struct since it is sent to userspace as
> a buffer.

That's no different than using structures in an ioctl, is it ?

> I see that the ipu3 also uses 'packed' for the metadata formats.

For data structures that are directly consumed (or produced) by the
hardware (or firmware) we often need the packed attribute. The
structures used to describe the stastics produced by the rkisp1 need to
be packed.

For structures that are consumed by software only, it's not necessarily
a requirement. Packing has drawbacks, for instance it means we have to
handle unaligned accesses explicitly on platforms that don't support
them.

> If we don't have the 'packed' it means we relay on that
> the userspace uses the same padding as the kernel for the struct.
> Is it ok to assume that?

Isn't that dictated by the C ABI ? We of course need to ensure that
there will be no 32/64-bit compatibility issues.

> > Some of the structures have a _cfg suffix, while others have a _config
> > suffix. Should we standardise on one of the two ?
> > 
> >> +
> >> +/*---------- PART2: Measurement Statistics ------------*/
> >> +
> >> +/**
> >> + * struct cifisp_bls_meas_val - AWB measured values
> > 
> > This should be cifisp_awb_meas
> > 
> >> + *
> >> + * @cnt: White pixel count, number of "white pixels" found during laster
> > 
> > s/laster/last/ ?
> > 
> >> + *	 measurement
> >> + * @mean_y_or_g: Mean value of Y within window and frames,
> >> + *		 Green if RGB is selected.
> >> + * @mean_cb_or_b: Mean value of Cb within window and frames,
> >> + *		  Blue if RGB is selected.
> >> + * @mean_cr_or_r: Mean value of Cr within window and frames,
> >> + *		  Red if RGB is selected.
> >> + */
> >> +struct cifisp_awb_meas {
> >> +	__u32 cnt;
> >> +	__u8 mean_y_or_g;
> >> +	__u8 mean_cb_or_b;
> >> +	__u8 mean_cr_or_r;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_awb_stat - statistics automatic white balance data
> >> + *
> >> + * @awb_mean: Mean measured data
> >> + */
> >> +struct cifisp_awb_stat {
> >> +	struct cifisp_awb_meas awb_mean[CIFISP_AWB_MAX_GRID];
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_bls_meas_val - BLS measured values
> >> + *
> >> + * @meas_r: Mean measured value for Bayer pattern R
> >> + * @meas_gr: Mean measured value for Bayer pattern Gr
> >> + * @meas_gb: Mean measured value for Bayer pattern Gb
> >> + * @meas_b: Mean measured value for Bayer pattern B
> >> + */
> >> +struct cifisp_bls_meas_val {
> >> +	__u16 meas_r;
> >> +	__u16 meas_gr;
> >> +	__u16 meas_gb;
> >> +	__u16 meas_b;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_ae_stat - statistics auto exposure data
> >> + *
> >> + * @exp_mean: Mean luminance value of block xx
> >> + * @bls_val:  BLS measured values
> >> + *
> >> + * Image is divided into 5x5 blocks.
> >> + */
> >> +struct cifisp_ae_stat {
> >> +	__u8 exp_mean[CIFISP_AE_MEAN_MAX];
> >> +	struct cifisp_bls_meas_val bls_val;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_af_meas_val - AF measured values
> >> + *
> >> + * @sum: sharpness, refer to REF_01 for definition
> >> + * @lum: luminance, refer to REF_01 for definition
> > 
> > That will be lovely to use without documentation...
> > 
> >> + */
> >> +struct cifisp_af_meas_val {
> >> +	__u32 sum;
> >> +	__u32 lum;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_af_stat - statistics auto focus data
> >> + *
> >> + * @window: AF measured value of window x
> >> + *
> >> + * The module measures the sharpness in 3 windows of selectable size via
> >> + * register settings(ISP_AFM_*_A/B/C)
> >> + */
> >> +struct cifisp_af_stat {
> >> +	struct cifisp_af_meas_val window[CIFISP_AFM_MAX_WINDOWS];
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct cifisp_hist_stat - statistics histogram data
> >> + *
> >> + * @hist_bins: measured bin counters
> >> + *
> >> + * Measurement window divided into 25 sub-windows, set
> >> + * with ISP_HIST_XXX
> > 
> > What is ISP_HIST_XXX ?
> > 
> >> + */
> >> +struct cifisp_hist_stat {
> >> +	__u16 hist_bins[CIFISP_HIST_BIN_N_MAX];
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Data
> >> + *
> >> + * @cifisp_awb_stat: statistics data for automatic white balance
> >> + * @cifisp_ae_stat: statistics data for auto exposure
> >> + * @cifisp_af_stat: statistics data for auto focus
> >> + * @cifisp_hist_stat: statistics histogram data
> >> + */
> >> +struct cifisp_stat {
> >> +	struct cifisp_awb_stat awb;
> >> +	struct cifisp_ae_stat ae;
> >> +	struct cifisp_af_stat af;
> >> +	struct cifisp_hist_stat hist;
> >> +} __attribute__ ((packed));
> >> +
> >> +/**
> >> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Meta Data
> >> + *
> >> + * @meas_type: measurement types (CIFISP_STAT_ definitions)
> >> + * @frame_id: frame ID for sync
> >> + * @params: statistics data
> >> + */
> >> +struct rkisp1_stat_buffer {
> >> +	__u32 meas_type;
> >> +	__u32 frame_id;
> >> +	struct cifisp_stat params;
> >> +} __attribute__ ((packed));
> >> +
> >> +#endif /* _UAPI_RKISP1_CONFIG_H */

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions
  2020-07-10 13:36       ` Laurent Pinchart
@ 2020-07-10 14:30         ` Dafna Hirschfeld
  0 siblings, 0 replies; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-07-10 14:30 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Helen Koike, hans.verkuil, sakari.ailus, zhengsq, mchehab,
	ezequiel, linux-arm-kernel, linux-media



On 10.07.20 15:36, Laurent Pinchart wrote:
> Hi Dafna,
> 
> On Fri, Jul 10, 2020 at 02:59:36PM +0200, Dafna Hirschfeld wrote:
>> On 15.08.19 20:46, Laurent Pinchart wrote:
>>> On Tue, Jul 30, 2019 at 03:42:45PM -0300, Helen Koike wrote:
>>>> From: Jeffy Chen <jeffy.chen@rock-chips.com>
>>>>
>>>> Add the header for userspace
>>>>
>>>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>>>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>>>> [update for upstream]
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>
>>>> ---
>>>> Hi,
>>>>
>>>> I don't have the "REF_01 - ISP_user_manual, Rev 2.57" document that was
>>>> mentioned in previous version, so I'm adding a TODO for now to improve
>>>> the docs once we collect the information.
>>>
>>> I would keep the document name in the TODO entry, for reference.
>>>
>>>> If Rockchip people could help here it would be great.
>>>
>>> As you don't have access to the documentation I won't focus on the
>>> issues that you can't address, but in general I think we need more
>>> documentation than what is provided here. In particular we need a
>>> description of the pipeline, with the order of the processing blocks.
>>>
>>>> Changes in v8: None
>>>> Changes in v7:
>>>> - Fix checkpatch errors (lines over 80 and SPDX)
>>>> - Add TODO to improve docs
>>>>
>>>>    include/uapi/linux/rkisp1-config.h | 816 +++++++++++++++++++++++++++++
>>>>    1 file changed, 816 insertions(+)
>>>>    create mode 100644 include/uapi/linux/rkisp1-config.h
>>>>
>>>> diff --git a/include/uapi/linux/rkisp1-config.h b/include/uapi/linux/rkisp1-config.h
>>>> new file mode 100644
>>>> index 000000000000..9ab979bb4adb
>>>> --- /dev/null
>>>> +++ b/include/uapi/linux/rkisp1-config.h
>>>> @@ -0,0 +1,816 @@
>>>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>>>> +/*
>>>> + * Rockchip isp1 driver
>>>
>>> Maybe "Rockchip ISP1 userspace API" ?
>>>
>>>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>>>> + */
>>>> +
>>>> +/*
>>>> + * TODO: Improve documentation, mostly regarding abbreviation and hardware
>>>> + * specificities.
>>>> + */
>>>> +
>>>> +#ifndef _UAPI_RKISP1_CONFIG_H
>>>> +#define _UAPI_RKISP1_CONFIG_H
>>>> +
>>>> +#include <linux/types.h>
>>>> +#include <linux/v4l2-controls.h>
>>>
>>> Is v4l2-controls.h needed ?
>>>
>>>> +
>>>> +#define CIFISP_MODULE_DPCC              (1 << 0)
>>>> +#define CIFISP_MODULE_BLS               (1 << 1)
>>>> +#define CIFISP_MODULE_SDG               (1 << 2)
>>>> +#define CIFISP_MODULE_HST               (1 << 3)
>>>> +#define CIFISP_MODULE_LSC               (1 << 4)
>>>> +#define CIFISP_MODULE_AWB_GAIN          (1 << 5)
>>>> +#define CIFISP_MODULE_FLT               (1 << 6)
>>>> +#define CIFISP_MODULE_BDM               (1 << 7)
>>>> +#define CIFISP_MODULE_CTK               (1 << 8)
>>>> +#define CIFISP_MODULE_GOC               (1 << 9)
>>>> +#define CIFISP_MODULE_CPROC             (1 << 10)
>>>> +#define CIFISP_MODULE_AFC               (1 << 11)
>>>> +#define CIFISP_MODULE_AWB               (1 << 12)
>>>> +#define CIFISP_MODULE_IE                (1 << 13)
>>>> +#define CIFISP_MODULE_AEC               (1 << 14)
>>>> +#define CIFISP_MODULE_WDR               (1 << 15)
>>>> +#define CIFISP_MODULE_DPF               (1 << 16)
>>>> +#define CIFISP_MODULE_DPF_STRENGTH      (1 << 17)
>>>
>>> Should we use BIT() here ?
>>>
>>>> +
>>>> +#define CIFISP_CTK_COEFF_MAX            0x100
>>>> +#define CIFISP_CTK_OFFSET_MAX           0x800
>>>> +
>>>> +#define CIFISP_AE_MEAN_MAX              25
>>>> +#define CIFISP_HIST_BIN_N_MAX           16
>>>> +#define CIFISP_AFM_MAX_WINDOWS          3
>>>> +#define CIFISP_DEGAMMA_CURVE_SIZE       17
>>>> +
>>>> +#define CIFISP_BDM_MAX_TH               0xFF
>>>
>>> Hex constants are usually expressed lower-case in the kernel.
>>>
>>>> +
>>>> +/*
>>>> + * Black level compensation
>>>> + */
>>>> +/* maximum value for horizontal start address */
>>>> +#define CIFISP_BLS_START_H_MAX             0x00000FFF
>>>> +/* maximum value for horizontal stop address */
>>>> +#define CIFISP_BLS_STOP_H_MAX              0x00000FFF
>>>> +/* maximum value for vertical start address */
>>>> +#define CIFISP_BLS_START_V_MAX             0x00000FFF
>>>> +/* maximum value for vertical stop address */
>>>> +#define CIFISP_BLS_STOP_V_MAX              0x00000FFF
>>>> +/* maximum is 2^18 = 262144*/
>>>> +#define CIFISP_BLS_SAMPLES_MAX             0x00000012
>>>> +/* maximum value for fixed black level */
>>>> +#define CIFISP_BLS_FIX_SUB_MAX             0x00000FFF
>>>> +/* minimum value for fixed black level */
>>>> +#define CIFISP_BLS_FIX_SUB_MIN             0xFFFFF000
>>>> +/* 13 bit range (signed)*/
>>>> +#define CIFISP_BLS_FIX_MASK                0x00001FFF
>>>> +
>>>> +/*
>>>> + * Automatic white balance measurments
>>>> + */
>>>> +#define CIFISP_AWB_MAX_GRID                1
>>>> +#define CIFISP_AWB_MAX_FRAMES              7
>>>> +
>>>> +/*
>>>> + * Gamma out
>>>> + */
>>>> +/* Maximum number of color samples supported */
>>>> +#define CIFISP_GAMMA_OUT_MAX_SAMPLES       17
>>>> +
>>>> +/*
>>>> + * Lens shade correction
>>>> + */
>>>> +#define CIFISP_LSC_GRAD_TBL_SIZE           8
>>>> +#define CIFISP_LSC_SIZE_TBL_SIZE           8
>>>> +/*
>>>> + * The following matches the tuning process,
>>>> + * not the max capabilities of the chip.
>>>> + * Last value unused.
>>>> + */
>>>> +#define	CIFISP_LSC_DATA_TBL_SIZE           290
>>>> +
>>>> +/*
>>>> + * Histogram calculation
>>>> + */
>>>> +/* Last 3 values unused. */
>>>> +#define CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE 28
>>>> +
>>>> +/*
>>>> + * Defect Pixel Cluster Correction
>>>> + */
>>>> +#define CIFISP_DPCC_METHODS_MAX       3
>>>> +
>>>> +/*
>>>> + * Denoising pre filter
>>>> + */
>>>> +#define CIFISP_DPF_MAX_NLF_COEFFS      17
>>>> +#define CIFISP_DPF_MAX_SPATIAL_COEFFS  6
>>>> +
>>>> +/*
>>>> + * Measurement types
>>>> + */
>>>> +#define CIFISP_STAT_AWB           (1 << 0)
>>>> +#define CIFISP_STAT_AUTOEXP       (1 << 1)
>>>> +#define CIFISP_STAT_AFM_FIN       (1 << 2)
>>>> +#define CIFISP_STAT_HIST          (1 << 3)
>>>> +
>>>> +enum cifisp_histogram_mode {
>>>> +	CIFISP_HISTOGRAM_MODE_DISABLE,
>>>> +	CIFISP_HISTOGRAM_MODE_RGB_COMBINED,
>>>> +	CIFISP_HISTOGRAM_MODE_R_HISTOGRAM,
>>>> +	CIFISP_HISTOGRAM_MODE_G_HISTOGRAM,
>>>> +	CIFISP_HISTOGRAM_MODE_B_HISTOGRAM,
>>>> +	CIFISP_HISTOGRAM_MODE_Y_HISTOGRAM
>>>> +};
>>>> +
>>>> +enum cifisp_awb_mode_type {
>>>> +	CIFISP_AWB_MODE_MANUAL,
>>>> +	CIFISP_AWB_MODE_RGB,
>>>> +	CIFISP_AWB_MODE_YCBCR
>>>> +};
>>>> +
>>>> +enum cifisp_flt_mode {
>>>> +	CIFISP_FLT_STATIC_MODE,
>>>> +	CIFISP_FLT_DYNAMIC_MODE
>>>> +};
>>>> +
>>>> +/**
>>>> + * enum cifisp_exp_ctrl_autostop - stop modes
>>>> + * @CIFISP_EXP_CTRL_AUTOSTOP_0: continuous measurement
>>>> + * @CIFISP_EXP_CTRL_AUTOSTOP_1: stop measuring after a complete frame
>>>> + */
>>>> +enum cifisp_exp_ctrl_autostop {
>>>> +	CIFISP_EXP_CTRL_AUTOSTOP_0 = 0,
>>>> +	CIFISP_EXP_CTRL_AUTOSTOP_1 = 1,
>>>> +};
>>>> +
>>>> +/**
>>>> + * enum cifisp_exp_meas_mode - Exposure measure mode
>>>> + * @CIFISP_EXP_MEASURING_MODE_0: Y = 16 + 0.25R + 0.5G + 0.1094B
>>>> + * @CIFISP_EXP_MEASURING_MODE_1: Y = (R + G + B) x (85/256)
>>>> + */
>>>> +enum cifisp_exp_meas_mode {
>>>> +	CIFISP_EXP_MEASURING_MODE_0,
>>>> +	CIFISP_EXP_MEASURING_MODE_1,
>>>> +};
>>>> +
>>>> +/*---------- PART1: Input Parameters ------------*/
>>>> +
>>>> +struct cifisp_window {
>>>> +	__u16 h_offs;
>>>> +	__u16 v_offs;
>>>> +	__u16 h_size;
>>>> +	__u16 v_size;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_bls_fixed_val - BLS fixed subtraction values
>>>> + *
>>>> + * The values will be subtracted from the sensor
>>>> + * values. Therefore a negative value means addition instead of subtraction!
>>>> + *
>>>> + * @r: Fixed (signed!) subtraction value for Bayer pattern R
>>>> + * @gr: Fixed (signed!) subtraction value for Bayer pattern Gr
>>>> + * @gb: Fixed (signed!) subtraction value for Bayer pattern Gb
>>>> + * @b: Fixed (signed!) subtraction value for Bayer pattern B
>>>> + */
>>>> +struct cifisp_bls_fixed_val {
>>>> +	__s16 r;
>>>> +	__s16 gr;
>>>> +	__s16 gb;
>>>> +	__s16 b;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_bls_config - Configuration used by black level subtraction
>>>> + *
>>>> + * @enable_auto: Automatic mode activated means that the measured values
>>>> + *		 are subtracted. Otherwise the fixed subtraction
>>>> + *		 values will be subtracted.
>>>> + * @en_windows: enabled window
>>>> + * @bls_window1: Measurement window 1 size
>>>> + * @bls_window2: Measurement window 2 size
>>>> + * @bls_samples: Set amount of measured pixels for each Bayer position
>>>> + *		 (A, B,C and D) to 2^bls_samples.
>>>> + * @cifisp_bls_fixed_val: Fixed subtraction values
>>>> + */
>>>> +struct cifisp_bls_config {
>>>> +	__u8 enable_auto;
>>>> +	__u8 en_windows;
>>>> +	struct cifisp_window bls_window1;
>>>> +	struct cifisp_window bls_window2;
>>>> +	__u8 bls_samples;
>>>> +	struct cifisp_bls_fixed_val fixed_val;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpcc_methods_config - Methods Configuration used by DPCC
>>>> + *
>>>> + * Methods Configuration used by Defect Pixel Cluster Correction
>>>> + *
>>>> + * @method: Method enable bits
>>>> + * @line_thresh: Line threshold
>>>> + * @line_mad_fac: Line MAD factor
>>>> + * @pg_fac: Peak gradient factor
>>>> + * @rnd_thresh: Rank Neighbor Difference threshold
>>>> + * @rg_fac: Rank gradient factor
>>>> + */
>>>> +struct cifisp_dpcc_methods_config {
>>>> +	__u32 method;
>>>> +	__u32 line_thresh;
>>>> +	__u32 line_mad_fac;
>>>> +	__u32 pg_fac;
>>>> +	__u32 rnd_thresh;
>>>> +	__u32 rg_fac;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpcc_methods_config - Configuration used by DPCC
>>>
>>> Should be cifisp_dpcc_config.
>>>
>>>> + *
>>>> + * Configuration used by Defect Pixel Cluster Correction
>>>> + *
>>>> + * @mode: dpcc output mode
>>>> + * @output_mode: whether use hard coded methods
>>>> + * @set_use: stage1 methods set
>>>> + * @methods: methods config
>>>> + * @ro_limits: rank order limits
>>>> + * @rnd_offs: differential rank offsets for rank neighbor difference
>>>> + */
>>>> +struct cifisp_dpcc_config {
>>>> +	__u32 mode;
>>>> +	__u32 output_mode;
>>>> +	__u32 set_use;
>>>> +	struct cifisp_dpcc_methods_config methods[CIFISP_DPCC_METHODS_MAX];
>>>> +	__u32 ro_limits;
>>>> +	__u32 rnd_offs;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +struct cifisp_gamma_corr_curve {
>>>> +	__u16 gamma_y[CIFISP_DEGAMMA_CURVE_SIZE];
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +struct cifisp_gamma_curve_x_axis_pnts {
>>>> +	__u32 gamma_dx0;
>>>> +	__u32 gamma_dx1;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_gamma_corr_curve - Configuration used by sensor degamma
>>>
>>> Should be cifisp_sdg_config
>>>
>>>> + *
>>>> + * @curve_x: gamma curve point definition axis for x
>>>> + * @xa_pnts: x increments
>>>> + */
>>>> +struct cifisp_sdg_config {
>>>> +	struct cifisp_gamma_corr_curve curve_r;
>>>> +	struct cifisp_gamma_corr_curve curve_g;
>>>> +	struct cifisp_gamma_corr_curve curve_b;
>>>> +	struct cifisp_gamma_curve_x_axis_pnts xa_pnts;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_lsc_config - Configuration used by Lens shading correction
>>>> + *
>>>> + * refer to REF_01 for details
>>>> + */
>>>> +struct cifisp_lsc_config {
>>>> +	__u32 r_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>>>> +	__u32 gr_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>>>> +	__u32 gb_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>>>> +	__u32 b_data_tbl[CIFISP_LSC_DATA_TBL_SIZE];
>>>> +
>>>> +	__u32 x_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
>>>> +	__u32 y_grad_tbl[CIFISP_LSC_GRAD_TBL_SIZE];
>>>> +
>>>> +	__u32 x_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
>>>> +	__u32 y_size_tbl[CIFISP_LSC_SIZE_TBL_SIZE];
>>>> +	__u16 config_width;
>>>> +	__u16 config_height;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_ie_config - Configuration used by image effects
>>>> + *
>>>> + * @eff_mat_1: 3x3 Matrix Coefficients for Emboss Effect 1
>>>> + * @eff_mat_2: 3x3 Matrix Coefficients for Emboss Effect 2
>>>> + * @eff_mat_3: 3x3 Matrix Coefficients for Emboss 3/Sketch 1
>>>> + * @eff_mat_4: 3x3 Matrix Coefficients for Sketch Effect 2
>>>> + * @eff_mat_5: 3x3 Matrix Coefficients for Sketch Effect 3
>>>> + * @eff_tint: Chrominance increment values of tint (used for sepia effect)
>>>> + */
>>>> +struct cifisp_ie_config {
>>>> +	__u16 effect;
>>>> +	__u16 color_sel;
>>>> +	__u16 eff_mat_1;
>>>> +	__u16 eff_mat_2;
>>>> +	__u16 eff_mat_3;
>>>> +	__u16 eff_mat_4;
>>>> +	__u16 eff_mat_5;
>>>> +	__u16 eff_tint;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_cproc_config - Configuration used by Color Processing
>>>> + *
>>>> + * @c_out_range: Chrominance pixel clipping range at output.
>>>> + *		 (0 for limit, 1 for full)
>>>> + * @y_in_range: Luminance pixel clipping range at output.
>>>> + * @y_out_range: Luminance pixel clipping range at output.
>>>> + * @contrast: 00~ff, 0.0~1.992
>>>> + * @brightness: 80~7F, -128~+127
>>>> + * @sat: saturation, 00~FF, 0.0~1.992
>>>> + * @hue: 80~7F, -90~+87.188
>>>> + */
>>>> +struct cifisp_cproc_config {
>>>> +	__u8 c_out_range;
>>>> +	__u8 y_in_range;
>>>> +	__u8 y_out_range;
>>>> +	__u8 contrast;
>>>> +	__u8 brightness;
>>>> +	__u8 sat;
>>>> +	__u8 hue;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_awb_meas_config - Configuration used by auto white balance
>>>> + *
>>>> + * @awb_wnd: white balance measurement window (in pixels)
>>>> + *	     (from enum cifisp_awb_mode_type)
>>>> + * @max_y: only pixels values < max_y contribute to awb measurement, set to 0
>>>> + *	   to disable this feature
>>>> + * @min_y: only pixels values > min_y contribute to awb measurement
>>>> + * @max_csum: Chrominance sum maximum value, only consider pixels with Cb+Cr,
>>>> + *	      smaller than threshold for awb measurements
>>>> + * @min_c: Chrominance minimum value, only consider pixels with Cb/Cr
>>>> + *	   each greater than threshold value for awb measurements
>>>> + * @frames: number of frames - 1 used for mean value calculation
>>>> + *	    (ucFrames=0 means 1 Frame)
>>>> + * @awb_ref_cr: reference Cr value for AWB regulation, target for AWB
>>>> + * @awb_ref_cb: reference Cb value for AWB regulation, target for AWB
>>>> + */
>>>> +struct cifisp_awb_meas_config {
>>>> +	/*
>>>> +	 * Note: currently the h and v offsets are mapped to grid offsets
>>>> +	 */
>>>> +	struct cifisp_window awb_wnd;
>>>> +	__u32 awb_mode;
>>>> +	__u8 max_y;
>>>> +	__u8 min_y;
>>>> +	__u8 max_csum;
>>>> +	__u8 min_c;
>>>> +	__u8 frames;
>>>> +	__u8 awb_ref_cr;
>>>> +	__u8 awb_ref_cb;
>>>> +	__u8 enable_ymax_cmp;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_awb_gain_config - Configuration used by auto white balance gain
>>>> + *
>>>> + * out_data_x = ( AWB_GEAIN_X * in_data + 128) >> 8
>>>> + */
>>>> +struct cifisp_awb_gain_config {
>>>> +	__u16 gain_red;
>>>> +	__u16 gain_green_r;
>>>> +	__u16 gain_blue;
>>>> +	__u16 gain_green_b;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_flt_config - Configuration used by ISP filtering
>>>> + *
>>>> + * @mode: ISP_FILT_MODE register fields (from enum cifisp_flt_mode)
>>>> + * @grn_stage1: ISP_FILT_MODE register fields
>>>> + * @chr_h_mode: ISP_FILT_MODE register fields
>>>> + * @chr_v_mode: ISP_FILT_MODE register fields
>>>> + *
>>>> + * refer to REF_01 for details.
>>>> + */
>>>> +
>>>> +struct cifisp_flt_config {
>>>> +	__u32 mode;
>>>> +	__u8 grn_stage1;
>>>> +	__u8 chr_h_mode;
>>>> +	__u8 chr_v_mode;
>>>> +	__u32 thresh_bl0;
>>>> +	__u32 thresh_bl1;
>>>> +	__u32 thresh_sh0;
>>>> +	__u32 thresh_sh1;
>>>> +	__u32 lum_weight;
>>>> +	__u32 fac_sh1;
>>>> +	__u32 fac_sh0;
>>>> +	__u32 fac_mid;
>>>> +	__u32 fac_bl0;
>>>> +	__u32 fac_bl1;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_bdm_config - Configuration used by Bayer DeMosaic
>>>> + *
>>>> + * @demosaic_th: threshod for bayer demosaicing texture detection
>>>> + */
>>>> +struct cifisp_bdm_config {
>>>> +	__u8 demosaic_th;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_ctk_config - Configuration used by Cross Talk correction
>>>> + *
>>>> + * @coeff: color correction matrix
>>>> + * @ct_offset_b: offset for the crosstalk correction matrix
>>>> + */
>>>> +struct cifisp_ctk_config {
>>>> +	__u16 coeff0;
>>>> +	__u16 coeff1;
>>>> +	__u16 coeff2;
>>>> +	__u16 coeff3;
>>>> +	__u16 coeff4;
>>>> +	__u16 coeff5;
>>>> +	__u16 coeff6;
>>>> +	__u16 coeff7;
>>>> +	__u16 coeff8;
>>>> +	__u16 ct_offset_r;
>>>> +	__u16 ct_offset_g;
>>>> +	__u16 ct_offset_b;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +enum cifisp_goc_mode {
>>>> +	CIFISP_GOC_MODE_LOGARITHMIC,
>>>> +	CIFISP_GOC_MODE_EQUIDISTANT
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct cifisp_goc_config - Configuration used by Gamma Out correction
>>>> + *
>>>> + * @mode: goc mode (from enum cifisp_goc_mode)
>>>> + * @gamma_y: gamma out curve y-axis for all color components
>>>> + */
>>>> +struct cifisp_goc_config {
>>>> +	__u32 mode;
>>>> +	__u16 gamma_y[CIFISP_GAMMA_OUT_MAX_SAMPLES];
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_hst_config - Configuration used by Histogram
>>>> + *
>>>> + * @mode: histogram mode (from enum cifisp_histogram_mode)
>>>> + * @histogram_predivider: process every stepsize pixel, all other pixels are
>>>> + *			  skipped
>>>> + * @meas_window: coordinates of the measure window
>>>> + * @hist_weight: weighting factor for sub-windows
>>>> + */
>>>> +struct cifisp_hst_config {
>>>> +	__u32 mode;
>>>> +	__u8 histogram_predivider;
>>>> +	struct cifisp_window meas_window;
>>>> +	__u8 hist_weight[CIFISP_HISTOGRAM_WEIGHT_GRIDS_SIZE];
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_aec_config - Configuration used by Auto Exposure Control
>>>> + *
>>>> + * @mode: Exposure measure mode (from enum cifisp_exp_meas_mode)
>>>> + * @autostop: stop mode (from enum cifisp_exp_ctrl_autostop)
>>>> + * @meas_window: coordinates of the measure window
>>>> + */
>>>> +struct cifisp_aec_config {
>>>> +	__u32 mode;
>>>> +	__u32 autostop;
>>>> +	struct cifisp_window meas_window;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_afc_config - Configuration used by Auto Focus Control
>>>> + *
>>>> + * @num_afm_win: max CIFISP_AFM_MAX_WINDOWS
>>>> + * @afm_win: coordinates of the meas window
>>>> + * @thres: threshold used for minimizing the influence of noise
>>>> + * @var_shift: the number of bits for the shift operation at the end of the
>>>> + *	       calculation chain.
>>>> + */
>>>> +struct cifisp_afc_config {
>>>> +	__u8 num_afm_win;
>>>> +	struct cifisp_window afm_win[CIFISP_AFM_MAX_WINDOWS];
>>>> +	__u32 thres;
>>>> +	__u32 var_shift;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * enum cifisp_dpf_gain_usage - dpf gain usage
>>>> + * @CIFISP_DPF_GAIN_USAGE_DISABLED: don't use any gains in preprocessing stage
>>>> + * @CIFISP_DPF_GAIN_USAGE_NF_GAINS: use only the noise function gains from
>>>> + *				    registers DPF_NF_GAIN_R, ...
>>>> + * @CIFISP_DPF_GAIN_USAGE_LSC_GAINS:  use only the gains from LSC module
>>>> + * @CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS: use the noise function gains and the
>>>> + *					gains from LSC module
>>>> + * @CIFISP_DPF_GAIN_USAGE_AWB_GAINS: use only the gains from AWB module
>>>> + * @CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS: use the gains from AWB and LSC module
>>>> + * @CIFISP_DPF_GAIN_USAGE_MAX: upper border (only for an internal evaluation)
>>>> + */
>>>> +enum cifisp_dpf_gain_usage {
>>>> +	CIFISP_DPF_GAIN_USAGE_DISABLED,
>>>> +	CIFISP_DPF_GAIN_USAGE_NF_GAINS,
>>>> +	CIFISP_DPF_GAIN_USAGE_LSC_GAINS,
>>>> +	CIFISP_DPF_GAIN_USAGE_NF_LSC_GAINS,
>>>> +	CIFISP_DPF_GAIN_USAGE_AWB_GAINS,
>>>> +	CIFISP_DPF_GAIN_USAGE_AWB_LSC_GAINS,
>>>> +	CIFISP_DPF_GAIN_USAGE_MAX
>>>> +};
>>>> +
>>>> +/**
>>>> + * enum cifisp_dpf_gain_usage - dpf gain usage
>>>
>>> This should be cifisp_dpf_rb_filtersize and the description should be
>>> updated.
>>>
>>>> + * @CIFISP_DPF_RB_FILTERSIZE_13x9: red and blue filter kernel size 13x9
>>>> + *				   (means 7x5 active pixel)
>>>> + * @CIFISP_DPF_RB_FILTERSIZE_9x9: red and blue filter kernel size 9x9
>>>> + *				   (means 5x5 active pixel)
>>>> + */
>>>> +enum cifisp_dpf_rb_filtersize {
>>>> +	CIFISP_DPF_RB_FILTERSIZE_13x9,
>>>> +	CIFISP_DPF_RB_FILTERSIZE_9x9,
>>>> +};
>>>> +
>>>> +/**
>>>> + * enum cifisp_dpf_nll_scale_mode - dpf noise level scale mode
>>>> + * @CIFISP_NLL_SCALE_LINEAR: use a linear scaling
>>>> + * @CIFISP_NLL_SCALE_LOGARITHMIC: use a logarithmic scaling
>>>> + */
>>>> +enum cifisp_dpf_nll_scale_mode {
>>>> +	CIFISP_NLL_SCALE_LINEAR,
>>>> +	CIFISP_NLL_SCALE_LOGARITHMIC,
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpf_nll - Noise level lookup
>>>> + *
>>>> + * @coeff: Noise level Lookup coefficient
>>>> + * @scale_mode: dpf noise level scale mode (from enum cifisp_dpf_nll_scale_mode)
>>>> + */
>>>> +struct cifisp_dpf_nll {
>>>> +	__u16 coeff[CIFISP_DPF_MAX_NLF_COEFFS];
>>>> +	__u32 scale_mode;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpf_rb_flt - Red blue filter config
>>>> + *
>>>> + * @fltsize: The filter size for the red and blue pixels
>>>> + *	     (from enum cifisp_dpf_rb_filtersize)
>>>> + * @spatial_coeff: Spatial weights
>>>> + * @r_enable: enable filter processing for red pixels
>>>> + * @b_enable: enable filter processing for blue pixels
>>>> + */
>>>> +struct cifisp_dpf_rb_flt {
>>>> +	__u32 fltsize;
>>>> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
>>>> +	__u8 r_enable;
>>>> +	__u8 b_enable;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpf_g_flt - Green filter Configuration
>>>> + *
>>>> + * @spatial_coeff: Spatial weights
>>>> + * @gr_enable: enable filter processing for green pixels in green/red lines
>>>> + * @gb_enable: enable filter processing for green pixels in green/blue lines
>>>> + */
>>>> +struct cifisp_dpf_g_flt {
>>>> +	__u8 spatial_coeff[CIFISP_DPF_MAX_SPATIAL_COEFFS];
>>>> +	__u8 gr_enable;
>>>> +	__u8 gb_enable;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpf_gain - Noise function Configuration
>>>> + *
>>>> + * @mode: dpf gain usage  (from enum cifisp_dpf_gain_usage)
>>>> + * @nf_r_gain: Noise function Gain that replaces the AWB gain for red pixels
>>>> + * @nf_b_gain: Noise function Gain that replaces the AWB gain for blue pixels
>>>> + * @nf_gr_gain: Noise function Gain that replaces the AWB gain
>>>> + *		for green pixels in a red line
>>>> + * @nf_gb_gain: Noise function Gain that replaces the AWB gain
>>>> + *		for green pixels in a blue line
>>>> + */
>>>> +struct cifisp_dpf_gain {
>>>> +	__u32 mode;
>>>> +	__u16 nf_r_gain;
>>>> +	__u16 nf_b_gain;
>>>> +	__u16 nf_gr_gain;
>>>> +	__u16 nf_gb_gain;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpf_config - Configuration used by De-noising pre-filter
>>>> + *
>>>> + * @gain: noise function gain
>>>> + * @g_flt: green filter config
>>>> + * @rb_flt: red blue filter config
>>>> + * @nll: noise level lookup
>>>> + */
>>>> +struct cifisp_dpf_config {
>>>> +	struct cifisp_dpf_gain gain;
>>>> +	struct cifisp_dpf_g_flt g_flt;
>>>> +	struct cifisp_dpf_rb_flt rb_flt;
>>>> +	struct cifisp_dpf_nll nll;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_dpf_strength_config - strength of the filter
>>>> + *
>>>> + * @r: filter strength of the RED filter
>>>> + * @g: filter strength of the GREEN filter
>>>> + * @b: filter strength of the BLUE filter
>>>> + */
>>>> +struct cifisp_dpf_strength_config {
>>>> +	__u8 r;
>>>> +	__u8 g;
>>>> +	__u8 b;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_isp_other_cfg - Parameters for some blocks in rockchip isp1
>>>> + *
>>>> + * @dpcc_config: Defect Pixel Cluster Correction config
>>>> + * @bls_config: Black Level Subtraction config
>>>> + * @sdg_config: sensor degamma config
>>>> + * @lsc_config: Lens Shade config
>>>> + * @awb_gain_config: Auto White balance gain config
>>>> + * @flt_config: filter config
>>>> + * @bdm_config: demosaic config
>>>> + * @ctk_config: cross talk config
>>>> + * @goc_config: gamma out config
>>>> + * @bls_config: black level subtraction config
>>>> + * @dpf_config: De-noising pre-filter config
>>>> + * @dpf_strength_config: dpf strength config
>>>> + * @cproc_config: color process config
>>>> + * @ie_config: image effects config
>>>> + */
>>>> +struct cifisp_isp_other_cfg {
>>>> +	struct cifisp_dpcc_config dpcc_config;
>>>> +	struct cifisp_bls_config bls_config;
>>>> +	struct cifisp_sdg_config sdg_config;
>>>> +	struct cifisp_lsc_config lsc_config;
>>>> +	struct cifisp_awb_gain_config awb_gain_config;
>>>> +	struct cifisp_flt_config flt_config;
>>>> +	struct cifisp_bdm_config bdm_config;
>>>> +	struct cifisp_ctk_config ctk_config;
>>>> +	struct cifisp_goc_config goc_config;
>>>> +	struct cifisp_dpf_config dpf_config;
>>>> +	struct cifisp_dpf_strength_config dpf_strength_config;
>>>> +	struct cifisp_cproc_config cproc_config;
>>>> +	struct cifisp_ie_config ie_config;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_isp_meas_cfg - Rockchip ISP1 Measure Parameters
>>>> + *
>>>> + * @awb_meas_config: auto white balance config
>>>> + * @hst_config: histogram config
>>>> + * @aec_config: auto exposure config
>>>> + * @afc_config: auto focus config
>>>> + */
>>>> +struct cifisp_isp_meas_cfg {
>>>> +	struct cifisp_awb_meas_config awb_meas_config;
>>>> +	struct cifisp_hst_config hst_config;
>>>> +	struct cifisp_aec_config aec_config;
>>>> +	struct cifisp_afc_config afc_config;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct rkisp1_isp_params_cfg - Rockchip ISP1 Input Parameters Meta Data
>>>> + *
>>>> + * @module_en_update: mask the enable bits of which module should be updated
>>>> + * @module_ens: mask the enable value of each module, only update the module
>>>> + *		which correspond bit was set in module_en_update
>>>> + * @module_cfg_update: mask the config bits of which module should be updated
>>>> + * @meas: measurement config
>>>> + * @others: other config
>>>> + */
>>>> +struct rkisp1_isp_params_cfg {
>>>> +	__u32 module_en_update;
>>>> +	__u32 module_ens;
>>>> +	__u32 module_cfg_update;
>>>> +
>>>> +	struct cifisp_isp_meas_cfg meas;
>>>> +	struct cifisp_isp_other_cfg others;
>>>> +} __attribute__ ((packed));
>>>
>>> As far as I understand most (all ?) of the structures defined above are
>>> not used directly by the hardware, is that right ? In that case, do they
>>> need to be packed ?
>>
>> Hi,
>> I guess the 'packed' is a mean to serialize the struct since it is sent to userspace as
>> a buffer.
> 
> That's no different than using structures in an ioctl, is it ?
> 
>> I see that the ipu3 also uses 'packed' for the metadata formats.
> 
> For data structures that are directly consumed (or produced) by the
> hardware (or firmware) we often need the packed attribute. The
> structures used to describe the stastics produced by the rkisp1 need to
> be packed.

The statistics is also not consumed directly from the hardware. The driver
reads the statistics registers and write them to fields defined in the statistic
structure.

> 
> For structures that are consumed by software only, it's not necessarily
> a requirement. Packing has drawbacks, for instance it means we have to
> handle unaligned accesses explicitly on platforms that don't support
> them.
> 
>> If we don't have the 'packed' it means we relay on that
>> the userspace uses the same padding as the kernel for the struct.
>> Is it ok to assume that?
> 
> Isn't that dictated by the C ABI ? We of course need to ensure that
> there will be no 32/64-bit compatibility issues.

ok, I was not sure about that.

Dafna,

> 
>>> Some of the structures have a _cfg suffix, while others have a _config
>>> suffix. Should we standardise on one of the two ?
>>>
>>>> +
>>>> +/*---------- PART2: Measurement Statistics ------------*/
>>>> +
>>>> +/**
>>>> + * struct cifisp_bls_meas_val - AWB measured values
>>>
>>> This should be cifisp_awb_meas
>>>
>>>> + *
>>>> + * @cnt: White pixel count, number of "white pixels" found during laster
>>>
>>> s/laster/last/ ?
>>>
>>>> + *	 measurement
>>>> + * @mean_y_or_g: Mean value of Y within window and frames,
>>>> + *		 Green if RGB is selected.
>>>> + * @mean_cb_or_b: Mean value of Cb within window and frames,
>>>> + *		  Blue if RGB is selected.
>>>> + * @mean_cr_or_r: Mean value of Cr within window and frames,
>>>> + *		  Red if RGB is selected.
>>>> + */
>>>> +struct cifisp_awb_meas {
>>>> +	__u32 cnt;
>>>> +	__u8 mean_y_or_g;
>>>> +	__u8 mean_cb_or_b;
>>>> +	__u8 mean_cr_or_r;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_awb_stat - statistics automatic white balance data
>>>> + *
>>>> + * @awb_mean: Mean measured data
>>>> + */
>>>> +struct cifisp_awb_stat {
>>>> +	struct cifisp_awb_meas awb_mean[CIFISP_AWB_MAX_GRID];
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_bls_meas_val - BLS measured values
>>>> + *
>>>> + * @meas_r: Mean measured value for Bayer pattern R
>>>> + * @meas_gr: Mean measured value for Bayer pattern Gr
>>>> + * @meas_gb: Mean measured value for Bayer pattern Gb
>>>> + * @meas_b: Mean measured value for Bayer pattern B
>>>> + */
>>>> +struct cifisp_bls_meas_val {
>>>> +	__u16 meas_r;
>>>> +	__u16 meas_gr;
>>>> +	__u16 meas_gb;
>>>> +	__u16 meas_b;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_ae_stat - statistics auto exposure data
>>>> + *
>>>> + * @exp_mean: Mean luminance value of block xx
>>>> + * @bls_val:  BLS measured values
>>>> + *
>>>> + * Image is divided into 5x5 blocks.
>>>> + */
>>>> +struct cifisp_ae_stat {
>>>> +	__u8 exp_mean[CIFISP_AE_MEAN_MAX];
>>>> +	struct cifisp_bls_meas_val bls_val;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_af_meas_val - AF measured values
>>>> + *
>>>> + * @sum: sharpness, refer to REF_01 for definition
>>>> + * @lum: luminance, refer to REF_01 for definition
>>>
>>> That will be lovely to use without documentation...
>>>
>>>> + */
>>>> +struct cifisp_af_meas_val {
>>>> +	__u32 sum;
>>>> +	__u32 lum;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_af_stat - statistics auto focus data
>>>> + *
>>>> + * @window: AF measured value of window x
>>>> + *
>>>> + * The module measures the sharpness in 3 windows of selectable size via
>>>> + * register settings(ISP_AFM_*_A/B/C)
>>>> + */
>>>> +struct cifisp_af_stat {
>>>> +	struct cifisp_af_meas_val window[CIFISP_AFM_MAX_WINDOWS];
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct cifisp_hist_stat - statistics histogram data
>>>> + *
>>>> + * @hist_bins: measured bin counters
>>>> + *
>>>> + * Measurement window divided into 25 sub-windows, set
>>>> + * with ISP_HIST_XXX
>>>
>>> What is ISP_HIST_XXX ?
>>>
>>>> + */
>>>> +struct cifisp_hist_stat {
>>>> +	__u16 hist_bins[CIFISP_HIST_BIN_N_MAX];
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Data
>>>> + *
>>>> + * @cifisp_awb_stat: statistics data for automatic white balance
>>>> + * @cifisp_ae_stat: statistics data for auto exposure
>>>> + * @cifisp_af_stat: statistics data for auto focus
>>>> + * @cifisp_hist_stat: statistics histogram data
>>>> + */
>>>> +struct cifisp_stat {
>>>> +	struct cifisp_awb_stat awb;
>>>> +	struct cifisp_ae_stat ae;
>>>> +	struct cifisp_af_stat af;
>>>> +	struct cifisp_hist_stat hist;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +/**
>>>> + * struct rkisp1_stat_buffer - Rockchip ISP1 Statistics Meta Data
>>>> + *
>>>> + * @meas_type: measurement types (CIFISP_STAT_ definitions)
>>>> + * @frame_id: frame ID for sync
>>>> + * @params: statistics data
>>>> + */
>>>> +struct rkisp1_stat_buffer {
>>>> +	__u32 meas_type;
>>>> +	__u32 frame_id;
>>>> +	struct cifisp_stat params;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +#endif /* _UAPI_RKISP1_CONFIG_H */
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2019-08-16  0:13   ` Laurent Pinchart
@ 2020-07-11 11:04     ` Dafna Hirschfeld
  2020-07-17  7:46       ` Dafna Hirschfeld
  2020-07-22 15:24       ` Tomasz Figa
  0 siblings, 2 replies; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-07-11 11:04 UTC (permalink / raw)
  To: Laurent Pinchart, Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, sakari.ailus, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi Laurent,

On 16.08.19 02:13, Laurent Pinchart wrote:
> Hello Helen,
> 
> Thank you for the patch.
> 
> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>
>> Add the subdev driver for rockchip isp1.
>>
>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
>> Signed-off-by: Jacob Chen <cc@rock-chips.com>
>> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>> [fixed unknown entity type / switched to PIXEL_RATE]
>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>> [update for upstream]
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>
>> ---
>>
>> Changes in v8: None
>> Changes in v7:
>> - fixed warning because of unknown entity type
>> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
>> and default values
>> - fix typo riksp1/rkisp1
>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>> isp subdevice in the media topology now. As a consequence, remove the
>> hack in mipidphy_g_mbus_config() where information from the sensor was
>> being propagated through the topology.
>> - From the old dphy:
>>          * cache get_remote_sensor() in s_stream
>>          * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
>> - Replace stream state with a boolean
>> - code styling and checkpatch fixes
>> - fix stop_stream (return after calling stop, do not reenable the stream)
>> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
>> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
>> - s/intput/input
>> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
>> reused by the capture
>>
>>   drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>>   drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>>   2 files changed, 1397 insertions(+)
>>   create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>>   create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>>
>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
>> new file mode 100644
>> index 000000000000..6d0c0ffb5e03
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
>> @@ -0,0 +1,1286 @@
>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>> +/*
>> + * Rockchip isp1 driver
> 
> Shouldn't each file describe what it contains ? Maybe
> 
>   * Rockchip ISP1 Driver - ISP Core
> 
> for this one ? Same for other .c or .h files.
> 
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#include <linux/iopoll.h>
>> +#include <linux/phy/phy.h>
>> +#include <linux/phy/phy-mipi-dphy.h>
>> +#include <linux/pm_runtime.h>
>> +#include <linux/videodev2.h>
>> +#include <linux/vmalloc.h>
>> +#include <media/v4l2-event.h>
>> +
>> +#include "common.h"
>> +#include "regs.h"
> 
> common.h and regs.h aren't available yet. This won't break bisection as
> this file isn't referenced from the makefile yet, but it makes it a bit
> annoying when reviewing patches in order :-S
> 
>> +
>> +#define CIF_ISP_INPUT_W_MAX		4032
>> +#define CIF_ISP_INPUT_H_MAX		3024
>> +#define CIF_ISP_INPUT_W_MIN		32
>> +#define CIF_ISP_INPUT_H_MIN		32
>> +#define CIF_ISP_OUTPUT_W_MAX		CIF_ISP_INPUT_W_MAX
>> +#define CIF_ISP_OUTPUT_H_MAX		CIF_ISP_INPUT_H_MAX
>> +#define CIF_ISP_OUTPUT_W_MIN		CIF_ISP_INPUT_W_MIN
>> +#define CIF_ISP_OUTPUT_H_MIN		CIF_ISP_INPUT_H_MIN
>> +
>> +/*
>> + * NOTE: MIPI controller and input MUX are also configured in this file,
>> + * because ISP Subdev is not only describe ISP submodule(input size,format,
>> + * output size, format), but also a virtual route device.
>> + */
>> +
>> +/*
>> + * There are many variables named with format/frame in below code,
>> + * please see here for their meaning.
>> + *
>> + * Cropping regions of ISP
>> + *
>> + * +---------------------------------------------------------+
>> + * | Sensor image                                            |
>> + * | +---------------------------------------------------+   |
>> + * | | ISP_ACQ (for black level)                         |   |
>> + * | | in_frm                                            |   |
>> + * | | +--------------------------------------------+    |   |
>> + * | | |    ISP_OUT                                 |    |   |
>> + * | | |    in_crop                                 |    |   |
> 
> in_crop at the ISP output ? That seems a bit weird. I'm guessing that
> this is really the ISP output, while ISP_IS is related to the resizer ?
> 
>> + * | | |    +---------------------------------+     |    |   |
>> + * | | |    |   ISP_IS                        |     |    |   |
>> + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
>> + * | | |    +---------------------------------+     |    |   |
>> + * | | +--------------------------------------------+    |   |
>> + * | +---------------------------------------------------+   |
>> + * +---------------------------------------------------------+
>> + */
>> +
>> +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
>> +{
>> +	return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
>> +}
>> +
>> +/* Get sensor by enabled media link */
>> +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
>> +{
>> +	struct media_pad *local, *remote;
>> +	struct media_entity *sensor_me;
>> +
>> +	local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
>> +	remote = media_entity_remote_pad(local);
>> +	if (!remote) {
>> +		v4l2_warn(sd, "No link between isp and sensor\n");
>> +		return NULL;
>> +	}
>> +
>> +	sensor_me = media_entity_remote_pad(local)->entity;
>> +	return media_entity_to_v4l2_subdev(sensor_me);
>> +}
>> +
>> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
>> +					       struct v4l2_subdev *sd)
>> +{
>> +	struct rkisp1_sensor *sensor;
>> +
>> +	list_for_each_entry(sensor, &dev->sensors, list)
>> +		if (sensor->sd == sd)
>> +			return sensor;
>> +
>> +	return NULL;
>> +}
>> +
>> +/****************  register operations ****************/
>> +
>> +/*
>> + * Image Stabilization.
>> + * This should only be called when configuring CIF
>> + * or at the frame end interrupt
>> + */
>> +static void rkisp1_config_ism(struct rkisp1_device *dev)
>> +{
>> +	void __iomem *base = dev->base_addr;
>> +	struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
>> +	u32 val;
>> +
>> +	writel(0, base + CIF_ISP_IS_RECENTER);
> 
> How about read/write wrappers that take a rkisp1_device pointer, a
> register offset and a value (for the write wrapper) and compute
> dev->base_addr + offset internally ? That would make the code easier to
> read.
> 
>> +	writel(0, base + CIF_ISP_IS_MAX_DX);
>> +	writel(0, base + CIF_ISP_IS_MAX_DY);
>> +	writel(0, base + CIF_ISP_IS_DISPLACE);
>> +	writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
>> +	writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
>> +	writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
>> +	writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
>> +
>> +	/* IS(Image Stabilization) is always on, working as output crop */
>> +	writel(1, base + CIF_ISP_IS_CTRL);
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD;
>> +	writel(val, base + CIF_ISP_CTRL);
>> +}
>> +
>> +/*
>> + * configure isp blocks with input format, size......
>> + */
>> +static int rkisp1_config_isp(struct rkisp1_device *dev)
>> +{
>> +	u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
>> +	struct v4l2_rect *out_crop, *in_crop;
>> +	void __iomem *base = dev->base_addr;
>> +	struct v4l2_mbus_framefmt *in_frm;
>> +	struct ispsd_out_fmt *out_fmt;
>> +	struct rkisp1_sensor *sensor;
>> +	struct ispsd_in_fmt *in_fmt;
>> +
>> +	sensor = dev->active_sensor;
>> +	in_frm = &dev->isp_sdev.in_frm;
>> +	in_fmt = &dev->isp_sdev.in_fmt;
>> +	out_fmt = &dev->isp_sdev.out_fmt;
>> +	out_crop = &dev->isp_sdev.out_crop;
>> +	in_crop = &dev->isp_sdev.in_crop;
>> +
>> +	if (in_fmt->fmt_type == FMT_BAYER) {
>> +		acq_mult = 1;
>> +		if (out_fmt->fmt_type == FMT_BAYER) {
>> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
>> +				isp_ctrl =
>> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
>> +			else
>> +				isp_ctrl =
>> +					CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
>> +		} else {
>> +			writel(CIF_ISP_DEMOSAIC_TH(0xc),
>> +			       base + CIF_ISP_DEMOSAIC);
>> +
>> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
>> +			else
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
>> +		}
>> +	} else if (in_fmt->fmt_type == FMT_YUV) {
>> +		acq_mult = 2;
>> +		if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>> +			isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
>> +		} else {
>> +			if (sensor->mbus.type == V4L2_MBUS_BT656)
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
>> +			else
>> +				isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
>> +
>> +		}
>> +
>> +		irq_mask |= CIF_ISP_DATA_LOSS;
>> +	}
>> +
>> +	/* Set up input acquisition properties */
>> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
>> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>> +		if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
>> +			signal = CIF_ISP_ACQ_PROP_POS_EDGE;
>> +	}
>> +
>> +	if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>> +		if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
>> +			signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
>> +
>> +		if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
>> +			signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
>> +	}
>> +
>> +	writel(isp_ctrl, base + CIF_ISP_CTRL);
>> +	writel(signal | in_fmt->yuv_seq |
>> +	       CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
>> +	       CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
>> +	writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
>> +
>> +	/* Acquisition Size */
>> +	writel(0, base + CIF_ISP_ACQ_H_OFFS);
>> +	writel(0, base + CIF_ISP_ACQ_V_OFFS);
>> +	writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
>> +	writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
>> +
>> +	/* ISP Out Area */
>> +	writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
>> +	writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
>> +	writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
>> +	writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
>> +
>> +	/* interrupt mask */
>> +	irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
>> +		    CIF_ISP_FRAME_IN;
>> +	writel(irq_mask, base + CIF_ISP_IMSC);
>> +
>> +	if (out_fmt->fmt_type == FMT_BAYER)
>> +		rkisp1_params_disable_isp(&dev->params_vdev);
>> +	else
>> +		rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
>> +					    dev->isp_sdev.quantization);
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_config_dvp(struct rkisp1_device *dev)
>> +{
>> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val, input_sel;
>> +
>> +	switch (in_fmt->bus_width) {
>> +	case 8:
>> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
>> +		break;
>> +	case 10:
>> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
>> +		break;
>> +	case 12:
>> +		input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
>> +		break;
>> +	default:
>> +		v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	val = readl(base + CIF_ISP_ACQ_PROP);
>> +	writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_config_mipi(struct rkisp1_device *dev)
>> +{
>> +	struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
>> +	struct rkisp1_sensor *sensor = dev->active_sensor;
>> +	void __iomem *base = dev->base_addr;
>> +	unsigned int lanes;
>> +	u32 mipi_ctrl;
>> +
>> +	/*
>> +	 * sensor->mbus is set in isp or d-phy notifier_bound function
>> +	 */
>> +	switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
>> +	case V4L2_MBUS_CSI2_4_LANE:
>> +		lanes = 4;
>> +		break;
>> +	case V4L2_MBUS_CSI2_3_LANE:
>> +		lanes = 3;
>> +		break;
>> +	case V4L2_MBUS_CSI2_2_LANE:
>> +		lanes = 2;
>> +		break;
>> +	case V4L2_MBUS_CSI2_1_LANE:
>> +		lanes = 1;
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
>> +		    CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
>> +		    CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
>> +		    CIF_MIPI_CTRL_CLOCKLANE_ENA;
>> +
>> +	writel(mipi_ctrl, base + CIF_MIPI_CTRL);
>> +
>> +	/* Configure Data Type and Virtual Channel */
>> +	writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
>> +	       base + CIF_MIPI_IMG_DATA_SEL);
>> +
>> +	/* Clear MIPI interrupts */
>> +	writel(~0, base + CIF_MIPI_ICR);
>> +	/*
>> +	 * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
>> +	 * isp bus may be dead when switch isp.
>> +	 */
>> +	writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
>> +	       CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
>> +	       base + CIF_MIPI_IMSC);
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
>> +		 "  MIPI_IMG_DATA_SEL 0x%08x\n"
>> +		 "  MIPI_STATUS 0x%08x\n"
>> +		 "  MIPI_IMSC 0x%08x\n",
>> +		 readl(base + CIF_MIPI_CTRL),
>> +		 readl(base + CIF_MIPI_IMG_DATA_SEL),
>> +		 readl(base + CIF_MIPI_STATUS),
>> +		 readl(base + CIF_MIPI_IMSC));
>> +
>> +	return 0;
>> +}
>> +
>> +/* Configure MUX */
>> +static int rkisp1_config_path(struct rkisp1_device *dev)
>> +{
>> +	struct rkisp1_sensor *sensor = dev->active_sensor;
>> +	u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
>> +	int ret = 0;
>> +
>> +	if (sensor->mbus.type == V4L2_MBUS_BT656 ||
>> +	    sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>> +		ret = rkisp1_config_dvp(dev);
>> +		dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
>> +	} else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>> +		ret = rkisp1_config_mipi(dev);
>> +		dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
>> +	}
>> +
>> +	writel(dpcl, dev->base_addr + CIF_VI_DPCL);
>> +
>> +	return ret;
>> +}
>> +
>> +/* Hareware configure Entry */
> 
> s/Hareware/Hardware/
> 
>> +static int rkisp1_config_cif(struct rkisp1_device *dev)
>> +{
>> +	u32 cif_id;
>> +	int ret;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming);
>> +
>> +	cif_id = readl(dev->base_addr + CIF_VI_ID);
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
>> +
>> +	ret = rkisp1_config_isp(dev);
>> +	if (ret < 0)
>> +		return ret;
>> +	ret = rkisp1_config_path(dev);
>> +	if (ret < 0)
>> +		return ret;
>> +	rkisp1_config_ism(dev);
>> +
>> +	return 0;
>> +}
>> +
>> +/* Mess register operations to stop isp */
> 
> Is it such a mess ? :-)
> 
> I would capitalise ISP in all comments.
> 
>> +static int rkisp1_isp_stop(struct rkisp1_device *dev)
>> +{
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming);
>> +
>> +	/*
>> +	 * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
>> +	 * Stop ISP(isp) ->wait for ISP isp off
>> +	 */
>> +	/* stop and clear MI, MIPI, and ISP interrupts */
>> +	writel(0, base + CIF_MIPI_IMSC);
>> +	writel(~0, base + CIF_MIPI_ICR);
>> +
>> +	writel(0, base + CIF_ISP_IMSC);
>> +	writel(~0, base + CIF_ISP_ICR);
>> +
>> +	writel(0, base + CIF_MI_IMSC);
>> +	writel(~0, base + CIF_MI_ICR);
>> +	val = readl(base + CIF_MIPI_CTRL);
>> +	writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
>> +	/* stop ISP */
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
>> +	writel(val, base + CIF_ISP_CTRL);
>> +
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
>> +
>> +	readx_poll_timeout(readl, base + CIF_ISP_RIS,
>> +			   val, val & CIF_ISP_OFF, 20, 100);
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		"streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming,
>> +		 readl(base + CIF_MI_CTRL),
>> +		 readl(base + CIF_ISP_CTRL),
>> +		 readl(base + CIF_MIPI_CTRL));
>> +
>> +	writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
>> +	writel(0x0, base + CIF_IRCL);
>> +
>> +	return 0;
>> +}
>> +
>> +/* Mess register operations to start isp */
>> +static int rkisp1_isp_start(struct rkisp1_device *dev)
>> +{
>> +	struct rkisp1_sensor *sensor = dev->active_sensor;
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming);
>> +
>> +	/* Activate MIPI */
>> +	if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>> +		val = readl(base + CIF_MIPI_CTRL);
>> +		writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
>> +	}
>> +	/* Activate ISP */
>> +	val = readl(base + CIF_ISP_CTRL);
>> +	val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
>> +	       CIF_ISP_CTRL_ISP_INFORM_ENABLE;
>> +	writel(val, base + CIF_ISP_CTRL);
>> +
>> +	/* XXX: Is the 1000us too long?
>> +	 * CIF spec says to wait for sufficient time after enabling
>> +	 * the MIPI interface and before starting the sensor output.
>> +	 */
>> +	usleep_range(1000, 1200);
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
>> +		 "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
>> +		 dev->stream[RKISP1_STREAM_SP].streaming,
>> +		 dev->stream[RKISP1_STREAM_MP].streaming,
>> +		 readl(base + CIF_MI_CTRL),
>> +		 readl(base + CIF_ISP_CTRL),
>> +		 readl(base + CIF_MIPI_CTRL));
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_config_clk(struct rkisp1_device *dev)
>> +{
>> +	u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
>> +		  CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
>> +		  CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
>> +
>> +	writel(val, dev->base_addr + CIF_ICCL);
>> +}
>> +
>> +/***************************** isp sub-devs *******************************/
>> +
>> +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
>> +	{
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_BGGR,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_RGGB,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_GBRG,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW10,
>> +		.bayer_pat	= RAW_GRBG,
>> +		.bus_width	= 10,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_RGGB,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
> 
> Is there a reason why the 10-bit Bayer format start with BGGR while the
> 12-bit formats start with RGGB ? Not a big deal, just OCD kicking in :-)
> 
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_BGGR,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_GBRG,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW12,
>> +		.bayer_pat	= RAW_GRBG,
>> +		.bus_width	= 12,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_RGGB,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_BGGR,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_GBRG,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +		.mipi_dt	= CIF_CSI2_DT_RAW8,
>> +		.bayer_pat	= RAW_GRBG,
>> +		.bus_width	= 8,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCBYCR,
>> +		.bus_width	= 16,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_YVYU8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_YCRYCB,
>> +		.bus_width	= 16,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_UYVY8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CBYCRY,
>> +		.bus_width	= 16,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_VYUY8_1X16,
>> +		.fmt_type	= FMT_YUV,
>> +		.mipi_dt	= CIF_CSI2_DT_YUV422_8b,
>> +		.yuv_seq	= CIF_ISP_ACQ_PROP_CRYCBY,
>> +		.bus_width	= 16,
>> +	},
>> +};
>> +
>> +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
>> +	{
>> +		.mbus_code	= MEDIA_BUS_FMT_YUYV8_2X8,
>> +		.fmt_type	= FMT_YUV,
>> +	}, {
> 
> This is the only entry not present in the previous table, so I'm
> wondering if it would make sense to merge the two tables and rename
> ispsd_in_fmt to rkisp1_format_info. You would need to add a field that
> tells, for each format, if it's valid as an input format, and output
> format, or both. Hmmmm and also make the enum logic a bit more complex.
> Maybe it's not worth it after all, but it bothers me a bit to have two
> tables :-) I'll let you decide what's best.
> 
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG12_1X12,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG10_1X10,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SRGGB8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SBGGR8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGBRG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	}, {
>> +		.mbus_code	= MEDIA_BUS_FMT_SGRBG8_1X8,
>> +		.fmt_type	= FMT_BAYER,
>> +	},
>> +};
>> +
>> +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
>> +{
>> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
>> +	const struct ispsd_in_fmt *fmt;
> 
> You can move this variable inside the loop.
> 
>> +
>> +	for (i = 0; i < array_size; i++) {
> 
> I would remove the array_size local variable, it doesn't improve
> readability. Same for the next function.
> 
>> +		fmt = &rkisp1_isp_input_formats[i];
>> +		if (fmt->mbus_code == mbus_code)
>> +			return fmt;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
>> +{
>> +	unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
>> +	const struct ispsd_out_fmt *fmt;
>> +
>> +	for (i = 0; i < array_size; i++) {
>> +		fmt = &rkisp1_isp_output_formats[i];
>> +		if (fmt->mbus_code == mbus_code)
>> +			return fmt;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
>> +					struct v4l2_subdev_pad_config *cfg,
>> +					struct v4l2_subdev_mbus_code_enum *code)
>> +{
>> +	unsigned int i = code->index;
>> +
>> +	if ((code->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>> +		if (i > 0)
>> +			return -EINVAL;
>> +		code->code = MEDIA_BUS_FMT_FIXED;
>> +		return 0;
>> +	}
>> +
>> +	if (code->pad == RKISP1_ISP_PAD_SINK) {
>> +		if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
>> +			return -EINVAL;
>> +		code->code = rkisp1_isp_input_formats[i].mbus_code;
>> +	} else {
>> +		if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
>> +			return -EINVAL;
>> +		code->code = rkisp1_isp_output_formats[i].mbus_code;
>> +	}
> 
> On the other hand, merging the two tables above into one, you could
> merge the two branches here and only consider formats indicated as valid
> for to the pad you want. Maybe the code would be cleaner in the end.
> 
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
>> +				     struct v4l2_subdev_pad_config *cfg)
>> +{
>> +	struct v4l2_rect *mf_in_crop, *mf_out_crop;
>> +	struct v4l2_mbus_framefmt *mf_in, *mf_out;
>> +
>> +	mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
>> +	mf_in->width = RKISP1_DEFAULT_WIDTH;
>> +	mf_in->height = RKISP1_DEFAULT_HEIGHT;
>> +	mf_in->field = V4L2_FIELD_NONE;
>> +	mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
>> +
>> +	mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
>> +	mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
>> +	mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
>> +	mf_in_crop->left = 0;
>> +	mf_in_crop->top = 0;
>> +
>> +	mf_out = v4l2_subdev_get_try_format(sd, cfg,
>> +					    RKISP1_ISP_PAD_SOURCE_PATH);
>> +	*mf_out = *mf_in;
>> +	mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
>> +	mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +
>> +	mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
>> +					       RKISP1_ISP_PAD_SOURCE_PATH);
>> +	*mf_out_crop = *mf_in_crop;
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
>> +				 struct v4l2_subdev_pad_config *cfg,
>> +				 struct v4l2_subdev_format *fmt)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>> +
>> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>> +		fmt->format.code = MEDIA_BUS_FMT_FIXED;
>> +		/*
>> +		 * NOTE: setting a format here doesn't make much sense
>> +		 * but v4l2-compliance complains
>> +		 */
> 
> For the params pad I agreed it makes no sense, and I think
> v4l2-compliance is at fault, so I'd set width and height to 0. For the
> stats pad we *could* use the size of the image from which stats are
> computed, but because v4l2_meta_format has no width/height, I think 0
> would also be appropriate.
> 
>> +		fmt->format.width = RKISP1_DEFAULT_WIDTH;
>> +		fmt->format.height = RKISP1_DEFAULT_HEIGHT;
>> +		fmt->format.field = V4L2_FIELD_NONE;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>> +		fmt->format = *mf;
>> +		return 0;
>> +	}
>> +
>> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>> +		*mf = isp_sd->in_frm;
>> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		/* format of source pad */
>> +		*mf = isp_sd->in_frm;
>> +		mf->code = isp_sd->out_fmt.mbus_code;
>> +		/* window size of source pad */
>> +		mf->width = isp_sd->out_crop.width;
>> +		mf->height = isp_sd->out_crop.height;
>> +		mf->quantization = isp_sd->quantization;
>> +	}
>> +	mf->field = V4L2_FIELD_NONE;
> 
> This can be simplified, please read through, or jump to the review of
> rkisp1_isp_subdev at the end.
> 
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
>> +				  unsigned int pad,
>> +				  struct v4l2_mbus_framefmt *fmt)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>> +	const struct ispsd_out_fmt *out_fmt;
>> +	const struct ispsd_in_fmt *in_fmt;
>> +
>> +	switch (pad) {
>> +	case RKISP1_ISP_PAD_SINK:
>> +		in_fmt = find_in_fmt(fmt->code);
>> +		if (in_fmt)
>> +			fmt->code = in_fmt->mbus_code;
>> +		else
>> +			fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
> 
> You write MEDIA_BUS_FMT_SRGGB10_1X10 explicitly here, while you use
> rkisp1_isp_output_formats[0].mbus_code below (and in other places). I
> would standardise on one of the two (explicit formats or array[0]), with
> a preference for the first as that would allow merging the input and
> output arrays more easily. I would then create two #define,
> RKISP1_DEF_INPUT_FORMAT and RKISP2_DEF_OUTPUT_FORMAT (or similar).
> Similar macros for the default width and height could also be useful, to
> make it easier to change them.
> 
>> +		fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
>> +				      CIF_ISP_INPUT_W_MAX);
>> +		fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
>> +				      CIF_ISP_INPUT_H_MAX);
>> +		break;
>> +	case RKISP1_ISP_PAD_SOURCE_PATH:
>> +		out_fmt = find_out_fmt(fmt->code);
>> +		if (out_fmt)
>> +			fmt->code = out_fmt->mbus_code;
>> +		else
>> +			fmt->code = rkisp1_isp_output_formats[0].mbus_code;
>> +		/* window size is set in s_selection */
>> +		fmt->width  = isp_sd->out_crop.width;
>> +		fmt->height = isp_sd->out_crop.height;
> 
> This function operates on the TRY configuration too, in which case you
> should use the TRY crop rectangle here, not the ACTIVE one. If you've
> already jumped to the review of rkisp1_isp_subdev you know my proposal
> to simplify this. Otherwise now may be a good time to do so :-)
> 
>> +		/* full range by default */
>> +		if (!fmt->quantization)
>> +			fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +		break;
>> +	}
>> +
>> +	fmt->field = V4L2_FIELD_NONE;
>> +}
>> +
>> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
>> +				 struct v4l2_subdev_pad_config *cfg,
>> +				 struct v4l2_subdev_format *fmt)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>> +	struct v4l2_mbus_framefmt *mf = &fmt->format;
>> +
>> +	if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>> +	    (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
>> +		return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
>> +
>> +	rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
>> +
>> +	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		struct v4l2_mbus_framefmt *try_mf;
>> +
>> +		try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>> +		*try_mf = *mf;
>> +		return 0;
> 
> When setting the format on the sink pad the crop rectangles need to be
> reset (here, and for the ACTIVE format below too).
> 
>> +	}
>> +
>> +	if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>> +		const struct ispsd_in_fmt *in_fmt;
>> +
>> +		in_fmt = find_in_fmt(mf->code);
>> +		isp_sd->in_fmt = *in_fmt;
>> +		isp_sd->in_frm = *mf;
>> +	} else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		const struct ispsd_out_fmt *out_fmt;
>> +
>> +		/* Ignore width/height */
>> +		out_fmt = find_out_fmt(mf->code);
>> +		isp_sd->out_fmt = *out_fmt;
> 
> I would return the in_fmt and out_fmt from rkisp1_isp_sd_try_fmt() as it
> already looks them up. If you merge the input and output tables, you'll
> have a single format info structure type, and rkisp1_isp_sd_try_fmt()
> could return the entry for the pad it operates on.
> 
>> +		/*
>> +		 * It is quantization for output,
>> +		 * isp use bt601 limit-range in internal
>> +		 */
>> +		isp_sd->quantization = mf->quantization;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
>> +				  struct v4l2_subdev_pad_config *cfg,
>> +				  struct v4l2_subdev_selection *sel)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
>> +	struct v4l2_rect in_crop = isp_sd->in_crop;
>> +	struct v4l2_rect *input = &sel->r;
>> +
>> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		in_frm = *v4l2_subdev_get_try_format(sd, cfg,
>> +						     RKISP1_ISP_PAD_SINK);
>> +		in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
>> +						    RKISP1_ISP_PAD_SINK);
>> +	}
>> +
>> +	input->left = ALIGN(input->left, 2);
>> +	input->width = ALIGN(input->width, 2);
>> +
>> +	if (sel->pad == RKISP1_ISP_PAD_SINK) {
>> +		input->left = clamp_t(u32, input->left, 0, in_frm.width);
>> +		input->top = clamp_t(u32, input->top, 0, in_frm.height);
>> +		input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
>> +				       in_frm.width - input->left);
>> +		input->height = clamp_t(u32, input->height,
>> +					CIF_ISP_INPUT_H_MIN,
>> +					in_frm.height - input->top);
>> +	} else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>> +		input->left = clamp_t(u32, input->left, 0, in_crop.width);
>> +		input->top = clamp_t(u32, input->top, 0, in_crop.height);
>> +		input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
>> +				       in_crop.width - input->left);
>> +		input->height = clamp_t(u32, input->height,
>> +					CIF_ISP_OUTPUT_H_MIN,
>> +					in_crop.height - input->top);
>> +	}
>> +}
>> +
>> +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_pad_config *cfg,
>> +				       struct v4l2_subdev_selection *sel)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct v4l2_mbus_framefmt *frm;
>> +	struct v4l2_rect *rect;
>> +
>> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
>> +	    sel->pad != RKISP1_ISP_PAD_SINK)
>> +		return -EINVAL;
>> +
>> +	switch (sel->target) {
>> +	case V4L2_SEL_TGT_CROP_BOUNDS:
>> +		if (sel->pad == RKISP1_ISP_PAD_SINK) {
>> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>> +				frm = v4l2_subdev_get_try_format(sd, cfg,
>> +								 sel->pad);
>> +			else
>> +				frm = &isp_sd->in_frm;
>> +
>> +			sel->r.height = frm->height;
>> +			sel->r.width = frm->width;
>> +			sel->r.left = 0;
>> +			sel->r.top = 0;
>> +		} else {
>> +			if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>> +				rect = v4l2_subdev_get_try_crop(sd, cfg,
>> +							RKISP1_ISP_PAD_SINK);
>> +			else
>> +				rect = &isp_sd->in_crop;
>> +			sel->r = *rect;
>> +		}
>> +		break;
>> +	case V4L2_SEL_TGT_CROP:
>> +		if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>> +			rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
>> +		else if (sel->pad == RKISP1_ISP_PAD_SINK)
>> +			rect = &isp_sd->in_crop;
>> +		else
>> +			rect = &isp_sd->out_crop;
>> +		sel->r = *rect;
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
>> +				       struct v4l2_subdev_pad_config *cfg,
>> +				       struct v4l2_subdev_selection *sel)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>> +	struct rkisp1_device *dev = sd_to_isp_dev(sd);
>> +
>> +	if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
>> +	    sel->pad != RKISP1_ISP_PAD_SINK)
>> +		return -EINVAL;
>> +	if (sel->target != V4L2_SEL_TGT_CROP)
>> +		return -EINVAL;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>> +		 "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
>> +		 sel->r.left, sel->r.top, sel->r.width, sel->r.height);
>> +	rkisp1_isp_sd_try_crop(sd, cfg, sel);
>> +
>> +	if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
>> +		struct v4l2_rect *try_sel =
>> +			v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
>> +
>> +		*try_sel = sel->r;
>> +		return 0;
> 
> When setting the crop rectangle on the sink pad the rectangle on the
> source pad should be reset (and below for the ACTIVE format too).
> 
> The resizer logic, through selection rectangles, seems to be missing. I
> assume it's configured through the video nodes, but that's not correct
> I'm afraid, scaling should be configured on subdevs, see [1]. I'm afraid
> this means that we'll need separate subdevs for the resizers :-S
> 
> [1] https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html#order-of-configuration-and-format-propagation
> 
>> +	}
>> +
>> +	if (sel->pad == RKISP1_ISP_PAD_SINK)
>> +		isp_sd->in_crop = sel->r;
>> +	else
>> +		isp_sd->out_crop = sel->r;
>> +
>> +	return 0;
>> +}
>> +
> 
> I'll skip the part related to power on/off below as Sakari requested it
> to be handled through runtime PM.
> 
>> +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
>> +				    struct rkisp1_sensor *sensor)
>> +{
>> +	union phy_configure_opts opts = { 0 };
>> +	struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
>> +	struct v4l2_ctrl *pixel_rate;
>> +	s64 pixel_clock;
>> +
>> +	pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
>> +				    V4L2_CID_PIXEL_RATE);
>> +	if (!pixel_rate) {
>> +		v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
>> +		return -EPIPE;
>> +	}
> 
> Would it make sense to retrieve (and cache) the control pointer (not its
> value of course) at probe time already ?
> 
>> +
>> +	pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
>> +	if (!pixel_clock) {
>> +		v4l2_err(sensor->sd, "Invalid pixel rate value\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
>> +					 sensor->lanes, cfg);
>> +	phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
>> +	phy_configure(sensor->dphy, &opts);
>> +	phy_power_on(sensor->dphy);
>> +
>> +	return 0;
>> +}
>> +
>> +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
>> +{
>> +	phy_power_off(sensor->dphy);
>> +}
>> +
>> +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	struct v4l2_subdev *sensor_sd;
>> +	int ret = 0;
>> +
>> +	if (!on) {
>> +		ret = rkisp1_isp_stop(isp_dev);
>> +		if (ret < 0)
>> +			return ret;
>> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
>> +		return 0;
>> +	}
>> +
>> +	sensor_sd = get_remote_sensor(sd);
>> +	if (!sensor_sd)
>> +		return -ENODEV;
>> +
>> +	isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
>> +	if (!isp_dev->active_sensor)
>> +		return -ENODEV;
>> +
>> +	atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
>> +	ret = rkisp1_config_cif(isp_dev);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* TODO: support other interfaces */
>> +	if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
>> +		return -EINVAL;
>> +
>> +	ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
>> +				       isp_dev->active_sensor);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	ret = rkisp1_isp_start(isp_dev);
>> +	if (ret)
>> +		mipi_csi2_s_stream_stop(isp_dev->active_sensor);
>> +
>> +	return ret;
>> +}
>> +
>> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> +	struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>> +	int ret;
>> +
>> +	v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
>> +
>> +	if (on) {
>> +		ret = pm_runtime_get_sync(isp_dev->dev);
>> +		if (ret < 0)
>> +			return ret;
>> +
>> +		rkisp1_config_clk(isp_dev);
>> +	} else {
>> +		ret = pm_runtime_put(isp_dev->dev);
>> +		if (ret < 0)
>> +			return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int rkisp1_subdev_link_validate(struct media_link *link)
>> +{
>> +	if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
>> +		return 0;
>> +
>> +	return v4l2_subdev_link_validate(link);
>> +}
>> +
>> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
>> +					struct media_link *link,
>> +					struct v4l2_subdev_format *source_fmt,
>> +					struct v4l2_subdev_format *sink_fmt)
>> +{
>> +	if (source_fmt->format.code != sink_fmt->format.code)
>> +		return -EINVAL;
>> +
>> +	/* Crop is available */
>> +	if (source_fmt->format.width < sink_fmt->format.width ||
>> +	    source_fmt->format.height < sink_fmt->format.height)
>> +		return -EINVAL;
> 
> Crop should be handled on pads, not on links. The size at the ends of
> the link should match. This will likely require a bit of a redesign of
> the format and selection rectangles handling, but I think it's also an
> opportunity to simplify the code.
> 
>> +
>> +	return 0;
>> +}
>> +
>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>> +{
>> +	struct v4l2_event event = {
>> +		.type = V4L2_EVENT_FRAME_SYNC,
>> +		.u.frame_sync.frame_sequence =
>> +			atomic_inc_return(&isp->frm_sync_seq) - 1,
> 
> I would move the increment to the caller, hiding it in this function is
> error-prone (and if you look at the caller I'm pointing out one possible
> error :-)).
> 
> In general usage of frm_sync_seq through the driver seems to be very
> race-prone. It's read in various IRQ handling functions, all coming from
> the same IRQ, so that part is fine (and wouldn't require an atomic
> variable), but when read from the buffer queue handlers I really get a
> red light flashing in my head. I'll try to investigate more when
> reviewing the next patches.

I see that the only place were 'frame_sequence' is read outside of the irq
handlers is in the capture in 'rkisp1_vb2_buf_queue':

	/*
          * If there's no next buffer assigned, queue this buffer directly
          * as the next buffer, and update the memory interface.
          */
         if (cap->is_streaming && !cap->buf.next &&
             atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
                 cap->buf.next = ispbuf;
                 rkisp1_set_next_buf(cap);
         } else {
                 list_add_tail(&ispbuf->queue, &cap->buf.queue);
         }
This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
from the above condition so that the next buffer is updated in case it is null not just before the first
v-start signal.





> 
>> +	};
>> +	v4l2_event_queue(isp->sd.devnode, &event);
>> +}
>> +
>> +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>> +				  struct v4l2_event_subscription *sub)
>> +{
>> +	if (sub->type != V4L2_EVENT_FRAME_SYNC)
>> +		return -EINVAL;
>> +
>> +	/* Line number. For now only zero accepted. */
> 
> Is the id a line number for V4L2_EVENT_FRAME_SYNC ? It's not mentioned
> for V4L2_EVENT_FRAME_SYNC in the V4L2 spec, so I think this check is
> correct, but the comment doesn't seem to be.
> 
>> +	if (sub->id != 0)
>> +		return -EINVAL;
>> +
>> +	return v4l2_event_subscribe(fh, sub, 0, NULL);
>> +}
>> +
>> +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
>> +	.enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
>> +	.get_selection = rkisp1_isp_sd_get_selection,
>> +	.set_selection = rkisp1_isp_sd_set_selection,
>> +	.init_cfg = rkisp1_isp_sd_init_config,
>> +	.get_fmt = rkisp1_isp_sd_get_fmt,
>> +	.set_fmt = rkisp1_isp_sd_set_fmt,
>> +	.link_validate = rkisp1_subdev_fmt_link_validate,
>> +};
>> +
>> +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
>> +	.link_validate = rkisp1_subdev_link_validate,
>> +};
>> +
>> +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
>> +	.s_stream = rkisp1_isp_sd_s_stream,
>> +};
>> +
>> +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
>> +	.subscribe_event = rkisp1_isp_sd_subs_evt,
>> +	.unsubscribe_event = v4l2_event_subdev_unsubscribe,
>> +	.s_power = rkisp1_isp_sd_s_power,
>> +};
>> +
>> +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {
> 
> static const
> 
>> +	.core = &rkisp1_isp_core_ops,
>> +	.video = &rkisp1_isp_sd_video_ops,
>> +	.pad = &rkisp1_isp_sd_pad_ops,
>> +};
>> +
>> +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
>> +{
>> +	struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
>> +	struct v4l2_rect *in_crop = &isp_sd->in_crop;
>> +	struct v4l2_rect *out_crop = &isp_sd->out_crop;
>> +	struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
>> +	struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
>> +
>> +	*in_fmt = rkisp1_isp_input_formats[0];
>> +	in_frm->width = RKISP1_DEFAULT_WIDTH;
>> +	in_frm->height = RKISP1_DEFAULT_HEIGHT;
>> +	in_frm->code = in_fmt->mbus_code;
>> +
>> +	in_crop->width = in_frm->width;
>> +	in_crop->height = in_frm->height;
>> +	in_crop->left = 0;
>> +	in_crop->top = 0;
>> +
>> +	/* propagate to source */
>> +	*out_crop = *in_crop;
>> +	*out_fmt = rkisp1_isp_output_formats[0];
>> +	isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> 
> I wonder, could this be implemented by sharing code with .init_cfg() if
> you store the active configuration in v4l2_subdev_pad_config instances
> (please see the comments about the rkisp1_isp_subdev structure below) ?
> 
>> +}
>> +
>> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
>> +			       struct v4l2_device *v4l2_dev)
>> +{
>> +	struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
>> +	struct v4l2_subdev *sd = &isp_sdev->sd;
>> +	int ret;
>> +
>> +	v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
>> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>> +	sd->entity.ops = &rkisp1_isp_sd_media_ops;
>> +	snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
>> +
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
>> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
>> +	isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
>> +	sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
>> +	ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
>> +				     isp_sdev->pads);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	sd->owner = THIS_MODULE;
>> +	v4l2_set_subdevdata(sd, isp_dev);
>> +
>> +	sd->grp_id = GRP_ID_ISP;
> 
> I think you can drop this, as well as all the GRP_ID_* macros, they're
> not used.
> 
>> +	ret = v4l2_device_register_subdev(v4l2_dev, sd);
>> +	if (ret < 0) {
>> +		v4l2_err(sd, "Failed to register isp subdev\n");
>> +		goto err_cleanup_media_entity;
>> +	}
>> +
>> +	rkisp1_isp_sd_init_default_fmt(isp_sdev);
>> +
>> +	return 0;
>> +err_cleanup_media_entity:
>> +	media_entity_cleanup(&sd->entity);
>> +	return ret;
>> +}
>> +
>> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
>> +{
>> +	struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
>> +
>> +	v4l2_device_unregister_subdev(sd);
>> +	media_entity_cleanup(&sd->entity);
>> +}
>> +
>> +/****************  Interrupter Handler ****************/
> 
> s/Handler/Handlers/
> 
>> +
>> +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
> 
> I would make the mis field an u32 as it contains register values. Should
> it also be renamed to status ? Same for rkisp1_isp_isr() below.
> 
>> +{
>> +	struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
>> +	void __iomem *base = dev->base_addr;
>> +	u32 val;
>> +
> 
> How about moving the CIF_MIPI_MIS read here and removing the mis
> argument to the function ? It would be more logical to group all
> register access related to interrupts in a single place. Same for the
> other interrupt handler functions.
> 
>> +	writel(~0, base + CIF_MIPI_ICR);
>> +
>> +	/*
>> +	 * Disable DPHY errctrl interrupt, because this dphy
>> +	 * erctrl signal is asserted until the next changes
>> +	 * of line state. This time is may be too long and cpu
>> +	 * is hold in this interrupt.
>> +	 */
>> +	if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
>> +		val = readl(base + CIF_MIPI_IMSC);
>> +		writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
>> +		dev->isp_sdev.dphy_errctrl_disabled = true;
>> +	}
>> +
>> +	/*
>> +	 * Enable DPHY errctrl interrupt again, if mipi have receive
>> +	 * the whole frame without any error.
>> +	 */
>> +	if (mis == CIF_MIPI_FRAME_END) {
>> +		/*
>> +		 * Enable DPHY errctrl interrupt again, if mipi have receive
>> +		 * the whole frame without any error.
>> +		 */
>> +		if (dev->isp_sdev.dphy_errctrl_disabled) {
>> +			val = readl(base + CIF_MIPI_IMSC);
>> +			val |= CIF_MIPI_ERR_CTRL(0x0f);
>> +			writel(val, base + CIF_MIPI_IMSC);
>> +			dev->isp_sdev.dphy_errctrl_disabled = false;
>> +		}
>> +	} else {
>> +		v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
>> +	}
>> +}
>> +
>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>> +{
>> +	void __iomem *base = dev->base_addr;
>> +	unsigned int isp_mis_tmp = 0;
> 
> _tmp are never good names :-S
> 
>> +	unsigned int isp_err = 0;
> 
> Neither of these variable need to be initialised to 0.
> 
>> +
>> +	/* start edge of v_sync */
>> +	if (isp_mis & CIF_ISP_V_START) {
>> +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> 
> This will increment the frame sequence number. What if the interrupt is
> slightly delayed and the next frame starts before we get a change to
> copy the sequence number to the buffers (before they will complete
> below) ?

Do you mean that we get two sequental v-start signals and then the next
frame-end signal in MI_MIS belongs to the first v-start signal of the two?
How can this be solved? I wonder if any v-start signal has a later signal
that correspond to the same frame so that we can follow it?

Maybe we should have one counter that is incremented on v-start signal,
and another counter that is incremented uppon some other signal?

> 
>> +
>> +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> 
> Do you need to clear all interrupt bits individually, can't you write
> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> all in one go ?
> 
>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>> +		if (isp_mis_tmp & CIF_ISP_V_START)
>> +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>> +				 isp_mis_tmp);
> 
> This require some explanation. It looks like a naive way to protect
> against something, but I think it could trigger under normal
> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> Same for the similar constructs below.
> 
>> +	}
>> +
>> +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>> +		/* Clear pic_size_error */
>> +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>> +		isp_err = readl(base + CIF_ISP_ERR);
>> +		v4l2_err(&dev->v4l2_dev,
>> +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> 
> What does this mean ?
> 
>> +		writel(isp_err, base + CIF_ISP_ERR_CLR);
>> +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> 
> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> 
>> +		/* Clear data_loss */
>> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>> +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>> +	}
>> +
>> +	/* sampled input frame is complete */
>> +	if (isp_mis & CIF_ISP_FRAME_IN) {
>> +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>> +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>> +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>> +				 isp_mis_tmp);
>> +	}
>> +
>> +	/* frame was completely put out */
> 
> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> ? The two comments could do with a bit of brush up, and I think the
> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> action.

Those two oneline comments are just copy-paste from the datasheet.

""
5 MIS_FRAME_IN sampled input frame is complete
1 MIS_FRAME frame was completely put out
""

Unfrotunately, the datasheet does not add any further explanation about those signals.


> 
>> +	if (isp_mis & CIF_ISP_FRAME) {
>> +		u32 isp_ris = 0;
> 
> No need to initialise this to 0.
> 
>> +		/* Clear Frame In (ISP) */
>> +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>> +		if (isp_mis_tmp & CIF_ISP_FRAME)
>> +			v4l2_err(&dev->v4l2_dev,
>> +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>> +
>> +		isp_ris = readl(base + CIF_ISP_RIS);
>> +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>> +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>> +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> 
> Is there a guarantee that the statistics will be fully written out
> before the video frame itself ? And doesn't this test if any of the
> statistics is complete, not all of them ? I think the logic is wrong, it

The datasheet does not add any explanation of what is expected to come first.
Should we wait until all statistics measurements are done? In the struct
sent to userspace there is a bitmaks for which of the statistics are read.
I think that if only part of the statistics are ready, we can already send the once
that are ready to userspace.

> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> own. It's hard to tell for sure without extra information though (for
> instance why are the stats-related bits read from CIF_ISP_RIS, when
> they seem to be documented as valid in CIF_ISP_ISR), but this should be
> validated, and most probably fixed. Care should be taken to keep
> synchronisation of sequence number between the different queues.

I see that the capture buffers are done before incrementing the frame_sequence with
the following explanation:

	/*
          * Call rkisp1_capture_isr() first to handle the frame that
          * potentially completed using the current frame_sequence number before
          * it is potentially incremented by rkisp1_isp_isr() in the vertical
          * sync.
          */

I think reading the stats/params should also be done before calling rkisp1_capture_isr
for the same reason. (so to match the correct frame_sequence)

Thanks,
Dafna


> 
>> +	}
>> +
>> +	/*
>> +	 * Then update changed configs. Some of them involve
>> +	 * lot of register writes. Do those only one per frame.
>> +	 * Do the updates in the order of the processing flow.
>> +	 */
>> +	rkisp1_params_isr(&dev->params_vdev, isp_mis);
> 
> You should pass dev to this function, and use it internally for the I/O
> accessors instead of going through the vdev.
> 
>> +}
>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
>> new file mode 100644
>> index 000000000000..b0366e354ec2
>> --- /dev/null
>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
>> @@ -0,0 +1,111 @@
>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>> +/*
>> + * Rockchip isp1 driver
>> + *
>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>> + */
>> +
>> +#ifndef _RKISP1_H
>> +#define _RKISP1_H
>> +
>> +#include <linux/platform_device.h>
>> +#include <media/v4l2-fwnode.h>
> 
> I don't think you need these headers. You however need
> 
> #include <media/v4l2-ctrls.h>
> #include <media/v4l2-subdev.h>
> 
> to make this header self-contained. Other headers could be added too
> (videodev2.h, v4l2-mediabus.h and media-entity.h), they're included
> through the above two headers at the moment.
> 
>> +
>> +#include "common.h"
>> +
>> +struct rkisp1_stream;
>> +
>> +/*
>> + * struct ispsd_in_fmt - ISP intput-pad format
>> + *
>> + * Translate mbus_code to hardware format values
>> + *
>> + * @bus_width: used for parallel
>> + */
>> +struct ispsd_in_fmt {
>> +	u32 mbus_code;
>> +	u8 fmt_type;
>> +	u32 mipi_dt;
>> +	u32 yuv_seq;
>> +	enum rkisp1_fmt_raw_pat_type bayer_pat;
>> +	u8 bus_width;
>> +};
> 
> Some of the structures are prefixed with ispsd_, some with rkisp1_. I
> think we should standardise on a single prefix.
> 
>> +
>> +struct ispsd_out_fmt {
>> +	u32 mbus_code;
>> +	u8 fmt_type;
>> +};
>> +
>> +struct rkisp1_ie_config {
>> +	unsigned int effect;
>> +};
> 
> This structure isn't used.
> 
>> +
>> +enum rkisp1_isp_pad {
>> +	RKISP1_ISP_PAD_SINK,
> 
> Should this be named SINK_VIDEO to match SINK_PARAMS ?
> 
>> +	RKISP1_ISP_PAD_SINK_PARAMS,
>> +	RKISP1_ISP_PAD_SOURCE_PATH,
> 
> And SOURCE_VIDEO ?
> 
>> +	RKISP1_ISP_PAD_SOURCE_STATS,
>> +	RKISP1_ISP_PAD_MAX
>> +};
>> +
>> +/*
>> + * struct rkisp1_isp_subdev - ISP sub-device
>> + *
>> + * See Cropping regions of ISP in rkisp1.c for details
> 
> Could you please document all fields from the structure ?
> 
>> + * @in_frm: input size, don't have to be equal to sensor size
>> + * @in_fmt: input format
>> + * @in_crop: crop for sink pad
>> + * @out_fmt: output format
>> + * @out_crop: output size
>> + *
>> + * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)
> 
> s/disabled/disabled /
> 
>> + * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
>> + * @quantization: output quantization
>> + */
>> +struct rkisp1_isp_subdev {
>> +	struct v4l2_subdev sd;
>> +	struct media_pad pads[RKISP1_ISP_PAD_MAX];
>> +	struct v4l2_ctrl_handler ctrl_handler;
>> +	struct v4l2_mbus_framefmt in_frm;
>> +	struct ispsd_in_fmt in_fmt;
> 
> I think this (and out_fmt) could be stored as pointers.
> 
>> +	struct v4l2_rect in_crop;
>> +	struct ispsd_out_fmt out_fmt;
>> +	struct v4l2_rect out_crop;
> 
> I think this could be greatly simplified if you stored all the data as
> v4l2_subdev_pad_config instances for both the sink and source video pads
> instead of using ad-hoc structures. In particular your get and set
> format handlers would become much simpler. I recommend looking at the
> mt9p031 sensor driver for an example. In a nutshell, the get handler
> would retrieve the format from the v4l2_subdev_pad_config instance
> either from rkisp1_isp_subdev (for V4L2_SUBDEV_FORMAT_ACTIVE) or from
> the passed cfg argument (for V4L2_SUBDEV_FORMAT_TRY). The set handler
> would operate similarly, with just a test one ACTIVE/TRY at the
> beginning, followed by common code that only operates on the cfg, and
> finally storing the format in rkisp1_isp_subdev for
> V4L2_SUBDEV_FORMAT_ACTIVE.
> 
>> +	bool dphy_errctrl_disabled;
>> +	atomic_t frm_sync_seq;
>> +	enum v4l2_quantization quantization;
> 
> The quantization would then be stored in the v4l2_subdev_pad_config too.
> 
>> +};
>> +
>> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
>> +			       struct v4l2_device *v4l2_dev);
>> +
>> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
>> +
>> +void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
>> +
>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
>> +
>> +static inline
>> +struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
>> +{
>> +	return &isp_sdev->out_fmt;
>> +}
>> +
>> +static inline
>> +struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
>> +{
>> +	return &isp_sdev->in_fmt;
>> +}
>> +
>> +static inline
>> +struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
>> +{
>> +	return &isp_sdev->out_crop;
>> +}
> 
> I agree with Sakari that accessing fields directly would be more
> readable.
> 
>> +
>> +static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
>> +{
>> +	return container_of(sd, struct rkisp1_isp_subdev, sd);
>> +}
>> +
>> +#endif /* _RKISP1_H */
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-11 11:04     ` Dafna Hirschfeld
@ 2020-07-17  7:46       ` Dafna Hirschfeld
  2020-07-22 16:01         ` Tomasz Figa
  2020-07-22 15:24       ` Tomasz Figa
  1 sibling, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-07-17  7:46 UTC (permalink / raw)
  To: Laurent Pinchart, Helen Koike
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, tfiga, linux-rockchip,
	Allon Huang, Jacob Chen, hans.verkuil, sakari.ailus, zhengsq,
	mchehab, ezequiel, linux-arm-kernel, linux-media

Hi,


On 11.07.20 13:04, Dafna Hirschfeld wrote:
> Hi Laurent,
> 
> On 16.08.19 02:13, Laurent Pinchart wrote:
>> Hello Helen,
>>
>> Thank you for the patch.
>>
>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>>> From: Jacob Chen <jacob2.chen@rock-chips.com>
>>>
>>> Add the subdev driver for rockchip isp1.
>>>
>>> Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
>>> Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
>>> Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
>>> Signed-off-by: Jacob Chen <cc@rock-chips.com>
>>> Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
>>> Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
>>> Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
>>> Signed-off-by: Tomasz Figa <tfiga@chromium.org>
>>> [fixed unknown entity type / switched to PIXEL_RATE]
>>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>>> [update for upstream]
>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>
>>> ---
>>>
>>> Changes in v8: None
>>> Changes in v7:
>>> - fixed warning because of unknown entity type
>>> - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
>>> and default values
>>> - fix typo riksp1/rkisp1
>>> - redesign: remove mipi/csi subdevice, sensors connect directly to the
>>> isp subdevice in the media topology now. As a consequence, remove the
>>> hack in mipidphy_g_mbus_config() where information from the sensor was
>>> being propagated through the topology.
>>> - From the old dphy:
>>>          * cache get_remote_sensor() in s_stream
>>>          * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
>>> - Replace stream state with a boolean
>>> - code styling and checkpatch fixes
>>> - fix stop_stream (return after calling stop, do not reenable the stream)
>>> - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
>>> - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
>>> - s/intput/input
>>> - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
>>> reused by the capture
>>>
>>>   drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
>>>   drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
>>>   2 files changed, 1397 insertions(+)
>>>   create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
>>>   create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
>>>
>>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
>>> new file mode 100644
>>> index 000000000000..6d0c0ffb5e03
>>> --- /dev/null
>>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
>>> @@ -0,0 +1,1286 @@
>>> +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>>> +/*
>>> + * Rockchip isp1 driver
>>
>> Shouldn't each file describe what it contains ? Maybe
>>
>>   * Rockchip ISP1 Driver - ISP Core
>>
>> for this one ? Same for other .c or .h files.
>>
>>> + *
>>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>>> + */
>>> +
>>> +#include <linux/iopoll.h>
>>> +#include <linux/phy/phy.h>
>>> +#include <linux/phy/phy-mipi-dphy.h>
>>> +#include <linux/pm_runtime.h>
>>> +#include <linux/videodev2.h>
>>> +#include <linux/vmalloc.h>
>>> +#include <media/v4l2-event.h>
>>> +
>>> +#include "common.h"
>>> +#include "regs.h"
>>
>> common.h and regs.h aren't available yet. This won't break bisection as
>> this file isn't referenced from the makefile yet, but it makes it a bit
>> annoying when reviewing patches in order :-S
>>
>>> +
>>> +#define CIF_ISP_INPUT_W_MAX        4032
>>> +#define CIF_ISP_INPUT_H_MAX        3024
>>> +#define CIF_ISP_INPUT_W_MIN        32
>>> +#define CIF_ISP_INPUT_H_MIN        32
>>> +#define CIF_ISP_OUTPUT_W_MAX        CIF_ISP_INPUT_W_MAX
>>> +#define CIF_ISP_OUTPUT_H_MAX        CIF_ISP_INPUT_H_MAX
>>> +#define CIF_ISP_OUTPUT_W_MIN        CIF_ISP_INPUT_W_MIN
>>> +#define CIF_ISP_OUTPUT_H_MIN        CIF_ISP_INPUT_H_MIN
>>> +
>>> +/*
>>> + * NOTE: MIPI controller and input MUX are also configured in this file,
>>> + * because ISP Subdev is not only describe ISP submodule(input size,format,
>>> + * output size, format), but also a virtual route device.
>>> + */
>>> +
>>> +/*
>>> + * There are many variables named with format/frame in below code,
>>> + * please see here for their meaning.
>>> + *
>>> + * Cropping regions of ISP
>>> + *
>>> + * +---------------------------------------------------------+
>>> + * | Sensor image                                            |
>>> + * | +---------------------------------------------------+   |
>>> + * | | ISP_ACQ (for black level)                         |   |
>>> + * | | in_frm                                            |   |
>>> + * | | +--------------------------------------------+    |   |
>>> + * | | |    ISP_OUT                                 |    |   |
>>> + * | | |    in_crop                                 |    |   |
>>
>> in_crop at the ISP output ? That seems a bit weird. I'm guessing that
>> this is really the ISP output, while ISP_IS is related to the resizer ?
>>
>>> + * | | |    +---------------------------------+     |    |   |
>>> + * | | |    |   ISP_IS                        |     |    |   |
>>> + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
>>> + * | | |    +---------------------------------+     |    |   |
>>> + * | | +--------------------------------------------+    |   |
>>> + * | +---------------------------------------------------+   |
>>> + * +---------------------------------------------------------+
>>> + */
>>> +
>>> +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
>>> +{
>>> +    return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
>>> +}
>>> +
>>> +/* Get sensor by enabled media link */
>>> +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
>>> +{
>>> +    struct media_pad *local, *remote;
>>> +    struct media_entity *sensor_me;
>>> +
>>> +    local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
>>> +    remote = media_entity_remote_pad(local);
>>> +    if (!remote) {
>>> +        v4l2_warn(sd, "No link between isp and sensor\n");
>>> +        return NULL;
>>> +    }
>>> +
>>> +    sensor_me = media_entity_remote_pad(local)->entity;
>>> +    return media_entity_to_v4l2_subdev(sensor_me);
>>> +}
>>> +
>>> +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
>>> +                           struct v4l2_subdev *sd)
>>> +{
>>> +    struct rkisp1_sensor *sensor;
>>> +
>>> +    list_for_each_entry(sensor, &dev->sensors, list)
>>> +        if (sensor->sd == sd)
>>> +            return sensor;
>>> +
>>> +    return NULL;
>>> +}
>>> +
>>> +/****************  register operations ****************/
>>> +
>>> +/*
>>> + * Image Stabilization.
>>> + * This should only be called when configuring CIF
>>> + * or at the frame end interrupt
>>> + */
>>> +static void rkisp1_config_ism(struct rkisp1_device *dev)
>>> +{
>>> +    void __iomem *base = dev->base_addr;
>>> +    struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
>>> +    u32 val;
>>> +
>>> +    writel(0, base + CIF_ISP_IS_RECENTER);
>>
>> How about read/write wrappers that take a rkisp1_device pointer, a
>> register offset and a value (for the write wrapper) and compute
>> dev->base_addr + offset internally ? That would make the code easier to
>> read.
>>
>>> +    writel(0, base + CIF_ISP_IS_MAX_DX);
>>> +    writel(0, base + CIF_ISP_IS_MAX_DY);
>>> +    writel(0, base + CIF_ISP_IS_DISPLACE);
>>> +    writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
>>> +    writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
>>> +    writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
>>> +    writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
>>> +
>>> +    /* IS(Image Stabilization) is always on, working as output crop */
>>> +    writel(1, base + CIF_ISP_IS_CTRL);
>>> +    val = readl(base + CIF_ISP_CTRL);
>>> +    val |= CIF_ISP_CTRL_ISP_CFG_UPD;
>>> +    writel(val, base + CIF_ISP_CTRL);
>>> +}
>>> +
>>> +/*
>>> + * configure isp blocks with input format, size......
>>> + */
>>> +static int rkisp1_config_isp(struct rkisp1_device *dev)
>>> +{
>>> +    u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
>>> +    struct v4l2_rect *out_crop, *in_crop;
>>> +    void __iomem *base = dev->base_addr;
>>> +    struct v4l2_mbus_framefmt *in_frm;
>>> +    struct ispsd_out_fmt *out_fmt;
>>> +    struct rkisp1_sensor *sensor;
>>> +    struct ispsd_in_fmt *in_fmt;
>>> +
>>> +    sensor = dev->active_sensor;
>>> +    in_frm = &dev->isp_sdev.in_frm;
>>> +    in_fmt = &dev->isp_sdev.in_fmt;
>>> +    out_fmt = &dev->isp_sdev.out_fmt;
>>> +    out_crop = &dev->isp_sdev.out_crop;
>>> +    in_crop = &dev->isp_sdev.in_crop;
>>> +
>>> +    if (in_fmt->fmt_type == FMT_BAYER) {
>>> +        acq_mult = 1;
>>> +        if (out_fmt->fmt_type == FMT_BAYER) {
>>> +            if (sensor->mbus.type == V4L2_MBUS_BT656)
>>> +                isp_ctrl =
>>> +                    CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
>>> +            else
>>> +                isp_ctrl =
>>> +                    CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
>>> +        } else {
>>> +            writel(CIF_ISP_DEMOSAIC_TH(0xc),
>>> +                   base + CIF_ISP_DEMOSAIC);
>>> +
>>> +            if (sensor->mbus.type == V4L2_MBUS_BT656)
>>> +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
>>> +            else
>>> +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
>>> +        }
>>> +    } else if (in_fmt->fmt_type == FMT_YUV) {
>>> +        acq_mult = 2;
>>> +        if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>>> +            isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
>>> +        } else {
>>> +            if (sensor->mbus.type == V4L2_MBUS_BT656)
>>> +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
>>> +            else
>>> +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
>>> +
>>> +        }
>>> +
>>> +        irq_mask |= CIF_ISP_DATA_LOSS;
>>> +    }
>>> +
>>> +    /* Set up input acquisition properties */
>>> +    if (sensor->mbus.type == V4L2_MBUS_BT656 ||
>>> +        sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>>> +        if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
>>> +            signal = CIF_ISP_ACQ_PROP_POS_EDGE;
>>> +    }
>>> +
>>> +    if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>>> +        if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
>>> +            signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
>>> +
>>> +        if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
>>> +            signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
>>> +    }
>>> +
>>> +    writel(isp_ctrl, base + CIF_ISP_CTRL);
>>> +    writel(signal | in_fmt->yuv_seq |
>>> +           CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
>>> +           CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
>>> +    writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
>>> +
>>> +    /* Acquisition Size */
>>> +    writel(0, base + CIF_ISP_ACQ_H_OFFS);
>>> +    writel(0, base + CIF_ISP_ACQ_V_OFFS);
>>> +    writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
>>> +    writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
>>> +
>>> +    /* ISP Out Area */
>>> +    writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
>>> +    writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
>>> +    writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
>>> +    writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
>>> +
>>> +    /* interrupt mask */
>>> +    irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
>>> +            CIF_ISP_FRAME_IN;
>>> +    writel(irq_mask, base + CIF_ISP_IMSC);
>>> +
>>> +    if (out_fmt->fmt_type == FMT_BAYER)
>>> +        rkisp1_params_disable_isp(&dev->params_vdev);
>>> +    else
>>> +        rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
>>> +                        dev->isp_sdev.quantization);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int rkisp1_config_dvp(struct rkisp1_device *dev)
>>> +{
>>> +    struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
>>> +    void __iomem *base = dev->base_addr;
>>> +    u32 val, input_sel;
>>> +
>>> +    switch (in_fmt->bus_width) {
>>> +    case 8:
>>> +        input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
>>> +        break;
>>> +    case 10:
>>> +        input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
>>> +        break;
>>> +    case 12:
>>> +        input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
>>> +        break;
>>> +    default:
>>> +        v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    val = readl(base + CIF_ISP_ACQ_PROP);
>>> +    writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int rkisp1_config_mipi(struct rkisp1_device *dev)
>>> +{
>>> +    struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
>>> +    struct rkisp1_sensor *sensor = dev->active_sensor;
>>> +    void __iomem *base = dev->base_addr;
>>> +    unsigned int lanes;
>>> +    u32 mipi_ctrl;
>>> +
>>> +    /*
>>> +     * sensor->mbus is set in isp or d-phy notifier_bound function
>>> +     */
>>> +    switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
>>> +    case V4L2_MBUS_CSI2_4_LANE:
>>> +        lanes = 4;
>>> +        break;
>>> +    case V4L2_MBUS_CSI2_3_LANE:
>>> +        lanes = 3;
>>> +        break;
>>> +    case V4L2_MBUS_CSI2_2_LANE:
>>> +        lanes = 2;
>>> +        break;
>>> +    case V4L2_MBUS_CSI2_1_LANE:
>>> +        lanes = 1;
>>> +        break;
>>> +    default:
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
>>> +            CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
>>> +            CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
>>> +            CIF_MIPI_CTRL_CLOCKLANE_ENA;
>>> +
>>> +    writel(mipi_ctrl, base + CIF_MIPI_CTRL);
>>> +
>>> +    /* Configure Data Type and Virtual Channel */
>>> +    writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
>>> +           base + CIF_MIPI_IMG_DATA_SEL);
>>> +
>>> +    /* Clear MIPI interrupts */
>>> +    writel(~0, base + CIF_MIPI_ICR);
>>> +    /*
>>> +     * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
>>> +     * isp bus may be dead when switch isp.
>>> +     */
>>> +    writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
>>> +           CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
>>> +           base + CIF_MIPI_IMSC);
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
>>> +         "  MIPI_IMG_DATA_SEL 0x%08x\n"
>>> +         "  MIPI_STATUS 0x%08x\n"
>>> +         "  MIPI_IMSC 0x%08x\n",
>>> +         readl(base + CIF_MIPI_CTRL),
>>> +         readl(base + CIF_MIPI_IMG_DATA_SEL),
>>> +         readl(base + CIF_MIPI_STATUS),
>>> +         readl(base + CIF_MIPI_IMSC));
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +/* Configure MUX */
>>> +static int rkisp1_config_path(struct rkisp1_device *dev)
>>> +{
>>> +    struct rkisp1_sensor *sensor = dev->active_sensor;
>>> +    u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
>>> +    int ret = 0;
>>> +
>>> +    if (sensor->mbus.type == V4L2_MBUS_BT656 ||
>>> +        sensor->mbus.type == V4L2_MBUS_PARALLEL) {
>>> +        ret = rkisp1_config_dvp(dev);
>>> +        dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
>>> +    } else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>>> +        ret = rkisp1_config_mipi(dev);
>>> +        dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
>>> +    }
>>> +
>>> +    writel(dpcl, dev->base_addr + CIF_VI_DPCL);
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +/* Hareware configure Entry */
>>
>> s/Hareware/Hardware/
>>
>>> +static int rkisp1_config_cif(struct rkisp1_device *dev)
>>> +{
>>> +    u32 cif_id;
>>> +    int ret;
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>>> +         "SP streaming = %d, MP streaming = %d\n",
>>> +         dev->stream[RKISP1_STREAM_SP].streaming,
>>> +         dev->stream[RKISP1_STREAM_MP].streaming);
>>> +
>>> +    cif_id = readl(dev->base_addr + CIF_VI_ID);
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
>>> +
>>> +    ret = rkisp1_config_isp(dev);
>>> +    if (ret < 0)
>>> +        return ret;
>>> +    ret = rkisp1_config_path(dev);
>>> +    if (ret < 0)
>>> +        return ret;
>>> +    rkisp1_config_ism(dev);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +/* Mess register operations to stop isp */
>>
>> Is it such a mess ? :-)
>>
>> I would capitalise ISP in all comments.
>>
>>> +static int rkisp1_isp_stop(struct rkisp1_device *dev)
>>> +{
>>> +    void __iomem *base = dev->base_addr;
>>> +    u32 val;
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>>> +         "SP streaming = %d, MP streaming = %d\n",
>>> +         dev->stream[RKISP1_STREAM_SP].streaming,
>>> +         dev->stream[RKISP1_STREAM_MP].streaming);
>>> +
>>> +    /*
>>> +     * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
>>> +     * Stop ISP(isp) ->wait for ISP isp off
>>> +     */
>>> +    /* stop and clear MI, MIPI, and ISP interrupts */
>>> +    writel(0, base + CIF_MIPI_IMSC);
>>> +    writel(~0, base + CIF_MIPI_ICR);
>>> +
>>> +    writel(0, base + CIF_ISP_IMSC);
>>> +    writel(~0, base + CIF_ISP_ICR);
>>> +
>>> +    writel(0, base + CIF_MI_IMSC);
>>> +    writel(~0, base + CIF_MI_ICR);
>>> +    val = readl(base + CIF_MIPI_CTRL);
>>> +    writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
>>> +    /* stop ISP */
>>> +    val = readl(base + CIF_ISP_CTRL);
>>> +    val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
>>> +    writel(val, base + CIF_ISP_CTRL);
>>> +
>>> +    val = readl(base + CIF_ISP_CTRL);
>>> +    writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
>>> +
>>> +    readx_poll_timeout(readl, base + CIF_ISP_RIS,
>>> +               val, val & CIF_ISP_OFF, 20, 100);
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>>> +        "streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
>>> +         dev->stream[RKISP1_STREAM_SP].streaming,
>>> +         dev->stream[RKISP1_STREAM_MP].streaming,
>>> +         readl(base + CIF_MI_CTRL),
>>> +         readl(base + CIF_ISP_CTRL),
>>> +         readl(base + CIF_MIPI_CTRL));
>>> +
>>> +    writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
>>> +    writel(0x0, base + CIF_IRCL);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +/* Mess register operations to start isp */
>>> +static int rkisp1_isp_start(struct rkisp1_device *dev)
>>> +{
>>> +    struct rkisp1_sensor *sensor = dev->active_sensor;
>>> +    void __iomem *base = dev->base_addr;
>>> +    u32 val;
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>>> +         "SP streaming = %d, MP streaming = %d\n",
>>> +         dev->stream[RKISP1_STREAM_SP].streaming,
>>> +         dev->stream[RKISP1_STREAM_MP].streaming);
>>> +
>>> +    /* Activate MIPI */
>>> +    if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
>>> +        val = readl(base + CIF_MIPI_CTRL);
>>> +        writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
>>> +    }
>>> +    /* Activate ISP */
>>> +    val = readl(base + CIF_ISP_CTRL);
>>> +    val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
>>> +           CIF_ISP_CTRL_ISP_INFORM_ENABLE;
>>> +    writel(val, base + CIF_ISP_CTRL);
>>> +
>>> +    /* XXX: Is the 1000us too long?
>>> +     * CIF spec says to wait for sufficient time after enabling
>>> +     * the MIPI interface and before starting the sensor output.
>>> +     */
>>> +    usleep_range(1000, 1200);
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>>> +         "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
>>> +         "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
>>> +         dev->stream[RKISP1_STREAM_SP].streaming,
>>> +         dev->stream[RKISP1_STREAM_MP].streaming,
>>> +         readl(base + CIF_MI_CTRL),
>>> +         readl(base + CIF_ISP_CTRL),
>>> +         readl(base + CIF_MIPI_CTRL));
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void rkisp1_config_clk(struct rkisp1_device *dev)
>>> +{
>>> +    u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
>>> +          CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
>>> +          CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
>>> +
>>> +    writel(val, dev->base_addr + CIF_ICCL);
>>> +}
>>> +
>>> +/***************************** isp sub-devs *******************************/
>>> +
>>> +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
>>> +    {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SBGGR10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW10,
>>> +        .bayer_pat    = RAW_BGGR,
>>> +        .bus_width    = 10,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SRGGB10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW10,
>>> +        .bayer_pat    = RAW_RGGB,
>>> +        .bus_width    = 10,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGBRG10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW10,
>>> +        .bayer_pat    = RAW_GBRG,
>>> +        .bus_width    = 10,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGRBG10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW10,
>>> +        .bayer_pat    = RAW_GRBG,
>>> +        .bus_width    = 10,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SRGGB12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW12,
>>> +        .bayer_pat    = RAW_RGGB,
>>> +        .bus_width    = 12,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SBGGR12_1X12,
>>
>> Is there a reason why the 10-bit Bayer format start with BGGR while the
>> 12-bit formats start with RGGB ? Not a big deal, just OCD kicking in :-)
>>
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW12,
>>> +        .bayer_pat    = RAW_BGGR,
>>> +        .bus_width    = 12,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGBRG12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW12,
>>> +        .bayer_pat    = RAW_GBRG,
>>> +        .bus_width    = 12,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGRBG12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW12,
>>> +        .bayer_pat    = RAW_GRBG,
>>> +        .bus_width    = 12,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SRGGB8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW8,
>>> +        .bayer_pat    = RAW_RGGB,
>>> +        .bus_width    = 8,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SBGGR8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW8,
>>> +        .bayer_pat    = RAW_BGGR,
>>> +        .bus_width    = 8,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGBRG8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW8,
>>> +        .bayer_pat    = RAW_GBRG,
>>> +        .bus_width    = 8,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGRBG8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +        .mipi_dt    = CIF_CSI2_DT_RAW8,
>>> +        .bayer_pat    = RAW_GRBG,
>>> +        .bus_width    = 8,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_YUYV8_1X16,
>>> +        .fmt_type    = FMT_YUV,
>>> +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
>>> +        .yuv_seq    = CIF_ISP_ACQ_PROP_YCBYCR,
>>> +        .bus_width    = 16,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_YVYU8_1X16,
>>> +        .fmt_type    = FMT_YUV,
>>> +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
>>> +        .yuv_seq    = CIF_ISP_ACQ_PROP_YCRYCB,
>>> +        .bus_width    = 16,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_UYVY8_1X16,
>>> +        .fmt_type    = FMT_YUV,
>>> +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
>>> +        .yuv_seq    = CIF_ISP_ACQ_PROP_CBYCRY,
>>> +        .bus_width    = 16,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_VYUY8_1X16,
>>> +        .fmt_type    = FMT_YUV,
>>> +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
>>> +        .yuv_seq    = CIF_ISP_ACQ_PROP_CRYCBY,
>>> +        .bus_width    = 16,
>>> +    },
>>> +};
>>> +
>>> +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
>>> +    {
>>> +        .mbus_code    = MEDIA_BUS_FMT_YUYV8_2X8,
>>> +        .fmt_type    = FMT_YUV,
>>> +    }, {
>>
>> This is the only entry not present in the previous table, so I'm
>> wondering if it would make sense to merge the two tables and rename
>> ispsd_in_fmt to rkisp1_format_info. You would need to add a field that
>> tells, for each format, if it's valid as an input format, and output
>> format, or both. Hmmmm and also make the enum logic a bit more complex.
>> Maybe it's not worth it after all, but it bothers me a bit to have two
>> tables :-) I'll let you decide what's best.
>>
>>> +        .mbus_code    = MEDIA_BUS_FMT_SRGGB12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SBGGR12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGBRG12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGRBG12_1X12,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SRGGB10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SBGGR10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGBRG10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGRBG10_1X10,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SRGGB8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SBGGR8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGBRG8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    }, {
>>> +        .mbus_code    = MEDIA_BUS_FMT_SGRBG8_1X8,
>>> +        .fmt_type    = FMT_BAYER,
>>> +    },
>>> +};
>>> +
>>> +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
>>> +{
>>> +    unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
>>> +    const struct ispsd_in_fmt *fmt;
>>
>> You can move this variable inside the loop.
>>
>>> +
>>> +    for (i = 0; i < array_size; i++) {
>>
>> I would remove the array_size local variable, it doesn't improve
>> readability. Same for the next function.
>>
>>> +        fmt = &rkisp1_isp_input_formats[i];
>>> +        if (fmt->mbus_code == mbus_code)
>>> +            return fmt;
>>> +    }
>>> +
>>> +    return NULL;
>>> +}
>>> +
>>> +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
>>> +{
>>> +    unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
>>> +    const struct ispsd_out_fmt *fmt;
>>> +
>>> +    for (i = 0; i < array_size; i++) {
>>> +        fmt = &rkisp1_isp_output_formats[i];
>>> +        if (fmt->mbus_code == mbus_code)
>>> +            return fmt;
>>> +    }
>>> +
>>> +    return NULL;
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
>>> +                    struct v4l2_subdev_pad_config *cfg,
>>> +                    struct v4l2_subdev_mbus_code_enum *code)
>>> +{
>>> +    unsigned int i = code->index;
>>> +
>>> +    if ((code->pad != RKISP1_ISP_PAD_SINK) &&
>>> +        (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>>> +        if (i > 0)
>>> +            return -EINVAL;
>>> +        code->code = MEDIA_BUS_FMT_FIXED;
>>> +        return 0;
>>> +    }
>>> +
>>> +    if (code->pad == RKISP1_ISP_PAD_SINK) {
>>> +        if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
>>> +            return -EINVAL;
>>> +        code->code = rkisp1_isp_input_formats[i].mbus_code;
>>> +    } else {
>>> +        if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
>>> +            return -EINVAL;
>>> +        code->code = rkisp1_isp_output_formats[i].mbus_code;
>>> +    }
>>
>> On the other hand, merging the two tables above into one, you could
>> merge the two branches here and only consider formats indicated as valid
>> for to the pad you want. Maybe the code would be cleaner in the end.
>>
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
>>> +                     struct v4l2_subdev_pad_config *cfg)
>>> +{
>>> +    struct v4l2_rect *mf_in_crop, *mf_out_crop;
>>> +    struct v4l2_mbus_framefmt *mf_in, *mf_out;
>>> +
>>> +    mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
>>> +    mf_in->width = RKISP1_DEFAULT_WIDTH;
>>> +    mf_in->height = RKISP1_DEFAULT_HEIGHT;
>>> +    mf_in->field = V4L2_FIELD_NONE;
>>> +    mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
>>> +
>>> +    mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
>>> +    mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
>>> +    mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
>>> +    mf_in_crop->left = 0;
>>> +    mf_in_crop->top = 0;
>>> +
>>> +    mf_out = v4l2_subdev_get_try_format(sd, cfg,
>>> +                        RKISP1_ISP_PAD_SOURCE_PATH);
>>> +    *mf_out = *mf_in;
>>> +    mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
>>> +    mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>> +
>>> +    mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>> +                           RKISP1_ISP_PAD_SOURCE_PATH);
>>> +    *mf_out_crop = *mf_in_crop;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
>>> +                 struct v4l2_subdev_pad_config *cfg,
>>> +                 struct v4l2_subdev_format *fmt)
>>> +{
>>> +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>>> +    struct v4l2_mbus_framefmt *mf = &fmt->format;
>>> +
>>> +    if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>>> +        (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
>>> +        fmt->format.code = MEDIA_BUS_FMT_FIXED;
>>> +        /*
>>> +         * NOTE: setting a format here doesn't make much sense
>>> +         * but v4l2-compliance complains
>>> +         */
>>
>> For the params pad I agreed it makes no sense, and I think
>> v4l2-compliance is at fault, so I'd set width and height to 0. For the
>> stats pad we *could* use the size of the image from which stats are
>> computed, but because v4l2_meta_format has no width/height, I think 0
>> would also be appropriate.
>>
>>> +        fmt->format.width = RKISP1_DEFAULT_WIDTH;
>>> +        fmt->format.height = RKISP1_DEFAULT_HEIGHT;
>>> +        fmt->format.field = V4L2_FIELD_NONE;
>>> +        return 0;
>>> +    }
>>> +
>>> +    if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>>> +        mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>>> +        fmt->format = *mf;
>>> +        return 0;
>>> +    }
>>> +
>>> +    if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>>> +        *mf = isp_sd->in_frm;
>>> +    } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>>> +        /* format of source pad */
>>> +        *mf = isp_sd->in_frm;
>>> +        mf->code = isp_sd->out_fmt.mbus_code;
>>> +        /* window size of source pad */
>>> +        mf->width = isp_sd->out_crop.width;
>>> +        mf->height = isp_sd->out_crop.height;
>>> +        mf->quantization = isp_sd->quantization;
>>> +    }
>>> +    mf->field = V4L2_FIELD_NONE;
>>
>> This can be simplified, please read through, or jump to the review of
>> rkisp1_isp_subdev at the end.
>>
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
>>> +                  unsigned int pad,
>>> +                  struct v4l2_mbus_framefmt *fmt)
>>> +{
>>> +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>>> +    struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>>> +    const struct ispsd_out_fmt *out_fmt;
>>> +    const struct ispsd_in_fmt *in_fmt;
>>> +
>>> +    switch (pad) {
>>> +    case RKISP1_ISP_PAD_SINK:
>>> +        in_fmt = find_in_fmt(fmt->code);
>>> +        if (in_fmt)
>>> +            fmt->code = in_fmt->mbus_code;
>>> +        else
>>> +            fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
>>
>> You write MEDIA_BUS_FMT_SRGGB10_1X10 explicitly here, while you use
>> rkisp1_isp_output_formats[0].mbus_code below (and in other places). I
>> would standardise on one of the two (explicit formats or array[0]), with
>> a preference for the first as that would allow merging the input and
>> output arrays more easily. I would then create two #define,
>> RKISP1_DEF_INPUT_FORMAT and RKISP2_DEF_OUTPUT_FORMAT (or similar).
>> Similar macros for the default width and height could also be useful, to
>> make it easier to change them.
>>
>>> +        fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
>>> +                      CIF_ISP_INPUT_W_MAX);
>>> +        fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
>>> +                      CIF_ISP_INPUT_H_MAX);
>>> +        break;
>>> +    case RKISP1_ISP_PAD_SOURCE_PATH:
>>> +        out_fmt = find_out_fmt(fmt->code);
>>> +        if (out_fmt)
>>> +            fmt->code = out_fmt->mbus_code;
>>> +        else
>>> +            fmt->code = rkisp1_isp_output_formats[0].mbus_code;
>>> +        /* window size is set in s_selection */
>>> +        fmt->width  = isp_sd->out_crop.width;
>>> +        fmt->height = isp_sd->out_crop.height;
>>
>> This function operates on the TRY configuration too, in which case you
>> should use the TRY crop rectangle here, not the ACTIVE one. If you've
>> already jumped to the review of rkisp1_isp_subdev you know my proposal
>> to simplify this. Otherwise now may be a good time to do so :-)
>>
>>> +        /* full range by default */
>>> +        if (!fmt->quantization)
>>> +            fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>> +        break;
>>> +    }
>>> +
>>> +    fmt->field = V4L2_FIELD_NONE;
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
>>> +                 struct v4l2_subdev_pad_config *cfg,
>>> +                 struct v4l2_subdev_format *fmt)
>>> +{
>>> +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>>> +    struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
>>> +    struct v4l2_mbus_framefmt *mf = &fmt->format;
>>> +
>>> +    if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
>>> +        (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
>>> +        return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
>>> +
>>> +    rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
>>> +
>>> +    if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
>>> +        struct v4l2_mbus_framefmt *try_mf;
>>> +
>>> +        try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
>>> +        *try_mf = *mf;
>>> +        return 0;
>>
>> When setting the format on the sink pad the crop rectangles need to be
>> reset (here, and for the ACTIVE format below too).
>>
>>> +    }
>>> +
>>> +    if (fmt->pad == RKISP1_ISP_PAD_SINK) {
>>> +        const struct ispsd_in_fmt *in_fmt;
>>> +
>>> +        in_fmt = find_in_fmt(mf->code);
>>> +        isp_sd->in_fmt = *in_fmt;
>>> +        isp_sd->in_frm = *mf;
>>> +    } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>>> +        const struct ispsd_out_fmt *out_fmt;
>>> +
>>> +        /* Ignore width/height */
>>> +        out_fmt = find_out_fmt(mf->code);
>>> +        isp_sd->out_fmt = *out_fmt;
>>
>> I would return the in_fmt and out_fmt from rkisp1_isp_sd_try_fmt() as it
>> already looks them up. If you merge the input and output tables, you'll
>> have a single format info structure type, and rkisp1_isp_sd_try_fmt()
>> could return the entry for the pad it operates on.
>>
>>> +        /*
>>> +         * It is quantization for output,
>>> +         * isp use bt601 limit-range in internal
>>> +         */
>>> +        isp_sd->quantization = mf->quantization;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
>>> +                  struct v4l2_subdev_pad_config *cfg,
>>> +                  struct v4l2_subdev_selection *sel)
>>> +{
>>> +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>>> +    struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
>>> +    struct v4l2_rect in_crop = isp_sd->in_crop;
>>> +    struct v4l2_rect *input = &sel->r;
>>> +
>>> +    if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
>>> +        in_frm = *v4l2_subdev_get_try_format(sd, cfg,
>>> +                             RKISP1_ISP_PAD_SINK);
>>> +        in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
>>> +                            RKISP1_ISP_PAD_SINK);
>>> +    }
>>> +
>>> +    input->left = ALIGN(input->left, 2);
>>> +    input->width = ALIGN(input->width, 2);
>>> +
>>> +    if (sel->pad == RKISP1_ISP_PAD_SINK) {
>>> +        input->left = clamp_t(u32, input->left, 0, in_frm.width);
>>> +        input->top = clamp_t(u32, input->top, 0, in_frm.height);
>>> +        input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
>>> +                       in_frm.width - input->left);
>>> +        input->height = clamp_t(u32, input->height,
>>> +                    CIF_ISP_INPUT_H_MIN,
>>> +                    in_frm.height - input->top);
>>> +    } else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
>>> +        input->left = clamp_t(u32, input->left, 0, in_crop.width);
>>> +        input->top = clamp_t(u32, input->top, 0, in_crop.height);
>>> +        input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
>>> +                       in_crop.width - input->left);
>>> +        input->height = clamp_t(u32, input->height,
>>> +                    CIF_ISP_OUTPUT_H_MIN,
>>> +                    in_crop.height - input->top);
>>> +    }
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
>>> +                       struct v4l2_subdev_pad_config *cfg,
>>> +                       struct v4l2_subdev_selection *sel)
>>> +{
>>> +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>>> +    struct v4l2_mbus_framefmt *frm;
>>> +    struct v4l2_rect *rect;
>>> +
>>> +    if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
>>> +        sel->pad != RKISP1_ISP_PAD_SINK)
>>> +        return -EINVAL;
>>> +
>>> +    switch (sel->target) {
>>> +    case V4L2_SEL_TGT_CROP_BOUNDS:
>>> +        if (sel->pad == RKISP1_ISP_PAD_SINK) {
>>> +            if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>>> +                frm = v4l2_subdev_get_try_format(sd, cfg,
>>> +                                 sel->pad);
>>> +            else
>>> +                frm = &isp_sd->in_frm;
>>> +
>>> +            sel->r.height = frm->height;
>>> +            sel->r.width = frm->width;
>>> +            sel->r.left = 0;
>>> +            sel->r.top = 0;
>>> +        } else {
>>> +            if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>>> +                rect = v4l2_subdev_get_try_crop(sd, cfg,
>>> +                            RKISP1_ISP_PAD_SINK);
>>> +            else
>>> +                rect = &isp_sd->in_crop;
>>> +            sel->r = *rect;
>>> +        }
>>> +        break;
>>> +    case V4L2_SEL_TGT_CROP:
>>> +        if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
>>> +            rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
>>> +        else if (sel->pad == RKISP1_ISP_PAD_SINK)
>>> +            rect = &isp_sd->in_crop;
>>> +        else
>>> +            rect = &isp_sd->out_crop;
>>> +        sel->r = *rect;
>>> +        break;
>>> +    default:
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
>>> +                       struct v4l2_subdev_pad_config *cfg,
>>> +                       struct v4l2_subdev_selection *sel)
>>> +{
>>> +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
>>> +    struct rkisp1_device *dev = sd_to_isp_dev(sd);
>>> +
>>> +    if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
>>> +        sel->pad != RKISP1_ISP_PAD_SINK)
>>> +        return -EINVAL;
>>> +    if (sel->target != V4L2_SEL_TGT_CROP)
>>> +        return -EINVAL;
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
>>> +         "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
>>> +         sel->r.left, sel->r.top, sel->r.width, sel->r.height);
>>> +    rkisp1_isp_sd_try_crop(sd, cfg, sel);
>>> +
>>> +    if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
>>> +        struct v4l2_rect *try_sel =
>>> +            v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
>>> +
>>> +        *try_sel = sel->r;
>>> +        return 0;
>>
>> When setting the crop rectangle on the sink pad the rectangle on the
>> source pad should be reset (and below for the ACTIVE format too).
>>
>> The resizer logic, through selection rectangles, seems to be missing. I
>> assume it's configured through the video nodes, but that's not correct
>> I'm afraid, scaling should be configured on subdevs, see [1]. I'm afraid
>> this means that we'll need separate subdevs for the resizers :-S
>>
>> [1] https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html#order-of-configuration-and-format-propagation
>>
>>> +    }
>>> +
>>> +    if (sel->pad == RKISP1_ISP_PAD_SINK)
>>> +        isp_sd->in_crop = sel->r;
>>> +    else
>>> +        isp_sd->out_crop = sel->r;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>
>> I'll skip the part related to power on/off below as Sakari requested it
>> to be handled through runtime PM.
>>
>>> +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
>>> +                    struct rkisp1_sensor *sensor)
>>> +{
>>> +    union phy_configure_opts opts = { 0 };
>>> +    struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
>>> +    struct v4l2_ctrl *pixel_rate;
>>> +    s64 pixel_clock;
>>> +
>>> +    pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
>>> +                    V4L2_CID_PIXEL_RATE);
>>> +    if (!pixel_rate) {
>>> +        v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
>>> +        return -EPIPE;
>>> +    }
>>
>> Would it make sense to retrieve (and cache) the control pointer (not its
>> value of course) at probe time already ?
>>
>>> +
>>> +    pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
>>> +    if (!pixel_clock) {
>>> +        v4l2_err(sensor->sd, "Invalid pixel rate value\n");
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
>>> +                     sensor->lanes, cfg);
>>> +    phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
>>> +    phy_configure(sensor->dphy, &opts);
>>> +    phy_power_on(sensor->dphy);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
>>> +{
>>> +    phy_power_off(sensor->dphy);
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
>>> +{
>>> +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>>> +    struct v4l2_subdev *sensor_sd;
>>> +    int ret = 0;
>>> +
>>> +    if (!on) {
>>> +        ret = rkisp1_isp_stop(isp_dev);
>>> +        if (ret < 0)
>>> +            return ret;
>>> +        mipi_csi2_s_stream_stop(isp_dev->active_sensor);
>>> +        return 0;
>>> +    }
>>> +
>>> +    sensor_sd = get_remote_sensor(sd);
>>> +    if (!sensor_sd)
>>> +        return -ENODEV;
>>> +
>>> +    isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
>>> +    if (!isp_dev->active_sensor)
>>> +        return -ENODEV;
>>> +
>>> +    atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
>>> +    ret = rkisp1_config_cif(isp_dev);
>>> +    if (ret < 0)
>>> +        return ret;
>>> +
>>> +    /* TODO: support other interfaces */
>>> +    if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
>>> +        return -EINVAL;
>>> +
>>> +    ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
>>> +                       isp_dev->active_sensor);
>>> +    if (ret < 0)
>>> +        return ret;
>>> +
>>> +    ret = rkisp1_isp_start(isp_dev);
>>> +    if (ret)
>>> +        mipi_csi2_s_stream_stop(isp_dev->active_sensor);
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
>>> +{
>>> +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
>>> +    int ret;
>>> +
>>> +    v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
>>> +
>>> +    if (on) {
>>> +        ret = pm_runtime_get_sync(isp_dev->dev);
>>> +        if (ret < 0)
>>> +            return ret;
>>> +
>>> +        rkisp1_config_clk(isp_dev);
>>> +    } else {
>>> +        ret = pm_runtime_put(isp_dev->dev);
>>> +        if (ret < 0)
>>> +            return ret;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int rkisp1_subdev_link_validate(struct media_link *link)
>>> +{
>>> +    if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
>>> +        return 0;
>>> +
>>> +    return v4l2_subdev_link_validate(link);
>>> +}
>>> +
>>> +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
>>> +                    struct media_link *link,
>>> +                    struct v4l2_subdev_format *source_fmt,
>>> +                    struct v4l2_subdev_format *sink_fmt)
>>> +{
>>> +    if (source_fmt->format.code != sink_fmt->format.code)
>>> +        return -EINVAL;
>>> +
>>> +    /* Crop is available */
>>> +    if (source_fmt->format.width < sink_fmt->format.width ||
>>> +        source_fmt->format.height < sink_fmt->format.height)
>>> +        return -EINVAL;
>>
>> Crop should be handled on pads, not on links. The size at the ends of
>> the link should match. This will likely require a bit of a redesign of
>> the format and selection rectangles handling, but I think it's also an
>> opportunity to simplify the code.
>>
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>>> +{
>>> +    struct v4l2_event event = {
>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>> +        .u.frame_sync.frame_sequence =
>>> +            atomic_inc_return(&isp->frm_sync_seq) - 1,
>>
>> I would move the increment to the caller, hiding it in this function is
>> error-prone (and if you look at the caller I'm pointing out one possible
>> error :-)).
>>
>> In general usage of frm_sync_seq through the driver seems to be very
>> race-prone. It's read in various IRQ handling functions, all coming from
>> the same IRQ, so that part is fine (and wouldn't require an atomic
>> variable), but when read from the buffer queue handlers I really get a
>> red light flashing in my head. I'll try to investigate more when
>> reviewing the next patches.
> 
> I see that the only place were 'frame_sequence' is read outside of the irq
> handlers is in the capture in 'rkisp1_vb2_buf_queue':
> 
>      /*
>           * If there's no next buffer assigned, queue this buffer directly
>           * as the next buffer, and update the memory interface.
>           */
>          if (cap->is_streaming && !cap->buf.next &&
>              atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>                  cap->buf.next = ispbuf;
>                  rkisp1_set_next_buf(cap);
>          } else {
>                  list_add_tail(&ispbuf->queue, &cap->buf.queue);
>          }
> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> from the above condition so that the next buffer is updated in case it is null not just before the first
> v-start signal.
> 
> 
> 
> 
> 
>>
>>> +    };
>>> +    v4l2_event_queue(isp->sd.devnode, &event);
>>> +}
>>> +
>>> +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
>>> +                  struct v4l2_event_subscription *sub)
>>> +{
>>> +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
>>> +        return -EINVAL;
>>> +
>>> +    /* Line number. For now only zero accepted. */
>>
>> Is the id a line number for V4L2_EVENT_FRAME_SYNC ? It's not mentioned
>> for V4L2_EVENT_FRAME_SYNC in the V4L2 spec, so I think this check is
>> correct, but the comment doesn't seem to be.
>>
>>> +    if (sub->id != 0)
>>> +        return -EINVAL;
>>> +
>>> +    return v4l2_event_subscribe(fh, sub, 0, NULL);
>>> +}
>>> +
>>> +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
>>> +    .enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
>>> +    .get_selection = rkisp1_isp_sd_get_selection,
>>> +    .set_selection = rkisp1_isp_sd_set_selection,
>>> +    .init_cfg = rkisp1_isp_sd_init_config,
>>> +    .get_fmt = rkisp1_isp_sd_get_fmt,
>>> +    .set_fmt = rkisp1_isp_sd_set_fmt,
>>> +    .link_validate = rkisp1_subdev_fmt_link_validate,
>>> +};
>>> +
>>> +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
>>> +    .link_validate = rkisp1_subdev_link_validate,
>>> +};
>>> +
>>> +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
>>> +    .s_stream = rkisp1_isp_sd_s_stream,
>>> +};
>>> +
>>> +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
>>> +    .subscribe_event = rkisp1_isp_sd_subs_evt,
>>> +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
>>> +    .s_power = rkisp1_isp_sd_s_power,
>>> +};
>>> +
>>> +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {
>>
>> static const
>>
>>> +    .core = &rkisp1_isp_core_ops,
>>> +    .video = &rkisp1_isp_sd_video_ops,
>>> +    .pad = &rkisp1_isp_sd_pad_ops,
>>> +};
>>> +
>>> +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
>>> +{
>>> +    struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
>>> +    struct v4l2_rect *in_crop = &isp_sd->in_crop;
>>> +    struct v4l2_rect *out_crop = &isp_sd->out_crop;
>>> +    struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
>>> +    struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
>>> +
>>> +    *in_fmt = rkisp1_isp_input_formats[0];
>>> +    in_frm->width = RKISP1_DEFAULT_WIDTH;
>>> +    in_frm->height = RKISP1_DEFAULT_HEIGHT;
>>> +    in_frm->code = in_fmt->mbus_code;
>>> +
>>> +    in_crop->width = in_frm->width;
>>> +    in_crop->height = in_frm->height;
>>> +    in_crop->left = 0;
>>> +    in_crop->top = 0;
>>> +
>>> +    /* propagate to source */
>>> +    *out_crop = *in_crop;
>>> +    *out_fmt = rkisp1_isp_output_formats[0];
>>> +    isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>
>> I wonder, could this be implemented by sharing code with .init_cfg() if
>> you store the active configuration in v4l2_subdev_pad_config instances
>> (please see the comments about the rkisp1_isp_subdev structure below) ?
>>
>>> +}
>>> +
>>> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
>>> +                   struct v4l2_device *v4l2_dev)
>>> +{
>>> +    struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
>>> +    struct v4l2_subdev *sd = &isp_sdev->sd;
>>> +    int ret;
>>> +
>>> +    v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
>>> +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
>>> +    sd->entity.ops = &rkisp1_isp_sd_media_ops;
>>> +    snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
>>> +
>>> +    isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
>>> +        MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
>>> +    isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
>>> +    isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
>>> +    isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
>>> +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
>>> +    ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
>>> +                     isp_sdev->pads);
>>> +    if (ret < 0)
>>> +        return ret;
>>> +
>>> +    sd->owner = THIS_MODULE;
>>> +    v4l2_set_subdevdata(sd, isp_dev);
>>> +
>>> +    sd->grp_id = GRP_ID_ISP;
>>
>> I think you can drop this, as well as all the GRP_ID_* macros, they're
>> not used.
>>
>>> +    ret = v4l2_device_register_subdev(v4l2_dev, sd);
>>> +    if (ret < 0) {
>>> +        v4l2_err(sd, "Failed to register isp subdev\n");
>>> +        goto err_cleanup_media_entity;
>>> +    }
>>> +
>>> +    rkisp1_isp_sd_init_default_fmt(isp_sdev);
>>> +
>>> +    return 0;
>>> +err_cleanup_media_entity:
>>> +    media_entity_cleanup(&sd->entity);
>>> +    return ret;
>>> +}
>>> +
>>> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
>>> +{
>>> +    struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
>>> +
>>> +    v4l2_device_unregister_subdev(sd);
>>> +    media_entity_cleanup(&sd->entity);
>>> +}
>>> +
>>> +/****************  Interrupter Handler ****************/
>>
>> s/Handler/Handlers/
>>
>>> +
>>> +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
>>
>> I would make the mis field an u32 as it contains register values. Should
>> it also be renamed to status ? Same for rkisp1_isp_isr() below.
>>
>>> +{
>>> +    struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
>>> +    void __iomem *base = dev->base_addr;
>>> +    u32 val;
>>> +
>>
>> How about moving the CIF_MIPI_MIS read here and removing the mis
>> argument to the function ? It would be more logical to group all
>> register access related to interrupts in a single place. Same for the
>> other interrupt handler functions.
>>
>>> +    writel(~0, base + CIF_MIPI_ICR);
>>> +
>>> +    /*
>>> +     * Disable DPHY errctrl interrupt, because this dphy
>>> +     * erctrl signal is asserted until the next changes
>>> +     * of line state. This time is may be too long and cpu
>>> +     * is hold in this interrupt.
>>> +     */
>>> +    if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
>>> +        val = readl(base + CIF_MIPI_IMSC);
>>> +        writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
>>> +        dev->isp_sdev.dphy_errctrl_disabled = true;
>>> +    }
>>> +
>>> +    /*
>>> +     * Enable DPHY errctrl interrupt again, if mipi have receive
>>> +     * the whole frame without any error.
>>> +     */
>>> +    if (mis == CIF_MIPI_FRAME_END) {
>>> +        /*
>>> +         * Enable DPHY errctrl interrupt again, if mipi have receive
>>> +         * the whole frame without any error.
>>> +         */
>>> +        if (dev->isp_sdev.dphy_errctrl_disabled) {
>>> +            val = readl(base + CIF_MIPI_IMSC);
>>> +            val |= CIF_MIPI_ERR_CTRL(0x0f);
>>> +            writel(val, base + CIF_MIPI_IMSC);
>>> +            dev->isp_sdev.dphy_errctrl_disabled = false;
>>> +        }
>>> +    } else {
>>> +        v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
>>> +    }
>>> +}
>>> +
>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>>> +{
>>> +    void __iomem *base = dev->base_addr;
>>> +    unsigned int isp_mis_tmp = 0;
>>
>> _tmp are never good names :-S
>>
>>> +    unsigned int isp_err = 0;
>>
>> Neither of these variable need to be initialised to 0.
>>
>>> +
>>> +    /* start edge of v_sync */
>>> +    if (isp_mis & CIF_ISP_V_START) {
>>> +        rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>>
>> This will increment the frame sequence number. What if the interrupt is
>> slightly delayed and the next frame starts before we get a change to
>> copy the sequence number to the buffers (before they will complete
>> below) ?
> 
> Do you mean that we get two sequental v-start signals and then the next
> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> How can this be solved? I wonder if any v-start signal has a later signal
> that correspond to the same frame so that we can follow it?
> 
> Maybe we should have one counter that is incremented on v-start signal,
> and another counter that is incremented uppon some other signal?
> 
>>
>>> +
>>> +        writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>>
>> Do you need to clear all interrupt bits individually, can't you write
>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
>> all in one go ?
>>
>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>> +        if (isp_mis_tmp & CIF_ISP_V_START)
>>> +            v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>>> +                 isp_mis_tmp);
>>
>> This require some explanation. It looks like a naive way to protect
>> against something, but I think it could trigger under normal
>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
>> Same for the similar constructs below.
>>
>>> +    }
>>> +
>>> +    if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>>> +        /* Clear pic_size_error */
>>> +        writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>>> +        isp_err = readl(base + CIF_ISP_ERR);
>>> +        v4l2_err(&dev->v4l2_dev,
>>> +             "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>>
>> What does this mean ?
>>
>>> +        writel(isp_err, base + CIF_ISP_ERR_CLR);
>>> +    } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>>
>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
>>
>>> +        /* Clear data_loss */
>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>> +        v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>> +    }
>>> +
>>> +    /* sampled input frame is complete */
>>> +    if (isp_mis & CIF_ISP_FRAME_IN) {
>>> +        writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>> +        if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>>> +            v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>>> +                 isp_mis_tmp);
>>> +    }
>>> +
>>> +    /* frame was completely put out */
>>
>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
>> ? The two comments could do with a bit of brush up, and I think the
>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
>> action.
> 
> Those two oneline comments are just copy-paste from the datasheet.
> 
> ""
> 5 MIS_FRAME_IN sampled input frame is complete
> 1 MIS_FRAME frame was completely put out
> ""
> 
> Unfrotunately, the datasheet does not add any further explanation about those signals.
> 
> 
>>
>>> +    if (isp_mis & CIF_ISP_FRAME) {
>>> +        u32 isp_ris = 0;
>>
>> No need to initialise this to 0.
>>
>>> +        /* Clear Frame In (ISP) */
>>> +        writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>> +        if (isp_mis_tmp & CIF_ISP_FRAME)
>>> +            v4l2_err(&dev->v4l2_dev,
>>> +                 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>>> +
>>> +        isp_ris = readl(base + CIF_ISP_RIS);
>>> +        if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>>> +                   CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>>> +            rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>>
>> Is there a guarantee that the statistics will be fully written out
>> before the video frame itself ? And doesn't this test if any of the
>> statistics is complete, not all of them ? I think the logic is wrong, it
> 
> The datasheet does not add any explanation of what is expected to come first.
> Should we wait until all statistics measurements are done? In the struct
> sent to userspace there is a bitmaks for which of the statistics are read.
> I think that if only part of the statistics are ready, we can already send the once
> that are ready to userspace.
> 
>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
>> own. It's hard to tell for sure without extra information though (for
>> instance why are the stats-related bits read from CIF_ISP_RIS, when
>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
>> validated, and most probably fixed. Care should be taken to keep
>> synchronisation of sequence number between the different queues.
> 
> I see that the capture buffers are done before incrementing the frame_sequence with
> the following explanation:
> 
>      /*
>           * Call rkisp1_capture_isr() first to handle the frame that
>           * potentially completed using the current frame_sequence number before
>           * it is potentially incremented by rkisp1_isp_isr() in the vertical
>           * sync.
>           */
> 
> I think reading the stats/params should also be done before calling rkisp1_capture_isr
> for the same reason. (so to match the correct frame_sequence)
> 
> Thanks,
> Dafna


I am not sure what should be the meaning of the sequence field for parameters buffers.
Currently the rkisp1-params entity returns the first buffer queued right away with the frame_sequence that it reads.
If streaming did not yet started then the first buffer is used as the initial configuration that overrides the driver's default.
In that case the first params buffer is not associated with any captured frame and has a sequence value '-1'
Is this a valid behavior?

Thanks,
Dafna

> 
> 
>>
>>> +    }
>>> +
>>> +    /*
>>> +     * Then update changed configs. Some of them involve
>>> +     * lot of register writes. Do those only one per frame.
>>> +     * Do the updates in the order of the processing flow.
>>> +     */
>>> +    rkisp1_params_isr(&dev->params_vdev, isp_mis);
>>
>> You should pass dev to this function, and use it internally for the I/O
>> accessors instead of going through the vdev.
>>
>>> +}
>>> diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.h b/drivers/media/platform/rockchip/isp1/rkisp1.h
>>> new file mode 100644
>>> index 000000000000..b0366e354ec2
>>> --- /dev/null
>>> +++ b/drivers/media/platform/rockchip/isp1/rkisp1.h
>>> @@ -0,0 +1,111 @@
>>> +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */
>>> +/*
>>> + * Rockchip isp1 driver
>>> + *
>>> + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
>>> + */
>>> +
>>> +#ifndef _RKISP1_H
>>> +#define _RKISP1_H
>>> +
>>> +#include <linux/platform_device.h>
>>> +#include <media/v4l2-fwnode.h>
>>
>> I don't think you need these headers. You however need
>>
>> #include <media/v4l2-ctrls.h>
>> #include <media/v4l2-subdev.h>
>>
>> to make this header self-contained. Other headers could be added too
>> (videodev2.h, v4l2-mediabus.h and media-entity.h), they're included
>> through the above two headers at the moment.
>>
>>> +
>>> +#include "common.h"
>>> +
>>> +struct rkisp1_stream;
>>> +
>>> +/*
>>> + * struct ispsd_in_fmt - ISP intput-pad format
>>> + *
>>> + * Translate mbus_code to hardware format values
>>> + *
>>> + * @bus_width: used for parallel
>>> + */
>>> +struct ispsd_in_fmt {
>>> +    u32 mbus_code;
>>> +    u8 fmt_type;
>>> +    u32 mipi_dt;
>>> +    u32 yuv_seq;
>>> +    enum rkisp1_fmt_raw_pat_type bayer_pat;
>>> +    u8 bus_width;
>>> +};
>>
>> Some of the structures are prefixed with ispsd_, some with rkisp1_. I
>> think we should standardise on a single prefix.
>>
>>> +
>>> +struct ispsd_out_fmt {
>>> +    u32 mbus_code;
>>> +    u8 fmt_type;
>>> +};
>>> +
>>> +struct rkisp1_ie_config {
>>> +    unsigned int effect;
>>> +};
>>
>> This structure isn't used.
>>
>>> +
>>> +enum rkisp1_isp_pad {
>>> +    RKISP1_ISP_PAD_SINK,
>>
>> Should this be named SINK_VIDEO to match SINK_PARAMS ?
>>
>>> +    RKISP1_ISP_PAD_SINK_PARAMS,
>>> +    RKISP1_ISP_PAD_SOURCE_PATH,
>>
>> And SOURCE_VIDEO ?
>>
>>> +    RKISP1_ISP_PAD_SOURCE_STATS,
>>> +    RKISP1_ISP_PAD_MAX
>>> +};
>>> +
>>> +/*
>>> + * struct rkisp1_isp_subdev - ISP sub-device
>>> + *
>>> + * See Cropping regions of ISP in rkisp1.c for details
>>
>> Could you please document all fields from the structure ?
>>
>>> + * @in_frm: input size, don't have to be equal to sensor size
>>> + * @in_fmt: input format
>>> + * @in_crop: crop for sink pad
>>> + * @out_fmt: output format
>>> + * @out_crop: output size
>>> + *
>>> + * @dphy_errctrl_disabled: if dphy errctrl is disabled(avoid endless interrupt)
>>
>> s/disabled/disabled /
>>
>>> + * @frm_sync_seq: frame sequence, to sync frame_id between video devices.
>>> + * @quantization: output quantization
>>> + */
>>> +struct rkisp1_isp_subdev {
>>> +    struct v4l2_subdev sd;
>>> +    struct media_pad pads[RKISP1_ISP_PAD_MAX];
>>> +    struct v4l2_ctrl_handler ctrl_handler;
>>> +    struct v4l2_mbus_framefmt in_frm;
>>> +    struct ispsd_in_fmt in_fmt;
>>
>> I think this (and out_fmt) could be stored as pointers.
>>
>>> +    struct v4l2_rect in_crop;
>>> +    struct ispsd_out_fmt out_fmt;
>>> +    struct v4l2_rect out_crop;
>>
>> I think this could be greatly simplified if you stored all the data as
>> v4l2_subdev_pad_config instances for both the sink and source video pads
>> instead of using ad-hoc structures. In particular your get and set
>> format handlers would become much simpler. I recommend looking at the
>> mt9p031 sensor driver for an example. In a nutshell, the get handler
>> would retrieve the format from the v4l2_subdev_pad_config instance
>> either from rkisp1_isp_subdev (for V4L2_SUBDEV_FORMAT_ACTIVE) or from
>> the passed cfg argument (for V4L2_SUBDEV_FORMAT_TRY). The set handler
>> would operate similarly, with just a test one ACTIVE/TRY at the
>> beginning, followed by common code that only operates on the cfg, and
>> finally storing the format in rkisp1_isp_subdev for
>> V4L2_SUBDEV_FORMAT_ACTIVE.
>>
>>> +    bool dphy_errctrl_disabled;
>>> +    atomic_t frm_sync_seq;
>>> +    enum v4l2_quantization quantization;
>>
>> The quantization would then be stored in the v4l2_subdev_pad_config too.
>>
>>> +};
>>> +
>>> +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
>>> +                   struct v4l2_device *v4l2_dev);
>>> +
>>> +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev);
>>> +
>>> +void rkisp1_mipi_isr(unsigned int mipi_mis, struct rkisp1_device *dev);
>>> +
>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev);
>>> +
>>> +static inline
>>> +struct ispsd_out_fmt *rkisp1_get_ispsd_out_fmt(struct rkisp1_isp_subdev *isp_sdev)
>>> +{
>>> +    return &isp_sdev->out_fmt;
>>> +}
>>> +
>>> +static inline
>>> +struct ispsd_in_fmt *rkisp1_get_ispsd_in_fmt(struct rkisp1_isp_subdev *isp_sdev)
>>> +{
>>> +    return &isp_sdev->in_fmt;
>>> +}
>>> +
>>> +static inline
>>> +struct v4l2_rect *rkisp1_get_isp_sd_win(struct rkisp1_isp_subdev *isp_sdev)
>>> +{
>>> +    return &isp_sdev->out_crop;
>>> +}
>>
>> I agree with Sakari that accessing fields directly would be more
>> readable.
>>
>>> +
>>> +static inline struct rkisp1_isp_subdev *sd_to_isp_sd(struct v4l2_subdev *sd)
>>> +{
>>> +    return container_of(sd, struct rkisp1_isp_subdev, sd);
>>> +}
>>> +
>>> +#endif /* _RKISP1_H */
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-11 11:04     ` Dafna Hirschfeld
  2020-07-17  7:46       ` Dafna Hirschfeld
@ 2020-07-22 15:24       ` Tomasz Figa
  2020-07-22 16:30         ` Laurent Pinchart
  2020-08-05 21:10         ` Dafna Hirschfeld
  1 sibling, 2 replies; 76+ messages in thread
From: Tomasz Figa @ 2020-07-22 15:24 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Allon Huang, linux-rockchip,
	Helen Koike, Jacob Chen, hans.verkuil, Laurent Pinchart,
	sakari.ailus, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media

Hi Dafna,

On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
> Hi Laurent,
> 
> On 16.08.19 02:13, Laurent Pinchart wrote:
> > Hello Helen,
> > 
> > Thank you for the patch.
> > 
> > On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
[snip]
> > > +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> > > +{
> > > +	struct v4l2_event event = {
> > > +		.type = V4L2_EVENT_FRAME_SYNC,
> > > +		.u.frame_sync.frame_sequence =
> > > +			atomic_inc_return(&isp->frm_sync_seq) - 1,
> > 
> > I would move the increment to the caller, hiding it in this function is
> > error-prone (and if you look at the caller I'm pointing out one possible
> > error :-)).
> > 
> > In general usage of frm_sync_seq through the driver seems to be very
> > race-prone. It's read in various IRQ handling functions, all coming from
> > the same IRQ, so that part is fine (and wouldn't require an atomic
> > variable), but when read from the buffer queue handlers I really get a
> > red light flashing in my head. I'll try to investigate more when
> > reviewing the next patches.
> 
> I see that the only place were 'frame_sequence' is read outside of the irq
> handlers is in the capture in 'rkisp1_vb2_buf_queue':
> 
> 	/*
>          * If there's no next buffer assigned, queue this buffer directly
>          * as the next buffer, and update the memory interface.
>          */
>         if (cap->is_streaming && !cap->buf.next &&
>             atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>                 cap->buf.next = ispbuf;
>                 rkisp1_set_next_buf(cap);
>         } else {
>                 list_add_tail(&ispbuf->queue, &cap->buf.queue);
>         }
> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> from the above condition so that the next buffer is updated in case it is null not just before the first
> v-start signal.
> 

We don't have this special case in the Chrome OS code.

I suppose it would make it possible to resume the capture 1 frame
earlier after a queue underrun, as otherwise the new buffer would be
only programmed after the next frame start interrupt and used for the
next-next frame.  However, it's racy, because programming of the buffer
addresses is not atomic and could end up with the hardware using few
plane addresses from the new buffer and few from the dummy buffer.

Given that and also the fact that a queue underrun is a very special
case, where the system was already having problems catching up, I'd just
remove this special case.

[snip]
> > > +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> > > +{
> > > +	void __iomem *base = dev->base_addr;
> > > +	unsigned int isp_mis_tmp = 0;
> > 
> > _tmp are never good names :-S
> > 
> > > +	unsigned int isp_err = 0;
> > 
> > Neither of these variable need to be initialised to 0.
> > 
> > > +
> > > +	/* start edge of v_sync */
> > > +	if (isp_mis & CIF_ISP_V_START) {
> > > +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> > 
> > This will increment the frame sequence number. What if the interrupt is
> > slightly delayed and the next frame starts before we get a change to
> > copy the sequence number to the buffers (before they will complete
> > below) ?
> 
> Do you mean that we get two sequental v-start signals and then the next
> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> How can this be solved? I wonder if any v-start signal has a later signal
> that correspond to the same frame so that we can follow it?
> 
> Maybe we should have one counter that is incremented on v-start signal,
> and another counter that is incremented uppon some other signal?
>

We're talking about a hard IRQ. I can't imagine the interrupt handler
being delayed for a time close to a full frame interval (~16ms for 60
fps) to trigger such scenario.

> > 
> > > +
> > > +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> > 
> > Do you need to clear all interrupt bits individually, can't you write
> > isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> > all in one go ?
> > 
> > > +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > +		if (isp_mis_tmp & CIF_ISP_V_START)
> > > +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> > > +				 isp_mis_tmp);
> > 
> > This require some explanation. It looks like a naive way to protect
> > against something, but I think it could trigger under normal
> > circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> > Same for the similar constructs below.
> > 
> > > +	}
> > > +
> > > +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> > > +		/* Clear pic_size_error */
> > > +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> > > +		isp_err = readl(base + CIF_ISP_ERR);
> > > +		v4l2_err(&dev->v4l2_dev,
> > > +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> > 
> > What does this mean ?
> > 
> > > +		writel(isp_err, base + CIF_ISP_ERR_CLR);
> > > +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> > 
> > Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> > 
> > > +		/* Clear data_loss */
> > > +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> > > +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > +	}
> > > +
> > > +	/* sampled input frame is complete */
> > > +	if (isp_mis & CIF_ISP_FRAME_IN) {
> > > +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> > > +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> > > +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> > > +				 isp_mis_tmp);
> > > +	}
> > > +
> > > +	/* frame was completely put out */
> > 
> > "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> > ? The two comments could do with a bit of brush up, and I think the
> > ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> > action.
> 
> Those two oneline comments are just copy-paste from the datasheet.
> 
> ""
> 5 MIS_FRAME_IN sampled input frame is complete
> 1 MIS_FRAME frame was completely put out
> ""
> 
> Unfrotunately, the datasheet does not add any further explanation about those signals.
> 
> 

My loose recollection is that the former is signaled when then frame
is fully input to the ISP and the latter when the ISP completes
outputting the frame to the next block in the pipeline, but someone
would need to verify this, for example by printing timestamps for all
the various interrupts.

> > 
> > > +	if (isp_mis & CIF_ISP_FRAME) {
> > > +		u32 isp_ris = 0;
> > 
> > No need to initialise this to 0.
> > 
> > > +		/* Clear Frame In (ISP) */
> > > +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> > > +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > +		if (isp_mis_tmp & CIF_ISP_FRAME)
> > > +			v4l2_err(&dev->v4l2_dev,
> > > +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> > > +
> > > +		isp_ris = readl(base + CIF_ISP_RIS);
> > > +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> > > +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> > > +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> > 
> > Is there a guarantee that the statistics will be fully written out
> > before the video frame itself ? And doesn't this test if any of the
> > statistics is complete, not all of them ? I think the logic is wrong, it
> 
> The datasheet does not add any explanation of what is expected to come first.
> Should we wait until all statistics measurements are done? In the struct
> sent to userspace there is a bitmaks for which of the statistics are read.
> I think that if only part of the statistics are ready, we can already send the once
> that are ready to userspace.
>

If we look further into the code, rkisp1_stats_isr() checks the
interrupt status mask passed to it and reads out only the parameters
with indicated completion. The statistics metadata buffer format
includes a bit mask which tells the userspace which measurements are
available.

However, I think I've spotted a bug there. At the beginning of
rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
regardless of the mask used later to decide which readouts need to be
done. This could mean that with an unfortunate timing, some measurements
would be lost. So at least the code should be fixed to only clear the
interrupts bits really handled.

As for whether to send separate buffers for each measurement, I guess
it's not a bad thing to let the userspace access the ones available
earlier. Now I only don't recall why we decided to put all the
measurements into one metadata structure, rather than splitting the 4
into their own structures and buffer queues...

> > seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> > own. It's hard to tell for sure without extra information though (for
> > instance why are the stats-related bits read from CIF_ISP_RIS, when
> > they seem to be documented as valid in CIF_ISP_ISR), but this should be
> > validated, and most probably fixed. Care should be taken to keep
> > synchronisation of sequence number between the different queues.
> 
> I see that the capture buffers are done before incrementing the frame_sequence with
> the following explanation:
> 
> 	/*
>          * Call rkisp1_capture_isr() first to handle the frame that
>          * potentially completed using the current frame_sequence number before
>          * it is potentially incremented by rkisp1_isp_isr() in the vertical
>          * sync.
>          */
> 
> I think reading the stats/params should also be done before calling rkisp1_capture_isr
> for the same reason. (so to match the correct frame_sequence)

My recollection of the sequence of interrupts in this hardware is like
this:

CIF_ISP_V_START (frame 0)
  CIF_ISP_FRAME_IN (frame 0)
    CIF_ISP_FRAME (frame 0)
      CIF_ISP_AWB_DONE
      CIF_ISP_AFM_FIN
      CIF_ISP_EXP_END
      CIF_ISP_HIST_MEASURE_RDY
      CIF_MI_FRAME*
      CIF_ISP_V_START (frame 1)
        CIF_ISP_FRAME_IN (frame 1)
          CIF_ISP_FRAME (frame 1)
            ...

where the interrupts at the same indentation level can happen
independently of each other. Again, someone would have to verify this.

Best regards,
Tomasz

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-17  7:46       ` Dafna Hirschfeld
@ 2020-07-22 16:01         ` Tomasz Figa
  0 siblings, 0 replies; 76+ messages in thread
From: Tomasz Figa @ 2020-07-22 16:01 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Allon Huang, linux-rockchip,
	Helen Koike, Jacob Chen, hans.verkuil, Laurent Pinchart,
	sakari.ailus, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media

On Fri, Jul 17, 2020 at 09:46:24AM +0200, Dafna Hirschfeld wrote:
> Hi,
> 
> 
> On 11.07.20 13:04, Dafna Hirschfeld wrote:
> > Hi Laurent,
> > 
> > On 16.08.19 02:13, Laurent Pinchart wrote:
> > > Hello Helen,
> > > 
> > > Thank you for the patch.
> > > 
> > > On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> > > > From: Jacob Chen <jacob2.chen@rock-chips.com>
> > > > 
> > > > Add the subdev driver for rockchip isp1.
> > > > 
> > > > Signed-off-by: Jacob Chen <jacob2.chen@rock-chips.com>
> > > > Signed-off-by: Shunqian Zheng <zhengsq@rock-chips.com>
> > > > Signed-off-by: Yichong Zhong <zyc@rock-chips.com>
> > > > Signed-off-by: Jacob Chen <cc@rock-chips.com>
> > > > Signed-off-by: Eddie Cai <eddie.cai.linux@gmail.com>
> > > > Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
> > > > Signed-off-by: Allon Huang <allon.huang@rock-chips.com>
> > > > Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > > > [fixed unknown entity type / switched to PIXEL_RATE]
> > > > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > > > [update for upstream]
> > > > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > > 
> > > > ---
> > > > 
> > > > Changes in v8: None
> > > > Changes in v7:
> > > > - fixed warning because of unknown entity type
> > > > - fixed v4l2-compliance errors regarding rkisp1 formats, try formats
> > > > and default values
> > > > - fix typo riksp1/rkisp1
> > > > - redesign: remove mipi/csi subdevice, sensors connect directly to the
> > > > isp subdevice in the media topology now. As a consequence, remove the
> > > > hack in mipidphy_g_mbus_config() where information from the sensor was
> > > > being propagated through the topology.
> > > > - From the old dphy:
> > > >          * cache get_remote_sensor() in s_stream
> > > >          * use V4L2_CID_PIXEL_RATE instead of V4L2_CID_LINK_FREQ
> > > > - Replace stream state with a boolean
> > > > - code styling and checkpatch fixes
> > > > - fix stop_stream (return after calling stop, do not reenable the stream)
> > > > - fix rkisp1_isp_sd_get_selection when V4L2_SUBDEV_FORMAT_TRY is set
> > > > - fix get format in output (isp_sd->out_fmt.mbus_code was being ignored)
> > > > - s/intput/input
> > > > - remove #define sd_to_isp_sd(_sd), add a static inline as it will be
> > > > reused by the capture
> > > > 
> > > >   drivers/media/platform/rockchip/isp1/rkisp1.c | 1286 +++++++++++++++++
> > > >   drivers/media/platform/rockchip/isp1/rkisp1.h |  111 ++
> > > >   2 files changed, 1397 insertions(+)
> > > >   create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.c
> > > >   create mode 100644 drivers/media/platform/rockchip/isp1/rkisp1.h
> > > > 
> > > > diff --git a/drivers/media/platform/rockchip/isp1/rkisp1.c b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > new file mode 100644
> > > > index 000000000000..6d0c0ffb5e03
> > > > --- /dev/null
> > > > +++ b/drivers/media/platform/rockchip/isp1/rkisp1.c
> > > > @@ -0,0 +1,1286 @@
> > > > +// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> > > > +/*
> > > > + * Rockchip isp1 driver
> > > 
> > > Shouldn't each file describe what it contains ? Maybe
> > > 
> > >   * Rockchip ISP1 Driver - ISP Core
> > > 
> > > for this one ? Same for other .c or .h files.
> > > 
> > > > + *
> > > > + * Copyright (C) 2017 Rockchip Electronics Co., Ltd.
> > > > + */
> > > > +
> > > > +#include <linux/iopoll.h>
> > > > +#include <linux/phy/phy.h>
> > > > +#include <linux/phy/phy-mipi-dphy.h>
> > > > +#include <linux/pm_runtime.h>
> > > > +#include <linux/videodev2.h>
> > > > +#include <linux/vmalloc.h>
> > > > +#include <media/v4l2-event.h>
> > > > +
> > > > +#include "common.h"
> > > > +#include "regs.h"
> > > 
> > > common.h and regs.h aren't available yet. This won't break bisection as
> > > this file isn't referenced from the makefile yet, but it makes it a bit
> > > annoying when reviewing patches in order :-S
> > > 
> > > > +
> > > > +#define CIF_ISP_INPUT_W_MAX        4032
> > > > +#define CIF_ISP_INPUT_H_MAX        3024
> > > > +#define CIF_ISP_INPUT_W_MIN        32
> > > > +#define CIF_ISP_INPUT_H_MIN        32
> > > > +#define CIF_ISP_OUTPUT_W_MAX        CIF_ISP_INPUT_W_MAX
> > > > +#define CIF_ISP_OUTPUT_H_MAX        CIF_ISP_INPUT_H_MAX
> > > > +#define CIF_ISP_OUTPUT_W_MIN        CIF_ISP_INPUT_W_MIN
> > > > +#define CIF_ISP_OUTPUT_H_MIN        CIF_ISP_INPUT_H_MIN
> > > > +
> > > > +/*
> > > > + * NOTE: MIPI controller and input MUX are also configured in this file,
> > > > + * because ISP Subdev is not only describe ISP submodule(input size,format,
> > > > + * output size, format), but also a virtual route device.
> > > > + */
> > > > +
> > > > +/*
> > > > + * There are many variables named with format/frame in below code,
> > > > + * please see here for their meaning.
> > > > + *
> > > > + * Cropping regions of ISP
> > > > + *
> > > > + * +---------------------------------------------------------+
> > > > + * | Sensor image                                            |
> > > > + * | +---------------------------------------------------+   |
> > > > + * | | ISP_ACQ (for black level)                         |   |
> > > > + * | | in_frm                                            |   |
> > > > + * | | +--------------------------------------------+    |   |
> > > > + * | | |    ISP_OUT                                 |    |   |
> > > > + * | | |    in_crop                                 |    |   |
> > > 
> > > in_crop at the ISP output ? That seems a bit weird. I'm guessing that
> > > this is really the ISP output, while ISP_IS is related to the resizer ?
> > > 
> > > > + * | | |    +---------------------------------+     |    |   |
> > > > + * | | |    |   ISP_IS                        |     |    |   |
> > > > + * | | |    |   rkisp1_isp_subdev: out_crop   |     |    |   |
> > > > + * | | |    +---------------------------------+     |    |   |
> > > > + * | | +--------------------------------------------+    |   |
> > > > + * | +---------------------------------------------------+   |
> > > > + * +---------------------------------------------------------+
> > > > + */
> > > > +
> > > > +static inline struct rkisp1_device *sd_to_isp_dev(struct v4l2_subdev *sd)
> > > > +{
> > > > +    return container_of(sd->v4l2_dev, struct rkisp1_device, v4l2_dev);
> > > > +}
> > > > +
> > > > +/* Get sensor by enabled media link */
> > > > +static struct v4l2_subdev *get_remote_sensor(struct v4l2_subdev *sd)
> > > > +{
> > > > +    struct media_pad *local, *remote;
> > > > +    struct media_entity *sensor_me;
> > > > +
> > > > +    local = &sd->entity.pads[RKISP1_ISP_PAD_SINK];
> > > > +    remote = media_entity_remote_pad(local);
> > > > +    if (!remote) {
> > > > +        v4l2_warn(sd, "No link between isp and sensor\n");
> > > > +        return NULL;
> > > > +    }
> > > > +
> > > > +    sensor_me = media_entity_remote_pad(local)->entity;
> > > > +    return media_entity_to_v4l2_subdev(sensor_me);
> > > > +}
> > > > +
> > > > +static struct rkisp1_sensor *sd_to_sensor(struct rkisp1_device *dev,
> > > > +                           struct v4l2_subdev *sd)
> > > > +{
> > > > +    struct rkisp1_sensor *sensor;
> > > > +
> > > > +    list_for_each_entry(sensor, &dev->sensors, list)
> > > > +        if (sensor->sd == sd)
> > > > +            return sensor;
> > > > +
> > > > +    return NULL;
> > > > +}
> > > > +
> > > > +/****************  register operations ****************/
> > > > +
> > > > +/*
> > > > + * Image Stabilization.
> > > > + * This should only be called when configuring CIF
> > > > + * or at the frame end interrupt
> > > > + */
> > > > +static void rkisp1_config_ism(struct rkisp1_device *dev)
> > > > +{
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    struct v4l2_rect *out_crop = &dev->isp_sdev.out_crop;
> > > > +    u32 val;
> > > > +
> > > > +    writel(0, base + CIF_ISP_IS_RECENTER);
> > > 
> > > How about read/write wrappers that take a rkisp1_device pointer, a
> > > register offset and a value (for the write wrapper) and compute
> > > dev->base_addr + offset internally ? That would make the code easier to
> > > read.
> > > 
> > > > +    writel(0, base + CIF_ISP_IS_MAX_DX);
> > > > +    writel(0, base + CIF_ISP_IS_MAX_DY);
> > > > +    writel(0, base + CIF_ISP_IS_DISPLACE);
> > > > +    writel(out_crop->left, base + CIF_ISP_IS_H_OFFS);
> > > > +    writel(out_crop->top, base + CIF_ISP_IS_V_OFFS);
> > > > +    writel(out_crop->width, base + CIF_ISP_IS_H_SIZE);
> > > > +    writel(out_crop->height, base + CIF_ISP_IS_V_SIZE);
> > > > +
> > > > +    /* IS(Image Stabilization) is always on, working as output crop */
> > > > +    writel(1, base + CIF_ISP_IS_CTRL);
> > > > +    val = readl(base + CIF_ISP_CTRL);
> > > > +    val |= CIF_ISP_CTRL_ISP_CFG_UPD;
> > > > +    writel(val, base + CIF_ISP_CTRL);
> > > > +}
> > > > +
> > > > +/*
> > > > + * configure isp blocks with input format, size......
> > > > + */
> > > > +static int rkisp1_config_isp(struct rkisp1_device *dev)
> > > > +{
> > > > +    u32 isp_ctrl = 0, irq_mask = 0, acq_mult = 0, signal = 0;
> > > > +    struct v4l2_rect *out_crop, *in_crop;
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    struct v4l2_mbus_framefmt *in_frm;
> > > > +    struct ispsd_out_fmt *out_fmt;
> > > > +    struct rkisp1_sensor *sensor;
> > > > +    struct ispsd_in_fmt *in_fmt;
> > > > +
> > > > +    sensor = dev->active_sensor;
> > > > +    in_frm = &dev->isp_sdev.in_frm;
> > > > +    in_fmt = &dev->isp_sdev.in_fmt;
> > > > +    out_fmt = &dev->isp_sdev.out_fmt;
> > > > +    out_crop = &dev->isp_sdev.out_crop;
> > > > +    in_crop = &dev->isp_sdev.in_crop;
> > > > +
> > > > +    if (in_fmt->fmt_type == FMT_BAYER) {
> > > > +        acq_mult = 1;
> > > > +        if (out_fmt->fmt_type == FMT_BAYER) {
> > > > +            if (sensor->mbus.type == V4L2_MBUS_BT656)
> > > > +                isp_ctrl =
> > > > +                    CIF_ISP_CTRL_ISP_MODE_RAW_PICT_ITU656;
> > > > +            else
> > > > +                isp_ctrl =
> > > > +                    CIF_ISP_CTRL_ISP_MODE_RAW_PICT;
> > > > +        } else {
> > > > +            writel(CIF_ISP_DEMOSAIC_TH(0xc),
> > > > +                   base + CIF_ISP_DEMOSAIC);
> > > > +
> > > > +            if (sensor->mbus.type == V4L2_MBUS_BT656)
> > > > +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU656;
> > > > +            else
> > > > +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_BAYER_ITU601;
> > > > +        }
> > > > +    } else if (in_fmt->fmt_type == FMT_YUV) {
> > > > +        acq_mult = 2;
> > > > +        if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> > > > +            isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> > > > +        } else {
> > > > +            if (sensor->mbus.type == V4L2_MBUS_BT656)
> > > > +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU656;
> > > > +            else
> > > > +                isp_ctrl = CIF_ISP_CTRL_ISP_MODE_ITU601;
> > > > +
> > > > +        }
> > > > +
> > > > +        irq_mask |= CIF_ISP_DATA_LOSS;
> > > > +    }
> > > > +
> > > > +    /* Set up input acquisition properties */
> > > > +    if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> > > > +        sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> > > > +        if (sensor->mbus.flags & V4L2_MBUS_PCLK_SAMPLE_RISING)
> > > > +            signal = CIF_ISP_ACQ_PROP_POS_EDGE;
> > > > +    }
> > > > +
> > > > +    if (sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> > > > +        if (sensor->mbus.flags & V4L2_MBUS_VSYNC_ACTIVE_LOW)
> > > > +            signal |= CIF_ISP_ACQ_PROP_VSYNC_LOW;
> > > > +
> > > > +        if (sensor->mbus.flags & V4L2_MBUS_HSYNC_ACTIVE_LOW)
> > > > +            signal |= CIF_ISP_ACQ_PROP_HSYNC_LOW;
> > > > +    }
> > > > +
> > > > +    writel(isp_ctrl, base + CIF_ISP_CTRL);
> > > > +    writel(signal | in_fmt->yuv_seq |
> > > > +           CIF_ISP_ACQ_PROP_BAYER_PAT(in_fmt->bayer_pat) |
> > > > +           CIF_ISP_ACQ_PROP_FIELD_SEL_ALL, base + CIF_ISP_ACQ_PROP);
> > > > +    writel(0, base + CIF_ISP_ACQ_NR_FRAMES);
> > > > +
> > > > +    /* Acquisition Size */
> > > > +    writel(0, base + CIF_ISP_ACQ_H_OFFS);
> > > > +    writel(0, base + CIF_ISP_ACQ_V_OFFS);
> > > > +    writel(acq_mult * in_frm->width, base + CIF_ISP_ACQ_H_SIZE);
> > > > +    writel(in_frm->height, base + CIF_ISP_ACQ_V_SIZE);
> > > > +
> > > > +    /* ISP Out Area */
> > > > +    writel(in_crop->left, base + CIF_ISP_OUT_H_OFFS);
> > > > +    writel(in_crop->top, base + CIF_ISP_OUT_V_OFFS);
> > > > +    writel(in_crop->width, base + CIF_ISP_OUT_H_SIZE);
> > > > +    writel(in_crop->height, base + CIF_ISP_OUT_V_SIZE);
> > > > +
> > > > +    /* interrupt mask */
> > > > +    irq_mask |= CIF_ISP_FRAME | CIF_ISP_V_START | CIF_ISP_PIC_SIZE_ERROR |
> > > > +            CIF_ISP_FRAME_IN;
> > > > +    writel(irq_mask, base + CIF_ISP_IMSC);
> > > > +
> > > > +    if (out_fmt->fmt_type == FMT_BAYER)
> > > > +        rkisp1_params_disable_isp(&dev->params_vdev);
> > > > +    else
> > > > +        rkisp1_params_configure_isp(&dev->params_vdev, in_fmt,
> > > > +                        dev->isp_sdev.quantization);
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rkisp1_config_dvp(struct rkisp1_device *dev)
> > > > +{
> > > > +    struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    u32 val, input_sel;
> > > > +
> > > > +    switch (in_fmt->bus_width) {
> > > > +    case 8:
> > > > +        input_sel = CIF_ISP_ACQ_PROP_IN_SEL_8B_ZERO;
> > > > +        break;
> > > > +    case 10:
> > > > +        input_sel = CIF_ISP_ACQ_PROP_IN_SEL_10B_ZERO;
> > > > +        break;
> > > > +    case 12:
> > > > +        input_sel = CIF_ISP_ACQ_PROP_IN_SEL_12B;
> > > > +        break;
> > > > +    default:
> > > > +        v4l2_err(&dev->v4l2_dev, "Invalid bus width\n");
> > > > +        return -EINVAL;
> > > > +    }
> > > > +
> > > > +    val = readl(base + CIF_ISP_ACQ_PROP);
> > > > +    writel(val | input_sel, base + CIF_ISP_ACQ_PROP);
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rkisp1_config_mipi(struct rkisp1_device *dev)
> > > > +{
> > > > +    struct ispsd_in_fmt *in_fmt = &dev->isp_sdev.in_fmt;
> > > > +    struct rkisp1_sensor *sensor = dev->active_sensor;
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    unsigned int lanes;
> > > > +    u32 mipi_ctrl;
> > > > +
> > > > +    /*
> > > > +     * sensor->mbus is set in isp or d-phy notifier_bound function
> > > > +     */
> > > > +    switch (sensor->mbus.flags & V4L2_MBUS_CSI2_LANES) {
> > > > +    case V4L2_MBUS_CSI2_4_LANE:
> > > > +        lanes = 4;
> > > > +        break;
> > > > +    case V4L2_MBUS_CSI2_3_LANE:
> > > > +        lanes = 3;
> > > > +        break;
> > > > +    case V4L2_MBUS_CSI2_2_LANE:
> > > > +        lanes = 2;
> > > > +        break;
> > > > +    case V4L2_MBUS_CSI2_1_LANE:
> > > > +        lanes = 1;
> > > > +        break;
> > > > +    default:
> > > > +        return -EINVAL;
> > > > +    }
> > > > +
> > > > +    mipi_ctrl = CIF_MIPI_CTRL_NUM_LANES(lanes - 1) |
> > > > +            CIF_MIPI_CTRL_SHUTDOWNLANES(0xf) |
> > > > +            CIF_MIPI_CTRL_ERR_SOT_SYNC_HS_SKIP |
> > > > +            CIF_MIPI_CTRL_CLOCKLANE_ENA;
> > > > +
> > > > +    writel(mipi_ctrl, base + CIF_MIPI_CTRL);
> > > > +
> > > > +    /* Configure Data Type and Virtual Channel */
> > > > +    writel(CIF_MIPI_DATA_SEL_DT(in_fmt->mipi_dt) | CIF_MIPI_DATA_SEL_VC(0),
> > > > +           base + CIF_MIPI_IMG_DATA_SEL);
> > > > +
> > > > +    /* Clear MIPI interrupts */
> > > > +    writel(~0, base + CIF_MIPI_ICR);
> > > > +    /*
> > > > +     * Disable CIF_MIPI_ERR_DPHY interrupt here temporary for
> > > > +     * isp bus may be dead when switch isp.
> > > > +     */
> > > > +    writel(CIF_MIPI_FRAME_END | CIF_MIPI_ERR_CSI | CIF_MIPI_ERR_DPHY |
> > > > +           CIF_MIPI_SYNC_FIFO_OVFLW(0x03) | CIF_MIPI_ADD_DATA_OVFLW,
> > > > +           base + CIF_MIPI_IMSC);
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "\n  MIPI_CTRL 0x%08x\n"
> > > > +         "  MIPI_IMG_DATA_SEL 0x%08x\n"
> > > > +         "  MIPI_STATUS 0x%08x\n"
> > > > +         "  MIPI_IMSC 0x%08x\n",
> > > > +         readl(base + CIF_MIPI_CTRL),
> > > > +         readl(base + CIF_MIPI_IMG_DATA_SEL),
> > > > +         readl(base + CIF_MIPI_STATUS),
> > > > +         readl(base + CIF_MIPI_IMSC));
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +/* Configure MUX */
> > > > +static int rkisp1_config_path(struct rkisp1_device *dev)
> > > > +{
> > > > +    struct rkisp1_sensor *sensor = dev->active_sensor;
> > > > +    u32 dpcl = readl(dev->base_addr + CIF_VI_DPCL);
> > > > +    int ret = 0;
> > > > +
> > > > +    if (sensor->mbus.type == V4L2_MBUS_BT656 ||
> > > > +        sensor->mbus.type == V4L2_MBUS_PARALLEL) {
> > > > +        ret = rkisp1_config_dvp(dev);
> > > > +        dpcl |= CIF_VI_DPCL_IF_SEL_PARALLEL;
> > > > +    } else if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> > > > +        ret = rkisp1_config_mipi(dev);
> > > > +        dpcl |= CIF_VI_DPCL_IF_SEL_MIPI;
> > > > +    }
> > > > +
> > > > +    writel(dpcl, dev->base_addr + CIF_VI_DPCL);
> > > > +
> > > > +    return ret;
> > > > +}
> > > > +
> > > > +/* Hareware configure Entry */
> > > 
> > > s/Hareware/Hardware/
> > > 
> > > > +static int rkisp1_config_cif(struct rkisp1_device *dev)
> > > > +{
> > > > +    u32 cif_id;
> > > > +    int ret;
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> > > > +         "SP streaming = %d, MP streaming = %d\n",
> > > > +         dev->stream[RKISP1_STREAM_SP].streaming,
> > > > +         dev->stream[RKISP1_STREAM_MP].streaming);
> > > > +
> > > > +    cif_id = readl(dev->base_addr + CIF_VI_ID);
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev, "CIF_ID 0x%08x\n", cif_id);
> > > > +
> > > > +    ret = rkisp1_config_isp(dev);
> > > > +    if (ret < 0)
> > > > +        return ret;
> > > > +    ret = rkisp1_config_path(dev);
> > > > +    if (ret < 0)
> > > > +        return ret;
> > > > +    rkisp1_config_ism(dev);
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +/* Mess register operations to stop isp */
> > > 
> > > Is it such a mess ? :-)
> > > 
> > > I would capitalise ISP in all comments.
> > > 
> > > > +static int rkisp1_isp_stop(struct rkisp1_device *dev)
> > > > +{
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    u32 val;
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> > > > +         "SP streaming = %d, MP streaming = %d\n",
> > > > +         dev->stream[RKISP1_STREAM_SP].streaming,
> > > > +         dev->stream[RKISP1_STREAM_MP].streaming);
> > > > +
> > > > +    /*
> > > > +     * ISP(mi) stop in mi frame end -> Stop ISP(mipi) ->
> > > > +     * Stop ISP(isp) ->wait for ISP isp off
> > > > +     */
> > > > +    /* stop and clear MI, MIPI, and ISP interrupts */
> > > > +    writel(0, base + CIF_MIPI_IMSC);
> > > > +    writel(~0, base + CIF_MIPI_ICR);
> > > > +
> > > > +    writel(0, base + CIF_ISP_IMSC);
> > > > +    writel(~0, base + CIF_ISP_ICR);
> > > > +
> > > > +    writel(0, base + CIF_MI_IMSC);
> > > > +    writel(~0, base + CIF_MI_ICR);
> > > > +    val = readl(base + CIF_MIPI_CTRL);
> > > > +    writel(val & (~CIF_MIPI_CTRL_OUTPUT_ENA), base + CIF_MIPI_CTRL);
> > > > +    /* stop ISP */
> > > > +    val = readl(base + CIF_ISP_CTRL);
> > > > +    val &= ~(CIF_ISP_CTRL_ISP_INFORM_ENABLE | CIF_ISP_CTRL_ISP_ENABLE);
> > > > +    writel(val, base + CIF_ISP_CTRL);
> > > > +
> > > > +    val = readl(base + CIF_ISP_CTRL);
> > > > +    writel(val | CIF_ISP_CTRL_ISP_CFG_UPD, base + CIF_ISP_CTRL);
> > > > +
> > > > +    readx_poll_timeout(readl, base + CIF_ISP_RIS,
> > > > +               val, val & CIF_ISP_OFF, 20, 100);
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> > > > +        "streaming(MP:%d, SP:%d), MI_CTRL:%x, ISP_CTRL:%x, MIPI_CTRL:%x\n",
> > > > +         dev->stream[RKISP1_STREAM_SP].streaming,
> > > > +         dev->stream[RKISP1_STREAM_MP].streaming,
> > > > +         readl(base + CIF_MI_CTRL),
> > > > +         readl(base + CIF_ISP_CTRL),
> > > > +         readl(base + CIF_MIPI_CTRL));
> > > > +
> > > > +    writel(CIF_IRCL_MIPI_SW_RST | CIF_IRCL_ISP_SW_RST, base + CIF_IRCL);
> > > > +    writel(0x0, base + CIF_IRCL);
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +/* Mess register operations to start isp */
> > > > +static int rkisp1_isp_start(struct rkisp1_device *dev)
> > > > +{
> > > > +    struct rkisp1_sensor *sensor = dev->active_sensor;
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    u32 val;
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> > > > +         "SP streaming = %d, MP streaming = %d\n",
> > > > +         dev->stream[RKISP1_STREAM_SP].streaming,
> > > > +         dev->stream[RKISP1_STREAM_MP].streaming);
> > > > +
> > > > +    /* Activate MIPI */
> > > > +    if (sensor->mbus.type == V4L2_MBUS_CSI2_DPHY) {
> > > > +        val = readl(base + CIF_MIPI_CTRL);
> > > > +        writel(val | CIF_MIPI_CTRL_OUTPUT_ENA, base + CIF_MIPI_CTRL);
> > > > +    }
> > > > +    /* Activate ISP */
> > > > +    val = readl(base + CIF_ISP_CTRL);
> > > > +    val |= CIF_ISP_CTRL_ISP_CFG_UPD | CIF_ISP_CTRL_ISP_ENABLE |
> > > > +           CIF_ISP_CTRL_ISP_INFORM_ENABLE;
> > > > +    writel(val, base + CIF_ISP_CTRL);
> > > > +
> > > > +    /* XXX: Is the 1000us too long?
> > > > +     * CIF spec says to wait for sufficient time after enabling
> > > > +     * the MIPI interface and before starting the sensor output.
> > > > +     */
> > > > +    usleep_range(1000, 1200);
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> > > > +         "SP streaming = %d, MP streaming = %d MI_CTRL 0x%08x\n"
> > > > +         "  ISP_CTRL 0x%08x MIPI_CTRL 0x%08x\n",
> > > > +         dev->stream[RKISP1_STREAM_SP].streaming,
> > > > +         dev->stream[RKISP1_STREAM_MP].streaming,
> > > > +         readl(base + CIF_MI_CTRL),
> > > > +         readl(base + CIF_ISP_CTRL),
> > > > +         readl(base + CIF_MIPI_CTRL));
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static void rkisp1_config_clk(struct rkisp1_device *dev)
> > > > +{
> > > > +    u32 val = CIF_ICCL_ISP_CLK | CIF_ICCL_CP_CLK | CIF_ICCL_MRSZ_CLK |
> > > > +          CIF_ICCL_SRSZ_CLK | CIF_ICCL_JPEG_CLK | CIF_ICCL_MI_CLK |
> > > > +          CIF_ICCL_IE_CLK | CIF_ICCL_MIPI_CLK | CIF_ICCL_DCROP_CLK;
> > > > +
> > > > +    writel(val, dev->base_addr + CIF_ICCL);
> > > > +}
> > > > +
> > > > +/***************************** isp sub-devs *******************************/
> > > > +
> > > > +static const struct ispsd_in_fmt rkisp1_isp_input_formats[] = {
> > > > +    {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SBGGR10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW10,
> > > > +        .bayer_pat    = RAW_BGGR,
> > > > +        .bus_width    = 10,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SRGGB10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW10,
> > > > +        .bayer_pat    = RAW_RGGB,
> > > > +        .bus_width    = 10,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGBRG10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW10,
> > > > +        .bayer_pat    = RAW_GBRG,
> > > > +        .bus_width    = 10,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGRBG10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW10,
> > > > +        .bayer_pat    = RAW_GRBG,
> > > > +        .bus_width    = 10,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SRGGB12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW12,
> > > > +        .bayer_pat    = RAW_RGGB,
> > > > +        .bus_width    = 12,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SBGGR12_1X12,
> > > 
> > > Is there a reason why the 10-bit Bayer format start with BGGR while the
> > > 12-bit formats start with RGGB ? Not a big deal, just OCD kicking in :-)
> > > 
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW12,
> > > > +        .bayer_pat    = RAW_BGGR,
> > > > +        .bus_width    = 12,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGBRG12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW12,
> > > > +        .bayer_pat    = RAW_GBRG,
> > > > +        .bus_width    = 12,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGRBG12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW12,
> > > > +        .bayer_pat    = RAW_GRBG,
> > > > +        .bus_width    = 12,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SRGGB8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW8,
> > > > +        .bayer_pat    = RAW_RGGB,
> > > > +        .bus_width    = 8,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SBGGR8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW8,
> > > > +        .bayer_pat    = RAW_BGGR,
> > > > +        .bus_width    = 8,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGBRG8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW8,
> > > > +        .bayer_pat    = RAW_GBRG,
> > > > +        .bus_width    = 8,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGRBG8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +        .mipi_dt    = CIF_CSI2_DT_RAW8,
> > > > +        .bayer_pat    = RAW_GRBG,
> > > > +        .bus_width    = 8,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_YUYV8_1X16,
> > > > +        .fmt_type    = FMT_YUV,
> > > > +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
> > > > +        .yuv_seq    = CIF_ISP_ACQ_PROP_YCBYCR,
> > > > +        .bus_width    = 16,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_YVYU8_1X16,
> > > > +        .fmt_type    = FMT_YUV,
> > > > +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
> > > > +        .yuv_seq    = CIF_ISP_ACQ_PROP_YCRYCB,
> > > > +        .bus_width    = 16,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_UYVY8_1X16,
> > > > +        .fmt_type    = FMT_YUV,
> > > > +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
> > > > +        .yuv_seq    = CIF_ISP_ACQ_PROP_CBYCRY,
> > > > +        .bus_width    = 16,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_VYUY8_1X16,
> > > > +        .fmt_type    = FMT_YUV,
> > > > +        .mipi_dt    = CIF_CSI2_DT_YUV422_8b,
> > > > +        .yuv_seq    = CIF_ISP_ACQ_PROP_CRYCBY,
> > > > +        .bus_width    = 16,
> > > > +    },
> > > > +};
> > > > +
> > > > +static const struct ispsd_out_fmt rkisp1_isp_output_formats[] = {
> > > > +    {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_YUYV8_2X8,
> > > > +        .fmt_type    = FMT_YUV,
> > > > +    }, {
> > > 
> > > This is the only entry not present in the previous table, so I'm
> > > wondering if it would make sense to merge the two tables and rename
> > > ispsd_in_fmt to rkisp1_format_info. You would need to add a field that
> > > tells, for each format, if it's valid as an input format, and output
> > > format, or both. Hmmmm and also make the enum logic a bit more complex.
> > > Maybe it's not worth it after all, but it bothers me a bit to have two
> > > tables :-) I'll let you decide what's best.
> > > 
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SRGGB12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SBGGR12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGBRG12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGRBG12_1X12,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SRGGB10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SBGGR10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGBRG10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGRBG10_1X10,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SRGGB8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SBGGR8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGBRG8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    }, {
> > > > +        .mbus_code    = MEDIA_BUS_FMT_SGRBG8_1X8,
> > > > +        .fmt_type    = FMT_BAYER,
> > > > +    },
> > > > +};
> > > > +
> > > > +static const struct ispsd_in_fmt *find_in_fmt(u32 mbus_code)
> > > > +{
> > > > +    unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_input_formats);
> > > > +    const struct ispsd_in_fmt *fmt;
> > > 
> > > You can move this variable inside the loop.
> > > 
> > > > +
> > > > +    for (i = 0; i < array_size; i++) {
> > > 
> > > I would remove the array_size local variable, it doesn't improve
> > > readability. Same for the next function.
> > > 
> > > > +        fmt = &rkisp1_isp_input_formats[i];
> > > > +        if (fmt->mbus_code == mbus_code)
> > > > +            return fmt;
> > > > +    }
> > > > +
> > > > +    return NULL;
> > > > +}
> > > > +
> > > > +static const struct ispsd_out_fmt *find_out_fmt(u32 mbus_code)
> > > > +{
> > > > +    unsigned int i, array_size = ARRAY_SIZE(rkisp1_isp_output_formats);
> > > > +    const struct ispsd_out_fmt *fmt;
> > > > +
> > > > +    for (i = 0; i < array_size; i++) {
> > > > +        fmt = &rkisp1_isp_output_formats[i];
> > > > +        if (fmt->mbus_code == mbus_code)
> > > > +            return fmt;
> > > > +    }
> > > > +
> > > > +    return NULL;
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_enum_mbus_code(struct v4l2_subdev *sd,
> > > > +                    struct v4l2_subdev_pad_config *cfg,
> > > > +                    struct v4l2_subdev_mbus_code_enum *code)
> > > > +{
> > > > +    unsigned int i = code->index;
> > > > +
> > > > +    if ((code->pad != RKISP1_ISP_PAD_SINK) &&
> > > > +        (code->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> > > > +        if (i > 0)
> > > > +            return -EINVAL;
> > > > +        code->code = MEDIA_BUS_FMT_FIXED;
> > > > +        return 0;
> > > > +    }
> > > > +
> > > > +    if (code->pad == RKISP1_ISP_PAD_SINK) {
> > > > +        if (i >= ARRAY_SIZE(rkisp1_isp_input_formats))
> > > > +            return -EINVAL;
> > > > +        code->code = rkisp1_isp_input_formats[i].mbus_code;
> > > > +    } else {
> > > > +        if (i >= ARRAY_SIZE(rkisp1_isp_output_formats))
> > > > +            return -EINVAL;
> > > > +        code->code = rkisp1_isp_output_formats[i].mbus_code;
> > > > +    }
> > > 
> > > On the other hand, merging the two tables above into one, you could
> > > merge the two branches here and only consider formats indicated as valid
> > > for to the pad you want. Maybe the code would be cleaner in the end.
> > > 
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_init_config(struct v4l2_subdev *sd,
> > > > +                     struct v4l2_subdev_pad_config *cfg)
> > > > +{
> > > > +    struct v4l2_rect *mf_in_crop, *mf_out_crop;
> > > > +    struct v4l2_mbus_framefmt *mf_in, *mf_out;
> > > > +
> > > > +    mf_in = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SINK);
> > > > +    mf_in->width = RKISP1_DEFAULT_WIDTH;
> > > > +    mf_in->height = RKISP1_DEFAULT_HEIGHT;
> > > > +    mf_in->field = V4L2_FIELD_NONE;
> > > > +    mf_in->code = rkisp1_isp_input_formats[0].mbus_code;
> > > > +
> > > > +    mf_in_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SINK);
> > > > +    mf_in_crop->width = RKISP1_DEFAULT_WIDTH;
> > > > +    mf_in_crop->height = RKISP1_DEFAULT_HEIGHT;
> > > > +    mf_in_crop->left = 0;
> > > > +    mf_in_crop->top = 0;
> > > > +
> > > > +    mf_out = v4l2_subdev_get_try_format(sd, cfg,
> > > > +                        RKISP1_ISP_PAD_SOURCE_PATH);
> > > > +    *mf_out = *mf_in;
> > > > +    mf_out->code = rkisp1_isp_output_formats[0].mbus_code;
> > > > +    mf_out->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > +
> > > > +    mf_out_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > > > +                           RKISP1_ISP_PAD_SOURCE_PATH);
> > > > +    *mf_out_crop = *mf_in_crop;
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_get_fmt(struct v4l2_subdev *sd,
> > > > +                 struct v4l2_subdev_pad_config *cfg,
> > > > +                 struct v4l2_subdev_format *fmt)
> > > > +{
> > > > +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > > +    struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > > +
> > > > +    if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> > > > +        (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH)) {
> > > > +        fmt->format.code = MEDIA_BUS_FMT_FIXED;
> > > > +        /*
> > > > +         * NOTE: setting a format here doesn't make much sense
> > > > +         * but v4l2-compliance complains
> > > > +         */
> > > 
> > > For the params pad I agreed it makes no sense, and I think
> > > v4l2-compliance is at fault, so I'd set width and height to 0. For the
> > > stats pad we *could* use the size of the image from which stats are
> > > computed, but because v4l2_meta_format has no width/height, I think 0
> > > would also be appropriate.
> > > 
> > > > +        fmt->format.width = RKISP1_DEFAULT_WIDTH;
> > > > +        fmt->format.height = RKISP1_DEFAULT_HEIGHT;
> > > > +        fmt->format.field = V4L2_FIELD_NONE;
> > > > +        return 0;
> > > > +    }
> > > > +
> > > > +    if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > > +        mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> > > > +        fmt->format = *mf;
> > > > +        return 0;
> > > > +    }
> > > > +
> > > > +    if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> > > > +        *mf = isp_sd->in_frm;
> > > > +    } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > > > +        /* format of source pad */
> > > > +        *mf = isp_sd->in_frm;
> > > > +        mf->code = isp_sd->out_fmt.mbus_code;
> > > > +        /* window size of source pad */
> > > > +        mf->width = isp_sd->out_crop.width;
> > > > +        mf->height = isp_sd->out_crop.height;
> > > > +        mf->quantization = isp_sd->quantization;
> > > > +    }
> > > > +    mf->field = V4L2_FIELD_NONE;
> > > 
> > > This can be simplified, please read through, or jump to the review of
> > > rkisp1_isp_subdev at the end.
> > > 
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static void rkisp1_isp_sd_try_fmt(struct v4l2_subdev *sd,
> > > > +                  unsigned int pad,
> > > > +                  struct v4l2_mbus_framefmt *fmt)
> > > > +{
> > > > +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > > +    struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> > > > +    const struct ispsd_out_fmt *out_fmt;
> > > > +    const struct ispsd_in_fmt *in_fmt;
> > > > +
> > > > +    switch (pad) {
> > > > +    case RKISP1_ISP_PAD_SINK:
> > > > +        in_fmt = find_in_fmt(fmt->code);
> > > > +        if (in_fmt)
> > > > +            fmt->code = in_fmt->mbus_code;
> > > > +        else
> > > > +            fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
> > > 
> > > You write MEDIA_BUS_FMT_SRGGB10_1X10 explicitly here, while you use
> > > rkisp1_isp_output_formats[0].mbus_code below (and in other places). I
> > > would standardise on one of the two (explicit formats or array[0]), with
> > > a preference for the first as that would allow merging the input and
> > > output arrays more easily. I would then create two #define,
> > > RKISP1_DEF_INPUT_FORMAT and RKISP2_DEF_OUTPUT_FORMAT (or similar).
> > > Similar macros for the default width and height could also be useful, to
> > > make it easier to change them.
> > > 
> > > > +        fmt->width  = clamp_t(u32, fmt->width, CIF_ISP_INPUT_W_MIN,
> > > > +                      CIF_ISP_INPUT_W_MAX);
> > > > +        fmt->height = clamp_t(u32, fmt->height, CIF_ISP_INPUT_H_MIN,
> > > > +                      CIF_ISP_INPUT_H_MAX);
> > > > +        break;
> > > > +    case RKISP1_ISP_PAD_SOURCE_PATH:
> > > > +        out_fmt = find_out_fmt(fmt->code);
> > > > +        if (out_fmt)
> > > > +            fmt->code = out_fmt->mbus_code;
> > > > +        else
> > > > +            fmt->code = rkisp1_isp_output_formats[0].mbus_code;
> > > > +        /* window size is set in s_selection */
> > > > +        fmt->width  = isp_sd->out_crop.width;
> > > > +        fmt->height = isp_sd->out_crop.height;
> > > 
> > > This function operates on the TRY configuration too, in which case you
> > > should use the TRY crop rectangle here, not the ACTIVE one. If you've
> > > already jumped to the review of rkisp1_isp_subdev you know my proposal
> > > to simplify this. Otherwise now may be a good time to do so :-)
> > > 
> > > > +        /* full range by default */
> > > > +        if (!fmt->quantization)
> > > > +            fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > +        break;
> > > > +    }
> > > > +
> > > > +    fmt->field = V4L2_FIELD_NONE;
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_set_fmt(struct v4l2_subdev *sd,
> > > > +                 struct v4l2_subdev_pad_config *cfg,
> > > > +                 struct v4l2_subdev_format *fmt)
> > > > +{
> > > > +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > > +    struct rkisp1_isp_subdev *isp_sd = &isp_dev->isp_sdev;
> > > > +    struct v4l2_mbus_framefmt *mf = &fmt->format;
> > > > +
> > > > +    if ((fmt->pad != RKISP1_ISP_PAD_SINK) &&
> > > > +        (fmt->pad != RKISP1_ISP_PAD_SOURCE_PATH))
> > > > +        return rkisp1_isp_sd_get_fmt(sd, cfg, fmt);
> > > > +
> > > > +    rkisp1_isp_sd_try_fmt(sd, fmt->pad, mf);
> > > > +
> > > > +    if (fmt->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > > +        struct v4l2_mbus_framefmt *try_mf;
> > > > +
> > > > +        try_mf = v4l2_subdev_get_try_format(sd, cfg, fmt->pad);
> > > > +        *try_mf = *mf;
> > > > +        return 0;
> > > 
> > > When setting the format on the sink pad the crop rectangles need to be
> > > reset (here, and for the ACTIVE format below too).
> > > 
> > > > +    }
> > > > +
> > > > +    if (fmt->pad == RKISP1_ISP_PAD_SINK) {
> > > > +        const struct ispsd_in_fmt *in_fmt;
> > > > +
> > > > +        in_fmt = find_in_fmt(mf->code);
> > > > +        isp_sd->in_fmt = *in_fmt;
> > > > +        isp_sd->in_frm = *mf;
> > > > +    } else if (fmt->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > > > +        const struct ispsd_out_fmt *out_fmt;
> > > > +
> > > > +        /* Ignore width/height */
> > > > +        out_fmt = find_out_fmt(mf->code);
> > > > +        isp_sd->out_fmt = *out_fmt;
> > > 
> > > I would return the in_fmt and out_fmt from rkisp1_isp_sd_try_fmt() as it
> > > already looks them up. If you merge the input and output tables, you'll
> > > have a single format info structure type, and rkisp1_isp_sd_try_fmt()
> > > could return the entry for the pad it operates on.
> > > 
> > > > +        /*
> > > > +         * It is quantization for output,
> > > > +         * isp use bt601 limit-range in internal
> > > > +         */
> > > > +        isp_sd->quantization = mf->quantization;
> > > > +    }
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static void rkisp1_isp_sd_try_crop(struct v4l2_subdev *sd,
> > > > +                  struct v4l2_subdev_pad_config *cfg,
> > > > +                  struct v4l2_subdev_selection *sel)
> > > > +{
> > > > +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > > +    struct v4l2_mbus_framefmt in_frm = isp_sd->in_frm;
> > > > +    struct v4l2_rect in_crop = isp_sd->in_crop;
> > > > +    struct v4l2_rect *input = &sel->r;
> > > > +
> > > > +    if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > > +        in_frm = *v4l2_subdev_get_try_format(sd, cfg,
> > > > +                             RKISP1_ISP_PAD_SINK);
> > > > +        in_crop = *v4l2_subdev_get_try_crop(sd, cfg,
> > > > +                            RKISP1_ISP_PAD_SINK);
> > > > +    }
> > > > +
> > > > +    input->left = ALIGN(input->left, 2);
> > > > +    input->width = ALIGN(input->width, 2);
> > > > +
> > > > +    if (sel->pad == RKISP1_ISP_PAD_SINK) {
> > > > +        input->left = clamp_t(u32, input->left, 0, in_frm.width);
> > > > +        input->top = clamp_t(u32, input->top, 0, in_frm.height);
> > > > +        input->width = clamp_t(u32, input->width, CIF_ISP_INPUT_W_MIN,
> > > > +                       in_frm.width - input->left);
> > > > +        input->height = clamp_t(u32, input->height,
> > > > +                    CIF_ISP_INPUT_H_MIN,
> > > > +                    in_frm.height - input->top);
> > > > +    } else if (sel->pad == RKISP1_ISP_PAD_SOURCE_PATH) {
> > > > +        input->left = clamp_t(u32, input->left, 0, in_crop.width);
> > > > +        input->top = clamp_t(u32, input->top, 0, in_crop.height);
> > > > +        input->width = clamp_t(u32, input->width, CIF_ISP_OUTPUT_W_MIN,
> > > > +                       in_crop.width - input->left);
> > > > +        input->height = clamp_t(u32, input->height,
> > > > +                    CIF_ISP_OUTPUT_H_MIN,
> > > > +                    in_crop.height - input->top);
> > > > +    }
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_get_selection(struct v4l2_subdev *sd,
> > > > +                       struct v4l2_subdev_pad_config *cfg,
> > > > +                       struct v4l2_subdev_selection *sel)
> > > > +{
> > > > +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > > +    struct v4l2_mbus_framefmt *frm;
> > > > +    struct v4l2_rect *rect;
> > > > +
> > > > +    if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> > > > +        sel->pad != RKISP1_ISP_PAD_SINK)
> > > > +        return -EINVAL;
> > > > +
> > > > +    switch (sel->target) {
> > > > +    case V4L2_SEL_TGT_CROP_BOUNDS:
> > > > +        if (sel->pad == RKISP1_ISP_PAD_SINK) {
> > > > +            if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> > > > +                frm = v4l2_subdev_get_try_format(sd, cfg,
> > > > +                                 sel->pad);
> > > > +            else
> > > > +                frm = &isp_sd->in_frm;
> > > > +
> > > > +            sel->r.height = frm->height;
> > > > +            sel->r.width = frm->width;
> > > > +            sel->r.left = 0;
> > > > +            sel->r.top = 0;
> > > > +        } else {
> > > > +            if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> > > > +                rect = v4l2_subdev_get_try_crop(sd, cfg,
> > > > +                            RKISP1_ISP_PAD_SINK);
> > > > +            else
> > > > +                rect = &isp_sd->in_crop;
> > > > +            sel->r = *rect;
> > > > +        }
> > > > +        break;
> > > > +    case V4L2_SEL_TGT_CROP:
> > > > +        if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
> > > > +            rect = v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> > > > +        else if (sel->pad == RKISP1_ISP_PAD_SINK)
> > > > +            rect = &isp_sd->in_crop;
> > > > +        else
> > > > +            rect = &isp_sd->out_crop;
> > > > +        sel->r = *rect;
> > > > +        break;
> > > > +    default:
> > > > +        return -EINVAL;
> > > > +    }
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_set_selection(struct v4l2_subdev *sd,
> > > > +                       struct v4l2_subdev_pad_config *cfg,
> > > > +                       struct v4l2_subdev_selection *sel)
> > > > +{
> > > > +    struct rkisp1_isp_subdev *isp_sd = sd_to_isp_sd(sd);
> > > > +    struct rkisp1_device *dev = sd_to_isp_dev(sd);
> > > > +
> > > > +    if (sel->pad != RKISP1_ISP_PAD_SOURCE_PATH &&
> > > > +        sel->pad != RKISP1_ISP_PAD_SINK)
> > > > +        return -EINVAL;
> > > > +    if (sel->target != V4L2_SEL_TGT_CROP)
> > > > +        return -EINVAL;
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &dev->v4l2_dev,
> > > > +         "%s: pad: %d sel(%d,%d)/%dx%d\n", __func__, sel->pad,
> > > > +         sel->r.left, sel->r.top, sel->r.width, sel->r.height);
> > > > +    rkisp1_isp_sd_try_crop(sd, cfg, sel);
> > > > +
> > > > +    if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
> > > > +        struct v4l2_rect *try_sel =
> > > > +            v4l2_subdev_get_try_crop(sd, cfg, sel->pad);
> > > > +
> > > > +        *try_sel = sel->r;
> > > > +        return 0;
> > > 
> > > When setting the crop rectangle on the sink pad the rectangle on the
> > > source pad should be reset (and below for the ACTIVE format too).
> > > 
> > > The resizer logic, through selection rectangles, seems to be missing. I
> > > assume it's configured through the video nodes, but that's not correct
> > > I'm afraid, scaling should be configured on subdevs, see [1]. I'm afraid
> > > this means that we'll need separate subdevs for the resizers :-S
> > > 
> > > [1] https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/dev-subdev.html#order-of-configuration-and-format-propagation
> > > 
> > > > +    }
> > > > +
> > > > +    if (sel->pad == RKISP1_ISP_PAD_SINK)
> > > > +        isp_sd->in_crop = sel->r;
> > > > +    else
> > > > +        isp_sd->out_crop = sel->r;
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > 
> > > I'll skip the part related to power on/off below as Sakari requested it
> > > to be handled through runtime PM.
> > > 
> > > > +static int mipi_csi2_s_stream_start(struct rkisp1_isp_subdev *isp_sd,
> > > > +                    struct rkisp1_sensor *sensor)
> > > > +{
> > > > +    union phy_configure_opts opts = { 0 };
> > > > +    struct phy_configure_opts_mipi_dphy *cfg = &opts.mipi_dphy;
> > > > +    struct v4l2_ctrl *pixel_rate;
> > > > +    s64 pixel_clock;
> > > > +
> > > > +    pixel_rate = v4l2_ctrl_find(sensor->sd->ctrl_handler,
> > > > +                    V4L2_CID_PIXEL_RATE);
> > > > +    if (!pixel_rate) {
> > > > +        v4l2_warn(sensor->sd, "No pixel rate control in subdev\n");
> > > > +        return -EPIPE;
> > > > +    }
> > > 
> > > Would it make sense to retrieve (and cache) the control pointer (not its
> > > value of course) at probe time already ?
> > > 
> > > > +
> > > > +    pixel_clock = v4l2_ctrl_g_ctrl_int64(pixel_rate);
> > > > +    if (!pixel_clock) {
> > > > +        v4l2_err(sensor->sd, "Invalid pixel rate value\n");
> > > > +        return -EINVAL;
> > > > +    }
> > > > +
> > > > +    phy_mipi_dphy_get_default_config(pixel_clock, isp_sd->in_fmt.bus_width,
> > > > +                     sensor->lanes, cfg);
> > > > +    phy_set_mode(sensor->dphy, PHY_MODE_MIPI_DPHY);
> > > > +    phy_configure(sensor->dphy, &opts);
> > > > +    phy_power_on(sensor->dphy);
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static void mipi_csi2_s_stream_stop(struct rkisp1_sensor *sensor)
> > > > +{
> > > > +    phy_power_off(sensor->dphy);
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_s_stream(struct v4l2_subdev *sd, int on)
> > > > +{
> > > > +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > > +    struct v4l2_subdev *sensor_sd;
> > > > +    int ret = 0;
> > > > +
> > > > +    if (!on) {
> > > > +        ret = rkisp1_isp_stop(isp_dev);
> > > > +        if (ret < 0)
> > > > +            return ret;
> > > > +        mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> > > > +        return 0;
> > > > +    }
> > > > +
> > > > +    sensor_sd = get_remote_sensor(sd);
> > > > +    if (!sensor_sd)
> > > > +        return -ENODEV;
> > > > +
> > > > +    isp_dev->active_sensor = sd_to_sensor(isp_dev, sensor_sd);
> > > > +    if (!isp_dev->active_sensor)
> > > > +        return -ENODEV;
> > > > +
> > > > +    atomic_set(&isp_dev->isp_sdev.frm_sync_seq, 0);
> > > > +    ret = rkisp1_config_cif(isp_dev);
> > > > +    if (ret < 0)
> > > > +        return ret;
> > > > +
> > > > +    /* TODO: support other interfaces */
> > > > +    if (isp_dev->active_sensor->mbus.type != V4L2_MBUS_CSI2_DPHY)
> > > > +        return -EINVAL;
> > > > +
> > > > +    ret = mipi_csi2_s_stream_start(&isp_dev->isp_sdev,
> > > > +                       isp_dev->active_sensor);
> > > > +    if (ret < 0)
> > > > +        return ret;
> > > > +
> > > > +    ret = rkisp1_isp_start(isp_dev);
> > > > +    if (ret)
> > > > +        mipi_csi2_s_stream_stop(isp_dev->active_sensor);
> > > > +
> > > > +    return ret;
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_s_power(struct v4l2_subdev *sd, int on)
> > > > +{
> > > > +    struct rkisp1_device *isp_dev = sd_to_isp_dev(sd);
> > > > +    int ret;
> > > > +
> > > > +    v4l2_dbg(1, rkisp1_debug, &isp_dev->v4l2_dev, "s_power: %d\n", on);
> > > > +
> > > > +    if (on) {
> > > > +        ret = pm_runtime_get_sync(isp_dev->dev);
> > > > +        if (ret < 0)
> > > > +            return ret;
> > > > +
> > > > +        rkisp1_config_clk(isp_dev);
> > > > +    } else {
> > > > +        ret = pm_runtime_put(isp_dev->dev);
> > > > +        if (ret < 0)
> > > > +            return ret;
> > > > +    }
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rkisp1_subdev_link_validate(struct media_link *link)
> > > > +{
> > > > +    if (link->source->index == RKISP1_ISP_PAD_SINK_PARAMS)
> > > > +        return 0;
> > > > +
> > > > +    return v4l2_subdev_link_validate(link);
> > > > +}
> > > > +
> > > > +static int rkisp1_subdev_fmt_link_validate(struct v4l2_subdev *sd,
> > > > +                    struct media_link *link,
> > > > +                    struct v4l2_subdev_format *source_fmt,
> > > > +                    struct v4l2_subdev_format *sink_fmt)
> > > > +{
> > > > +    if (source_fmt->format.code != sink_fmt->format.code)
> > > > +        return -EINVAL;
> > > > +
> > > > +    /* Crop is available */
> > > > +    if (source_fmt->format.width < sink_fmt->format.width ||
> > > > +        source_fmt->format.height < sink_fmt->format.height)
> > > > +        return -EINVAL;
> > > 
> > > Crop should be handled on pads, not on links. The size at the ends of
> > > the link should match. This will likely require a bit of a redesign of
> > > the format and selection rectangles handling, but I think it's also an
> > > opportunity to simplify the code.
> > > 
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> > > > +{
> > > > +    struct v4l2_event event = {
> > > > +        .type = V4L2_EVENT_FRAME_SYNC,
> > > > +        .u.frame_sync.frame_sequence =
> > > > +            atomic_inc_return(&isp->frm_sync_seq) - 1,
> > > 
> > > I would move the increment to the caller, hiding it in this function is
> > > error-prone (and if you look at the caller I'm pointing out one possible
> > > error :-)).
> > > 
> > > In general usage of frm_sync_seq through the driver seems to be very
> > > race-prone. It's read in various IRQ handling functions, all coming from
> > > the same IRQ, so that part is fine (and wouldn't require an atomic
> > > variable), but when read from the buffer queue handlers I really get a
> > > red light flashing in my head. I'll try to investigate more when
> > > reviewing the next patches.
> > 
> > I see that the only place were 'frame_sequence' is read outside of the irq
> > handlers is in the capture in 'rkisp1_vb2_buf_queue':
> > 
> >      /*
> >           * If there's no next buffer assigned, queue this buffer directly
> >           * as the next buffer, and update the memory interface.
> >           */
> >          if (cap->is_streaming && !cap->buf.next &&
> >              atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
> >                  cap->buf.next = ispbuf;
> >                  rkisp1_set_next_buf(cap);
> >          } else {
> >                  list_add_tail(&ispbuf->queue, &cap->buf.queue);
> >          }
> > This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> > I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> > from the above condition so that the next buffer is updated in case it is null not just before the first
> > v-start signal.
> > 
> > 
> > 
> > 
> > 
> > > 
> > > > +    };
> > > > +    v4l2_event_queue(isp->sd.devnode, &event);
> > > > +}
> > > > +
> > > > +static int rkisp1_isp_sd_subs_evt(struct v4l2_subdev *sd, struct v4l2_fh *fh,
> > > > +                  struct v4l2_event_subscription *sub)
> > > > +{
> > > > +    if (sub->type != V4L2_EVENT_FRAME_SYNC)
> > > > +        return -EINVAL;
> > > > +
> > > > +    /* Line number. For now only zero accepted. */
> > > 
> > > Is the id a line number for V4L2_EVENT_FRAME_SYNC ? It's not mentioned
> > > for V4L2_EVENT_FRAME_SYNC in the V4L2 spec, so I think this check is
> > > correct, but the comment doesn't seem to be.
> > > 
> > > > +    if (sub->id != 0)
> > > > +        return -EINVAL;
> > > > +
> > > > +    return v4l2_event_subscribe(fh, sub, 0, NULL);
> > > > +}
> > > > +
> > > > +static const struct v4l2_subdev_pad_ops rkisp1_isp_sd_pad_ops = {
> > > > +    .enum_mbus_code = rkisp1_isp_sd_enum_mbus_code,
> > > > +    .get_selection = rkisp1_isp_sd_get_selection,
> > > > +    .set_selection = rkisp1_isp_sd_set_selection,
> > > > +    .init_cfg = rkisp1_isp_sd_init_config,
> > > > +    .get_fmt = rkisp1_isp_sd_get_fmt,
> > > > +    .set_fmt = rkisp1_isp_sd_set_fmt,
> > > > +    .link_validate = rkisp1_subdev_fmt_link_validate,
> > > > +};
> > > > +
> > > > +static const struct media_entity_operations rkisp1_isp_sd_media_ops = {
> > > > +    .link_validate = rkisp1_subdev_link_validate,
> > > > +};
> > > > +
> > > > +static const struct v4l2_subdev_video_ops rkisp1_isp_sd_video_ops = {
> > > > +    .s_stream = rkisp1_isp_sd_s_stream,
> > > > +};
> > > > +
> > > > +static const struct v4l2_subdev_core_ops rkisp1_isp_core_ops = {
> > > > +    .subscribe_event = rkisp1_isp_sd_subs_evt,
> > > > +    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
> > > > +    .s_power = rkisp1_isp_sd_s_power,
> > > > +};
> > > > +
> > > > +static struct v4l2_subdev_ops rkisp1_isp_sd_ops = {
> > > 
> > > static const
> > > 
> > > > +    .core = &rkisp1_isp_core_ops,
> > > > +    .video = &rkisp1_isp_sd_video_ops,
> > > > +    .pad = &rkisp1_isp_sd_pad_ops,
> > > > +};
> > > > +
> > > > +static void rkisp1_isp_sd_init_default_fmt(struct rkisp1_isp_subdev *isp_sd)
> > > > +{
> > > > +    struct v4l2_mbus_framefmt *in_frm = &isp_sd->in_frm;
> > > > +    struct v4l2_rect *in_crop = &isp_sd->in_crop;
> > > > +    struct v4l2_rect *out_crop = &isp_sd->out_crop;
> > > > +    struct ispsd_in_fmt *in_fmt = &isp_sd->in_fmt;
> > > > +    struct ispsd_out_fmt *out_fmt = &isp_sd->out_fmt;
> > > > +
> > > > +    *in_fmt = rkisp1_isp_input_formats[0];
> > > > +    in_frm->width = RKISP1_DEFAULT_WIDTH;
> > > > +    in_frm->height = RKISP1_DEFAULT_HEIGHT;
> > > > +    in_frm->code = in_fmt->mbus_code;
> > > > +
> > > > +    in_crop->width = in_frm->width;
> > > > +    in_crop->height = in_frm->height;
> > > > +    in_crop->left = 0;
> > > > +    in_crop->top = 0;
> > > > +
> > > > +    /* propagate to source */
> > > > +    *out_crop = *in_crop;
> > > > +    *out_fmt = rkisp1_isp_output_formats[0];
> > > > +    isp_sd->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > 
> > > I wonder, could this be implemented by sharing code with .init_cfg() if
> > > you store the active configuration in v4l2_subdev_pad_config instances
> > > (please see the comments about the rkisp1_isp_subdev structure below) ?
> > > 
> > > > +}
> > > > +
> > > > +int rkisp1_register_isp_subdev(struct rkisp1_device *isp_dev,
> > > > +                   struct v4l2_device *v4l2_dev)
> > > > +{
> > > > +    struct rkisp1_isp_subdev *isp_sdev = &isp_dev->isp_sdev;
> > > > +    struct v4l2_subdev *sd = &isp_sdev->sd;
> > > > +    int ret;
> > > > +
> > > > +    v4l2_subdev_init(sd, &rkisp1_isp_sd_ops);
> > > > +    sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
> > > > +    sd->entity.ops = &rkisp1_isp_sd_media_ops;
> > > > +    snprintf(sd->name, sizeof(sd->name), "rkisp1-isp-subdev");
> > > > +
> > > > +    isp_sdev->pads[RKISP1_ISP_PAD_SINK].flags =
> > > > +        MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_MUST_CONNECT;
> > > > +    isp_sdev->pads[RKISP1_ISP_PAD_SINK_PARAMS].flags = MEDIA_PAD_FL_SINK;
> > > > +    isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_PATH].flags = MEDIA_PAD_FL_SOURCE;
> > > > +    isp_sdev->pads[RKISP1_ISP_PAD_SOURCE_STATS].flags = MEDIA_PAD_FL_SOURCE;
> > > > +    sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> > > > +    ret = media_entity_pads_init(&sd->entity, RKISP1_ISP_PAD_MAX,
> > > > +                     isp_sdev->pads);
> > > > +    if (ret < 0)
> > > > +        return ret;
> > > > +
> > > > +    sd->owner = THIS_MODULE;
> > > > +    v4l2_set_subdevdata(sd, isp_dev);
> > > > +
> > > > +    sd->grp_id = GRP_ID_ISP;
> > > 
> > > I think you can drop this, as well as all the GRP_ID_* macros, they're
> > > not used.
> > > 
> > > > +    ret = v4l2_device_register_subdev(v4l2_dev, sd);
> > > > +    if (ret < 0) {
> > > > +        v4l2_err(sd, "Failed to register isp subdev\n");
> > > > +        goto err_cleanup_media_entity;
> > > > +    }
> > > > +
> > > > +    rkisp1_isp_sd_init_default_fmt(isp_sdev);
> > > > +
> > > > +    return 0;
> > > > +err_cleanup_media_entity:
> > > > +    media_entity_cleanup(&sd->entity);
> > > > +    return ret;
> > > > +}
> > > > +
> > > > +void rkisp1_unregister_isp_subdev(struct rkisp1_device *isp_dev)
> > > > +{
> > > > +    struct v4l2_subdev *sd = &isp_dev->isp_sdev.sd;
> > > > +
> > > > +    v4l2_device_unregister_subdev(sd);
> > > > +    media_entity_cleanup(&sd->entity);
> > > > +}
> > > > +
> > > > +/****************  Interrupter Handler ****************/
> > > 
> > > s/Handler/Handlers/
> > > 
> > > > +
> > > > +void rkisp1_mipi_isr(unsigned int mis, struct rkisp1_device *dev)
> > > 
> > > I would make the mis field an u32 as it contains register values. Should
> > > it also be renamed to status ? Same for rkisp1_isp_isr() below.
> > > 
> > > > +{
> > > > +    struct v4l2_device *v4l2_dev = &dev->v4l2_dev;
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    u32 val;
> > > > +
> > > 
> > > How about moving the CIF_MIPI_MIS read here and removing the mis
> > > argument to the function ? It would be more logical to group all
> > > register access related to interrupts in a single place. Same for the
> > > other interrupt handler functions.
> > > 
> > > > +    writel(~0, base + CIF_MIPI_ICR);
> > > > +
> > > > +    /*
> > > > +     * Disable DPHY errctrl interrupt, because this dphy
> > > > +     * erctrl signal is asserted until the next changes
> > > > +     * of line state. This time is may be too long and cpu
> > > > +     * is hold in this interrupt.
> > > > +     */
> > > > +    if (mis & CIF_MIPI_ERR_CTRL(0x0f)) {
> > > > +        val = readl(base + CIF_MIPI_IMSC);
> > > > +        writel(val & ~CIF_MIPI_ERR_CTRL(0x0f), base + CIF_MIPI_IMSC);
> > > > +        dev->isp_sdev.dphy_errctrl_disabled = true;
> > > > +    }
> > > > +
> > > > +    /*
> > > > +     * Enable DPHY errctrl interrupt again, if mipi have receive
> > > > +     * the whole frame without any error.
> > > > +     */
> > > > +    if (mis == CIF_MIPI_FRAME_END) {
> > > > +        /*
> > > > +         * Enable DPHY errctrl interrupt again, if mipi have receive
> > > > +         * the whole frame without any error.
> > > > +         */
> > > > +        if (dev->isp_sdev.dphy_errctrl_disabled) {
> > > > +            val = readl(base + CIF_MIPI_IMSC);
> > > > +            val |= CIF_MIPI_ERR_CTRL(0x0f);
> > > > +            writel(val, base + CIF_MIPI_IMSC);
> > > > +            dev->isp_sdev.dphy_errctrl_disabled = false;
> > > > +        }
> > > > +    } else {
> > > > +        v4l2_warn(v4l2_dev, "MIPI mis error: 0x%08x\n", mis);
> > > > +    }
> > > > +}
> > > > +
> > > > +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> > > > +{
> > > > +    void __iomem *base = dev->base_addr;
> > > > +    unsigned int isp_mis_tmp = 0;
> > > 
> > > _tmp are never good names :-S
> > > 
> > > > +    unsigned int isp_err = 0;
> > > 
> > > Neither of these variable need to be initialised to 0.
> > > 
> > > > +
> > > > +    /* start edge of v_sync */
> > > > +    if (isp_mis & CIF_ISP_V_START) {
> > > > +        rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> > > 
> > > This will increment the frame sequence number. What if the interrupt is
> > > slightly delayed and the next frame starts before we get a change to
> > > copy the sequence number to the buffers (before they will complete
> > > below) ?
> > 
> > Do you mean that we get two sequental v-start signals and then the next
> > frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> > How can this be solved? I wonder if any v-start signal has a later signal
> > that correspond to the same frame so that we can follow it?
> > 
> > Maybe we should have one counter that is incremented on v-start signal,
> > and another counter that is incremented uppon some other signal?
> > 
> > > 
> > > > +
> > > > +        writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> > > 
> > > Do you need to clear all interrupt bits individually, can't you write
> > > isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> > > all in one go ?
> > > 
> > > > +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > +        if (isp_mis_tmp & CIF_ISP_V_START)
> > > > +            v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> > > > +                 isp_mis_tmp);
> > > 
> > > This require some explanation. It looks like a naive way to protect
> > > against something, but I think it could trigger under normal
> > > circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> > > Same for the similar constructs below.
> > > 
> > > > +    }
> > > > +
> > > > +    if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> > > > +        /* Clear pic_size_error */
> > > > +        writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> > > > +        isp_err = readl(base + CIF_ISP_ERR);
> > > > +        v4l2_err(&dev->v4l2_dev,
> > > > +             "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> > > 
> > > What does this mean ?
> > > 
> > > > +        writel(isp_err, base + CIF_ISP_ERR_CLR);
> > > > +    } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> > > 
> > > Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> > > 
> > > > +        /* Clear data_loss */
> > > > +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > +        v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> > > > +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > +    }
> > > > +
> > > > +    /* sampled input frame is complete */
> > > > +    if (isp_mis & CIF_ISP_FRAME_IN) {
> > > > +        writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> > > > +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > +        if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> > > > +            v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> > > > +                 isp_mis_tmp);
> > > > +    }
> > > > +
> > > > +    /* frame was completely put out */
> > > 
> > > "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> > > ? The two comments could do with a bit of brush up, and I think the
> > > ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> > > action.
> > 
> > Those two oneline comments are just copy-paste from the datasheet.
> > 
> > ""
> > 5 MIS_FRAME_IN sampled input frame is complete
> > 1 MIS_FRAME frame was completely put out
> > ""
> > 
> > Unfrotunately, the datasheet does not add any further explanation about those signals.
> > 
> > 
> > > 
> > > > +    if (isp_mis & CIF_ISP_FRAME) {
> > > > +        u32 isp_ris = 0;
> > > 
> > > No need to initialise this to 0.
> > > 
> > > > +        /* Clear Frame In (ISP) */
> > > > +        writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> > > > +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > +        if (isp_mis_tmp & CIF_ISP_FRAME)
> > > > +            v4l2_err(&dev->v4l2_dev,
> > > > +                 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> > > > +
> > > > +        isp_ris = readl(base + CIF_ISP_RIS);
> > > > +        if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> > > > +                   CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> > > > +            rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> > > 
> > > Is there a guarantee that the statistics will be fully written out
> > > before the video frame itself ? And doesn't this test if any of the
> > > statistics is complete, not all of them ? I think the logic is wrong, it
> > 
> > The datasheet does not add any explanation of what is expected to come first.
> > Should we wait until all statistics measurements are done? In the struct
> > sent to userspace there is a bitmaks for which of the statistics are read.
> > I think that if only part of the statistics are ready, we can already send the once
> > that are ready to userspace.
> > 
> > > seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> > > own. It's hard to tell for sure without extra information though (for
> > > instance why are the stats-related bits read from CIF_ISP_RIS, when
> > > they seem to be documented as valid in CIF_ISP_ISR), but this should be
> > > validated, and most probably fixed. Care should be taken to keep
> > > synchronisation of sequence number between the different queues.
> > 
> > I see that the capture buffers are done before incrementing the frame_sequence with
> > the following explanation:
> > 
> >      /*
> >           * Call rkisp1_capture_isr() first to handle the frame that
> >           * potentially completed using the current frame_sequence number before
> >           * it is potentially incremented by rkisp1_isp_isr() in the vertical
> >           * sync.
> >           */
> > 
> > I think reading the stats/params should also be done before calling rkisp1_capture_isr
> > for the same reason. (so to match the correct frame_sequence)
> > 
> > Thanks,
> > Dafna
> 
> 
> I am not sure what should be the meaning of the sequence field for parameters buffers.
> Currently the rkisp1-params entity returns the first buffer queued right away with the frame_sequence that it reads.
> If streaming did not yet started then the first buffer is used as the initial configuration that overrides the driver's default.
> In that case the first params buffer is not associated with any captured frame and has a sequence value '-1'
> Is this a valid behavior?

-1 certainly doesn't sound like a valid sequence number. I think it
should correspond to the frame that the parameters were (or are about to
be) applied to.

Best regards,
Tomasz

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-22 15:24       ` Tomasz Figa
@ 2020-07-22 16:30         ` Laurent Pinchart
  2020-07-22 17:12           ` Tomasz Figa
  2020-08-05 21:10         ` Dafna Hirschfeld
  1 sibling, 1 reply; 76+ messages in thread
From: Laurent Pinchart @ 2020-07-22 16:30 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, eddie.cai.linux, kernel, Dafna Hirschfeld, heiko,
	jacob2.chen, jeffy.chen, zyc, linux-kernel, Allon Huang,
	linux-rockchip, Helen Koike, Jacob Chen, hans.verkuil,
	sakari.ailus, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media

Hi Dafna and Tomasz,

On Wed, Jul 22, 2020 at 03:24:59PM +0000, Tomasz Figa wrote:
> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
> > On 16.08.19 02:13, Laurent Pinchart wrote:
> > > On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>
> [snip]
>
> > > > +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> > > > +{
> > > > +	struct v4l2_event event = {
> > > > +		.type = V4L2_EVENT_FRAME_SYNC,
> > > > +		.u.frame_sync.frame_sequence =
> > > > +			atomic_inc_return(&isp->frm_sync_seq) - 1,
> > > 
> > > I would move the increment to the caller, hiding it in this function is
> > > error-prone (and if you look at the caller I'm pointing out one possible
> > > error :-)).
> > > 
> > > In general usage of frm_sync_seq through the driver seems to be very
> > > race-prone. It's read in various IRQ handling functions, all coming from
> > > the same IRQ, so that part is fine (and wouldn't require an atomic
> > > variable), but when read from the buffer queue handlers I really get a
> > > red light flashing in my head. I'll try to investigate more when
> > > reviewing the next patches.
> > 
> > I see that the only place were 'frame_sequence' is read outside of the irq
> > handlers is in the capture in 'rkisp1_vb2_buf_queue':
> > 
> > 	/*
> >          * If there's no next buffer assigned, queue this buffer directly
> >          * as the next buffer, and update the memory interface.
> >          */
> >         if (cap->is_streaming && !cap->buf.next &&
> >             atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
> >                 cap->buf.next = ispbuf;
> >                 rkisp1_set_next_buf(cap);
> >         } else {
> >                 list_add_tail(&ispbuf->queue, &cap->buf.queue);
> >         }
> >
> > This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> > I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> > from the above condition so that the next buffer is updated in case it is null not just before the first
> > v-start signal.
> 
> We don't have this special case in the Chrome OS code.
> 
> I suppose it would make it possible to resume the capture 1 frame
> earlier after a queue underrun, as otherwise the new buffer would be
> only programmed after the next frame start interrupt and used for the
> next-next frame.  However, it's racy, because programming of the buffer
> addresses is not atomic and could end up with the hardware using few
> plane addresses from the new buffer and few from the dummy buffer.
> 
> Given that and also the fact that a queue underrun is a very special
> case, where the system was already having problems catching up, I'd just
> remove this special case.
> 
> [snip]
>
> > > > +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> > > > +{
> > > > +	void __iomem *base = dev->base_addr;
> > > > +	unsigned int isp_mis_tmp = 0;
> > > 
> > > _tmp are never good names :-S
> > > 
> > > > +	unsigned int isp_err = 0;
> > > 
> > > Neither of these variable need to be initialised to 0.
> > > 
> > > > +
> > > > +	/* start edge of v_sync */
> > > > +	if (isp_mis & CIF_ISP_V_START) {
> > > > +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> > > 
> > > This will increment the frame sequence number. What if the interrupt is
> > > slightly delayed and the next frame starts before we get a change to
> > > copy the sequence number to the buffers (before they will complete
> > > below) ?
> > 
> > Do you mean that we get two sequental v-start signals and then the next
> > frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> > How can this be solved? I wonder if any v-start signal has a later signal
> > that correspond to the same frame so that we can follow it?
> > 
> > Maybe we should have one counter that is incremented on v-start signal,
> > and another counter that is incremented uppon some other signal?
> 
> We're talking about a hard IRQ. I can't imagine the interrupt handler
> being delayed for a time close to a full frame interval (~16ms for 60
> fps) to trigger such scenario.

I've been burnt too many times by making such statements and then seeing
a wifi driver disablign interrupts for 40ms... :-S We can only perform
as well as the system and the hardware allow us to, I understand we
can't solve all issues related to long interrupt delays as that would
require more hardware support. I'm not sure what an appropriate best
effort level is though.

> > > > +
> > > > +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> > > 
> > > Do you need to clear all interrupt bits individually, can't you write
> > > isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> > > all in one go ?
> > > 
> > > > +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > +		if (isp_mis_tmp & CIF_ISP_V_START)
> > > > +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> > > > +				 isp_mis_tmp);
> > > 
> > > This require some explanation. It looks like a naive way to protect
> > > against something, but I think it could trigger under normal
> > > circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> > > Same for the similar constructs below.
> > > 
> > > > +	}
> > > > +
> > > > +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> > > > +		/* Clear pic_size_error */
> > > > +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> > > > +		isp_err = readl(base + CIF_ISP_ERR);
> > > > +		v4l2_err(&dev->v4l2_dev,
> > > > +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> > > 
> > > What does this mean ?
> > > 
> > > > +		writel(isp_err, base + CIF_ISP_ERR_CLR);
> > > > +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> > > 
> > > Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> > > 
> > > > +		/* Clear data_loss */
> > > > +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> > > > +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > +	}
> > > > +
> > > > +	/* sampled input frame is complete */
> > > > +	if (isp_mis & CIF_ISP_FRAME_IN) {
> > > > +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> > > > +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> > > > +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> > > > +				 isp_mis_tmp);
> > > > +	}
> > > > +
> > > > +	/* frame was completely put out */
> > > 
> > > "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> > > ? The two comments could do with a bit of brush up, and I think the
> > > ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> > > action.
> > 
> > Those two oneline comments are just copy-paste from the datasheet.
> > 
> > ""
> > 5 MIS_FRAME_IN sampled input frame is complete
> > 1 MIS_FRAME frame was completely put out
> > ""
> > 
> > Unfrotunately, the datasheet does not add any further explanation about those signals.
> 
> My loose recollection is that the former is signaled when then frame
> is fully input to the ISP and the latter when the ISP completes
> outputting the frame to the next block in the pipeline, but someone
> would need to verify this, for example by printing timestamps for all
> the various interrupts.
> 
> > > 
> > > > +	if (isp_mis & CIF_ISP_FRAME) {
> > > > +		u32 isp_ris = 0;
> > > 
> > > No need to initialise this to 0.
> > > 
> > > > +		/* Clear Frame In (ISP) */
> > > > +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> > > > +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > +		if (isp_mis_tmp & CIF_ISP_FRAME)
> > > > +			v4l2_err(&dev->v4l2_dev,
> > > > +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> > > > +
> > > > +		isp_ris = readl(base + CIF_ISP_RIS);
> > > > +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> > > > +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> > > > +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> > > 
> > > Is there a guarantee that the statistics will be fully written out
> > > before the video frame itself ? And doesn't this test if any of the
> > > statistics is complete, not all of them ? I think the logic is wrong, it
> > 
> > The datasheet does not add any explanation of what is expected to come first.
> > Should we wait until all statistics measurements are done? In the struct
> > sent to userspace there is a bitmaks for which of the statistics are read.
> > I think that if only part of the statistics are ready, we can already send the once
> > that are ready to userspace.
> 
> If we look further into the code, rkisp1_stats_isr() checks the
> interrupt status mask passed to it and reads out only the parameters
> with indicated completion. The statistics metadata buffer format
> includes a bit mask which tells the userspace which measurements are
> available.
> 
> However, I think I've spotted a bug there. At the beginning of
> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
> regardless of the mask used later to decide which readouts need to be
> done. This could mean that with an unfortunate timing, some measurements
> would be lost. So at least the code should be fixed to only clear the
> interrupts bits really handled.
> 
> As for whether to send separate buffers for each measurement, I guess
> it's not a bad thing to let the userspace access the ones available
> earlier. Now I only don't recall why we decided to put all the
> measurements into one metadata structure, rather than splitting the 4
> into their own structures and buffer queues...
> 
> > > seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> > > own. It's hard to tell for sure without extra information though (for
> > > instance why are the stats-related bits read from CIF_ISP_RIS, when
> > > they seem to be documented as valid in CIF_ISP_ISR), but this should be
> > > validated, and most probably fixed. Care should be taken to keep
> > > synchronisation of sequence number between the different queues.
> > 
> > I see that the capture buffers are done before incrementing the frame_sequence with
> > the following explanation:
> > 
> > 	/*
> >          * Call rkisp1_capture_isr() first to handle the frame that
> >          * potentially completed using the current frame_sequence number before
> >          * it is potentially incremented by rkisp1_isp_isr() in the vertical
> >          * sync.
> >          */
> > 
> > I think reading the stats/params should also be done before calling rkisp1_capture_isr
> > for the same reason. (so to match the correct frame_sequence)
> 
> My recollection of the sequence of interrupts in this hardware is like
> this:
> 
> CIF_ISP_V_START (frame 0)
>   CIF_ISP_FRAME_IN (frame 0)
>     CIF_ISP_FRAME (frame 0)
>       CIF_ISP_AWB_DONE
>       CIF_ISP_AFM_FIN
>       CIF_ISP_EXP_END
>       CIF_ISP_HIST_MEASURE_RDY
>       CIF_MI_FRAME*
>       CIF_ISP_V_START (frame 1)
>         CIF_ISP_FRAME_IN (frame 1)
>           CIF_ISP_FRAME (frame 1)
>             ...
> 
> where the interrupts at the same indentation level can happen
> independently of each other. Again, someone would have to verify this.

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-22 16:30         ` Laurent Pinchart
@ 2020-07-22 17:12           ` Tomasz Figa
  2020-07-22 17:50             ` Laurent Pinchart
  0 siblings, 1 reply; 76+ messages in thread
From: Tomasz Figa @ 2020-07-22 17:12 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: linux-devicetree, Eddie Cai, kernel, Dafna Hirschfeld,
	Heiko Stübner, Chen Jacob, Jeffy, 钟以崇,
	Linux Kernel Mailing List, Allon Huang,
	open list:ARM/Rockchip SoC...,
	Helen Koike, Jacob Chen, Hans Verkuil, Sakari Ailus,
	Shunqian Zheng, Mauro Carvalho Chehab, Ezequiel Garcia,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

On Wed, Jul 22, 2020 at 6:30 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dafna and Tomasz,
>
> On Wed, Jul 22, 2020 at 03:24:59PM +0000, Tomasz Figa wrote:
> > On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
> > > On 16.08.19 02:13, Laurent Pinchart wrote:
> > > > On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> >
> > [snip]
> >
> > > > > +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> > > > > +{
> > > > > +       struct v4l2_event event = {
> > > > > +               .type = V4L2_EVENT_FRAME_SYNC,
> > > > > +               .u.frame_sync.frame_sequence =
> > > > > +                       atomic_inc_return(&isp->frm_sync_seq) - 1,
> > > >
> > > > I would move the increment to the caller, hiding it in this function is
> > > > error-prone (and if you look at the caller I'm pointing out one possible
> > > > error :-)).
> > > >
> > > > In general usage of frm_sync_seq through the driver seems to be very
> > > > race-prone. It's read in various IRQ handling functions, all coming from
> > > > the same IRQ, so that part is fine (and wouldn't require an atomic
> > > > variable), but when read from the buffer queue handlers I really get a
> > > > red light flashing in my head. I'll try to investigate more when
> > > > reviewing the next patches.
> > >
> > > I see that the only place were 'frame_sequence' is read outside of the irq
> > > handlers is in the capture in 'rkisp1_vb2_buf_queue':
> > >
> > >     /*
> > >          * If there's no next buffer assigned, queue this buffer directly
> > >          * as the next buffer, and update the memory interface.
> > >          */
> > >         if (cap->is_streaming && !cap->buf.next &&
> > >             atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
> > >                 cap->buf.next = ispbuf;
> > >                 rkisp1_set_next_buf(cap);
> > >         } else {
> > >                 list_add_tail(&ispbuf->queue, &cap->buf.queue);
> > >         }
> > >
> > > This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> > > I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> > > from the above condition so that the next buffer is updated in case it is null not just before the first
> > > v-start signal.
> >
> > We don't have this special case in the Chrome OS code.
> >
> > I suppose it would make it possible to resume the capture 1 frame
> > earlier after a queue underrun, as otherwise the new buffer would be
> > only programmed after the next frame start interrupt and used for the
> > next-next frame.  However, it's racy, because programming of the buffer
> > addresses is not atomic and could end up with the hardware using few
> > plane addresses from the new buffer and few from the dummy buffer.
> >
> > Given that and also the fact that a queue underrun is a very special
> > case, where the system was already having problems catching up, I'd just
> > remove this special case.
> >
> > [snip]
> >
> > > > > +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> > > > > +{
> > > > > +       void __iomem *base = dev->base_addr;
> > > > > +       unsigned int isp_mis_tmp = 0;
> > > >
> > > > _tmp are never good names :-S
> > > >
> > > > > +       unsigned int isp_err = 0;
> > > >
> > > > Neither of these variable need to be initialised to 0.
> > > >
> > > > > +
> > > > > +       /* start edge of v_sync */
> > > > > +       if (isp_mis & CIF_ISP_V_START) {
> > > > > +               rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> > > >
> > > > This will increment the frame sequence number. What if the interrupt is
> > > > slightly delayed and the next frame starts before we get a change to
> > > > copy the sequence number to the buffers (before they will complete
> > > > below) ?
> > >
> > > Do you mean that we get two sequental v-start signals and then the next
> > > frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> > > How can this be solved? I wonder if any v-start signal has a later signal
> > > that correspond to the same frame so that we can follow it?
> > >
> > > Maybe we should have one counter that is incremented on v-start signal,
> > > and another counter that is incremented uppon some other signal?
> >
> > We're talking about a hard IRQ. I can't imagine the interrupt handler
> > being delayed for a time close to a full frame interval (~16ms for 60
> > fps) to trigger such scenario.
>
> I've been burnt too many times by making such statements and then seeing
> a wifi driver disablign interrupts for 40ms... :-S We can only perform
> as well as the system and the hardware allow us to, I understand we
> can't solve all issues related to long interrupt delays as that would
> require more hardware support. I'm not sure what an appropriate best
> effort level is though.
>

In that case most of the driver would just stop working, because this
hardware requires reprogramming every frame. I think it's a problem of
that wifi driver and it shouldn't be worked around in other drivers.

Of course we still have to ensure that nothing catastrophic happens,
like the DMA writing to random addresses or the driver crashing. We
could also attempt to detect the case and print a warning.

> > > > > +
> > > > > +               writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> > > >
> > > > Do you need to clear all interrupt bits individually, can't you write
> > > > isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> > > > all in one go ?
> > > >
> > > > > +               isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > > +               if (isp_mis_tmp & CIF_ISP_V_START)
> > > > > +                       v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> > > > > +                                isp_mis_tmp);
> > > >
> > > > This require some explanation. It looks like a naive way to protect
> > > > against something, but I think it could trigger under normal
> > > > circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> > > > Same for the similar constructs below.
> > > >
> > > > > +       }
> > > > > +
> > > > > +       if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> > > > > +               /* Clear pic_size_error */
> > > > > +               writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> > > > > +               isp_err = readl(base + CIF_ISP_ERR);
> > > > > +               v4l2_err(&dev->v4l2_dev,
> > > > > +                        "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> > > >
> > > > What does this mean ?
> > > >
> > > > > +               writel(isp_err, base + CIF_ISP_ERR_CLR);
> > > > > +       } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> > > >
> > > > Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> > > >
> > > > > +               /* Clear data_loss */
> > > > > +               writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > > +               v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> > > > > +               writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > > +       }
> > > > > +
> > > > > +       /* sampled input frame is complete */
> > > > > +       if (isp_mis & CIF_ISP_FRAME_IN) {
> > > > > +               writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> > > > > +               isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > > +               if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> > > > > +                       v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> > > > > +                                isp_mis_tmp);
> > > > > +       }
> > > > > +
> > > > > +       /* frame was completely put out */
> > > >
> > > > "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> > > > ? The two comments could do with a bit of brush up, and I think the
> > > > ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> > > > action.
> > >
> > > Those two oneline comments are just copy-paste from the datasheet.
> > >
> > > ""
> > > 5 MIS_FRAME_IN sampled input frame is complete
> > > 1 MIS_FRAME frame was completely put out
> > > ""
> > >
> > > Unfrotunately, the datasheet does not add any further explanation about those signals.
> >
> > My loose recollection is that the former is signaled when then frame
> > is fully input to the ISP and the latter when the ISP completes
> > outputting the frame to the next block in the pipeline, but someone
> > would need to verify this, for example by printing timestamps for all
> > the various interrupts.
> >
> > > >
> > > > > +       if (isp_mis & CIF_ISP_FRAME) {
> > > > > +               u32 isp_ris = 0;
> > > >
> > > > No need to initialise this to 0.
> > > >
> > > > > +               /* Clear Frame In (ISP) */
> > > > > +               writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> > > > > +               isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > > +               if (isp_mis_tmp & CIF_ISP_FRAME)
> > > > > +                       v4l2_err(&dev->v4l2_dev,
> > > > > +                                "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> > > > > +
> > > > > +               isp_ris = readl(base + CIF_ISP_RIS);
> > > > > +               if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> > > > > +                              CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> > > > > +                       rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> > > >
> > > > Is there a guarantee that the statistics will be fully written out
> > > > before the video frame itself ? And doesn't this test if any of the
> > > > statistics is complete, not all of them ? I think the logic is wrong, it
> > >
> > > The datasheet does not add any explanation of what is expected to come first.
> > > Should we wait until all statistics measurements are done? In the struct
> > > sent to userspace there is a bitmaks for which of the statistics are read.
> > > I think that if only part of the statistics are ready, we can already send the once
> > > that are ready to userspace.
> >
> > If we look further into the code, rkisp1_stats_isr() checks the
> > interrupt status mask passed to it and reads out only the parameters
> > with indicated completion. The statistics metadata buffer format
> > includes a bit mask which tells the userspace which measurements are
> > available.
> >
> > However, I think I've spotted a bug there. At the beginning of
> > rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
> > regardless of the mask used later to decide which readouts need to be
> > done. This could mean that with an unfortunate timing, some measurements
> > would be lost. So at least the code should be fixed to only clear the
> > interrupts bits really handled.
> >
> > As for whether to send separate buffers for each measurement, I guess
> > it's not a bad thing to let the userspace access the ones available
> > earlier. Now I only don't recall why we decided to put all the
> > measurements into one metadata structure, rather than splitting the 4
> > into their own structures and buffer queues...
> >
> > > > seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> > > > own. It's hard to tell for sure without extra information though (for
> > > > instance why are the stats-related bits read from CIF_ISP_RIS, when
> > > > they seem to be documented as valid in CIF_ISP_ISR), but this should be
> > > > validated, and most probably fixed. Care should be taken to keep
> > > > synchronisation of sequence number between the different queues.
> > >
> > > I see that the capture buffers are done before incrementing the frame_sequence with
> > > the following explanation:
> > >
> > >     /*
> > >          * Call rkisp1_capture_isr() first to handle the frame that
> > >          * potentially completed using the current frame_sequence number before
> > >          * it is potentially incremented by rkisp1_isp_isr() in the vertical
> > >          * sync.
> > >          */
> > >
> > > I think reading the stats/params should also be done before calling rkisp1_capture_isr
> > > for the same reason. (so to match the correct frame_sequence)
> >
> > My recollection of the sequence of interrupts in this hardware is like
> > this:
> >
> > CIF_ISP_V_START (frame 0)
> >   CIF_ISP_FRAME_IN (frame 0)
> >     CIF_ISP_FRAME (frame 0)
> >       CIF_ISP_AWB_DONE
> >       CIF_ISP_AFM_FIN
> >       CIF_ISP_EXP_END
> >       CIF_ISP_HIST_MEASURE_RDY
> >       CIF_MI_FRAME*
> >       CIF_ISP_V_START (frame 1)
> >         CIF_ISP_FRAME_IN (frame 1)
> >           CIF_ISP_FRAME (frame 1)
> >             ...
> >
> > where the interrupts at the same indentation level can happen
> > independently of each other. Again, someone would have to verify this.
>
> --
> Regards,
>
> Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-22 17:12           ` Tomasz Figa
@ 2020-07-22 17:50             ` Laurent Pinchart
  0 siblings, 0 replies; 76+ messages in thread
From: Laurent Pinchart @ 2020-07-22 17:50 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: linux-devicetree, Eddie Cai, kernel, Dafna Hirschfeld,
	Heiko Stübner, Chen Jacob, Jeffy, 钟以崇,
	Linux Kernel Mailing List, Allon Huang,
	open list:ARM/Rockchip SoC...,
	Helen Koike, Jacob Chen, Hans Verkuil, Sakari Ailus,
	Shunqian Zheng, Mauro Carvalho Chehab, Ezequiel Garcia,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

Hi Tomasz,

On Wed, Jul 22, 2020 at 07:12:12PM +0200, Tomasz Figa wrote:
> On Wed, Jul 22, 2020 at 6:30 PM Laurent Pinchart wrote:
> > On Wed, Jul 22, 2020 at 03:24:59PM +0000, Tomasz Figa wrote:
> > > On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
> > > > On 16.08.19 02:13, Laurent Pinchart wrote:
> > > > > On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> > >
> > > [snip]
> > >
> > > > > > +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> > > > > > +{
> > > > > > +       struct v4l2_event event = {
> > > > > > +               .type = V4L2_EVENT_FRAME_SYNC,
> > > > > > +               .u.frame_sync.frame_sequence =
> > > > > > +                       atomic_inc_return(&isp->frm_sync_seq) - 1,
> > > > >
> > > > > I would move the increment to the caller, hiding it in this function is
> > > > > error-prone (and if you look at the caller I'm pointing out one possible
> > > > > error :-)).
> > > > >
> > > > > In general usage of frm_sync_seq through the driver seems to be very
> > > > > race-prone. It's read in various IRQ handling functions, all coming from
> > > > > the same IRQ, so that part is fine (and wouldn't require an atomic
> > > > > variable), but when read from the buffer queue handlers I really get a
> > > > > red light flashing in my head. I'll try to investigate more when
> > > > > reviewing the next patches.
> > > >
> > > > I see that the only place were 'frame_sequence' is read outside of the irq
> > > > handlers is in the capture in 'rkisp1_vb2_buf_queue':
> > > >
> > > >     /*
> > > >          * If there's no next buffer assigned, queue this buffer directly
> > > >          * as the next buffer, and update the memory interface.
> > > >          */
> > > >         if (cap->is_streaming && !cap->buf.next &&
> > > >             atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
> > > >                 cap->buf.next = ispbuf;
> > > >                 rkisp1_set_next_buf(cap);
> > > >         } else {
> > > >                 list_add_tail(&ispbuf->queue, &cap->buf.queue);
> > > >         }
> > > >
> > > > This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> > > > I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> > > > from the above condition so that the next buffer is updated in case it is null not just before the first
> > > > v-start signal.
> > >
> > > We don't have this special case in the Chrome OS code.
> > >
> > > I suppose it would make it possible to resume the capture 1 frame
> > > earlier after a queue underrun, as otherwise the new buffer would be
> > > only programmed after the next frame start interrupt and used for the
> > > next-next frame.  However, it's racy, because programming of the buffer
> > > addresses is not atomic and could end up with the hardware using few
> > > plane addresses from the new buffer and few from the dummy buffer.
> > >
> > > Given that and also the fact that a queue underrun is a very special
> > > case, where the system was already having problems catching up, I'd just
> > > remove this special case.
> > >
> > > [snip]
> > >
> > > > > > +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> > > > > > +{
> > > > > > +       void __iomem *base = dev->base_addr;
> > > > > > +       unsigned int isp_mis_tmp = 0;
> > > > >
> > > > > _tmp are never good names :-S
> > > > >
> > > > > > +       unsigned int isp_err = 0;
> > > > >
> > > > > Neither of these variable need to be initialised to 0.
> > > > >
> > > > > > +
> > > > > > +       /* start edge of v_sync */
> > > > > > +       if (isp_mis & CIF_ISP_V_START) {
> > > > > > +               rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> > > > >
> > > > > This will increment the frame sequence number. What if the interrupt is
> > > > > slightly delayed and the next frame starts before we get a change to
> > > > > copy the sequence number to the buffers (before they will complete
> > > > > below) ?
> > > >
> > > > Do you mean that we get two sequental v-start signals and then the next
> > > > frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> > > > How can this be solved? I wonder if any v-start signal has a later signal
> > > > that correspond to the same frame so that we can follow it?
> > > >
> > > > Maybe we should have one counter that is incremented on v-start signal,
> > > > and another counter that is incremented uppon some other signal?
> > >
> > > We're talking about a hard IRQ. I can't imagine the interrupt handler
> > > being delayed for a time close to a full frame interval (~16ms for 60
> > > fps) to trigger such scenario.
> >
> > I've been burnt too many times by making such statements and then seeing
> > a wifi driver disablign interrupts for 40ms... :-S We can only perform
> > as well as the system and the hardware allow us to, I understand we
> > can't solve all issues related to long interrupt delays as that would
> > require more hardware support. I'm not sure what an appropriate best
> > effort level is though.
> 
> In that case most of the driver would just stop working, because this
> hardware requires reprogramming every frame. I think it's a problem of
> that wifi driver and it shouldn't be worked around in other drivers.
>
> Of course we still have to ensure that nothing catastrophic happens,
> like the DMA writing to random addresses or the driver crashing. We
> could also attempt to detect the case and print a warning.

Absolutely, I strongly believe that at the kernel level drivers need to
cooperate and be good citizens overall, otherwise we can't guarantee
proper operation. Avoiding catastrophes, and if possible detecting the
issues, should be good enough. In general I try to get drivers to
support userspace with recovery in such cases, depending on how much can
be reasonably achieved.

I've worked with a system that would disable interrupts (actually
preempting the CPU from an execution level higher than the kernel's) for
500ms when a crypto operation was offloaded to the hardware crypto
engine. We ended up performing the crypto operations in software :-)

> > > > > > +
> > > > > > +               writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> > > > >
> > > > > Do you need to clear all interrupt bits individually, can't you write
> > > > > isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> > > > > all in one go ?
> > > > >
> > > > > > +               isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > > > +               if (isp_mis_tmp & CIF_ISP_V_START)
> > > > > > +                       v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> > > > > > +                                isp_mis_tmp);
> > > > >
> > > > > This require some explanation. It looks like a naive way to protect
> > > > > against something, but I think it could trigger under normal
> > > > > circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> > > > > Same for the similar constructs below.
> > > > >
> > > > > > +       }
> > > > > > +
> > > > > > +       if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> > > > > > +               /* Clear pic_size_error */
> > > > > > +               writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> > > > > > +               isp_err = readl(base + CIF_ISP_ERR);
> > > > > > +               v4l2_err(&dev->v4l2_dev,
> > > > > > +                        "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> > > > >
> > > > > What does this mean ?
> > > > >
> > > > > > +               writel(isp_err, base + CIF_ISP_ERR_CLR);
> > > > > > +       } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> > > > >
> > > > > Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> > > > >
> > > > > > +               /* Clear data_loss */
> > > > > > +               writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > > > +               v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> > > > > > +               writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> > > > > > +       }
> > > > > > +
> > > > > > +       /* sampled input frame is complete */
> > > > > > +       if (isp_mis & CIF_ISP_FRAME_IN) {
> > > > > > +               writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> > > > > > +               isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > > > +               if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> > > > > > +                       v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> > > > > > +                                isp_mis_tmp);
> > > > > > +       }
> > > > > > +
> > > > > > +       /* frame was completely put out */
> > > > >
> > > > > "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> > > > > ? The two comments could do with a bit of brush up, and I think the
> > > > > ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> > > > > action.
> > > >
> > > > Those two oneline comments are just copy-paste from the datasheet.
> > > >
> > > > ""
> > > > 5 MIS_FRAME_IN sampled input frame is complete
> > > > 1 MIS_FRAME frame was completely put out
> > > > ""
> > > >
> > > > Unfrotunately, the datasheet does not add any further explanation about those signals.
> > >
> > > My loose recollection is that the former is signaled when then frame
> > > is fully input to the ISP and the latter when the ISP completes
> > > outputting the frame to the next block in the pipeline, but someone
> > > would need to verify this, for example by printing timestamps for all
> > > the various interrupts.
> > >
> > > > >
> > > > > > +       if (isp_mis & CIF_ISP_FRAME) {
> > > > > > +               u32 isp_ris = 0;
> > > > >
> > > > > No need to initialise this to 0.
> > > > >
> > > > > > +               /* Clear Frame In (ISP) */
> > > > > > +               writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> > > > > > +               isp_mis_tmp = readl(base + CIF_ISP_MIS);
> > > > > > +               if (isp_mis_tmp & CIF_ISP_FRAME)
> > > > > > +                       v4l2_err(&dev->v4l2_dev,
> > > > > > +                                "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> > > > > > +
> > > > > > +               isp_ris = readl(base + CIF_ISP_RIS);
> > > > > > +               if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> > > > > > +                              CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> > > > > > +                       rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> > > > >
> > > > > Is there a guarantee that the statistics will be fully written out
> > > > > before the video frame itself ? And doesn't this test if any of the
> > > > > statistics is complete, not all of them ? I think the logic is wrong, it
> > > >
> > > > The datasheet does not add any explanation of what is expected to come first.
> > > > Should we wait until all statistics measurements are done? In the struct
> > > > sent to userspace there is a bitmaks for which of the statistics are read.
> > > > I think that if only part of the statistics are ready, we can already send the once
> > > > that are ready to userspace.
> > >
> > > If we look further into the code, rkisp1_stats_isr() checks the
> > > interrupt status mask passed to it and reads out only the parameters
> > > with indicated completion. The statistics metadata buffer format
> > > includes a bit mask which tells the userspace which measurements are
> > > available.
> > >
> > > However, I think I've spotted a bug there. At the beginning of
> > > rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
> > > regardless of the mask used later to decide which readouts need to be
> > > done. This could mean that with an unfortunate timing, some measurements
> > > would be lost. So at least the code should be fixed to only clear the
> > > interrupts bits really handled.
> > >
> > > As for whether to send separate buffers for each measurement, I guess
> > > it's not a bad thing to let the userspace access the ones available
> > > earlier. Now I only don't recall why we decided to put all the
> > > measurements into one metadata structure, rather than splitting the 4
> > > into their own structures and buffer queues...
> > >
> > > > > seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> > > > > own. It's hard to tell for sure without extra information though (for
> > > > > instance why are the stats-related bits read from CIF_ISP_RIS, when
> > > > > they seem to be documented as valid in CIF_ISP_ISR), but this should be
> > > > > validated, and most probably fixed. Care should be taken to keep
> > > > > synchronisation of sequence number between the different queues.
> > > >
> > > > I see that the capture buffers are done before incrementing the frame_sequence with
> > > > the following explanation:
> > > >
> > > >     /*
> > > >          * Call rkisp1_capture_isr() first to handle the frame that
> > > >          * potentially completed using the current frame_sequence number before
> > > >          * it is potentially incremented by rkisp1_isp_isr() in the vertical
> > > >          * sync.
> > > >          */
> > > >
> > > > I think reading the stats/params should also be done before calling rkisp1_capture_isr
> > > > for the same reason. (so to match the correct frame_sequence)
> > >
> > > My recollection of the sequence of interrupts in this hardware is like
> > > this:
> > >
> > > CIF_ISP_V_START (frame 0)
> > >   CIF_ISP_FRAME_IN (frame 0)
> > >     CIF_ISP_FRAME (frame 0)
> > >       CIF_ISP_AWB_DONE
> > >       CIF_ISP_AFM_FIN
> > >       CIF_ISP_EXP_END
> > >       CIF_ISP_HIST_MEASURE_RDY
> > >       CIF_MI_FRAME*
> > >       CIF_ISP_V_START (frame 1)
> > >         CIF_ISP_FRAME_IN (frame 1)
> > >           CIF_ISP_FRAME (frame 1)
> > >             ...
> > >
> > > where the interrupts at the same indentation level can happen
> > > independently of each other. Again, someone would have to verify this.

-- 
Regards,

Laurent Pinchart

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-07-22 15:24       ` Tomasz Figa
  2020-07-22 16:30         ` Laurent Pinchart
@ 2020-08-05 21:10         ` Dafna Hirschfeld
  2020-08-06  9:21           ` Dafna Hirschfeld
  2020-08-06 12:08           ` Tomasz Figa
  1 sibling, 2 replies; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-08-05 21:10 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Allon Huang, linux-rockchip,
	Helen Koike, Jacob Chen, hans.verkuil, Laurent Pinchart,
	sakari.ailus, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media

Hi

On 22.07.20 17:24, Tomasz Figa wrote:
> Hi Dafna,
> 
> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
>> Hi Laurent,
>>
>> On 16.08.19 02:13, Laurent Pinchart wrote:
>>> Hello Helen,
>>>
>>> Thank you for the patch.
>>>
>>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> [snip]
>>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>>>> +{
>>>> +	struct v4l2_event event = {
>>>> +		.type = V4L2_EVENT_FRAME_SYNC,
>>>> +		.u.frame_sync.frame_sequence =
>>>> +			atomic_inc_return(&isp->frm_sync_seq) - 1,
>>>
>>> I would move the increment to the caller, hiding it in this function is
>>> error-prone (and if you look at the caller I'm pointing out one possible
>>> error :-)).
>>>
>>> In general usage of frm_sync_seq through the driver seems to be very
>>> race-prone. It's read in various IRQ handling functions, all coming from
>>> the same IRQ, so that part is fine (and wouldn't require an atomic
>>> variable), but when read from the buffer queue handlers I really get a
>>> red light flashing in my head. I'll try to investigate more when
>>> reviewing the next patches.
>>
>> I see that the only place were 'frame_sequence' is read outside of the irq
>> handlers is in the capture in 'rkisp1_vb2_buf_queue':
>>
>> 	/*
>>           * If there's no next buffer assigned, queue this buffer directly
>>           * as the next buffer, and update the memory interface.
>>           */
>>          if (cap->is_streaming && !cap->buf.next &&
>>              atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>>                  cap->buf.next = ispbuf;
>>                  rkisp1_set_next_buf(cap);
>>          } else {
>>                  list_add_tail(&ispbuf->queue, &cap->buf.queue);
>>          }
>> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
>> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
>> from the above condition so that the next buffer is updated in case it is null not just before the first
>> v-start signal.
>>
> 
> We don't have this special case in the Chrome OS code.
> 
> I suppose it would make it possible to resume the capture 1 frame
> earlier after a queue underrun, as otherwise the new buffer would be
> only programmed after the next frame start interrupt and used for the
> next-next frame.  However, it's racy, because programming of the buffer
> addresses is not atomic and could end up with the hardware using few
> plane addresses from the new buffer and few from the dummy buffer.
> 
> Given that and also the fact that a queue underrun is a very special
> case, where the system was already having problems catching up, I'd just
> remove this special case.
> 
> [snip]
>>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>>>> +{
>>>> +	void __iomem *base = dev->base_addr;
>>>> +	unsigned int isp_mis_tmp = 0;
>>>
>>> _tmp are never good names :-S
>>>
>>>> +	unsigned int isp_err = 0;
>>>
>>> Neither of these variable need to be initialised to 0.
>>>
>>>> +
>>>> +	/* start edge of v_sync */
>>>> +	if (isp_mis & CIF_ISP_V_START) {
>>>> +		rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>>>
>>> This will increment the frame sequence number. What if the interrupt is
>>> slightly delayed and the next frame starts before we get a change to
>>> copy the sequence number to the buffers (before they will complete
>>> below) ?
>>
>> Do you mean that we get two sequental v-start signals and then the next
>> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
>> How can this be solved? I wonder if any v-start signal has a later signal
>> that correspond to the same frame so that we can follow it?
>>
>> Maybe we should have one counter that is incremented on v-start signal,
>> and another counter that is incremented uppon some other signal?
>>
> 
> We're talking about a hard IRQ. I can't imagine the interrupt handler
> being delayed for a time close to a full frame interval (~16ms for 60
> fps) to trigger such scenario.
> 
>>>
>>>> +
>>>> +		writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>>>
>>> Do you need to clear all interrupt bits individually, can't you write
>>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
>>> all in one go ?
>>>
>>>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>> +		if (isp_mis_tmp & CIF_ISP_V_START)
>>>> +			v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>>>> +				 isp_mis_tmp);
>>>
>>> This require some explanation. It looks like a naive way to protect
>>> against something, but I think it could trigger under normal
>>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
>>> Same for the similar constructs below.
>>>
>>>> +	}
>>>> +
>>>> +	if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>>>> +		/* Clear pic_size_error */
>>>> +		writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>>>> +		isp_err = readl(base + CIF_ISP_ERR);
>>>> +		v4l2_err(&dev->v4l2_dev,
>>>> +			 "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>>>
>>> What does this mean ?
>>>
>>>> +		writel(isp_err, base + CIF_ISP_ERR_CLR);
>>>> +	} else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>>>
>>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
>>>
>>>> +		/* Clear data_loss */
>>>> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>> +		v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>>>> +		writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>> +	}
>>>> +
>>>> +	/* sampled input frame is complete */
>>>> +	if (isp_mis & CIF_ISP_FRAME_IN) {
>>>> +		writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>>>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>> +		if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>>>> +			v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>>>> +				 isp_mis_tmp);
>>>> +	}
>>>> +
>>>> +	/* frame was completely put out */
>>>
>>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
>>> ? The two comments could do with a bit of brush up, and I think the
>>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
>>> action.
>>
>> Those two oneline comments are just copy-paste from the datasheet.
>>
>> ""
>> 5 MIS_FRAME_IN sampled input frame is complete
>> 1 MIS_FRAME frame was completely put out
>> ""
>>
>> Unfrotunately, the datasheet does not add any further explanation about those signals.
>>
>>
> 
> My loose recollection is that the former is signaled when then frame
> is fully input to the ISP and the latter when the ISP completes
> outputting the frame to the next block in the pipeline, but someone
> would need to verify this, for example by printing timestamps for all
> the various interrupts.
> 
>>>
>>>> +	if (isp_mis & CIF_ISP_FRAME) {
>>>> +		u32 isp_ris = 0;
>>>
>>> No need to initialise this to 0.
>>>
>>>> +		/* Clear Frame In (ISP) */
>>>> +		writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>>>> +		isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>> +		if (isp_mis_tmp & CIF_ISP_FRAME)
>>>> +			v4l2_err(&dev->v4l2_dev,
>>>> +				 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>>>> +
>>>> +		isp_ris = readl(base + CIF_ISP_RIS);
>>>> +		if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>>>> +			       CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>>>> +			rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>>>
>>> Is there a guarantee that the statistics will be fully written out
>>> before the video frame itself ? And doesn't this test if any of the
>>> statistics is complete, not all of them ? I think the logic is wrong, it
>>
>> The datasheet does not add any explanation of what is expected to come first.
>> Should we wait until all statistics measurements are done? In the struct
>> sent to userspace there is a bitmaks for which of the statistics are read.
>> I think that if only part of the statistics are ready, we can already send the once
>> that are ready to userspace.
>>
> 
> If we look further into the code, rkisp1_stats_isr() checks the
> interrupt status mask passed to it and reads out only the parameters
> with indicated completion. The statistics metadata buffer format
> includes a bit mask which tells the userspace which measurements are
> available.
> 
> However, I think I've spotted a bug there. At the beginning of
> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
> regardless of the mask used later to decide which readouts need to be
> done. This could mean that with an unfortunate timing, some measurements
> would be lost. So at least the code should be fixed to only clear the
> interrupts bits really handled.

I'll fix that

> 
> As for whether to send separate buffers for each measurement, I guess
> it's not a bad thing to let the userspace access the ones available
> earlier. Now I only don't recall why we decided to put all the
> measurements into one metadata structure, rather than splitting the 4
> into their own structures and buffer queues...

Is it possible to have several queues to the same video node?

> 
>>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
>>> own. It's hard to tell for sure without extra information though (for
>>> instance why are the stats-related bits read from CIF_ISP_RIS, when
>>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
>>> validated, and most probably fixed. Care should be taken to keep
>>> synchronisation of sequence number between the different queues.
>>
>> I see that the capture buffers are done before incrementing the frame_sequence with
>> the following explanation:
>>
>> 	/*
>>           * Call rkisp1_capture_isr() first to handle the frame that
>>           * potentially completed using the current frame_sequence number before
>>           * it is potentially incremented by rkisp1_isp_isr() in the vertical
>>           * sync.
>>           */
>>
>> I think reading the stats/params should also be done before calling rkisp1_capture_isr
>> for the same reason. (so to match the correct frame_sequence)
> 
> My recollection of the sequence of interrupts in this hardware is like
> this:
> 
> CIF_ISP_V_START (frame 0)
>    CIF_ISP_FRAME_IN (frame 0)
>      CIF_ISP_FRAME (frame 0)
>        CIF_ISP_AWB_DONE
>        CIF_ISP_AFM_FIN
>        CIF_ISP_EXP_END
>        CIF_ISP_HIST_MEASURE_RDY
>        CIF_MI_FRAME*
>        CIF_ISP_V_START (frame 1)
>          CIF_ISP_FRAME_IN (frame 1)
>            CIF_ISP_FRAME (frame 1)
>              ...
> 
> where the interrupts at the same indentation level can happen
> independently of each other. Again, someone would have to verify this.

I wrote this patch to print the interrupts and the time difference between interrupts:
https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5

I got this output: http://ix.io/2tl8,
there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
(FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.

I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
I think that for that we need a code that keep track of the previous interrupt.

Thanks,
Dafna


> 
> Best regards,
> Tomasz
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-08-05 21:10         ` Dafna Hirschfeld
@ 2020-08-06  9:21           ` Dafna Hirschfeld
  2020-08-06 12:22             ` Tomasz Figa
  2020-08-06 12:08           ` Tomasz Figa
  1 sibling, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-08-06  9:21 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: devicetree, eddie.cai.linux, kernel, heiko, jacob2.chen,
	jeffy.chen, zyc, linux-kernel, Allon Huang, linux-rockchip,
	Helen Koike, Jacob Chen, hans.verkuil, Laurent Pinchart,
	sakari.ailus, zhengsq, mchehab, ezequiel, linux-arm-kernel,
	linux-media



Am 05.08.20 um 23:10 schrieb Dafna Hirschfeld:
> Hi
> 
> On 22.07.20 17:24, Tomasz Figa wrote:
>> Hi Dafna,
>>
>> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
>>> Hi Laurent,
>>>
>>> On 16.08.19 02:13, Laurent Pinchart wrote:
>>>> Hello Helen,
>>>>
>>>> Thank you for the patch.
>>>>
>>>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>> [snip]
>>>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>>>>> +{
>>>>> +    struct v4l2_event event = {
>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>>>> +        .u.frame_sync.frame_sequence =
>>>>> +            atomic_inc_return(&isp->frm_sync_seq) - 1,
>>>>
>>>> I would move the increment to the caller, hiding it in this function is
>>>> error-prone (and if you look at the caller I'm pointing out one possible
>>>> error :-)).
>>>>
>>>> In general usage of frm_sync_seq through the driver seems to be very
>>>> race-prone. It's read in various IRQ handling functions, all coming from
>>>> the same IRQ, so that part is fine (and wouldn't require an atomic
>>>> variable), but when read from the buffer queue handlers I really get a
>>>> red light flashing in my head. I'll try to investigate more when
>>>> reviewing the next patches.
>>>
>>> I see that the only place were 'frame_sequence' is read outside of the irq
>>> handlers is in the capture in 'rkisp1_vb2_buf_queue':
>>>
>>>     /*
>>>           * If there's no next buffer assigned, queue this buffer directly
>>>           * as the next buffer, and update the memory interface.
>>>           */
>>>          if (cap->is_streaming && !cap->buf.next &&
>>>              atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>>>                  cap->buf.next = ispbuf;
>>>                  rkisp1_set_next_buf(cap);
>>>          } else {
>>>                  list_add_tail(&ispbuf->queue, &cap->buf.queue);
>>>          }
>>> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
>>> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
>>> from the above condition so that the next buffer is updated in case it is null not just before the first
>>> v-start signal.
>>>
>>
>> We don't have this special case in the Chrome OS code.
>>
>> I suppose it would make it possible to resume the capture 1 frame
>> earlier after a queue underrun, as otherwise the new buffer would be
>> only programmed after the next frame start interrupt and used for the
>> next-next frame.  However, it's racy, because programming of the buffer
>> addresses is not atomic and could end up with the hardware using few
>> plane addresses from the new buffer and few from the dummy buffer.
>>
>> Given that and also the fact that a queue underrun is a very special
>> case, where the system was already having problems catching up, I'd just
>> remove this special case.
>>
>> [snip]
>>>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>>>>> +{
>>>>> +    void __iomem *base = dev->base_addr;
>>>>> +    unsigned int isp_mis_tmp = 0;
>>>>
>>>> _tmp are never good names :-S
>>>>
>>>>> +    unsigned int isp_err = 0;
>>>>
>>>> Neither of these variable need to be initialised to 0.
>>>>
>>>>> +
>>>>> +    /* start edge of v_sync */
>>>>> +    if (isp_mis & CIF_ISP_V_START) {
>>>>> +        rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>>>>
>>>> This will increment the frame sequence number. What if the interrupt is
>>>> slightly delayed and the next frame starts before we get a change to
>>>> copy the sequence number to the buffers (before they will complete
>>>> below) ?
>>>
>>> Do you mean that we get two sequental v-start signals and then the next
>>> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
>>> How can this be solved? I wonder if any v-start signal has a later signal
>>> that correspond to the same frame so that we can follow it?
>>>
>>> Maybe we should have one counter that is incremented on v-start signal,
>>> and another counter that is incremented uppon some other signal?
>>>
>>
>> We're talking about a hard IRQ. I can't imagine the interrupt handler
>> being delayed for a time close to a full frame interval (~16ms for 60
>> fps) to trigger such scenario.
>>
>>>>
>>>>> +
>>>>> +        writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>>>>
>>>> Do you need to clear all interrupt bits individually, can't you write
>>>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
>>>> all in one go ?
>>>>
>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>> +        if (isp_mis_tmp & CIF_ISP_V_START)
>>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>>>>> +                 isp_mis_tmp);
>>>>
>>>> This require some explanation. It looks like a naive way to protect
>>>> against something, but I think it could trigger under normal
>>>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
>>>> Same for the similar constructs below.
>>>>
>>>>> +    }
>>>>> +
>>>>> +    if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>>>>> +        /* Clear pic_size_error */
>>>>> +        writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>>>>> +        isp_err = readl(base + CIF_ISP_ERR);
>>>>> +        v4l2_err(&dev->v4l2_dev,
>>>>> +             "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>>>>
>>>> What does this mean ?
>>>>
>>>>> +        writel(isp_err, base + CIF_ISP_ERR_CLR);
>>>>> +    } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>>>>
>>>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
>>>>
>>>>> +        /* Clear data_loss */
>>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>> +        v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>> +    }
>>>>> +
>>>>> +    /* sampled input frame is complete */
>>>>> +    if (isp_mis & CIF_ISP_FRAME_IN) {
>>>>> +        writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>>>>> +                 isp_mis_tmp);
>>>>> +    }
>>>>> +
>>>>> +    /* frame was completely put out */
>>>>
>>>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
>>>> ? The two comments could do with a bit of brush up, and I think the
>>>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
>>>> action.
>>>
>>> Those two oneline comments are just copy-paste from the datasheet.
>>>
>>> ""
>>> 5 MIS_FRAME_IN sampled input frame is complete
>>> 1 MIS_FRAME frame was completely put out
>>> ""
>>>
>>> Unfrotunately, the datasheet does not add any further explanation about those signals.
>>>
>>>
>>
>> My loose recollection is that the former is signaled when then frame
>> is fully input to the ISP and the latter when the ISP completes
>> outputting the frame to the next block in the pipeline, but someone
>> would need to verify this, for example by printing timestamps for all
>> the various interrupts.
>>
>>>>
>>>>> +    if (isp_mis & CIF_ISP_FRAME) {
>>>>> +        u32 isp_ris = 0;
>>>>
>>>> No need to initialise this to 0.
>>>>
>>>>> +        /* Clear Frame In (ISP) */
>>>>> +        writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME)
>>>>> +            v4l2_err(&dev->v4l2_dev,
>>>>> +                 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>>>>> +
>>>>> +        isp_ris = readl(base + CIF_ISP_RIS);
>>>>> +        if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>>>>> +                   CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>>>>> +            rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>>>>
>>>> Is there a guarantee that the statistics will be fully written out
>>>> before the video frame itself ? And doesn't this test if any of the
>>>> statistics is complete, not all of them ? I think the logic is wrong, it
>>>
>>> The datasheet does not add any explanation of what is expected to come first.
>>> Should we wait until all statistics measurements are done? In the struct
>>> sent to userspace there is a bitmaks for which of the statistics are read.
>>> I think that if only part of the statistics are ready, we can already send the once
>>> that are ready to userspace.
>>>
>>
>> If we look further into the code, rkisp1_stats_isr() checks the
>> interrupt status mask passed to it and reads out only the parameters
>> with indicated completion. The statistics metadata buffer format
>> includes a bit mask which tells the userspace which measurements are
>> available.
>>
>> However, I think I've spotted a bug there. At the beginning of
>> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
>> regardless of the mask used later to decide which readouts need to be
>> done. This could mean that with an unfortunate timing, some measurements
>> would be lost. So at least the code should be fixed to only clear the
>> interrupts bits really handled.
> 
> I'll fix that

I actually don't think this is a bug. The statistics interrupts are not
enabled and are read from the raw interrupts register. This means
that if we missed a statistics for the current frame and we don't reset it
then we will read it only when the next frame comes out, so it will be
wrongly set as statistics for the next frame although it is actually for the
current frame.

Thanks,
Dafna

> 
>>
>> As for whether to send separate buffers for each measurement, I guess
>> it's not a bad thing to let the userspace access the ones available
>> earlier. Now I only don't recall why we decided to put all the
>> measurements into one metadata structure, rather than splitting the 4
>> into their own structures and buffer queues...
> 
> Is it possible to have several queues to the same video node?
> 
>>
>>>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
>>>> own. It's hard to tell for sure without extra information though (for
>>>> instance why are the stats-related bits read from CIF_ISP_RIS, when
>>>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
>>>> validated, and most probably fixed. Care should be taken to keep
>>>> synchronisation of sequence number between the different queues.
>>>
>>> I see that the capture buffers are done before incrementing the frame_sequence with
>>> the following explanation:
>>>
>>>     /*
>>>           * Call rkisp1_capture_isr() first to handle the frame that
>>>           * potentially completed using the current frame_sequence number before
>>>           * it is potentially incremented by rkisp1_isp_isr() in the vertical
>>>           * sync.
>>>           */
>>>
>>> I think reading the stats/params should also be done before calling rkisp1_capture_isr
>>> for the same reason. (so to match the correct frame_sequence)
>>
>> My recollection of the sequence of interrupts in this hardware is like
>> this:
>>
>> CIF_ISP_V_START (frame 0)
>>    CIF_ISP_FRAME_IN (frame 0)
>>      CIF_ISP_FRAME (frame 0)
>>        CIF_ISP_AWB_DONE
>>        CIF_ISP_AFM_FIN
>>        CIF_ISP_EXP_END
>>        CIF_ISP_HIST_MEASURE_RDY
>>        CIF_MI_FRAME*
>>        CIF_ISP_V_START (frame 1)
>>          CIF_ISP_FRAME_IN (frame 1)
>>            CIF_ISP_FRAME (frame 1)
>>              ...
>>
>> where the interrupts at the same indentation level can happen
>> independently of each other. Again, someone would have to verify this.
> 
> I wrote this patch to print the interrupts and the time difference between interrupts:
> https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5
> 
> I got this output: http://ix.io/2tl8,
> there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
> (FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.
> 
> I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
> we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
> I think that for that we need a code that keep track of the previous interrupt.
> 
> Thanks,
> Dafna
> 
> 
>>
>> Best regards,
>> Tomasz
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-08-05 21:10         ` Dafna Hirschfeld
  2020-08-06  9:21           ` Dafna Hirschfeld
@ 2020-08-06 12:08           ` Tomasz Figa
  2020-08-07 16:02             ` Dafna Hirschfeld
  1 sibling, 1 reply; 76+ messages in thread
From: Tomasz Figa @ 2020-08-06 12:08 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: linux-devicetree, Eddie Cai, kernel, Heiko Stübner,
	Chen Jacob, Jeffy, 钟以崇,
	Linux Kernel Mailing List, Allon Huang,
	open list:ARM/Rockchip SoC...,
	Helen Koike, Jacob Chen, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Shunqian Zheng, Mauro Carvalho Chehab,
	Ezequiel Garcia,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

On Wed, Aug 5, 2020 at 11:10 PM Dafna Hirschfeld
<dafna.hirschfeld@collabora.com> wrote:
>
> Hi
>
> On 22.07.20 17:24, Tomasz Figa wrote:
> > Hi Dafna,
> >
> > On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
> >> Hi Laurent,
> >>
> >> On 16.08.19 02:13, Laurent Pinchart wrote:
> >>> Hello Helen,
> >>>
> >>> Thank you for the patch.
> >>>
> >>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> > [snip]
> >>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> >>>> +{
> >>>> +  struct v4l2_event event = {
> >>>> +          .type = V4L2_EVENT_FRAME_SYNC,
> >>>> +          .u.frame_sync.frame_sequence =
> >>>> +                  atomic_inc_return(&isp->frm_sync_seq) - 1,
> >>>
> >>> I would move the increment to the caller, hiding it in this function is
> >>> error-prone (and if you look at the caller I'm pointing out one possible
> >>> error :-)).
> >>>
> >>> In general usage of frm_sync_seq through the driver seems to be very
> >>> race-prone. It's read in various IRQ handling functions, all coming from
> >>> the same IRQ, so that part is fine (and wouldn't require an atomic
> >>> variable), but when read from the buffer queue handlers I really get a
> >>> red light flashing in my head. I'll try to investigate more when
> >>> reviewing the next patches.
> >>
> >> I see that the only place were 'frame_sequence' is read outside of the irq
> >> handlers is in the capture in 'rkisp1_vb2_buf_queue':
> >>
> >>      /*
> >>           * If there's no next buffer assigned, queue this buffer directly
> >>           * as the next buffer, and update the memory interface.
> >>           */
> >>          if (cap->is_streaming && !cap->buf.next &&
> >>              atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
> >>                  cap->buf.next = ispbuf;
> >>                  rkisp1_set_next_buf(cap);
> >>          } else {
> >>                  list_add_tail(&ispbuf->queue, &cap->buf.queue);
> >>          }
> >> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> >> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> >> from the above condition so that the next buffer is updated in case it is null not just before the first
> >> v-start signal.
> >>
> >
> > We don't have this special case in the Chrome OS code.
> >
> > I suppose it would make it possible to resume the capture 1 frame
> > earlier after a queue underrun, as otherwise the new buffer would be
> > only programmed after the next frame start interrupt and used for the
> > next-next frame.  However, it's racy, because programming of the buffer
> > addresses is not atomic and could end up with the hardware using few
> > plane addresses from the new buffer and few from the dummy buffer.
> >
> > Given that and also the fact that a queue underrun is a very special
> > case, where the system was already having problems catching up, I'd just
> > remove this special case.
> >
> > [snip]
> >>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> >>>> +{
> >>>> +  void __iomem *base = dev->base_addr;
> >>>> +  unsigned int isp_mis_tmp = 0;
> >>>
> >>> _tmp are never good names :-S
> >>>
> >>>> +  unsigned int isp_err = 0;
> >>>
> >>> Neither of these variable need to be initialised to 0.
> >>>
> >>>> +
> >>>> +  /* start edge of v_sync */
> >>>> +  if (isp_mis & CIF_ISP_V_START) {
> >>>> +          rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> >>>
> >>> This will increment the frame sequence number. What if the interrupt is
> >>> slightly delayed and the next frame starts before we get a change to
> >>> copy the sequence number to the buffers (before they will complete
> >>> below) ?
> >>
> >> Do you mean that we get two sequental v-start signals and then the next
> >> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> >> How can this be solved? I wonder if any v-start signal has a later signal
> >> that correspond to the same frame so that we can follow it?
> >>
> >> Maybe we should have one counter that is incremented on v-start signal,
> >> and another counter that is incremented uppon some other signal?
> >>
> >
> > We're talking about a hard IRQ. I can't imagine the interrupt handler
> > being delayed for a time close to a full frame interval (~16ms for 60
> > fps) to trigger such scenario.
> >
> >>>
> >>>> +
> >>>> +          writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> >>>
> >>> Do you need to clear all interrupt bits individually, can't you write
> >>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> >>> all in one go ?
> >>>
> >>>> +          isp_mis_tmp = readl(base + CIF_ISP_MIS);
> >>>> +          if (isp_mis_tmp & CIF_ISP_V_START)
> >>>> +                  v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> >>>> +                           isp_mis_tmp);
> >>>
> >>> This require some explanation. It looks like a naive way to protect
> >>> against something, but I think it could trigger under normal
> >>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> >>> Same for the similar constructs below.
> >>>
> >>>> +  }
> >>>> +
> >>>> +  if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> >>>> +          /* Clear pic_size_error */
> >>>> +          writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> >>>> +          isp_err = readl(base + CIF_ISP_ERR);
> >>>> +          v4l2_err(&dev->v4l2_dev,
> >>>> +                   "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> >>>
> >>> What does this mean ?
> >>>
> >>>> +          writel(isp_err, base + CIF_ISP_ERR_CLR);
> >>>> +  } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> >>>
> >>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> >>>
> >>>> +          /* Clear data_loss */
> >>>> +          writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> >>>> +          v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> >>>> +          writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> >>>> +  }
> >>>> +
> >>>> +  /* sampled input frame is complete */
> >>>> +  if (isp_mis & CIF_ISP_FRAME_IN) {
> >>>> +          writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> >>>> +          isp_mis_tmp = readl(base + CIF_ISP_MIS);
> >>>> +          if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> >>>> +                  v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> >>>> +                           isp_mis_tmp);
> >>>> +  }
> >>>> +
> >>>> +  /* frame was completely put out */
> >>>
> >>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> >>> ? The two comments could do with a bit of brush up, and I think the
> >>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> >>> action.
> >>
> >> Those two oneline comments are just copy-paste from the datasheet.
> >>
> >> ""
> >> 5 MIS_FRAME_IN sampled input frame is complete
> >> 1 MIS_FRAME frame was completely put out
> >> ""
> >>
> >> Unfrotunately, the datasheet does not add any further explanation about those signals.
> >>
> >>
> >
> > My loose recollection is that the former is signaled when then frame
> > is fully input to the ISP and the latter when the ISP completes
> > outputting the frame to the next block in the pipeline, but someone
> > would need to verify this, for example by printing timestamps for all
> > the various interrupts.
> >
> >>>
> >>>> +  if (isp_mis & CIF_ISP_FRAME) {
> >>>> +          u32 isp_ris = 0;
> >>>
> >>> No need to initialise this to 0.
> >>>
> >>>> +          /* Clear Frame In (ISP) */
> >>>> +          writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> >>>> +          isp_mis_tmp = readl(base + CIF_ISP_MIS);
> >>>> +          if (isp_mis_tmp & CIF_ISP_FRAME)
> >>>> +                  v4l2_err(&dev->v4l2_dev,
> >>>> +                           "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> >>>> +
> >>>> +          isp_ris = readl(base + CIF_ISP_RIS);
> >>>> +          if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> >>>> +                         CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> >>>> +                  rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> >>>
> >>> Is there a guarantee that the statistics will be fully written out
> >>> before the video frame itself ? And doesn't this test if any of the
> >>> statistics is complete, not all of them ? I think the logic is wrong, it
> >>
> >> The datasheet does not add any explanation of what is expected to come first.
> >> Should we wait until all statistics measurements are done? In the struct
> >> sent to userspace there is a bitmaks for which of the statistics are read.
> >> I think that if only part of the statistics are ready, we can already send the once
> >> that are ready to userspace.
> >>
> >
> > If we look further into the code, rkisp1_stats_isr() checks the
> > interrupt status mask passed to it and reads out only the parameters
> > with indicated completion. The statistics metadata buffer format
> > includes a bit mask which tells the userspace which measurements are
> > available.
> >
> > However, I think I've spotted a bug there. At the beginning of
> > rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
> > regardless of the mask used later to decide which readouts need to be
> > done. This could mean that with an unfortunate timing, some measurements
> > would be lost. So at least the code should be fixed to only clear the
> > interrupts bits really handled.
>
> I'll fix that
>
> >
> > As for whether to send separate buffers for each measurement, I guess
> > it's not a bad thing to let the userspace access the ones available
> > earlier. Now I only don't recall why we decided to put all the
> > measurements into one metadata structure, rather than splitting the 4
> > into their own structures and buffer queues...
>
> Is it possible to have several queues to the same video node?
>
> >
> >>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> >>> own. It's hard to tell for sure without extra information though (for
> >>> instance why are the stats-related bits read from CIF_ISP_RIS, when
> >>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
> >>> validated, and most probably fixed. Care should be taken to keep
> >>> synchronisation of sequence number between the different queues.
> >>
> >> I see that the capture buffers are done before incrementing the frame_sequence with
> >> the following explanation:
> >>
> >>      /*
> >>           * Call rkisp1_capture_isr() first to handle the frame that
> >>           * potentially completed using the current frame_sequence number before
> >>           * it is potentially incremented by rkisp1_isp_isr() in the vertical
> >>           * sync.
> >>           */
> >>
> >> I think reading the stats/params should also be done before calling rkisp1_capture_isr
> >> for the same reason. (so to match the correct frame_sequence)
> >
> > My recollection of the sequence of interrupts in this hardware is like
> > this:
> >
> > CIF_ISP_V_START (frame 0)
> >    CIF_ISP_FRAME_IN (frame 0)
> >      CIF_ISP_FRAME (frame 0)
> >        CIF_ISP_AWB_DONE
> >        CIF_ISP_AFM_FIN
> >        CIF_ISP_EXP_END
> >        CIF_ISP_HIST_MEASURE_RDY
> >        CIF_MI_FRAME*
> >        CIF_ISP_V_START (frame 1)
> >          CIF_ISP_FRAME_IN (frame 1)
> >            CIF_ISP_FRAME (frame 1)
> >              ...
> >
> > where the interrupts at the same indentation level can happen
> > independently of each other. Again, someone would have to verify this.
>
> I wrote this patch to print the interrupts and the time difference between interrupts:
> https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5
>
> I got this output: http://ix.io/2tl8,
> there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
> (FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.
>
> I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
> we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
> I think that for that we need a code that keep track of the previous interrupt.
>

Okay, thanks for double checking this. We are still missing the
CIF_ISP_FRAME_IN interrupt and the interrupts from the CIF_ISP_RIS
register in the picture, though.

Regardless of that, looking at how this hardware seems to work, I'd
imagine that what we need to do is:

1) On an MI interrupt:
  a) Program MI shadow registers for frame N+2
  b) Signal completion of frame N to the user

2) On an RKISP1_CIF_ISP_FRAME interrupt:
  a) Program ISP parameter shadow registers for frame N+2

3) On one of the 4 interrupts from the CIF_ISP_RIS register:
  a) Read out the corresponding ISP statistics
  b.1) Return the partial statistics buffer to the userspace, OR
  b.2) If all the 4 statistics have been read, return the full
statistics buffer to the userspace.

All the 3 cases don't care about each other. Every time each interrupt
is received, the driver does the work for the sequential frame. We
basically assume that no interrupts are ever lost.

If we were to detect interrupt lost cases, we would have to either
look at some hardware frame counter (if one exists) or use some
heuristics based on the system timer and configured frame interval. I
don't think the extra complexity is worth it, because if the system
suffers from interrupt latency high enough to cause lost interrupts
(i.e. close to the shortest frame intervals supported by this ISP),
then it is going to have other problems as well and the underlying
cause of the latency should be fixed instead.

One more note about the current implementation is that it seems to
assume that CIF_ISP_RIS interrupts are already done when the
RKISP1_CIF_ISP_FRAME interrupt arrives. If that's how the hardware
works indeed, then it's a good thing to do, because it decreases the
number of signalled interrupts and thus the total interrupt handling
overhead.

Another thing to note is that when both MP and SP are enabled, I
recall seeing their interrupts sometimes arriving separately.

Now one thing that needs to be done to handle the 3 cases correctly is
to make sure they match the input parameters, output capture buffers
and statistics to the right frame. Since the MIs are asynchronous from
the ISP and each other, I suppose we could make the code less fragile
by making the MIs have their own frame counters, so they don't have to
rely on the order of interrupt handling. Perhaps that would be enough
to address your concern?

Best regards,
Tomasz

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-08-06  9:21           ` Dafna Hirschfeld
@ 2020-08-06 12:22             ` Tomasz Figa
  2020-08-07 16:08               ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Tomasz Figa @ 2020-08-06 12:22 UTC (permalink / raw)
  To: Dafna Hirschfeld
  Cc: linux-devicetree, Eddie Cai, kernel, Heiko Stübner,
	Chen Jacob, Jeffy, 钟以崇,
	Linux Kernel Mailing List, Allon Huang,
	open list:ARM/Rockchip SoC...,
	Helen Koike, Jacob Chen, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Shunqian Zheng, Mauro Carvalho Chehab,
	Ezequiel Garcia,
	list@263.net:IOMMU DRIVERS
	<iommu@lists.linux-foundation.org>,
	Joerg Roedel <joro@8bytes.org>, ,
	Linux Media Mailing List

On Thu, Aug 6, 2020 at 11:21 AM Dafna Hirschfeld
<dafna.hirschfeld@collabora.com> wrote:
>
>
>
> Am 05.08.20 um 23:10 schrieb Dafna Hirschfeld:
> > Hi
> >
> > On 22.07.20 17:24, Tomasz Figa wrote:
> >> Hi Dafna,
> >>
> >> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
> >>> Hi Laurent,
> >>>
> >>> On 16.08.19 02:13, Laurent Pinchart wrote:
> >>>> Hello Helen,
> >>>>
> >>>> Thank you for the patch.
> >>>>
> >>>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
> >> [snip]
> >>>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
> >>>>> +{
> >>>>> +    struct v4l2_event event = {
> >>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
> >>>>> +        .u.frame_sync.frame_sequence =
> >>>>> +            atomic_inc_return(&isp->frm_sync_seq) - 1,
> >>>>
> >>>> I would move the increment to the caller, hiding it in this function is
> >>>> error-prone (and if you look at the caller I'm pointing out one possible
> >>>> error :-)).
> >>>>
> >>>> In general usage of frm_sync_seq through the driver seems to be very
> >>>> race-prone. It's read in various IRQ handling functions, all coming from
> >>>> the same IRQ, so that part is fine (and wouldn't require an atomic
> >>>> variable), but when read from the buffer queue handlers I really get a
> >>>> red light flashing in my head. I'll try to investigate more when
> >>>> reviewing the next patches.
> >>>
> >>> I see that the only place were 'frame_sequence' is read outside of the irq
> >>> handlers is in the capture in 'rkisp1_vb2_buf_queue':
> >>>
> >>>     /*
> >>>           * If there's no next buffer assigned, queue this buffer directly
> >>>           * as the next buffer, and update the memory interface.
> >>>           */
> >>>          if (cap->is_streaming && !cap->buf.next &&
> >>>              atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
> >>>                  cap->buf.next = ispbuf;
> >>>                  rkisp1_set_next_buf(cap);
> >>>          } else {
> >>>                  list_add_tail(&ispbuf->queue, &cap->buf.queue);
> >>>          }
> >>> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
> >>> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
> >>> from the above condition so that the next buffer is updated in case it is null not just before the first
> >>> v-start signal.
> >>>
> >>
> >> We don't have this special case in the Chrome OS code.
> >>
> >> I suppose it would make it possible to resume the capture 1 frame
> >> earlier after a queue underrun, as otherwise the new buffer would be
> >> only programmed after the next frame start interrupt and used for the
> >> next-next frame.  However, it's racy, because programming of the buffer
> >> addresses is not atomic and could end up with the hardware using few
> >> plane addresses from the new buffer and few from the dummy buffer.
> >>
> >> Given that and also the fact that a queue underrun is a very special
> >> case, where the system was already having problems catching up, I'd just
> >> remove this special case.
> >>
> >> [snip]
> >>>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
> >>>>> +{
> >>>>> +    void __iomem *base = dev->base_addr;
> >>>>> +    unsigned int isp_mis_tmp = 0;
> >>>>
> >>>> _tmp are never good names :-S
> >>>>
> >>>>> +    unsigned int isp_err = 0;
> >>>>
> >>>> Neither of these variable need to be initialised to 0.
> >>>>
> >>>>> +
> >>>>> +    /* start edge of v_sync */
> >>>>> +    if (isp_mis & CIF_ISP_V_START) {
> >>>>> +        rkisp1_isp_queue_event_sof(&dev->isp_sdev);
> >>>>
> >>>> This will increment the frame sequence number. What if the interrupt is
> >>>> slightly delayed and the next frame starts before we get a change to
> >>>> copy the sequence number to the buffers (before they will complete
> >>>> below) ?
> >>>
> >>> Do you mean that we get two sequental v-start signals and then the next
> >>> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
> >>> How can this be solved? I wonder if any v-start signal has a later signal
> >>> that correspond to the same frame so that we can follow it?
> >>>
> >>> Maybe we should have one counter that is incremented on v-start signal,
> >>> and another counter that is incremented uppon some other signal?
> >>>
> >>
> >> We're talking about a hard IRQ. I can't imagine the interrupt handler
> >> being delayed for a time close to a full frame interval (~16ms for 60
> >> fps) to trigger such scenario.
> >>
> >>>>
> >>>>> +
> >>>>> +        writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
> >>>>
> >>>> Do you need to clear all interrupt bits individually, can't you write
> >>>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
> >>>> all in one go ?
> >>>>
> >>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
> >>>>> +        if (isp_mis_tmp & CIF_ISP_V_START)
> >>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
> >>>>> +                 isp_mis_tmp);
> >>>>
> >>>> This require some explanation. It looks like a naive way to protect
> >>>> against something, but I think it could trigger under normal
> >>>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
> >>>> Same for the similar constructs below.
> >>>>
> >>>>> +    }
> >>>>> +
> >>>>> +    if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
> >>>>> +        /* Clear pic_size_error */
> >>>>> +        writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
> >>>>> +        isp_err = readl(base + CIF_ISP_ERR);
> >>>>> +        v4l2_err(&dev->v4l2_dev,
> >>>>> +             "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
> >>>>
> >>>> What does this mean ?
> >>>>
> >>>>> +        writel(isp_err, base + CIF_ISP_ERR_CLR);
> >>>>> +    } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
> >>>>
> >>>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
> >>>>
> >>>>> +        /* Clear data_loss */
> >>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> >>>>> +        v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
> >>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
> >>>>> +    }
> >>>>> +
> >>>>> +    /* sampled input frame is complete */
> >>>>> +    if (isp_mis & CIF_ISP_FRAME_IN) {
> >>>>> +        writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
> >>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
> >>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME_IN)
> >>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
> >>>>> +                 isp_mis_tmp);
> >>>>> +    }
> >>>>> +
> >>>>> +    /* frame was completely put out */
> >>>>
> >>>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
> >>>> ? The two comments could do with a bit of brush up, and I think the
> >>>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
> >>>> action.
> >>>
> >>> Those two oneline comments are just copy-paste from the datasheet.
> >>>
> >>> ""
> >>> 5 MIS_FRAME_IN sampled input frame is complete
> >>> 1 MIS_FRAME frame was completely put out
> >>> ""
> >>>
> >>> Unfrotunately, the datasheet does not add any further explanation about those signals.
> >>>
> >>>
> >>
> >> My loose recollection is that the former is signaled when then frame
> >> is fully input to the ISP and the latter when the ISP completes
> >> outputting the frame to the next block in the pipeline, but someone
> >> would need to verify this, for example by printing timestamps for all
> >> the various interrupts.
> >>
> >>>>
> >>>>> +    if (isp_mis & CIF_ISP_FRAME) {
> >>>>> +        u32 isp_ris = 0;
> >>>>
> >>>> No need to initialise this to 0.
> >>>>
> >>>>> +        /* Clear Frame In (ISP) */
> >>>>> +        writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
> >>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
> >>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME)
> >>>>> +            v4l2_err(&dev->v4l2_dev,
> >>>>> +                 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
> >>>>> +
> >>>>> +        isp_ris = readl(base + CIF_ISP_RIS);
> >>>>> +        if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
> >>>>> +                   CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
> >>>>> +            rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
> >>>>
> >>>> Is there a guarantee that the statistics will be fully written out
> >>>> before the video frame itself ? And doesn't this test if any of the
> >>>> statistics is complete, not all of them ? I think the logic is wrong, it
> >>>
> >>> The datasheet does not add any explanation of what is expected to come first.
> >>> Should we wait until all statistics measurements are done? In the struct
> >>> sent to userspace there is a bitmaks for which of the statistics are read.
> >>> I think that if only part of the statistics are ready, we can already send the once
> >>> that are ready to userspace.
> >>>
> >>
> >> If we look further into the code, rkisp1_stats_isr() checks the
> >> interrupt status mask passed to it and reads out only the parameters
> >> with indicated completion. The statistics metadata buffer format
> >> includes a bit mask which tells the userspace which measurements are
> >> available.
> >>
> >> However, I think I've spotted a bug there. At the beginning of
> >> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
> >> regardless of the mask used later to decide which readouts need to be
> >> done. This could mean that with an unfortunate timing, some measurements
> >> would be lost. So at least the code should be fixed to only clear the
> >> interrupts bits really handled.
> >
> > I'll fix that
>
> I actually don't think this is a bug. The statistics interrupts are not
> enabled and are read from the raw interrupts register. This means
> that if we missed a statistics for the current frame and we don't reset it
> then we will read it only when the next frame comes out, so it will be
> wrongly set as statistics for the next frame although it is actually for the
> current frame.

Yes, I noticed that the driver attempts to reduce the number of
interrupts by assuming that the ISP statistics can be read after the
MIS_FRAME interrupt. However, in this case, I don't think we can ever
miss statistics for a frame (unless the system is broken and has
unacceptable interrupt latencies) nor the unfortunate timing I
suggested before could ever take place.

Best regards,
Tomasz

>
> Thanks,
> Dafna
>
> >
> >>
> >> As for whether to send separate buffers for each measurement, I guess
> >> it's not a bad thing to let the userspace access the ones available
> >> earlier. Now I only don't recall why we decided to put all the
> >> measurements into one metadata structure, rather than splitting the 4
> >> into their own structures and buffer queues...
> >
> > Is it possible to have several queues to the same video node?
> >
> >>
> >>>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
> >>>> own. It's hard to tell for sure without extra information though (for
> >>>> instance why are the stats-related bits read from CIF_ISP_RIS, when
> >>>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
> >>>> validated, and most probably fixed. Care should be taken to keep
> >>>> synchronisation of sequence number between the different queues.
> >>>
> >>> I see that the capture buffers are done before incrementing the frame_sequence with
> >>> the following explanation:
> >>>
> >>>     /*
> >>>           * Call rkisp1_capture_isr() first to handle the frame that
> >>>           * potentially completed using the current frame_sequence number before
> >>>           * it is potentially incremented by rkisp1_isp_isr() in the vertical
> >>>           * sync.
> >>>           */
> >>>
> >>> I think reading the stats/params should also be done before calling rkisp1_capture_isr
> >>> for the same reason. (so to match the correct frame_sequence)
> >>
> >> My recollection of the sequence of interrupts in this hardware is like
> >> this:
> >>
> >> CIF_ISP_V_START (frame 0)
> >>    CIF_ISP_FRAME_IN (frame 0)
> >>      CIF_ISP_FRAME (frame 0)
> >>        CIF_ISP_AWB_DONE
> >>        CIF_ISP_AFM_FIN
> >>        CIF_ISP_EXP_END
> >>        CIF_ISP_HIST_MEASURE_RDY
> >>        CIF_MI_FRAME*
> >>        CIF_ISP_V_START (frame 1)
> >>          CIF_ISP_FRAME_IN (frame 1)
> >>            CIF_ISP_FRAME (frame 1)
> >>              ...
> >>
> >> where the interrupts at the same indentation level can happen
> >> independently of each other. Again, someone would have to verify this.
> >
> > I wrote this patch to print the interrupts and the time difference between interrupts:
> > https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5
> >
> > I got this output: http://ix.io/2tl8,
> > there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
> > (FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.
> >
> > I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
> > we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
> > I think that for that we need a code that keep track of the previous interrupt.
> >
> > Thanks,
> > Dafna
> >
> >
> >>
> >> Best regards,
> >> Tomasz
> >>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-08-06 12:08           ` Tomasz Figa
@ 2020-08-07 16:02             ` Dafna Hirschfeld
  0 siblings, 0 replies; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-08-07 16:02 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Eddie Cai, Heiko Stübner, Laurent Pinchart, kernel,
	Joerg Roedel, 钟以崇,
	open list:ARM/Rockchip SoC...,
	Hans Verkuil, Allon Huang, Shunqian Zheng,
	Linux Media Mailing List, linux-devicetree, Jacob Chen, Jeffy,
	Helen Koike, Mauro Carvalho Chehab, Ezequiel Garcia,
	linux-arm-kernel, Chen Jacob, Linux Kernel Mailing List,
	list@263.net:IOMMU DRIVERS, Sakari Ailus

Hi,


Am 06.08.20 um 14:08 schrieb Tomasz Figa:
> On Wed, Aug 5, 2020 at 11:10 PM Dafna Hirschfeld
> <dafna.hirschfeld@collabora.com> wrote:
>>
>> Hi
>>
>> On 22.07.20 17:24, Tomasz Figa wrote:
>>> Hi Dafna,
>>>
>>> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
>>>> Hi Laurent,
>>>>
>>>> On 16.08.19 02:13, Laurent Pinchart wrote:
>>>>> Hello Helen,
>>>>>
>>>>> Thank you for the patch.
>>>>>
>>>>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>>> [snip]
>>>>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>>>>>> +{
>>>>>> +  struct v4l2_event event = {
>>>>>> +          .type = V4L2_EVENT_FRAME_SYNC,
>>>>>> +          .u.frame_sync.frame_sequence =
>>>>>> +                  atomic_inc_return(&isp->frm_sync_seq) - 1,
>>>>>
>>>>> I would move the increment to the caller, hiding it in this function is
>>>>> error-prone (and if you look at the caller I'm pointing out one possible
>>>>> error :-)).
>>>>>
>>>>> In general usage of frm_sync_seq through the driver seems to be very
>>>>> race-prone. It's read in various IRQ handling functions, all coming from
>>>>> the same IRQ, so that part is fine (and wouldn't require an atomic
>>>>> variable), but when read from the buffer queue handlers I really get a
>>>>> red light flashing in my head. I'll try to investigate more when
>>>>> reviewing the next patches.
>>>>
>>>> I see that the only place were 'frame_sequence' is read outside of the irq
>>>> handlers is in the capture in 'rkisp1_vb2_buf_queue':
>>>>
>>>>       /*
>>>>            * If there's no next buffer assigned, queue this buffer directly
>>>>            * as the next buffer, and update the memory interface.
>>>>            */
>>>>           if (cap->is_streaming && !cap->buf.next &&
>>>>               atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>>>>                   cap->buf.next = ispbuf;
>>>>                   rkisp1_set_next_buf(cap);
>>>>           } else {
>>>>                   list_add_tail(&ispbuf->queue, &cap->buf.queue);
>>>>           }
>>>> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
>>>> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
>>>> from the above condition so that the next buffer is updated in case it is null not just before the first
>>>> v-start signal.
>>>>
>>>
>>> We don't have this special case in the Chrome OS code.
>>>
>>> I suppose it would make it possible to resume the capture 1 frame
>>> earlier after a queue underrun, as otherwise the new buffer would be
>>> only programmed after the next frame start interrupt and used for the
>>> next-next frame.  However, it's racy, because programming of the buffer
>>> addresses is not atomic and could end up with the hardware using few
>>> plane addresses from the new buffer and few from the dummy buffer.
>>>
>>> Given that and also the fact that a queue underrun is a very special
>>> case, where the system was already having problems catching up, I'd just
>>> remove this special case.
>>>
>>> [snip]
>>>>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>>>>>> +{
>>>>>> +  void __iomem *base = dev->base_addr;
>>>>>> +  unsigned int isp_mis_tmp = 0;
>>>>>
>>>>> _tmp are never good names :-S
>>>>>
>>>>>> +  unsigned int isp_err = 0;
>>>>>
>>>>> Neither of these variable need to be initialised to 0.
>>>>>
>>>>>> +
>>>>>> +  /* start edge of v_sync */
>>>>>> +  if (isp_mis & CIF_ISP_V_START) {
>>>>>> +          rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>>>>>
>>>>> This will increment the frame sequence number. What if the interrupt is
>>>>> slightly delayed and the next frame starts before we get a change to
>>>>> copy the sequence number to the buffers (before they will complete
>>>>> below) ?
>>>>
>>>> Do you mean that we get two sequental v-start signals and then the next
>>>> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
>>>> How can this be solved? I wonder if any v-start signal has a later signal
>>>> that correspond to the same frame so that we can follow it?
>>>>
>>>> Maybe we should have one counter that is incremented on v-start signal,
>>>> and another counter that is incremented uppon some other signal?
>>>>
>>>
>>> We're talking about a hard IRQ. I can't imagine the interrupt handler
>>> being delayed for a time close to a full frame interval (~16ms for 60
>>> fps) to trigger such scenario.
>>>
>>>>>
>>>>>> +
>>>>>> +          writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>>>>>
>>>>> Do you need to clear all interrupt bits individually, can't you write
>>>>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
>>>>> all in one go ?
>>>>>
>>>>>> +          isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>> +          if (isp_mis_tmp & CIF_ISP_V_START)
>>>>>> +                  v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>>>>>> +                           isp_mis_tmp);
>>>>>
>>>>> This require some explanation. It looks like a naive way to protect
>>>>> against something, but I think it could trigger under normal
>>>>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
>>>>> Same for the similar constructs below.
>>>>>
>>>>>> +  }
>>>>>> +
>>>>>> +  if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>>>>>> +          /* Clear pic_size_error */
>>>>>> +          writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>>>>>> +          isp_err = readl(base + CIF_ISP_ERR);
>>>>>> +          v4l2_err(&dev->v4l2_dev,
>>>>>> +                   "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>>>>>
>>>>> What does this mean ?
>>>>>
>>>>>> +          writel(isp_err, base + CIF_ISP_ERR_CLR);
>>>>>> +  } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>>>>>
>>>>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
>>>>>
>>>>>> +          /* Clear data_loss */
>>>>>> +          writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>>> +          v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>>>>>> +          writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>>> +  }
>>>>>> +
>>>>>> +  /* sampled input frame is complete */
>>>>>> +  if (isp_mis & CIF_ISP_FRAME_IN) {
>>>>>> +          writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>>>>>> +          isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>> +          if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>>>>>> +                  v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>>>>>> +                           isp_mis_tmp);
>>>>>> +  }
>>>>>> +
>>>>>> +  /* frame was completely put out */
>>>>>
>>>>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
>>>>> ? The two comments could do with a bit of brush up, and I think the
>>>>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
>>>>> action.
>>>>
>>>> Those two oneline comments are just copy-paste from the datasheet.
>>>>
>>>> ""
>>>> 5 MIS_FRAME_IN sampled input frame is complete
>>>> 1 MIS_FRAME frame was completely put out
>>>> ""
>>>>
>>>> Unfrotunately, the datasheet does not add any further explanation about those signals.
>>>>
>>>>
>>>
>>> My loose recollection is that the former is signaled when then frame
>>> is fully input to the ISP and the latter when the ISP completes
>>> outputting the frame to the next block in the pipeline, but someone
>>> would need to verify this, for example by printing timestamps for all
>>> the various interrupts.
>>>
>>>>>
>>>>>> +  if (isp_mis & CIF_ISP_FRAME) {
>>>>>> +          u32 isp_ris = 0;
>>>>>
>>>>> No need to initialise this to 0.
>>>>>
>>>>>> +          /* Clear Frame In (ISP) */
>>>>>> +          writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>>>>>> +          isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>> +          if (isp_mis_tmp & CIF_ISP_FRAME)
>>>>>> +                  v4l2_err(&dev->v4l2_dev,
>>>>>> +                           "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>>>>>> +
>>>>>> +          isp_ris = readl(base + CIF_ISP_RIS);
>>>>>> +          if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>>>>>> +                         CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>>>>>> +                  rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>>>>>
>>>>> Is there a guarantee that the statistics will be fully written out
>>>>> before the video frame itself ? And doesn't this test if any of the
>>>>> statistics is complete, not all of them ? I think the logic is wrong, it
>>>>
>>>> The datasheet does not add any explanation of what is expected to come first.
>>>> Should we wait until all statistics measurements are done? In the struct
>>>> sent to userspace there is a bitmaks for which of the statistics are read.
>>>> I think that if only part of the statistics are ready, we can already send the once
>>>> that are ready to userspace.
>>>>
>>>
>>> If we look further into the code, rkisp1_stats_isr() checks the
>>> interrupt status mask passed to it and reads out only the parameters
>>> with indicated completion. The statistics metadata buffer format
>>> includes a bit mask which tells the userspace which measurements are
>>> available.
>>>
>>> However, I think I've spotted a bug there. At the beginning of
>>> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
>>> regardless of the mask used later to decide which readouts need to be
>>> done. This could mean that with an unfortunate timing, some measurements
>>> would be lost. So at least the code should be fixed to only clear the
>>> interrupts bits really handled.
>>
>> I'll fix that
>>
>>>
>>> As for whether to send separate buffers for each measurement, I guess
>>> it's not a bad thing to let the userspace access the ones available
>>> earlier. Now I only don't recall why we decided to put all the
>>> measurements into one metadata structure, rather than splitting the 4
>>> into their own structures and buffer queues...
>>
>> Is it possible to have several queues to the same video node?
>>
>>>
>>>>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
>>>>> own. It's hard to tell for sure without extra information though (for
>>>>> instance why are the stats-related bits read from CIF_ISP_RIS, when
>>>>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
>>>>> validated, and most probably fixed. Care should be taken to keep
>>>>> synchronisation of sequence number between the different queues.
>>>>
>>>> I see that the capture buffers are done before incrementing the frame_sequence with
>>>> the following explanation:
>>>>
>>>>       /*
>>>>            * Call rkisp1_capture_isr() first to handle the frame that
>>>>            * potentially completed using the current frame_sequence number before
>>>>            * it is potentially incremented by rkisp1_isp_isr() in the vertical
>>>>            * sync.
>>>>            */
>>>>
>>>> I think reading the stats/params should also be done before calling rkisp1_capture_isr
>>>> for the same reason. (so to match the correct frame_sequence)
>>>
>>> My recollection of the sequence of interrupts in this hardware is like
>>> this:
>>>
>>> CIF_ISP_V_START (frame 0)
>>>     CIF_ISP_FRAME_IN (frame 0)
>>>       CIF_ISP_FRAME (frame 0)
>>>         CIF_ISP_AWB_DONE
>>>         CIF_ISP_AFM_FIN
>>>         CIF_ISP_EXP_END
>>>         CIF_ISP_HIST_MEASURE_RDY
>>>         CIF_MI_FRAME*
>>>         CIF_ISP_V_START (frame 1)
>>>           CIF_ISP_FRAME_IN (frame 1)
>>>             CIF_ISP_FRAME (frame 1)
>>>               ...
>>>
>>> where the interrupts at the same indentation level can happen
>>> independently of each other. Again, someone would have to verify this.
>>
>> I wrote this patch to print the interrupts and the time difference between interrupts:
>> https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5
>>
>> I got this output: http://ix.io/2tl8,
>> there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
>> (FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.
>>
>> I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
>> we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
>> I think that for that we need a code that keep track of the previous interrupt.
>>
> 
> Okay, thanks for double checking this. We are still missing the
> CIF_ISP_FRAME_IN interrupt and the interrupts from the CIF_ISP_RIS
> register in the picture, though.
> 
> Regardless of that, looking at how this hardware seems to work, I'd
> imagine that what we need to do is:
> 
> 1) On an MI interrupt:
>    a) Program MI shadow registers for frame N+2
>    b) Signal completion of frame N to the user
> 
> 2) On an RKISP1_CIF_ISP_FRAME interrupt:
>    a) Program ISP parameter shadow registers for frame N+2

I see that the current implementation of rkisp1_params_isr,
the shadow registers are updated immediately, you think this is wrong?
Probably the assumption is that they are updated before the next frame arrives to the isp.
In that case, if we want the frame_sequence of the params to be the
frame to which the params are applied, then we should set it
to "rkisp1->isp.frame_sequence + 1" since they are always applied
to the next frame.
  
> 
> 3) On one of the 4 interrupts from the CIF_ISP_RIS register:
>    a) Read out the corresponding ISP statistics
>    b.1) Return the partial statistics buffer to the userspace, OR
>    b.2) If all the 4 statistics have been read, return the full
> statistics buffer to the userspace.
> 
> All the 3 cases don't care about each other. Every time each interrupt
> is received, the driver does the work for the sequential frame. We
> basically assume that no interrupts are ever lost.
> 
> If we were to detect interrupt lost cases, we would have to either
> look at some hardware frame counter (if one exists) or use some
> heuristics based on the system timer and configured frame interval. I
> don't think the extra complexity is worth it, because if the system
> suffers from interrupt latency high enough to cause lost interrupts
> (i.e. close to the shortest frame intervals supported by this ISP),
> then it is going to have other problems as well and the underlying
> cause of the latency should be fixed instead.

So we assume that v-start interrupt should not arrive together with FRAME,FRAME_IN interrupts?

> 
> One more note about the current implementation is that it seems to
> assume that CIF_ISP_RIS interrupts are already done when the
> RKISP1_CIF_ISP_FRAME interrupt arrives. If that's how the hardware
> works indeed, then it's a good thing to do, because it decreases the
> number of signalled interrupts and thus the total interrupt handling
> overhead.
> 
> Another thing to note is that when both MP and SP are enabled, I
> recall seeing their interrupts sometimes arriving separately.
> 
> Now one thing that needs to be done to handle the 3 cases correctly is
> to make sure they match the input parameters, output capture buffers
> and statistics to the right frame. Since the MIs are asynchronous from
> the ISP and each other, I suppose we could make the code less fragile
> by making the MIs have their own frame counters, so they don't have to
> rely on the order of interrupt handling. Perhaps that would be enough
> to address your concern?

It is possible solve it with separate counters. In that case shouldn't we
also have separate counters for the stats and params? because also handling the 'FRAME'
interrupt is dependent on the order of handling.
Unless we assume that the v-start interrupts always arrive without the FRAME,FRAME_IN interrupts.
In that case there is no problem.

Thanks,
Dafna

> 
> Best regards,
> Tomasz
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-08-06 12:22             ` Tomasz Figa
@ 2020-08-07 16:08               ` Dafna Hirschfeld
  2020-08-13  6:17                 ` Dafna Hirschfeld
  0 siblings, 1 reply; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-08-07 16:08 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Eddie Cai, Heiko Stübner, Laurent Pinchart, kernel,
	Joerg Roedel, 钟以崇,
	open list:ARM/Rockchip SoC...,
	Hans Verkuil, Allon Huang, Shunqian Zheng,
	Linux Media Mailing List, linux-devicetree, Jacob Chen, Jeffy,
	Helen Koike, Mauro Carvalho Chehab, Ezequiel Garcia,
	linux-arm-kernel, Chen Jacob, Linux Kernel Mailing List,
	list@263.net:IOMMU DRIVERS, Sakari Ailus

Hi

Am 06.08.20 um 14:22 schrieb Tomasz Figa:
> On Thu, Aug 6, 2020 at 11:21 AM Dafna Hirschfeld
> <dafna.hirschfeld@collabora.com> wrote:
>>
>>
>>
>> Am 05.08.20 um 23:10 schrieb Dafna Hirschfeld:
>>> Hi
>>>
>>> On 22.07.20 17:24, Tomasz Figa wrote:
>>>> Hi Dafna,
>>>>
>>>> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
>>>>> Hi Laurent,
>>>>>
>>>>> On 16.08.19 02:13, Laurent Pinchart wrote:
>>>>>> Hello Helen,
>>>>>>
>>>>>> Thank you for the patch.
>>>>>>
>>>>>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>>>> [snip]
>>>>>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>>>>>>> +{
>>>>>>> +    struct v4l2_event event = {
>>>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>>>>>> +        .u.frame_sync.frame_sequence =
>>>>>>> +            atomic_inc_return(&isp->frm_sync_seq) - 1,
>>>>>>
>>>>>> I would move the increment to the caller, hiding it in this function is
>>>>>> error-prone (and if you look at the caller I'm pointing out one possible
>>>>>> error :-)).
>>>>>>
>>>>>> In general usage of frm_sync_seq through the driver seems to be very
>>>>>> race-prone. It's read in various IRQ handling functions, all coming from
>>>>>> the same IRQ, so that part is fine (and wouldn't require an atomic
>>>>>> variable), but when read from the buffer queue handlers I really get a
>>>>>> red light flashing in my head. I'll try to investigate more when
>>>>>> reviewing the next patches.
>>>>>
>>>>> I see that the only place were 'frame_sequence' is read outside of the irq
>>>>> handlers is in the capture in 'rkisp1_vb2_buf_queue':
>>>>>
>>>>>      /*
>>>>>            * If there's no next buffer assigned, queue this buffer directly
>>>>>            * as the next buffer, and update the memory interface.
>>>>>            */
>>>>>           if (cap->is_streaming && !cap->buf.next &&
>>>>>               atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>>>>>                   cap->buf.next = ispbuf;
>>>>>                   rkisp1_set_next_buf(cap);
>>>>>           } else {
>>>>>                   list_add_tail(&ispbuf->queue, &cap->buf.queue);
>>>>>           }
>>>>> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
>>>>> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
>>>>> from the above condition so that the next buffer is updated in case it is null not just before the first
>>>>> v-start signal.
>>>>>
>>>>
>>>> We don't have this special case in the Chrome OS code.
>>>>
>>>> I suppose it would make it possible to resume the capture 1 frame
>>>> earlier after a queue underrun, as otherwise the new buffer would be
>>>> only programmed after the next frame start interrupt and used for the
>>>> next-next frame.  However, it's racy, because programming of the buffer
>>>> addresses is not atomic and could end up with the hardware using few
>>>> plane addresses from the new buffer and few from the dummy buffer.
>>>>
>>>> Given that and also the fact that a queue underrun is a very special
>>>> case, where the system was already having problems catching up, I'd just
>>>> remove this special case.
>>>>
>>>> [snip]
>>>>>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>>>>>>> +{
>>>>>>> +    void __iomem *base = dev->base_addr;
>>>>>>> +    unsigned int isp_mis_tmp = 0;
>>>>>>
>>>>>> _tmp are never good names :-S
>>>>>>
>>>>>>> +    unsigned int isp_err = 0;
>>>>>>
>>>>>> Neither of these variable need to be initialised to 0.
>>>>>>
>>>>>>> +
>>>>>>> +    /* start edge of v_sync */
>>>>>>> +    if (isp_mis & CIF_ISP_V_START) {
>>>>>>> +        rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>>>>>>
>>>>>> This will increment the frame sequence number. What if the interrupt is
>>>>>> slightly delayed and the next frame starts before we get a change to
>>>>>> copy the sequence number to the buffers (before they will complete
>>>>>> below) ?
>>>>>
>>>>> Do you mean that we get two sequental v-start signals and then the next
>>>>> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
>>>>> How can this be solved? I wonder if any v-start signal has a later signal
>>>>> that correspond to the same frame so that we can follow it?
>>>>>
>>>>> Maybe we should have one counter that is incremented on v-start signal,
>>>>> and another counter that is incremented uppon some other signal?
>>>>>
>>>>
>>>> We're talking about a hard IRQ. I can't imagine the interrupt handler
>>>> being delayed for a time close to a full frame interval (~16ms for 60
>>>> fps) to trigger such scenario.
>>>>
>>>>>>
>>>>>>> +
>>>>>>> +        writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>>>>>>
>>>>>> Do you need to clear all interrupt bits individually, can't you write
>>>>>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
>>>>>> all in one go ?
>>>>>>
>>>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>>> +        if (isp_mis_tmp & CIF_ISP_V_START)
>>>>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>>>>>>> +                 isp_mis_tmp);
>>>>>>
>>>>>> This require some explanation. It looks like a naive way to protect
>>>>>> against something, but I think it could trigger under normal
>>>>>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
>>>>>> Same for the similar constructs below.
>>>>>>
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>>>>>>> +        /* Clear pic_size_error */
>>>>>>> +        writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>>>>>>> +        isp_err = readl(base + CIF_ISP_ERR);
>>>>>>> +        v4l2_err(&dev->v4l2_dev,
>>>>>>> +             "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>>>>>>
>>>>>> What does this mean ?
>>>>>>
>>>>>>> +        writel(isp_err, base + CIF_ISP_ERR_CLR);
>>>>>>> +    } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>>>>>>
>>>>>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
>>>>>>
>>>>>>> +        /* Clear data_loss */
>>>>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>>>> +        v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>>>>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    /* sampled input frame is complete */
>>>>>>> +    if (isp_mis & CIF_ISP_FRAME_IN) {
>>>>>>> +        writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>>>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>>>>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>>>>>>> +                 isp_mis_tmp);
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    /* frame was completely put out */
>>>>>>
>>>>>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
>>>>>> ? The two comments could do with a bit of brush up, and I think the
>>>>>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
>>>>>> action.
>>>>>
>>>>> Those two oneline comments are just copy-paste from the datasheet.
>>>>>
>>>>> ""
>>>>> 5 MIS_FRAME_IN sampled input frame is complete
>>>>> 1 MIS_FRAME frame was completely put out
>>>>> ""
>>>>>
>>>>> Unfrotunately, the datasheet does not add any further explanation about those signals.
>>>>>
>>>>>
>>>>
>>>> My loose recollection is that the former is signaled when then frame
>>>> is fully input to the ISP and the latter when the ISP completes
>>>> outputting the frame to the next block in the pipeline, but someone
>>>> would need to verify this, for example by printing timestamps for all
>>>> the various interrupts.
>>>>
>>>>>>
>>>>>>> +    if (isp_mis & CIF_ISP_FRAME) {
>>>>>>> +        u32 isp_ris = 0;
>>>>>>
>>>>>> No need to initialise this to 0.
>>>>>>
>>>>>>> +        /* Clear Frame In (ISP) */
>>>>>>> +        writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>>>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME)
>>>>>>> +            v4l2_err(&dev->v4l2_dev,
>>>>>>> +                 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>>>>>>> +
>>>>>>> +        isp_ris = readl(base + CIF_ISP_RIS);
>>>>>>> +        if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>>>>>>> +                   CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>>>>>>> +            rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>>>>>>
>>>>>> Is there a guarantee that the statistics will be fully written out
>>>>>> before the video frame itself ? And doesn't this test if any of the
>>>>>> statistics is complete, not all of them ? I think the logic is wrong, it
>>>>>
>>>>> The datasheet does not add any explanation of what is expected to come first.
>>>>> Should we wait until all statistics measurements are done? In the struct
>>>>> sent to userspace there is a bitmaks for which of the statistics are read.
>>>>> I think that if only part of the statistics are ready, we can already send the once
>>>>> that are ready to userspace.
>>>>>
>>>>
>>>> If we look further into the code, rkisp1_stats_isr() checks the
>>>> interrupt status mask passed to it and reads out only the parameters
>>>> with indicated completion. The statistics metadata buffer format
>>>> includes a bit mask which tells the userspace which measurements are
>>>> available.
>>>>
>>>> However, I think I've spotted a bug there. At the beginning of
>>>> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
>>>> regardless of the mask used later to decide which readouts need to be
>>>> done. This could mean that with an unfortunate timing, some measurements
>>>> would be lost. So at least the code should be fixed to only clear the
>>>> interrupts bits really handled.
>>>
>>> I'll fix that
>>
>> I actually don't think this is a bug. The statistics interrupts are not
>> enabled and are read from the raw interrupts register. This means
>> that if we missed a statistics for the current frame and we don't reset it
>> then we will read it only when the next frame comes out, so it will be
>> wrongly set as statistics for the next frame although it is actually for the
>> current frame.
> 
> Yes, I noticed that the driver attempts to reduce the number of
> interrupts by assuming that the ISP statistics can be read after the
> MIS_FRAME interrupt. However, in this case, I don't think we can ever
> miss statistics for a frame (unless the system is broken and has
> unacceptable interrupt latencies) nor the unfortunate timing I
> suggested before could ever take place.

So we actually don't even need the `meas_type` bitmask that tells which
statistics are in in the struct. Should I send a patch removing it?
Maybe just to be on the safe side I can add a WARNING in case not all
statistics are ready or or at least a debugfs variable.

Thanks,
Dafna

> 
> Best regards,
> Tomasz
> 
>>
>> Thanks,
>> Dafna
>>
>>>
>>>>
>>>> As for whether to send separate buffers for each measurement, I guess
>>>> it's not a bad thing to let the userspace access the ones available
>>>> earlier. Now I only don't recall why we decided to put all the
>>>> measurements into one metadata structure, rather than splitting the 4
>>>> into their own structures and buffer queues...
>>>
>>> Is it possible to have several queues to the same video node?
>>>
>>>>
>>>>>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
>>>>>> own. It's hard to tell for sure without extra information though (for
>>>>>> instance why are the stats-related bits read from CIF_ISP_RIS, when
>>>>>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
>>>>>> validated, and most probably fixed. Care should be taken to keep
>>>>>> synchronisation of sequence number between the different queues.
>>>>>
>>>>> I see that the capture buffers are done before incrementing the frame_sequence with
>>>>> the following explanation:
>>>>>
>>>>>      /*
>>>>>            * Call rkisp1_capture_isr() first to handle the frame that
>>>>>            * potentially completed using the current frame_sequence number before
>>>>>            * it is potentially incremented by rkisp1_isp_isr() in the vertical
>>>>>            * sync.
>>>>>            */
>>>>>
>>>>> I think reading the stats/params should also be done before calling rkisp1_capture_isr
>>>>> for the same reason. (so to match the correct frame_sequence)
>>>>
>>>> My recollection of the sequence of interrupts in this hardware is like
>>>> this:
>>>>
>>>> CIF_ISP_V_START (frame 0)
>>>>     CIF_ISP_FRAME_IN (frame 0)
>>>>       CIF_ISP_FRAME (frame 0)
>>>>         CIF_ISP_AWB_DONE
>>>>         CIF_ISP_AFM_FIN
>>>>         CIF_ISP_EXP_END
>>>>         CIF_ISP_HIST_MEASURE_RDY
>>>>         CIF_MI_FRAME*
>>>>         CIF_ISP_V_START (frame 1)
>>>>           CIF_ISP_FRAME_IN (frame 1)
>>>>             CIF_ISP_FRAME (frame 1)
>>>>               ...
>>>>
>>>> where the interrupts at the same indentation level can happen
>>>> independently of each other. Again, someone would have to verify this.
>>>
>>> I wrote this patch to print the interrupts and the time difference between interrupts:
>>> https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5
>>>
>>> I got this output: http://ix.io/2tl8,
>>> there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
>>> (FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.
>>>
>>> I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
>>> we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
>>> I think that for that we need a code that keep track of the previous interrupt.
>>>
>>> Thanks,
>>> Dafna
>>>
>>>
>>>>
>>>> Best regards,
>>>> Tomasz
>>>>
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

* Re: [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver
  2020-08-07 16:08               ` Dafna Hirschfeld
@ 2020-08-13  6:17                 ` Dafna Hirschfeld
  0 siblings, 0 replies; 76+ messages in thread
From: Dafna Hirschfeld @ 2020-08-13  6:17 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Eddie Cai, Heiko Stübner, Laurent Pinchart, kernel,
	Joerg Roedel, 钟以崇,
	open list:ARM/Rockchip SoC...,
	Hans Verkuil, Allon Huang, Shunqian Zheng,
	Linux Media Mailing List, linux-devicetree, Jacob Chen, Jeffy,
	Helen Koike, Mauro Carvalho Chehab, Ezequiel Garcia,
	linux-arm-kernel, Chen Jacob, Linux Kernel Mailing List,
	list@263.net:IOMMU DRIVERS, Sakari Ailus



Am 07.08.20 um 18:08 schrieb Dafna Hirschfeld:
> Hi
> 
> Am 06.08.20 um 14:22 schrieb Tomasz Figa:
>> On Thu, Aug 6, 2020 at 11:21 AM Dafna Hirschfeld
>> <dafna.hirschfeld@collabora.com> wrote:
>>>
>>>
>>>
>>> Am 05.08.20 um 23:10 schrieb Dafna Hirschfeld:
>>>> Hi
>>>>
>>>> On 22.07.20 17:24, Tomasz Figa wrote:
>>>>> Hi Dafna,
>>>>>
>>>>> On Sat, Jul 11, 2020 at 01:04:31PM +0200, Dafna Hirschfeld wrote:
>>>>>> Hi Laurent,
>>>>>>
>>>>>> On 16.08.19 02:13, Laurent Pinchart wrote:
>>>>>>> Hello Helen,
>>>>>>>
>>>>>>> Thank you for the patch.
>>>>>>>
>>>>>>> On Tue, Jul 30, 2019 at 03:42:47PM -0300, Helen Koike wrote:
>>>>> [snip]
>>>>>>>> +static void rkisp1_isp_queue_event_sof(struct rkisp1_isp_subdev *isp)
>>>>>>>> +{
>>>>>>>> +    struct v4l2_event event = {
>>>>>>>> +        .type = V4L2_EVENT_FRAME_SYNC,
>>>>>>>> +        .u.frame_sync.frame_sequence =
>>>>>>>> +            atomic_inc_return(&isp->frm_sync_seq) - 1,
>>>>>>>
>>>>>>> I would move the increment to the caller, hiding it in this function is
>>>>>>> error-prone (and if you look at the caller I'm pointing out one possible
>>>>>>> error :-)).
>>>>>>>
>>>>>>> In general usage of frm_sync_seq through the driver seems to be very
>>>>>>> race-prone. It's read in various IRQ handling functions, all coming from
>>>>>>> the same IRQ, so that part is fine (and wouldn't require an atomic
>>>>>>> variable), but when read from the buffer queue handlers I really get a
>>>>>>> red light flashing in my head. I'll try to investigate more when
>>>>>>> reviewing the next patches.
>>>>>>
>>>>>> I see that the only place were 'frame_sequence' is read outside of the irq
>>>>>> handlers is in the capture in 'rkisp1_vb2_buf_queue':
>>>>>>
>>>>>>      /*
>>>>>>            * If there's no next buffer assigned, queue this buffer directly
>>>>>>            * as the next buffer, and update the memory interface.
>>>>>>            */
>>>>>>           if (cap->is_streaming && !cap->buf.next &&
>>>>>>               atomic_read(&cap->rkisp1->isp.frame_sequence) == -1) {
>>>>>>                   cap->buf.next = ispbuf;
>>>>>>                   rkisp1_set_next_buf(cap);
>>>>>>           } else {
>>>>>>                   list_add_tail(&ispbuf->queue, &cap->buf.queue);
>>>>>>           }
>>>>>> This "if" condition seems very specific, a case where we already stream but v-start was not yet received.
>>>>>> I think it is possible to remove the test 'atomic_read(&cap->rkisp1->isp.frame_sequence) == -1'
>>>>>> from the above condition so that the next buffer is updated in case it is null not just before the first
>>>>>> v-start signal.
>>>>>>
>>>>>
>>>>> We don't have this special case in the Chrome OS code.
>>>>>
>>>>> I suppose it would make it possible to resume the capture 1 frame
>>>>> earlier after a queue underrun, as otherwise the new buffer would be
>>>>> only programmed after the next frame start interrupt and used for the
>>>>> next-next frame.  However, it's racy, because programming of the buffer
>>>>> addresses is not atomic and could end up with the hardware using few
>>>>> plane addresses from the new buffer and few from the dummy buffer.
>>>>>
>>>>> Given that and also the fact that a queue underrun is a very special
>>>>> case, where the system was already having problems catching up, I'd just
>>>>> remove this special case.
>>>>>
>>>>> [snip]
>>>>>>>> +void rkisp1_isp_isr(unsigned int isp_mis, struct rkisp1_device *dev)
>>>>>>>> +{
>>>>>>>> +    void __iomem *base = dev->base_addr;
>>>>>>>> +    unsigned int isp_mis_tmp = 0;
>>>>>>>
>>>>>>> _tmp are never good names :-S
>>>>>>>
>>>>>>>> +    unsigned int isp_err = 0;
>>>>>>>
>>>>>>> Neither of these variable need to be initialised to 0.
>>>>>>>
>>>>>>>> +
>>>>>>>> +    /* start edge of v_sync */
>>>>>>>> +    if (isp_mis & CIF_ISP_V_START) {
>>>>>>>> +        rkisp1_isp_queue_event_sof(&dev->isp_sdev);
>>>>>>>
>>>>>>> This will increment the frame sequence number. What if the interrupt is
>>>>>>> slightly delayed and the next frame starts before we get a change to
>>>>>>> copy the sequence number to the buffers (before they will complete
>>>>>>> below) ?
>>>>>>
>>>>>> Do you mean that we get two sequental v-start signals and then the next
>>>>>> frame-end signal in MI_MIS belongs to the first v-start signal of the two?
>>>>>> How can this be solved? I wonder if any v-start signal has a later signal
>>>>>> that correspond to the same frame so that we can follow it?
>>>>>>
>>>>>> Maybe we should have one counter that is incremented on v-start signal,
>>>>>> and another counter that is incremented uppon some other signal?
>>>>>>
>>>>>
>>>>> We're talking about a hard IRQ. I can't imagine the interrupt handler
>>>>> being delayed for a time close to a full frame interval (~16ms for 60
>>>>> fps) to trigger such scenario.
>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> +        writel(CIF_ISP_V_START, base + CIF_ISP_ICR);
>>>>>>>
>>>>>>> Do you need to clear all interrupt bits individually, can't you write
>>>>>>> isp_mis to CIF_ISP_ICR at the beginning of the function to clear them
>>>>>>> all in one go ?
>>>>>>>
>>>>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>>>> +        if (isp_mis_tmp & CIF_ISP_V_START)
>>>>>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr v_statr err: 0x%x\n",
>>>>>>>> +                 isp_mis_tmp);
>>>>>>>
>>>>>>> This require some explanation. It looks like a naive way to protect
>>>>>>> against something, but I think it could trigger under normal
>>>>>>> circumstances if IRQ handling is delayed, and wouldn't do much anyway.
>>>>>>> Same for the similar constructs below.
>>>>>>>
>>>>>>>> +    }
>>>>>>>> +
>>>>>>>> +    if ((isp_mis & CIF_ISP_PIC_SIZE_ERROR)) {
>>>>>>>> +        /* Clear pic_size_error */
>>>>>>>> +        writel(CIF_ISP_PIC_SIZE_ERROR, base + CIF_ISP_ICR);
>>>>>>>> +        isp_err = readl(base + CIF_ISP_ERR);
>>>>>>>> +        v4l2_err(&dev->v4l2_dev,
>>>>>>>> +             "CIF_ISP_PIC_SIZE_ERROR (0x%08x)", isp_err);
>>>>>>>
>>>>>>> What does this mean ?
>>>>>>>
>>>>>>>> +        writel(isp_err, base + CIF_ISP_ERR_CLR);
>>>>>>>> +    } else if ((isp_mis & CIF_ISP_DATA_LOSS)) {
>>>>>>>
>>>>>>> Are CIF_ISP_PIC_SIZE_ERROR and CIF_ISP_DATA_LOSS mutually exclusive ?
>>>>>>>
>>>>>>>> +        /* Clear data_loss */
>>>>>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>>>>> +        v4l2_err(&dev->v4l2_dev, "CIF_ISP_DATA_LOSS\n");
>>>>>>>> +        writel(CIF_ISP_DATA_LOSS, base + CIF_ISP_ICR);
>>>>>>>> +    }
>>>>>>>> +
>>>>>>>> +    /* sampled input frame is complete */
>>>>>>>> +    if (isp_mis & CIF_ISP_FRAME_IN) {
>>>>>>>> +        writel(CIF_ISP_FRAME_IN, base + CIF_ISP_ICR);
>>>>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME_IN)
>>>>>>>> +            v4l2_err(&dev->v4l2_dev, "isp icr frame_in err: 0x%x\n",
>>>>>>>> +                 isp_mis_tmp);
>>>>>>>> +    }
>>>>>>>> +
>>>>>>>> +    /* frame was completely put out */
>>>>>>>
>>>>>>> "put out" ? :-) What's the difference between ISP_FRAME_IN and ISP_FRAME
>>>>>>> ? The two comments could do with a bit of brush up, and I think the
>>>>>>> ISP_FRAME_IN interrupt could be disabled as it doesn't perform any
>>>>>>> action.
>>>>>>
>>>>>> Those two oneline comments are just copy-paste from the datasheet.
>>>>>>
>>>>>> ""
>>>>>> 5 MIS_FRAME_IN sampled input frame is complete
>>>>>> 1 MIS_FRAME frame was completely put out
>>>>>> ""
>>>>>>
>>>>>> Unfrotunately, the datasheet does not add any further explanation about those signals.
>>>>>>
>>>>>>
>>>>>
>>>>> My loose recollection is that the former is signaled when then frame
>>>>> is fully input to the ISP and the latter when the ISP completes
>>>>> outputting the frame to the next block in the pipeline, but someone
>>>>> would need to verify this, for example by printing timestamps for all
>>>>> the various interrupts.
>>>>>
>>>>>>>
>>>>>>>> +    if (isp_mis & CIF_ISP_FRAME) {
>>>>>>>> +        u32 isp_ris = 0;
>>>>>>>
>>>>>>> No need to initialise this to 0.
>>>>>>>
>>>>>>>> +        /* Clear Frame In (ISP) */
>>>>>>>> +        writel(CIF_ISP_FRAME, base + CIF_ISP_ICR);
>>>>>>>> +        isp_mis_tmp = readl(base + CIF_ISP_MIS);
>>>>>>>> +        if (isp_mis_tmp & CIF_ISP_FRAME)
>>>>>>>> +            v4l2_err(&dev->v4l2_dev,
>>>>>>>> +                 "isp icr frame end err: 0x%x\n", isp_mis_tmp);
>>>>>>>> +
>>>>>>>> +        isp_ris = readl(base + CIF_ISP_RIS);
>>>>>>>> +        if (isp_ris & (CIF_ISP_AWB_DONE | CIF_ISP_AFM_FIN |
>>>>>>>> +                   CIF_ISP_EXP_END | CIF_ISP_HIST_MEASURE_RDY))
>>>>>>>> +            rkisp1_stats_isr(&dev->stats_vdev, isp_ris);
>>>>>>>
>>>>>>> Is there a guarantee that the statistics will be fully written out
>>>>>>> before the video frame itself ? And doesn't this test if any of the
>>>>>>> statistics is complete, not all of them ? I think the logic is wrong, it
>>>>>>
>>>>>> The datasheet does not add any explanation of what is expected to come first.
>>>>>> Should we wait until all statistics measurements are done? In the struct
>>>>>> sent to userspace there is a bitmaks for which of the statistics are read.
>>>>>> I think that if only part of the statistics are ready, we can already send the once
>>>>>> that are ready to userspace.
>>>>>>
>>>>>
>>>>> If we look further into the code, rkisp1_stats_isr() checks the
>>>>> interrupt status mask passed to it and reads out only the parameters
>>>>> with indicated completion. The statistics metadata buffer format
>>>>> includes a bit mask which tells the userspace which measurements are
>>>>> available.
>>>>>
>>>>> However, I think I've spotted a bug there. At the beginning of
>>>>> rkisp1_stats_isr(), all the 4 interrupt status bits are cleared,
>>>>> regardless of the mask used later to decide which readouts need to be
>>>>> done. This could mean that with an unfortunate timing, some measurements
>>>>> would be lost. So at least the code should be fixed to only clear the
>>>>> interrupts bits really handled.
>>>>
>>>> I'll fix that
>>>
>>> I actually don't think this is a bug. The statistics interrupts are not
>>> enabled and are read from the raw interrupts register. This means
>>> that if we missed a statistics for the current frame and we don't reset it
>>> then we will read it only when the next frame comes out, so it will be
>>> wrongly set as statistics for the next frame although it is actually for the
>>> current frame.
>>
>> Yes, I noticed that the driver attempts to reduce the number of
>> interrupts by assuming that the ISP statistics can be read after the
>> MIS_FRAME interrupt. However, in this case, I don't think we can ever
>> miss statistics for a frame (unless the system is broken and has
>> unacceptable interrupt latencies) nor the unfortunate timing I
>> suggested before could ever take place.
> 
> So we actually don't even need the `meas_type` bitmask that tells which
> statistics are in in the struct. Should I send a patch removing it?
> Maybe just to be on the safe side I can add a WARNING in case not all
> statistics are ready or or at least a debugfs variable.

I see that userspace can disable statistics through the params configuration
so we need that bitmask after all,

Thanks,
Dafna


> 
> Thanks,
> Dafna
> 
>>
>> Best regards,
>> Tomasz
>>
>>>
>>> Thanks,
>>> Dafna
>>>
>>>>
>>>>>
>>>>> As for whether to send separate buffers for each measurement, I guess
>>>>> it's not a bad thing to let the userspace access the ones available
>>>>> earlier. Now I only don't recall why we decided to put all the
>>>>> measurements into one metadata structure, rather than splitting the 4
>>>>> into their own structures and buffer queues...
>>>>
>>>> Is it possible to have several queues to the same video node?
>>>>
>>>>>
>>>>>>> seems it should be moved out of the CIF_ISP_FRAME test, to a test of its
>>>>>>> own. It's hard to tell for sure without extra information though (for
>>>>>>> instance why are the stats-related bits read from CIF_ISP_RIS, when
>>>>>>> they seem to be documented as valid in CIF_ISP_ISR), but this should be
>>>>>>> validated, and most probably fixed. Care should be taken to keep
>>>>>>> synchronisation of sequence number between the different queues.
>>>>>>
>>>>>> I see that the capture buffers are done before incrementing the frame_sequence with
>>>>>> the following explanation:
>>>>>>
>>>>>>      /*
>>>>>>            * Call rkisp1_capture_isr() first to handle the frame that
>>>>>>            * potentially completed using the current frame_sequence number before
>>>>>>            * it is potentially incremented by rkisp1_isp_isr() in the vertical
>>>>>>            * sync.
>>>>>>            */
>>>>>>
>>>>>> I think reading the stats/params should also be done before calling rkisp1_capture_isr
>>>>>> for the same reason. (so to match the correct frame_sequence)
>>>>>
>>>>> My recollection of the sequence of interrupts in this hardware is like
>>>>> this:
>>>>>
>>>>> CIF_ISP_V_START (frame 0)
>>>>>     CIF_ISP_FRAME_IN (frame 0)
>>>>>       CIF_ISP_FRAME (frame 0)
>>>>>         CIF_ISP_AWB_DONE
>>>>>         CIF_ISP_AFM_FIN
>>>>>         CIF_ISP_EXP_END
>>>>>         CIF_ISP_HIST_MEASURE_RDY
>>>>>         CIF_MI_FRAME*
>>>>>         CIF_ISP_V_START (frame 1)
>>>>>           CIF_ISP_FRAME_IN (frame 1)
>>>>>             CIF_ISP_FRAME (frame 1)
>>>>>               ...
>>>>>
>>>>> where the interrupts at the same indentation level can happen
>>>>> independently of each other. Again, someone would have to verify this.
>>>>
>>>> I wrote this patch to print the interrupts and the time difference between interrupts:
>>>> https://gitlab.collabora.com/dafna/linux/-/commit/9b9c5ddc2f06a6b87d2c1b210219f69de83296c5
>>>>
>>>> I got this output: http://ix.io/2tl8,
>>>> there is a repeating pattern where only v-start interrupt is sent, indicated by the prints "isp mis 0x00000040" then about 23 milisec later are the other interrupts
>>>> (FRAME_IN, FRAME, MI_FRAME* ) and about 10 milisec the v-start interrupt again.
>>>>
>>>> I am still not sure why the mi_frame interrupt should be handled first. If it happen for example that all the interrupts arrive at once, how can
>>>> we know that the MI_FRAME interrupt relates to the previous v-start interrupt and not the current one?
>>>> I think that for that we need a code that keep track of the previous interrupt.
>>>>
>>>> Thanks,
>>>> Dafna
>>>>
>>>>
>>>>>
>>>>> Best regards,
>>>>> Tomasz
>>>>>
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 76+ messages in thread

end of thread, other threads:[~2020-08-13  6:20 UTC | newest]

Thread overview: 76+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-30 18:42 [PATCH v8 00/14] Rockchip ISP1 Driver Helen Koike
2019-07-30 18:42 ` [PATCH v8 01/14] media: videodev2.h, v4l2-ioctl: add rkisp1 meta buffer format Helen Koike
2019-08-15 13:30   ` Laurent Pinchart
2019-07-30 18:42 ` [PATCH v8 02/14] media: doc: add document for " Helen Koike
2019-08-07 13:09   ` Sakari Ailus
2019-08-15 13:51   ` Laurent Pinchart
2019-07-30 18:42 ` [PATCH v8 03/14] media: rkisp1: Add user space ABI definitions Helen Koike
2019-08-15 18:46   ` Laurent Pinchart
2020-07-10 12:59     ` Dafna Hirschfeld
2020-07-10 13:36       ` Laurent Pinchart
2020-07-10 14:30         ` Dafna Hirschfeld
2019-07-30 18:42 ` [PATCH v8 04/14] media: rkisp1: add Rockchip MIPI Synopsys DPHY driver Helen Koike
2019-08-07 13:05   ` Sakari Ailus
2019-08-07 13:37     ` Helen Koike
2019-08-15 17:54       ` Laurent Pinchart
2019-08-15 18:26         ` Heiko Stübner
2019-08-21 21:46         ` Helen Koike
2019-08-22  2:32           ` Laurent Pinchart
2019-07-30 18:42 ` [PATCH v8 05/14] media: rkisp1: add Rockchip ISP1 subdev driver Helen Koike
2019-08-06 18:51   ` Helen Koike
2019-08-07 10:39     ` Hans Verkuil
2019-08-15 19:35       ` Laurent Pinchart
2020-03-25  6:34         ` Dafna Hirschfeld
2020-03-25  7:11           ` Laurent Pinchart
2020-03-25  8:51             ` Dafna Hirschfeld
2020-03-25  9:11               ` Laurent Pinchart
2019-08-08  9:14   ` Sakari Ailus
2019-08-15  0:58     ` Helen Koike
2019-08-15  8:24       ` Sakari Ailus
2019-08-15 10:29         ` Tomasz Figa
2019-08-15 10:45           ` Sakari Ailus
2019-08-15 13:17         ` Sakari Ailus
2020-01-31 19:38           ` Dafna Hirschfeld
2020-02-12 21:13             ` Sakari Ailus
2020-02-13 12:50               ` Dafna Hirschfeld
2019-08-16  0:13   ` Laurent Pinchart
2020-07-11 11:04     ` Dafna Hirschfeld
2020-07-17  7:46       ` Dafna Hirschfeld
2020-07-22 16:01         ` Tomasz Figa
2020-07-22 15:24       ` Tomasz Figa
2020-07-22 16:30         ` Laurent Pinchart
2020-07-22 17:12           ` Tomasz Figa
2020-07-22 17:50             ` Laurent Pinchart
2020-08-05 21:10         ` Dafna Hirschfeld
2020-08-06  9:21           ` Dafna Hirschfeld
2020-08-06 12:22             ` Tomasz Figa
2020-08-07 16:08               ` Dafna Hirschfeld
2020-08-13  6:17                 ` Dafna Hirschfeld
2020-08-06 12:08           ` Tomasz Figa
2020-08-07 16:02             ` Dafna Hirschfeld
2019-07-30 18:42 ` [PATCH v8 06/14] media: rkisp1: add ISP1 statistics driver Helen Koike
2019-08-08  9:37   ` Sakari Ailus
2019-07-30 18:42 ` [PATCH v8 07/14] media: rkisp1: add ISP1 params driver Helen Koike
2019-07-30 18:42 ` [PATCH v8 09/14] media: rkisp1: add rockchip isp1 core driver Helen Koike
2019-08-07 15:27   ` Sakari Ailus
2019-08-08 21:59     ` Helen Koike
2019-08-09 12:05       ` Sakari Ailus
2019-08-07 15:36   ` Sakari Ailus
2019-07-30 18:42 ` [PATCH v8 10/14] dt-bindings: Document the Rockchip ISP1 bindings Helen Koike
2019-08-16  0:21   ` Laurent Pinchart
2019-07-30 18:42 ` [PATCH v8 11/14] dt-bindings: Document the Rockchip MIPI RX D-PHY bindings Helen Koike
2019-08-15 18:14   ` Laurent Pinchart
2019-07-30 18:42 ` [PATCH v8 12/14] arm64: dts: rockchip: add isp0 node for rk3399 Helen Koike
2019-07-30 18:42 ` [PATCH v8 13/14] arm64: dts: rockchip: add rx0 mipi-phy " Helen Koike
2019-07-30 18:42 ` [PATCH v8 14/14] MAINTAINERS: add entry for Rockchip ISP1 driver Helen Koike
2019-08-15 13:56   ` Laurent Pinchart
2019-07-30 20:15 ` [PATCH v8 00/14] Rockchip ISP1 Driver Hans Verkuil
2019-07-30 20:50   ` Helen Koike
2019-07-31  0:08     ` Helen Koike
2019-07-31  4:29       ` Hans Verkuil
2019-07-31  4:33         ` Hans Verkuil
2019-07-31  4:55           ` Hans Verkuil
2019-07-31 14:42             ` Helen Koike
2019-08-07 15:37 ` Sakari Ailus
2019-08-07 17:57   ` Helen Koike
2019-08-09 18:45 ` Manivannan Sadhasivam

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).