* [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
@ 2020-11-10 17:40 Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA Jacopo Mondi
` (5 more replies)
0 siblings, 6 replies; 31+ messages in thread
From: Jacopo Mondi @ 2020-11-10 17:40 UTC (permalink / raw)
To: linux-media
Cc: Jacopo Mondi, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
sakari.ailus, nsaenzjulienne, mchehab+huawei
Hello,
new iteration following
v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
Major changelog:
- Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
- Address Hans' comments on queue setup and metadata format handling function
- s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
Hans and rebase on Dafna's fixed metadata format patch
- Add a TODO file to explain why the driver is in staging.
- Conditionally register the unicam-embedded video device node on the presence
of the sensor's metadata source pad.
The media graph, when connected to a sensor that does not report metadata
looks like:
Media controller API version 5.10.0
Media device information
------------------------
driver unicam
model unicam
serial
bus info platform:fe801000.csi
hw revision 0x0
driver version 5.10.0
Device topology
- entity 1: ov5647 10-0036 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev0
pad0: Source
[fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
crop.bounds:(16,16)/2592x1944
crop:(32,16)/2560x1920]
-> "unicam-image":0 [ENABLED,IMMUTABLE]
- entity 3: unicam-image (1 pad, 1 link)
type Node subtype V4L flags 1
device node name /dev/video0
pad0: Sink
<- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
If the sensor reports an additional metadata pad:
Media controller API version 5.10.0
Media device information
------------------------
driver unicam
model unicam
serial
bus info platform:fe801000.csi
hw revision 0x0
driver version 5.10.0
Device topology
- entity 1: imx219 10-0010 (2 pads, 2 links)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev0
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
crop:(0,0)/3280x2464]
-> "unicam-image":0 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:unknown/16384x1 field:none
crop:(0,0)/3280x2464]
-> "unicam-embedded":0 [ENABLED,IMMUTABLE]
- entity 4: unicam-image (1 pad, 1 link)
type Node subtype V4L flags 1
device node name /dev/video0
pad0: Sink
<- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
- entity 10: unicam-embedded (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video1
pad0: Sink
<- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
Conditionally registering the metadata video node allows to simplify the
code in the driver as well, removing the 'sensor_embedded_data' flag.
An additional note: this version will break the libcamera pipeline handler
which assume the unicam-embedded video device node to always be there.
From Dave's reply to Dafna's comments I get instead that for the existing
applications ecosystem, having the metadata node not registered if the sensor
does not support it is not an issue.
The v4l2-compliance output is reported below, with one test failing, again
on the selection API, and again I'm not sure about the root cause. Seems like
the compliance tool complains because a list of discrete formats is reported and
the selection APIs are supported at the same time. Seems unlikely though.
-------------------------------------------------------------------------------
v4l2-compliance 1.21.0-4679, 32 bits, 32-bit time_t
v4l2-compliance SHA: 225c6c2a17be 2020-10-30 15:13:07
Compliance test for unicam device /dev/media0:
Media Driver Info:
Driver name : unicam
Model : unicam
Serial :
Bus info : platform:fe801000.csi
Media version : 5.10.0
Hardware revision: 0x00000000 (0)
Driver version : 5.10.0
Required ioctls:
test MEDIA_IOC_DEVICE_INFO: OK
Allow for multiple opens:
test second /dev/media0 open: OK
test MEDIA_IOC_DEVICE_INFO: OK
test for unlimited opens: OK
Media Controller ioctls:
test MEDIA_IOC_G_TOPOLOGY: OK
Entities: 2 Interfaces: 2 Pads: 2 Links: 3
test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
test MEDIA_IOC_SETUP_LINK: OK
test invalid ioctls: OK
Total for unicam device /dev/media0: 8, Succeeded: 8, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for unicam device /dev/video0:
Driver Info:
Driver name : unicam
Card type : unicam
Bus info : platform:fe801000.csi
Driver version : 5.10.0
Capabilities : 0x85a00001
Video Capture
Metadata Capture
Read/Write
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x05200001
Video Capture
Read/Write
Streaming
Extended Pix Format
Media Driver Info:
Driver name : unicam
Model : unicam
Serial :
Bus info : platform:fe801000.csi
Media version : 5.10.0
Hardware revision: 0x00000000 (0)
Driver version : 5.10.0
Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:
ID : 0x00000003 (3)
Name : unicam-image
Function : V4L2 I/O
Flags : default
Pad 0x01000004 : 0: Sink
Link 0x02000007: from remote pad 0x1000002 of entity 'ov5647 10-0036': Data, Enabled, Immutable
Required ioctls:
test MC information (see 'Media Driver Info' above): OK
test VIDIOC_QUERYCAP: OK
Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
test invalid ioctls: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 12 Private Controls: 0
Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
fail: v4l2-test-formats.cpp(1505): node->frmsizes_count[pixfmt] > 1
test Cropping: FAIL
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported)
Test input 0:
Streaming ioctls:
test read/write: OK
test blocking wait: OK
test MMAP (no poll): OK
test MMAP (select): OK
test MMAP (epoll): OK
test USERPTR (no poll): OK (Not Supported)
test USERPTR (select): OK (Not Supported)
test DMABUF: Cannot test, specify --expbuf-device
Stream using all formats:
test MMAP for Format pBAA, Frame Size 2592x1944:
Crop 2592x1944@16x16, Stride 3264, Field None: OK
Crop 2592x1944@16x16, Stride 3328, Field None: OK
test MMAP for Format pBAA, Frame Size 1920x1080:
Crop 1928x1080@364x450, Stride 2400, Field None: OK
Crop 1928x1080@364x450, Stride 2464, Field None: OK
test MMAP for Format pBAA, Frame Size 1296x972:
Crop 2592x1944@16x16, Stride 1632, Field None: OK
Crop 2592x1944@16x16, Stride 1696, Field None: OK
test MMAP for Format pBAA, Frame Size 640x480:
Crop 2560x1920@32x16, Stride 800, Field None: OK
Crop 2560x1920@32x16, Stride 864, Field None: OK
test MMAP for Format BG10, Frame Size 2592x1944:
Crop 2592x1944@16x16, Stride 5184, Field None: OK
Crop 2592x1944@16x16, Stride 5248, Field None: OK
test MMAP for Format BG10, Frame Size 1920x1080:
Crop 1928x1080@364x450, Stride 3840, Field None: OK
Crop 1928x1080@364x450, Stride 3904, Field None: OK
test MMAP for Format BG10, Frame Size 1296x972:
Crop 2592x1944@16x16, Stride 2592, Field None: OK
Crop 2592x1944@16x16, Stride 2656, Field None: OK
test MMAP for Format BG10, Frame Size 640x480:
Crop 2560x1920@32x16, Stride 1280, Field None: OK
Crop 2560x1920@32x16, Stride 1344, Field None: OK
Total for unicam device /dev/video0: 69, Succeeded: 68, Failed: 1, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for device /dev/v4l-subdev0:
Required ioctls:
Allow for multiple opens:
test second /dev/v4l-subdev0 open: OK
test for unlimited opens: OK
test invalid ioctls: OK
Debug ioctls:
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 12 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK (Not Supported)
test VIDIOC_TRY_FMT: OK (Not Supported)
test VIDIOC_S_FMT: OK (Not Supported)
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
test VIDIOC_EXPBUF: OK (Not Supported)
test Requests: OK (Not Supported)
Total for device /dev/v4l-subdev0: 41, Succeeded: 41, Failed: 0, Warnings: 0
Grand Total for unicam device /dev/media0: 118, Succeeded: 117, Failed: 1, Warnings: 0
-------------------------------------------------------------------------------
Tested by capturing frames from RPi Camera Module v1 (ov5647) and v2 (imx219)
in 10-bit formats and 8-bit formats (imx219 only).
Image are captured in a format I cannot inspect (raw2rgbpnm and 7yuv do not
support the sensor's Bayern pattern), but I can spot shapes when converting to
a format with a different pattern permutation, so I assume data is there.
Thanks
j
Dave Stevenson (1):
dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
Jacopo Mondi (1):
media: bcm2835-unicam: Add TODO file
Laurent Pinchart (1):
ARM: dts: bcm2711: Add Unicam DT nodes
Naushir Patuck (2):
media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
.../bindings/media/brcm,bcm2835-unicam.yaml | 155 +
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../media/v4l/pixfmt-meta-sensor-data.rst | 24 +
.../media/v4l/subdev-formats.rst | 29 +
MAINTAINERS | 7 +
arch/arm/boot/dts/bcm2711.dtsi | 41 +
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
drivers/staging/media/Kconfig | 2 +
drivers/staging/media/Makefile | 1 +
drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
drivers/staging/media/bcm2835-unicam/Makefile | 3 +
drivers/staging/media/bcm2835-unicam/TODO | 37 +
.../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
.../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
include/uapi/linux/media-bus-format.h | 3 +
include/uapi/linux/videodev2.h | 1 +
16 files changed, 3329 insertions(+)
create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
--
2.29.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
@ 2020-11-10 17:40 ` Jacopo Mondi
2020-12-02 21:16 ` Sakari Ailus
2020-11-10 17:40 ` [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver Jacopo Mondi
` (4 subsequent siblings)
5 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-11-10 17:40 UTC (permalink / raw)
To: linux-media
Cc: Jacopo Mondi, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
sakari.ailus, nsaenzjulienne, mchehab+huawei
From: Naushir Patuck <naush@raspberrypi.com>
Add V4L2_META_FMT_CUSTOM_SENSOR_DATA format.
This new format will be used to return camera sensor embedded data.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
---
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../media/v4l/pixfmt-meta-sensor-data.rst | 24 +++++++++++++++
.../media/v4l/subdev-formats.rst | 29 +++++++++++++++++++
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
include/uapi/linux/media-bus-format.h | 3 ++
include/uapi/linux/videodev2.h | 1 +
6 files changed, 59 insertions(+)
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index fff25357fe860..b2201d1524eb6 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -15,6 +15,7 @@ These formats are used for the :ref:`metadata` interface only.
pixfmt-meta-d4xx
pixfmt-meta-intel-ipu3
pixfmt-meta-rkisp1
+ pixfmt-meta-sensor-data
pixfmt-meta-uvc
pixfmt-meta-vsp1-hgo
pixfmt-meta-vsp1-hgt
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
new file mode 100644
index 0000000000000..cc3929f595389
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
@@ -0,0 +1,24 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+
+.. _v4l2-meta-fmt-sensor-data:
+
+***********************************
+V4L2_META_FMT_SENSOR_DATA ('SENS')
+***********************************
+
+Sensor Ancillary Metadata
+
+Description
+===========
+
+This format describes ancillary data generated by a camera sensor and
+transmitted over a stream on the camera bus. Most sensor vendors have their
+own custom format for this ancillary data. Some vendors follow a generic
+CSI-2/SMIA embedded data format as described in the `CSI-2 specification.
+<https://mipi.org/specifications/csi-2>`_
+
+The size of the embedded buffer is defined as a single line with a pixel width
+specified in bytes and obtained by a call to the :c:type:`VIDIOC_SUBDEV_G_FMT`
+ioctl on the sensor sub-device. Note that this size is fixed and cannot be
+modified with a call to :c:type:`VIDIOC_SUBDEV_S_FMT`.
+
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index 7f16cbe46e5c2..99e270bbdd223 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -7926,3 +7926,32 @@ The following table lists the existing metadata formats.
both sides of the link and the bus format is a fixed
metadata format that is not configurable from userspace.
Width and height will be set to 0 for this format.
+
+.. _v4l2-mbus-sensor-data:
+
+Sensor Ancillary Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This section lists ancillary data generated by a camera sensor and
+transmitted over a stream on the camera bus.
+
+The following table lists the existing sensor ancillary metadata formats:
+
+.. _v4l2-mbus-pixelcode-sensor-metadata:
+
+.. tabularcolumns:: |p{8.0cm}|p{1.4cm}|p{7.7cm}|
+
+.. flat-table:: Sensor ancillary metadata formats
+ :header-rows: 1
+ :stub-columns: 0
+
+ * - Identifier
+ - Code
+ - Comments
+ * .. _MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA:
+
+ - MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
+ - 0x7002
+ - Sensor vendor specific ancillary metadata. Some vendors follow a generic
+ CSI-2/SMIA embedded data format as described in the `CSI-2 specification.
+ <https://mipi.org/specifications/csi-2>`_
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index f0f6906a879de..83288a00f28e4 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1403,6 +1403,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
case V4L2_META_FMT_UVC: descr = "UVC Payload Header Metadata"; break;
case V4L2_META_FMT_D4XX: descr = "Intel D4xx UVC Metadata"; break;
case V4L2_META_FMT_VIVID: descr = "Vivid Metadata"; break;
+ case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
default:
/* Compressed formats */
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 2ce3d891d3447..726557120fc03 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -164,4 +164,7 @@
*/
#define MEDIA_BUS_FMT_METADATA_FIXED 0x7001
+/* Sensor ancillary metadata formats - next is 0x7003 */
+#define MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA 0x7002
+
#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 927075fa9099e..d1ef094090a3b 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -767,6 +767,7 @@ struct v4l2_pix_format {
#define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
#define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
#define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
+#define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
/* priv field value to indicates that subsequent fields are valid. */
#define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
--
2.29.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA Jacopo Mondi
@ 2020-11-10 17:40 ` Jacopo Mondi
2020-12-02 21:20 ` Sakari Ailus
2020-11-10 17:40 ` [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface Jacopo Mondi
` (3 subsequent siblings)
5 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-11-10 17:40 UTC (permalink / raw)
To: linux-media
Cc: Jacopo Mondi, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
sakari.ailus, nsaenzjulienne, mchehab+huawei
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
Unicam) on BCM283x SoCs.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
---
.../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
1 file changed, 155 insertions(+)
create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
new file mode 100644
index 0000000000000..6ffc900e8ae8f
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
@@ -0,0 +1,155 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Broadcom BCM283x Camera Interface (Unicam)
+
+maintainers:
+ - Dave Stevenson <dave.stevenson@raspberrypi.com>
+ - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
+
+description:
+ The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
+ data from image sensors or similar devices.
+
+ The main platform using this SoC is the Raspberry Pi family of boards. On the
+ Pi the VideoCore firmware can also control this hardware block, and driving
+ it from two different processors will cause issues. To avoid this, the
+ firmware checks the device tree configuration during boot. If it finds device
+ tree nodes whose name starts with "csi" then it will stop the firmware
+ accessing the block, and it can then safely be used via the device tree
+ binding.
+
+properties:
+ compatible:
+ const: brcm,bcm2835-unicam
+
+ reg:
+ items:
+ - description: Main registers block
+ - description: Clock registers block
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ minItems: 1
+ maxItems: 2
+
+ clock-names:
+ minItems: 1
+ items:
+ - const: lp
+ - const: core
+
+ power-domains:
+ maxItems: 1
+
+ brcm,num-data-lanes:
+ description:
+ The number of data lanes supported by this Unicam instance. It may be
+ larger than the number of data lanes routed on the board, as described by
+ the data-lanes property of the endpoint.
+ allOf:
+ - $ref: "/schemas/types.yaml#/definitions/uint32"
+ - enum: [1, 2, 4]
+
+ port:
+ type: object
+ description:
+ Input port node, as described in video-interfaces.txt.
+
+ properties:
+ endpoint:
+ type: object
+
+ properties:
+ clock-lanes:
+ items:
+ - const: 0
+
+ data-lanes:
+ description:
+ Lane reordering is not supported, items shall be in order,
+ starting at 1.
+ allOf:
+ - $ref: "/schemas/types.yaml#/definitions/uint32-array"
+ - maxItems: 1
+ items:
+ minItems: 1
+ maxItems: 4
+ items:
+ - const: 1
+ - const: 2
+ - const: 3
+ - const: 4
+
+ lane-polarities:
+ description:
+ Lane inversion is not supported. If the property is specified, it
+ shall contain all 0's.
+ allOf:
+ - $ref: "/schemas/types.yaml#/definitions/uint32-array"
+ - maxItems: 1
+ items:
+ minItems: 2
+ maxItems: 5
+ items:
+ - const: 0
+ - const: 0
+ - const: 0
+ - const: 0
+ - const: 0
+
+ remote-endpoint: true
+
+ required:
+ - data-lanes
+ - remote-endpoint
+
+ additionalProperties: false
+
+ required:
+ - endpoint
+
+ additionalProperties: false
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+ - clock-names
+ - power-domains
+ - brcm,num-data-lanes
+ - port
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/bcm2835.h>
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/power/raspberrypi-power.h>
+
+ csi@7e801000 {
+ compatible = "brcm,bcm2835-unicam";
+ reg = <0x7e801000 0x800>,
+ <0x7e802004 0x4>;
+ interrupts = <2 7>;
+ clocks = <&clocks BCM2835_CLOCK_CAM1>;
+ clock-names = "lp";
+ power-domains = <&power RPI_POWER_DOMAIN_UNICAM1>;
+ brcm,num-data-lanes = <4>;
+
+ port {
+ csi1_ep: endpoint {
+ remote-endpoint = <&imx219_0>;
+ data-lanes = <1 2>;
+ };
+ };
+ };
+...
--
2.29.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver Jacopo Mondi
@ 2020-11-10 17:40 ` Jacopo Mondi
2020-12-02 21:53 ` Sakari Ailus
2020-12-02 22:35 ` Kieran Bingham
2020-11-10 17:40 ` [PATCH v4 4/5] ARM: dts: bcm2711: Add Unicam DT nodes Jacopo Mondi
` (2 subsequent siblings)
5 siblings, 2 replies; 31+ messages in thread
From: Jacopo Mondi @ 2020-11-10 17:40 UTC (permalink / raw)
To: linux-media
Cc: Jacopo Mondi, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
sakari.ailus, nsaenzjulienne, mchehab+huawei
From: Naushir Patuck <naush@raspberrypi.com>
Add a driver for the Unicam camera receiver block on BCM283x processors.
Compared to the bcm2835-camera driver present in staging, this driver
handles the Unicam block only (CSI-2 receiver), and doesn't depend on
the VC4 firmware running on the VPU.
The commit is made up of a series of changes cherry-picked from the
rpi-5.9.y branch of https://github.com/raspberrypi/linux/ at revision
commit be371a8ffc012
("media: bcm2835-unicam: change minimum number of vb2_queue buffers to 1")
with additional enhancements, forward-ported to the mainline kernel.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
---
MAINTAINERS | 7 +
drivers/staging/media/Kconfig | 2 +
drivers/staging/media/Makefile | 1 +
drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
drivers/staging/media/bcm2835-unicam/Makefile | 3 +
.../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
.../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
7 files changed, 3037 insertions(+)
create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 811db1d3ca33b..69d55ed67e1cf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3411,6 +3411,13 @@ N: bcm113*
N: bcm216*
N: kona
+BROADCOM BCM2835 CAMERA DRIVER
+M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
+L: linux-media@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
+F: drivers/staging/media/bcm2835/
+
BROADCOM BCM47XX MIPS ARCHITECTURE
M: Hauke Mehrtens <hauke@hauke-m.de>
M: Rafał Miłecki <zajec5@gmail.com>
diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
index 747c6cf1d795e..b734e882f63c9 100644
--- a/drivers/staging/media/Kconfig
+++ b/drivers/staging/media/Kconfig
@@ -46,4 +46,6 @@ source "drivers/staging/media/ipu3/Kconfig"
source "drivers/staging/media/rkisp1/Kconfig"
+source "drivers/staging/media/bcm2835-unicam/Kconfig"
+
endif
diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
index b59571826ba69..93d819ef243f1 100644
--- a/drivers/staging/media/Makefile
+++ b/drivers/staging/media/Makefile
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_VIDEO_ALLEGRO_DVT) += allegro-dvt/
obj-$(CONFIG_INTEL_ATOMISP) += atomisp/
+obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam/
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx/
obj-$(CONFIG_VIDEO_MESON_VDEC) += meson/vdec/
obj-$(CONFIG_VIDEO_OMAP4) += omap4iss/
diff --git a/drivers/staging/media/bcm2835-unicam/Kconfig b/drivers/staging/media/bcm2835-unicam/Kconfig
new file mode 100644
index 0000000000000..bd13701996509
--- /dev/null
+++ b/drivers/staging/media/bcm2835-unicam/Kconfig
@@ -0,0 +1,21 @@
+# Broadcom VideoCore4 V4L2 camera support
+
+config VIDEO_BCM2835_UNICAM
+ tristate "Broadcom BCM283x/BCM271x Unicam video capture driver"
+ depends on VIDEO_V4L2
+ depends on ARCH_BCM2835 || COMPILE_TEST
+ select VIDEO_V4L2_SUBDEV_API
+ select MEDIA_CONTROLLER
+ select VIDEOBUF2_DMA_CONTIG
+ select V4L2_FWNODE
+ help
+ Say Y here to enable support for the BCM283x/BCM271x CSI-2 receiver.
+ This is a V4L2 driver that controls the CSI-2 receiver directly,
+ independently from the VC4 firmware.
+ This driver is mutually exclusive with the use of bcm2835-camera. The
+ firmware will disable all access to the peripheral from within the
+ firmware if it finds a DT node using it, and bcm2835-camera will
+ therefore fail to probe.
+
+ To compile this driver as a module, choose M here. The module will be
+ called bcm2835-unicam.
diff --git a/drivers/staging/media/bcm2835-unicam/Makefile b/drivers/staging/media/bcm2835-unicam/Makefile
new file mode 100644
index 0000000000000..a98aba03598ab
--- /dev/null
+++ b/drivers/staging/media/bcm2835-unicam/Makefile
@@ -0,0 +1,3 @@
+# Makefile for BCM2835 Unicam driver
+
+obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam.o
diff --git a/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
new file mode 100644
index 0000000000000..3e365baa8197f
--- /dev/null
+++ b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
@@ -0,0 +1,2750 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * BCM283x / BCM271x Unicam Capture Driver
+ *
+ * Copyright (C) 2017-2020 - Raspberry Pi (Trading) Ltd.
+ *
+ * Dave Stevenson <dave.stevenson@raspberrypi.com>
+ *
+ * Based on TI am437x driver by
+ * Benoit Parrot <bparrot@ti.com>
+ * Lad, Prabhakar <prabhakar.csengg@gmail.com>
+ *
+ * and TI CAL camera interface driver by
+ * Benoit Parrot <bparrot@ti.com>
+ *
+ *
+ * There are two camera drivers in the kernel for BCM283x - this one
+ * and bcm2835-camera (currently in staging).
+ *
+ * This driver directly controls the Unicam peripheral - there is no
+ * involvement with the VideoCore firmware. Unicam receives CSI-2 or
+ * CCP2 data and writes it into SDRAM.
+ * The only potential processing options are to repack Bayer data into an
+ * alternate format, and applying windowing.
+ * The repacking does not shift the data, so can repack V4L2_PIX_FMT_Sxxxx10P
+ * to V4L2_PIX_FMT_Sxxxx10, or V4L2_PIX_FMT_Sxxxx12P to V4L2_PIX_FMT_Sxxxx12,
+ * but not generically up to V4L2_PIX_FMT_Sxxxx16. The driver will add both
+ * formats where the relevant formats are defined, and will automatically
+ * configure the repacking as required.
+ * Support for windowing may be added later.
+ *
+ * It should be possible to connect this driver to any sensor with a
+ * suitable output interface and V4L2 subdevice driver.
+ *
+ * bcm2835-camera uses the VideoCore firmware to control the sensor,
+ * Unicam, ISP, and all tuner control loops. Fully processed frames are
+ * delivered to the driver by the firmware. It only has sensor drivers
+ * for Omnivision OV5647, and Sony IMX219 sensors.
+ *
+ * The two drivers are mutually exclusive for the same Unicam instance.
+ * The VideoCore firmware checks the device tree configuration during boot.
+ * If it finds device tree nodes called csi0 or csi1 it will block the
+ * firmware from accessing the peripheral, and bcm2835-camera will
+ * not be able to stream data.
+ */
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/of_graph.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/videodev2.h>
+
+#include <media/v4l2-common.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-dv-timings.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-fwnode.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "vc4-regs-unicam.h"
+
+#define UNICAM_MODULE_NAME "unicam"
+#define UNICAM_VERSION "0.1.0"
+
+/*
+ * Unicam must request a minimum of 250Mhz from the VPU clock.
+ * Otherwise the input FIFOs overrun and cause image corruption.
+ */
+#define MIN_VPU_CLOCK_RATE (250 * 1000 * 1000)
+/*
+ * To protect against a dodgy sensor driver never returning an error from
+ * enum_mbus_code, set a maximum index value to be used.
+ */
+#define MAX_ENUM_MBUS_CODE 128
+
+/*
+ * Stride is a 16 bit register, but also has to be a multiple of 32.
+ */
+#define BPL_ALIGNMENT 32
+#define MAX_BYTESPERLINE ((1 << 16) - BPL_ALIGNMENT)
+/*
+ * Max width is therefore determined by the max stride divided by
+ * the number of bits per pixel. Take 32bpp as a
+ * worst case.
+ * No imposed limit on the height, so adopt a square image for want
+ * of anything better.
+ */
+#define MAX_WIDTH (MAX_BYTESPERLINE / 4)
+#define MAX_HEIGHT MAX_WIDTH
+/* Define a nominal minimum image size */
+#define MIN_WIDTH 16
+#define MIN_HEIGHT 16
+
+/*
+ * Size of the dummy buffer. Can be any size really, but the DMA
+ * allocation works in units of page sizes.
+ */
+#define DUMMY_BUF_SIZE (PAGE_SIZE)
+
+/* Enumeration of the video device node identifiers. */
+enum unicam_node_ids {
+ IMAGE_NODE,
+ METADATA_NODE,
+ MAX_NODES
+};
+
+/*
+ * Associate video device nodes with the subdevice's source pad ids.
+ * The image video device is connected to the subdevice pad #0;
+ * The metadata video device is connected to the subdevice pad #1;
+ */
+unsigned int subdev_pads[2] = {
+ [IMAGE_NODE] = 0,
+ [METADATA_NODE] = 1,
+};
+
+/*
+ * struct unicam_fmt - Unicam media bus format information
+ * @pixelformat: V4L2 pixel format FCC identifier. 0 if n/a.
+ * @repacked_fourcc: V4L2 pixel format FCC identifier if the data is expanded
+ * out to 16bpp. 0 if n/a.
+ * @code: V4L2 media bus format code.
+ * @depth: Bits per pixel as delivered from the source.
+ * @csi_dt: CSI data type.
+ * @check_variants: Flag to denote that there are multiple mediabus formats
+ * still in the list that could match this V4L2 format.
+ */
+struct unicam_fmt {
+ u32 fourcc;
+ u32 repacked_fourcc;
+ u32 code;
+ u8 depth;
+ u8 csi_dt;
+ u8 check_variants;
+};
+
+static const struct unicam_fmt formats[] = {
+ /* YUV Formats */
+ {
+ .fourcc = V4L2_PIX_FMT_YUYV,
+ .code = MEDIA_BUS_FMT_YUYV8_2X8,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ .check_variants = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_UYVY,
+ .code = MEDIA_BUS_FMT_UYVY8_2X8,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ .check_variants = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVYU,
+ .code = MEDIA_BUS_FMT_YVYU8_2X8,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ .check_variants = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_VYUY,
+ .code = MEDIA_BUS_FMT_VYUY8_2X8,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ .check_variants = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YUYV,
+ .code = MEDIA_BUS_FMT_YUYV8_1X16,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ }, {
+ .fourcc = V4L2_PIX_FMT_UYVY,
+ .code = MEDIA_BUS_FMT_UYVY8_1X16,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVYU,
+ .code = MEDIA_BUS_FMT_YVYU8_1X16,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ }, {
+ .fourcc = V4L2_PIX_FMT_VYUY,
+ .code = MEDIA_BUS_FMT_VYUY8_1X16,
+ .depth = 16,
+ .csi_dt = 0x1e,
+ }, {
+ /* RGB Formats */
+ .fourcc = V4L2_PIX_FMT_RGB565, /* gggbbbbb rrrrrggg */
+ .code = MEDIA_BUS_FMT_RGB565_2X8_LE,
+ .depth = 16,
+ .csi_dt = 0x22,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB565X, /* rrrrrggg gggbbbbb */
+ .code = MEDIA_BUS_FMT_RGB565_2X8_BE,
+ .depth = 16,
+ .csi_dt = 0x22
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB555, /* gggbbbbb arrrrrgg */
+ .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE,
+ .depth = 16,
+ .csi_dt = 0x21,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB555X, /* arrrrrgg gggbbbbb */
+ .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE,
+ .depth = 16,
+ .csi_dt = 0x21,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB24, /* rgb */
+ .code = MEDIA_BUS_FMT_RGB888_1X24,
+ .depth = 24,
+ .csi_dt = 0x24,
+ }, {
+ .fourcc = V4L2_PIX_FMT_BGR24, /* bgr */
+ .code = MEDIA_BUS_FMT_BGR888_1X24,
+ .depth = 24,
+ .csi_dt = 0x24,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB32, /* argb */
+ .code = MEDIA_BUS_FMT_ARGB8888_1X32,
+ .depth = 32,
+ .csi_dt = 0x0,
+ }, {
+ /* Bayer Formats */
+ .fourcc = V4L2_PIX_FMT_SBGGR8,
+ .code = MEDIA_BUS_FMT_SBGGR8_1X8,
+ .depth = 8,
+ .csi_dt = 0x2a,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG8,
+ .code = MEDIA_BUS_FMT_SGBRG8_1X8,
+ .depth = 8,
+ .csi_dt = 0x2a,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG8,
+ .code = MEDIA_BUS_FMT_SGRBG8_1X8,
+ .depth = 8,
+ .csi_dt = 0x2a,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SRGGB8,
+ .code = MEDIA_BUS_FMT_SRGGB8_1X8,
+ .depth = 8,
+ .csi_dt = 0x2a,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10P,
+ .repacked_fourcc = V4L2_PIX_FMT_SBGGR10,
+ .code = MEDIA_BUS_FMT_SBGGR10_1X10,
+ .depth = 10,
+ .csi_dt = 0x2b,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10P,
+ .repacked_fourcc = V4L2_PIX_FMT_SGBRG10,
+ .code = MEDIA_BUS_FMT_SGBRG10_1X10,
+ .depth = 10,
+ .csi_dt = 0x2b,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10P,
+ .repacked_fourcc = V4L2_PIX_FMT_SGRBG10,
+ .code = MEDIA_BUS_FMT_SGRBG10_1X10,
+ .depth = 10,
+ .csi_dt = 0x2b,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SRGGB10P,
+ .repacked_fourcc = V4L2_PIX_FMT_SRGGB10,
+ .code = MEDIA_BUS_FMT_SRGGB10_1X10,
+ .depth = 10,
+ .csi_dt = 0x2b,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12P,
+ .repacked_fourcc = V4L2_PIX_FMT_SBGGR12,
+ .code = MEDIA_BUS_FMT_SBGGR12_1X12,
+ .depth = 12,
+ .csi_dt = 0x2c,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12P,
+ .repacked_fourcc = V4L2_PIX_FMT_SGBRG12,
+ .code = MEDIA_BUS_FMT_SGBRG12_1X12,
+ .depth = 12,
+ .csi_dt = 0x2c,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12P,
+ .repacked_fourcc = V4L2_PIX_FMT_SGRBG12,
+ .code = MEDIA_BUS_FMT_SGRBG12_1X12,
+ .depth = 12,
+ .csi_dt = 0x2c,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SRGGB12P,
+ .repacked_fourcc = V4L2_PIX_FMT_SRGGB12,
+ .code = MEDIA_BUS_FMT_SRGGB12_1X12,
+ .depth = 12,
+ .csi_dt = 0x2c,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14P,
+ .code = MEDIA_BUS_FMT_SBGGR14_1X14,
+ .depth = 14,
+ .csi_dt = 0x2d,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14P,
+ .code = MEDIA_BUS_FMT_SGBRG14_1X14,
+ .depth = 14,
+ .csi_dt = 0x2d,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14P,
+ .code = MEDIA_BUS_FMT_SGRBG14_1X14,
+ .depth = 14,
+ .csi_dt = 0x2d,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SRGGB14P,
+ .code = MEDIA_BUS_FMT_SRGGB14_1X14,
+ .depth = 14,
+ .csi_dt = 0x2d,
+ }, {
+ /*
+ * 16 bit Bayer formats could be supported, but there is no CSI2
+ * data_type defined for raw 16, and no sensors that produce it at
+ * present.
+ */
+
+ /* Greyscale formats */
+ .fourcc = V4L2_PIX_FMT_GREY,
+ .code = MEDIA_BUS_FMT_Y8_1X8,
+ .depth = 8,
+ .csi_dt = 0x2a,
+ }, {
+ .fourcc = V4L2_PIX_FMT_Y10P,
+ .repacked_fourcc = V4L2_PIX_FMT_Y10,
+ .code = MEDIA_BUS_FMT_Y10_1X10,
+ .depth = 10,
+ .csi_dt = 0x2b,
+ }, {
+ /* NB There is no packed V4L2 fourcc for this format. */
+ .repacked_fourcc = V4L2_PIX_FMT_Y12,
+ .code = MEDIA_BUS_FMT_Y12_1X12,
+ .depth = 12,
+ .csi_dt = 0x2c,
+ },
+ /* Embedded data format */
+ {
+ .fourcc = V4L2_META_FMT_SENSOR_DATA,
+ .code = MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA,
+ .depth = 8,
+ }
+};
+
+struct unicam_buffer {
+ struct vb2_v4l2_buffer vb;
+ struct list_head list;
+};
+
+static inline struct unicam_buffer *to_unicam_buffer(struct vb2_buffer *vb)
+{
+ return container_of(vb, struct unicam_buffer, vb.vb2_buf);
+}
+
+struct unicam_node {
+ bool registered;
+ int open;
+ bool streaming;
+ unsigned int node_id;
+ /* Pointer pointing to current v4l2_buffer */
+ struct unicam_buffer *cur_frm;
+ /* Pointer pointing to next v4l2_buffer */
+ struct unicam_buffer *next_frm;
+ /* video capture */
+ const struct unicam_fmt *fmt;
+ /* Used to store current pixel format */
+ struct v4l2_format v_fmt;
+ /* Used to store current mbus frame format */
+ struct v4l2_mbus_framefmt m_fmt;
+ /* Buffer queue used in video-buf */
+ struct vb2_queue buffer_queue;
+ /* Queue of filled frames */
+ struct list_head dma_queue;
+ /* IRQ lock for DMA queue */
+ spinlock_t dma_queue_lock;
+ /* lock used to access this structure */
+ struct mutex lock;
+ /* Identifies video device for this channel */
+ struct video_device video_dev;
+ /* Pointer to the parent handle */
+ struct unicam_device *dev;
+ struct media_pad pad;
+ /*
+ * Dummy buffer intended to be used by unicam
+ * if we have no other queued buffers to swap to.
+ */
+ void *dummy_buf_cpu_addr;
+ dma_addr_t dummy_buf_dma_addr;
+};
+
+struct unicam_device {
+ struct device *dev;
+
+ /* V4l2 specific parameters */
+ struct v4l2_async_subdev asd;
+
+ /* peripheral base address */
+ void __iomem *base;
+ /* clock gating base address */
+ void __iomem *clk_gate_base;
+ /* lp clock handle */
+ struct clk *clock;
+ /* vpu clock handle */
+ struct clk *vpu_clock;
+ /* V4l2 device */
+ struct v4l2_device v4l2_dev;
+ struct media_device mdev;
+
+ /* subdevice async Notifier */
+ struct v4l2_async_notifier notifier;
+ unsigned int sequence;
+
+ /* ptr to sub device */
+ struct v4l2_subdev *sensor;
+ /* Pad config for the sensor */
+ struct v4l2_subdev_pad_config *sensor_config;
+
+ enum v4l2_mbus_type bus_type;
+ /*
+ * Stores bus.mipi_csi2.flags for CSI2 sensors, or
+ * bus.mipi_csi1.strobe for CCP2.
+ */
+ unsigned int bus_flags;
+ unsigned int max_data_lanes;
+ unsigned int active_data_lanes;
+
+ struct unicam_node node[MAX_NODES];
+ struct v4l2_ctrl_handler ctrl_handler;
+};
+
+static inline struct unicam_device *
+to_unicam_device(struct v4l2_device *v4l2_dev)
+{
+ return container_of(v4l2_dev, struct unicam_device, v4l2_dev);
+}
+
+static char *print_fourcc(u32 fmt)
+{
+ static char code[16];
+
+ code[0] = (unsigned char)(fmt & 0xff);
+ code[1] = (unsigned char)((fmt >> 8) & 0xff);
+ code[2] = (unsigned char)((fmt >> 16) & 0xff);
+ code[3] = (unsigned char)((fmt >> 24) & 0xff);
+ snprintf(&code[4], 12, "=0x%8x", fmt);
+
+ return code;
+}
+
+/* Hardware access */
+static inline void clk_write(struct unicam_device *dev, u32 val)
+{
+ writel(val | 0x5a000000, dev->clk_gate_base);
+}
+
+static inline u32 reg_read(struct unicam_device *dev, u32 offset)
+{
+ return readl(dev->base + offset);
+}
+
+static inline void reg_write(struct unicam_device *dev, u32 offset, u32 val)
+{
+ writel(val, dev->base + offset);
+}
+
+static inline int get_field(u32 value, u32 mask)
+{
+ return (value & mask) >> __ffs(mask);
+}
+
+static inline void set_field(u32 *valp, u32 field, u32 mask)
+{
+ u32 val = *valp;
+
+ val &= ~mask;
+ val |= (field << __ffs(mask)) & mask;
+ *valp = val;
+}
+
+static inline u32 reg_read_field(struct unicam_device *dev, u32 offset,
+ u32 mask)
+{
+ return get_field(reg_read(dev, offset), mask);
+}
+
+static inline void reg_write_field(struct unicam_device *dev, u32 offset,
+ u32 field, u32 mask)
+{
+ u32 val = reg_read(dev, offset);
+
+ set_field(&val, field, mask);
+ reg_write(dev, offset, val);
+}
+
+/* Format setup functions */
+static const struct unicam_fmt *find_format_by_code(u32 code)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(formats); i++) {
+ if (formats[i].code == code)
+ return &formats[i];
+ }
+
+ return NULL;
+}
+
+static int check_mbus_format(struct unicam_device *dev,
+ const struct unicam_fmt *format)
+{
+ unsigned int i;
+ int ret = 0;
+
+ for (i = 0; !ret && i < MAX_ENUM_MBUS_CODE; i++) {
+ struct v4l2_subdev_mbus_code_enum mbus_code = {
+ .index = i,
+ .pad = subdev_pads[IMAGE_NODE],
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ };
+
+ ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code,
+ NULL, &mbus_code);
+
+ if (!ret && mbus_code.code == format->code)
+ return 1;
+ }
+
+ return 0;
+}
+
+static const struct unicam_fmt *find_format_by_pix(struct unicam_device *dev,
+ u32 pixelformat)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(formats); i++) {
+ if (formats[i].fourcc == pixelformat ||
+ formats[i].repacked_fourcc == pixelformat) {
+ if (formats[i].check_variants &&
+ !check_mbus_format(dev, &formats[i]))
+ continue;
+ return &formats[i];
+ }
+ }
+
+ return NULL;
+}
+
+static unsigned int bytes_per_line(u32 width, const struct unicam_fmt *fmt,
+ u32 v4l2_fourcc)
+{
+ if (v4l2_fourcc == fmt->repacked_fourcc)
+ /* Repacking always goes to 16bpp */
+ return ALIGN(width << 1, BPL_ALIGNMENT);
+ else
+ return ALIGN((width * fmt->depth) >> 3, BPL_ALIGNMENT);
+}
+
+static int __subdev_get_format(struct unicam_device *dev,
+ struct v4l2_mbus_framefmt *fmt, int node_id)
+{
+ struct v4l2_subdev_format sd_fmt = {
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ .pad = subdev_pads[node_id],
+ };
+ int ret;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, get_fmt, dev->sensor_config,
+ &sd_fmt);
+ if (ret < 0)
+ return ret;
+
+ *fmt = sd_fmt.format;
+
+ dev_dbg(dev->dev, "%s %dx%d code:%04x\n", __func__,
+ fmt->width, fmt->height, fmt->code);
+
+ return 0;
+}
+
+static int __subdev_set_format(struct unicam_device *dev,
+ struct v4l2_mbus_framefmt *fmt, int node_id)
+{
+ struct v4l2_subdev_format sd_fmt = {
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ .pad = subdev_pads[node_id],
+ };
+ int ret;
+
+ sd_fmt.format = *fmt;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_config,
+ &sd_fmt);
+ if (ret < 0)
+ return ret;
+
+ *fmt = sd_fmt.format;
+
+ if (node_id == IMAGE_NODE)
+ dev_dbg(dev->dev, "%s %dx%d code:%04x\n", __func__, fmt->width,
+ fmt->height, fmt->code);
+ else
+ dev_dbg(dev->dev, "%s Embedded data code:%04x\n", __func__,
+ sd_fmt.format.code);
+
+ return 0;
+}
+
+static int unicam_calc_format_size_bpl(struct unicam_device *dev,
+ const struct unicam_fmt *fmt,
+ struct v4l2_format *f)
+{
+ unsigned int min_bytesperline;
+
+ v4l_bound_align_image(&f->fmt.pix.width, MIN_WIDTH, MAX_WIDTH, 2,
+ &f->fmt.pix.height, MIN_HEIGHT, MAX_HEIGHT, 0,
+ 0);
+
+ min_bytesperline = bytes_per_line(f->fmt.pix.width, fmt,
+ f->fmt.pix.pixelformat);
+
+ if (f->fmt.pix.bytesperline > min_bytesperline &&
+ f->fmt.pix.bytesperline <= MAX_BYTESPERLINE)
+ f->fmt.pix.bytesperline = ALIGN(f->fmt.pix.bytesperline,
+ BPL_ALIGNMENT);
+ else
+ f->fmt.pix.bytesperline = min_bytesperline;
+
+ f->fmt.pix.sizeimage = f->fmt.pix.height * f->fmt.pix.bytesperline;
+
+ dev_dbg(dev->dev, "%s: fourcc: %s size: %dx%d bpl:%d img_size:%d\n",
+ __func__, print_fourcc(f->fmt.pix.pixelformat),
+ f->fmt.pix.width, f->fmt.pix.height, f->fmt.pix.bytesperline,
+ f->fmt.pix.sizeimage);
+
+ return 0;
+}
+
+static int unicam_reset_format(struct unicam_node *node)
+{
+ struct unicam_device *dev = node->dev;
+ struct v4l2_mbus_framefmt mbus_fmt;
+ int ret;
+
+ ret = __subdev_get_format(dev, &mbus_fmt, node->node_id);
+ if (ret) {
+ dev_err(dev->dev, "Failed to get_format - ret %d\n", ret);
+ return ret;
+ }
+
+ if (mbus_fmt.code != node->fmt->code) {
+ dev_err(dev->dev,
+ "code mismatch - fmt->code %08x, mbus_fmt.code %08x\n",
+ node->fmt->code, mbus_fmt.code);
+ return ret;
+ }
+
+ if (node->node_id == IMAGE_NODE) {
+ v4l2_fill_pix_format(&node->v_fmt.fmt.pix, &mbus_fmt);
+ node->v_fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ unicam_calc_format_size_bpl(dev, node->fmt, &node->v_fmt);
+ } else {
+ node->v_fmt.type = V4L2_BUF_TYPE_META_CAPTURE;
+ node->v_fmt.fmt.meta.dataformat = V4L2_META_FMT_SENSOR_DATA;
+ node->v_fmt.fmt.meta.buffersize = mbus_fmt.width
+ * mbus_fmt.height;
+ }
+
+ node->m_fmt = mbus_fmt;
+ return 0;
+}
+
+static void unicam_wr_dma_addr(struct unicam_device *dev, dma_addr_t dmaaddr,
+ unsigned int buffer_size, int node_id)
+{
+ dma_addr_t endaddr = dmaaddr + buffer_size;
+
+ if (node_id == IMAGE_NODE) {
+ reg_write(dev, UNICAM_IBSA0, dmaaddr);
+ reg_write(dev, UNICAM_IBEA0, endaddr);
+ } else {
+ reg_write(dev, UNICAM_DBSA0, dmaaddr);
+ reg_write(dev, UNICAM_DBEA0, endaddr);
+ }
+}
+
+static unsigned int unicam_get_lines_done(struct unicam_device *dev)
+{
+ dma_addr_t start_addr, cur_addr;
+ unsigned int stride = dev->node[IMAGE_NODE].v_fmt.fmt.pix.bytesperline;
+ struct unicam_buffer *frm = dev->node[IMAGE_NODE].cur_frm;
+
+ if (!frm)
+ return 0;
+
+ start_addr = vb2_dma_contig_plane_dma_addr(&frm->vb.vb2_buf, 0);
+ cur_addr = reg_read(dev, UNICAM_IBWP);
+ return (unsigned int)(cur_addr - start_addr) / stride;
+}
+
+static void unicam_schedule_next_buffer(struct unicam_node *node)
+{
+ struct unicam_device *dev = node->dev;
+ struct unicam_buffer *buf;
+ unsigned int size;
+ dma_addr_t addr;
+
+ buf = list_first_entry(&node->dma_queue, struct unicam_buffer, list);
+ node->next_frm = buf;
+ list_del(&buf->list);
+
+ addr = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
+ size = (node->node_id == IMAGE_NODE) ?
+ node->v_fmt.fmt.pix.sizeimage :
+ node->v_fmt.fmt.meta.buffersize;
+
+ unicam_wr_dma_addr(dev, addr, size, node->node_id);
+}
+
+static void unicam_schedule_dummy_buffer(struct unicam_node *node)
+{
+ struct unicam_device *dev = node->dev;
+
+ dev_dbg(dev->dev, "Scheduling dummy buffer for node %d\n",
+ node->node_id);
+
+ unicam_wr_dma_addr(dev, node->dummy_buf_dma_addr, DUMMY_BUF_SIZE,
+ node->node_id);
+ node->next_frm = NULL;
+}
+
+static void unicam_process_buffer_complete(struct unicam_node *node,
+ unsigned int sequence)
+{
+ node->cur_frm->vb.field = node->m_fmt.field;
+ node->cur_frm->vb.sequence = sequence;
+
+ vb2_buffer_done(&node->cur_frm->vb.vb2_buf, VB2_BUF_STATE_DONE);
+}
+
+static void unicam_queue_event_sof(struct unicam_device *unicam)
+{
+ struct v4l2_event event = {
+ .type = V4L2_EVENT_FRAME_SYNC,
+ .u.frame_sync.frame_sequence = unicam->sequence,
+ };
+
+ v4l2_event_queue(&unicam->node[IMAGE_NODE].video_dev, &event);
+}
+
+/*
+ * unicam_isr : ISR handler for unicam capture
+ * @irq: irq number
+ * @dev_id: dev_id ptr
+ *
+ * It changes status of the captured buffer, takes next buffer from the queue
+ * and sets its address in unicam registers
+ */
+static irqreturn_t unicam_isr(int irq, void *dev)
+{
+ struct unicam_device *unicam = dev;
+ unsigned int lines_done = unicam_get_lines_done(dev);
+ unsigned int sequence = unicam->sequence;
+ unsigned int i;
+ u32 ista, sta;
+ u64 ts;
+
+ sta = reg_read(unicam, UNICAM_STA);
+ /* Write value back to clear the interrupts */
+ reg_write(unicam, UNICAM_STA, sta);
+
+ ista = reg_read(unicam, UNICAM_ISTA);
+ /* Write value back to clear the interrupts */
+ reg_write(unicam, UNICAM_ISTA, ista);
+
+ dev_dbg(unicam->dev,
+ "ISR: ISTA: 0x%X, STA: 0x%X, sequence %d, lines done %d",
+ ista, sta, sequence, lines_done);
+
+ if (!(sta & (UNICAM_IS | UNICAM_PI0)))
+ return IRQ_HANDLED;
+
+ /*
+ * We must run the frame end handler first. If we have a valid next_frm
+ * and we get a simultaneout FE + FS interrupt, running the FS handler
+ * first would null out the next_frm ptr and we would have lost the
+ * buffer forever.
+ */
+ if (ista & UNICAM_FEI || sta & UNICAM_PI0) {
+ /*
+ * Ensure we have swapped buffers already as we can't
+ * stop the peripheral. If no buffer is available, use a
+ * dummy buffer to dump out frames until we get a new buffer
+ * to use.
+ */
+ for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
+ if (!unicam->node[i].streaming)
+ continue;
+
+ if (unicam->node[i].cur_frm)
+ unicam_process_buffer_complete(&unicam->node[i],
+ sequence);
+ unicam->node[i].cur_frm = unicam->node[i].next_frm;
+ }
+ unicam->sequence++;
+ }
+
+ if (ista & UNICAM_FSI) {
+ /*
+ * Timestamp is to be when the first data byte was captured,
+ * aka frame start.
+ */
+ ts = ktime_get_ns();
+ for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
+ if (!unicam->node[i].streaming)
+ continue;
+
+ if (unicam->node[i].cur_frm)
+ unicam->node[i].cur_frm->vb.vb2_buf.timestamp =
+ ts;
+ /*
+ * Set the next frame output to go to a dummy frame
+ * if we have not managed to obtain another frame
+ * from the queue.
+ */
+ unicam_schedule_dummy_buffer(&unicam->node[i]);
+ }
+
+ unicam_queue_event_sof(unicam);
+ }
+
+ /*
+ * Cannot swap buffer at frame end, there may be a race condition
+ * where the HW does not actually swap it if the new frame has
+ * already started.
+ */
+ if (ista & (UNICAM_FSI | UNICAM_LCI) && !(ista & UNICAM_FEI)) {
+ for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
+ if (!unicam->node[i].streaming)
+ continue;
+
+ spin_lock(&unicam->node[i].dma_queue_lock);
+ if (!list_empty(&unicam->node[i].dma_queue) &&
+ !unicam->node[i].next_frm)
+ unicam_schedule_next_buffer(&unicam->node[i]);
+ spin_unlock(&unicam->node[i].dma_queue_lock);
+ }
+ }
+
+ if (reg_read(unicam, UNICAM_ICTL) & UNICAM_FCM) {
+ /* Switch out of trigger mode if selected */
+ reg_write_field(unicam, UNICAM_ICTL, 1, UNICAM_TFC);
+ reg_write_field(unicam, UNICAM_ICTL, 0, UNICAM_FCM);
+ }
+ return IRQ_HANDLED;
+}
+
+static int unicam_querycap(struct file *file, void *priv,
+ struct v4l2_capability *cap)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ strscpy(cap->driver, UNICAM_MODULE_NAME, sizeof(cap->driver));
+ strscpy(cap->card, UNICAM_MODULE_NAME, sizeof(cap->card));
+
+ snprintf(cap->bus_info, sizeof(cap->bus_info),
+ "platform:%s", dev_name(dev->dev));
+
+ cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_VIDEO_CAPTURE |
+ V4L2_CAP_META_CAPTURE | V4L2_CAP_READWRITE |
+ V4L2_CAP_STREAMING;
+ return 0;
+}
+
+static int unicam_enum_fmt_vid_cap(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ unsigned int index = 0;
+ unsigned int i;
+ int ret = 0;
+
+ if (node->node_id != IMAGE_NODE)
+ return -EINVAL;
+
+ for (i = 0; !ret && i < MAX_ENUM_MBUS_CODE; i++) {
+ struct v4l2_subdev_mbus_code_enum mbus_code = {
+ .index = i,
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ };
+ const struct unicam_fmt *fmt;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code,
+ NULL, &mbus_code);
+ if (ret < 0) {
+ dev_dbg(dev->dev,
+ "subdev->enum_mbus_code idx %d = %d\n",
+ i, ret);
+ return -EINVAL;
+ }
+
+ fmt = find_format_by_code(mbus_code.code);
+ if (fmt) {
+ if (fmt->fourcc) {
+ if (index == f->index) {
+ f->pixelformat = fmt->fourcc;
+ break;
+ }
+ index++;
+ }
+ if (fmt->repacked_fourcc) {
+ if (index == f->index) {
+ f->pixelformat = fmt->repacked_fourcc;
+ break;
+ }
+ index++;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int unicam_g_fmt_vid_cap(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct v4l2_mbus_framefmt mbus_fmt = {0};
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ const struct unicam_fmt *fmt = NULL;
+ int ret;
+
+ if (node->node_id != IMAGE_NODE)
+ return -EINVAL;
+
+ /*
+ * If a flip has occurred in the sensor, the fmt code might have
+ * changed. So we will need to re-fetch the format from the subdevice.
+ */
+ ret = __subdev_get_format(dev, &mbus_fmt, node->node_id);
+ if (ret)
+ return -EINVAL;
+
+ /* Find the V4L2 format from mbus code. We must match a known format. */
+ fmt = find_format_by_code(mbus_fmt.code);
+ if (!fmt)
+ return -EINVAL;
+
+ if (node->fmt != fmt) {
+ /*
+ * The sensor format has changed so the pixelformat needs to
+ * be updated. Try and retain the packed/unpacked choice if
+ * at all possible.
+ */
+ if (node->fmt->repacked_fourcc ==
+ node->v_fmt.fmt.pix.pixelformat)
+ /* Using the repacked format */
+ node->v_fmt.fmt.pix.pixelformat = fmt->repacked_fourcc;
+ else
+ /* Using the native format */
+ node->v_fmt.fmt.pix.pixelformat = fmt->fourcc;
+
+ node->fmt = fmt;
+ }
+
+ *f = node->v_fmt;
+
+ return 0;
+}
+
+static const struct unicam_fmt *
+get_first_supported_format(struct unicam_device *dev)
+{
+ struct v4l2_subdev_mbus_code_enum mbus_code;
+ const struct unicam_fmt *fmt = NULL;
+ unsigned int i;
+ int ret = 0;
+
+ for (i = 0; ret != -EINVAL && ret != -ENOIOCTLCMD; ++i) {
+ memset(&mbus_code, 0, sizeof(mbus_code));
+ mbus_code.index = i;
+ mbus_code.pad = subdev_pads[IMAGE_NODE];
+ mbus_code.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code, NULL,
+ &mbus_code);
+ if (ret < 0) {
+ dev_dbg(dev->dev,
+ "subdev->enum_mbus_code idx %u = %d - continue\n",
+ i, ret);
+ continue;
+ }
+
+ dev_dbg(dev->dev, "subdev %s: code: 0x%08x idx: %u\n",
+ dev->sensor->name, mbus_code.code, i);
+
+ fmt = find_format_by_code(mbus_code.code);
+ dev_dbg(dev->dev,
+ "fmt 0x%08x returned as %p, V4L2 FOURCC %s, csi_dt 0x%02x\n",
+ mbus_code.code, fmt,
+ print_fourcc(fmt ? fmt->fourcc : 0),
+ fmt ? fmt->csi_dt : 0);
+ if (fmt)
+ return fmt;
+ }
+
+ return NULL;
+}
+
+static int unicam_try_fmt_vid_cap(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ struct v4l2_subdev_format sd_fmt = {
+ .which = V4L2_SUBDEV_FORMAT_TRY,
+ .pad = subdev_pads[IMAGE_NODE],
+ };
+ struct v4l2_mbus_framefmt *mbus_fmt = &sd_fmt.format;
+ const struct unicam_fmt *fmt;
+ int ret;
+
+ if (node->node_id != IMAGE_NODE)
+ return -EINVAL;
+
+ fmt = find_format_by_pix(dev, f->fmt.pix.pixelformat);
+ if (!fmt) {
+ /*
+ * Pixel format not supported by unicam. Choose the first
+ * supported format, and let the sensor choose something else.
+ */
+ dev_dbg(dev->dev, "Fourcc %s not found. Use first one.\n",
+ print_fourcc(f->fmt.pix.pixelformat));
+
+ fmt = &formats[0];
+ f->fmt.pix.pixelformat = fmt->fourcc;
+ }
+
+ v4l2_fill_mbus_format(mbus_fmt, &f->fmt.pix, fmt->code);
+ /*
+ * No support for receiving interlaced video, so never
+ * request it from the sensor subdev.
+ */
+ mbus_fmt->field = V4L2_FIELD_NONE;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_config,
+ &sd_fmt);
+ if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV)
+ return ret;
+
+ if (mbus_fmt->field != V4L2_FIELD_NONE)
+ dev_info(dev->dev,
+ "Sensor trying to send interlaced video - results may be unpredictable\n");
+
+ v4l2_fill_pix_format(&f->fmt.pix, &sd_fmt.format);
+ if (mbus_fmt->code != fmt->code) {
+ /* Sensor has returned an alternate format */
+ fmt = find_format_by_code(mbus_fmt->code);
+ if (!fmt) {
+ /*
+ * The alternate format is one unicam can't support.
+ * Find the first format that is supported by both, and
+ * then set that.
+ */
+ fmt = get_first_supported_format(dev);
+ mbus_fmt->code = fmt->code;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, set_fmt,
+ dev->sensor_config, &sd_fmt);
+ if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV)
+ return ret;
+
+ if (mbus_fmt->field != V4L2_FIELD_NONE)
+ dev_info(dev->dev,
+ "Sensor trying to send interlaced video - results may be unpredictable\n");
+
+ v4l2_fill_pix_format(&f->fmt.pix, &sd_fmt.format);
+
+ if (mbus_fmt->code != fmt->code) {
+ /*
+ * We've set a format that the sensor reports
+ * as being supported, but it refuses to set it.
+ * Not much else we can do.
+ * Assume that the sensor driver may accept the
+ * format when it is set (rather than tried).
+ */
+ dev_err(dev->dev,
+ "Sensor won't accept default format, and Unicam can't support sensor default\n");
+ }
+ }
+
+ if (fmt->fourcc)
+ f->fmt.pix.pixelformat = fmt->fourcc;
+ else
+ f->fmt.pix.pixelformat = fmt->repacked_fourcc;
+ }
+
+ return unicam_calc_format_size_bpl(dev, fmt, f);
+}
+
+static int unicam_s_fmt_vid_cap(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ struct vb2_queue *q = &node->buffer_queue;
+ struct v4l2_mbus_framefmt mbus_fmt = {0};
+ const struct unicam_fmt *fmt;
+ int ret;
+
+ if (vb2_is_busy(q))
+ return -EBUSY;
+
+ ret = unicam_try_fmt_vid_cap(file, priv, f);
+ if (ret < 0)
+ return ret;
+
+ fmt = find_format_by_pix(dev, f->fmt.pix.pixelformat);
+ if (!fmt) {
+ /*
+ * Unknown pixel format - adopt a default.
+ * This shouldn't happen as try_fmt should have resolved any
+ * issues first.
+ */
+ fmt = get_first_supported_format(dev);
+ if (!fmt)
+ /*
+ * It shouldn't be possible to get here with no
+ * supported formats
+ */
+ return -EINVAL;
+ f->fmt.pix.pixelformat = fmt->fourcc;
+ return -EINVAL;
+ }
+
+ v4l2_fill_mbus_format(&mbus_fmt, &f->fmt.pix, fmt->code);
+
+ ret = __subdev_set_format(dev, &mbus_fmt, node->node_id);
+ if (ret) {
+ dev_err(dev->dev, "%s set_format failed %d\n", __func__, ret);
+ return ret;
+ }
+
+ /* Just double check nothing has gone wrong */
+ if (mbus_fmt.code != fmt->code) {
+ dev_dbg(dev->dev,
+ "%s subdev changed format, this should not happen\n",
+ __func__);
+ return -EINVAL;
+ }
+
+ node->fmt = fmt;
+ node->v_fmt.fmt.pix.pixelformat = f->fmt.pix.pixelformat;
+ node->v_fmt.fmt.pix.bytesperline = f->fmt.pix.bytesperline;
+ unicam_reset_format(node);
+
+ dev_dbg(dev->dev,
+ "%s %dx%d, mbus_fmt 0x%08X, V4L2 pix %s.\n",
+ __func__, node->v_fmt.fmt.pix.width,
+ node->v_fmt.fmt.pix.height, mbus_fmt.code,
+ print_fourcc(node->v_fmt.fmt.pix.pixelformat));
+
+ *f = node->v_fmt;
+
+ return 0;
+}
+
+static int unicam_enum_fmt_meta_cap(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ const struct unicam_fmt *fmt;
+ u32 code;
+ int ret;
+
+ struct v4l2_subdev_mbus_code_enum mbus_code = {
+ .index = f->index,
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ .pad = subdev_pads[METADATA_NODE],
+ };
+
+ if (node->node_id != METADATA_NODE || f->index != 0)
+ return -EINVAL;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code, NULL,
+ &mbus_code);
+ if (ret < 0) {
+ dev_err(dev->dev,
+ "subdev->enum_mbus_code idx 0 = %d - invalid\n", ret);
+ return -EINVAL;
+ }
+
+ code = mbus_code.code;
+
+ fmt = find_format_by_code(code);
+ if (fmt)
+ f->pixelformat = fmt->fourcc;
+
+ return 0;
+}
+
+static int unicam_fmt_meta_cap(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct unicam_node *node = video_drvdata(file);
+
+ if (node->node_id != METADATA_NODE)
+ return -EINVAL;
+
+ *f = node->v_fmt;
+
+ return 0;
+}
+
+static int unicam_queue_setup(struct vb2_queue *vq,
+ unsigned int *nbuffers,
+ unsigned int *nplanes,
+ unsigned int sizes[],
+ struct device *alloc_devs[])
+{
+ struct unicam_node *node = vb2_get_drv_priv(vq);
+ unsigned int size = node->node_id == IMAGE_NODE ?
+ node->v_fmt.fmt.pix.sizeimage :
+ node->v_fmt.fmt.meta.buffersize;
+
+ if (vq->num_buffers + *nbuffers < 3)
+ *nbuffers = 3 - vq->num_buffers;
+
+ if (*nplanes)
+ return sizes[0] < size ? -EINVAL : 0;
+
+ *nplanes = 1;
+ sizes[0] = size;
+
+ return 0;
+}
+
+static int unicam_buffer_prepare(struct vb2_buffer *vb)
+{
+ struct unicam_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct unicam_device *dev = node->dev;
+ struct unicam_buffer *buf = to_unicam_buffer(vb);
+ unsigned long size;
+
+ if (WARN_ON(!node->fmt))
+ return -EINVAL;
+
+ size = node->node_id == IMAGE_NODE ? node->v_fmt.fmt.pix.sizeimage :
+ node->v_fmt.fmt.meta.buffersize;
+ if (vb2_plane_size(vb, 0) < size) {
+ dev_err(dev->dev, "data will not fit into plane (%lu < %lu)\n",
+ vb2_plane_size(vb, 0), size);
+ return -EINVAL;
+ }
+
+ vb2_set_plane_payload(&buf->vb.vb2_buf, 0, size);
+ return 0;
+}
+
+static void unicam_buffer_queue(struct vb2_buffer *vb)
+{
+ struct unicam_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct unicam_buffer *buf = to_unicam_buffer(vb);
+ unsigned long flags;
+
+ spin_lock_irqsave(&node->dma_queue_lock, flags);
+ list_add_tail(&buf->list, &node->dma_queue);
+ spin_unlock_irqrestore(&node->dma_queue_lock, flags);
+}
+
+static void unicam_set_packing_config(struct unicam_device *dev)
+{
+ u32 pack, unpack;
+ u32 val;
+
+ if (dev->node[IMAGE_NODE].v_fmt.fmt.pix.pixelformat ==
+ dev->node[IMAGE_NODE].fmt->fourcc) {
+ unpack = UNICAM_PUM_NONE;
+ pack = UNICAM_PPM_NONE;
+ } else {
+ switch (dev->node[IMAGE_NODE].fmt->depth) {
+ case 8:
+ unpack = UNICAM_PUM_UNPACK8;
+ break;
+ case 10:
+ unpack = UNICAM_PUM_UNPACK10;
+ break;
+ case 12:
+ unpack = UNICAM_PUM_UNPACK12;
+ break;
+ case 14:
+ unpack = UNICAM_PUM_UNPACK14;
+ break;
+ case 16:
+ unpack = UNICAM_PUM_UNPACK16;
+ break;
+ default:
+ unpack = UNICAM_PUM_NONE;
+ break;
+ }
+
+ /* Repacking is always to 16bpp */
+ pack = UNICAM_PPM_PACK16;
+ }
+
+ val = 0;
+ set_field(&val, unpack, UNICAM_PUM_MASK);
+ set_field(&val, pack, UNICAM_PPM_MASK);
+ reg_write(dev, UNICAM_IPIPE, val);
+}
+
+static void unicam_cfg_image_id(struct unicam_device *dev)
+{
+ if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
+ /* CSI2 mode, hardcode VC 0 for now. */
+ reg_write(dev, UNICAM_IDI0,
+ (0 << 6) | dev->node[IMAGE_NODE].fmt->csi_dt);
+ } else {
+ /* CCP2 mode */
+ reg_write(dev, UNICAM_IDI0,
+ 0x80 | dev->node[IMAGE_NODE].fmt->csi_dt);
+ }
+}
+
+static void unicam_enable_ed(struct unicam_device *dev)
+{
+ u32 val = reg_read(dev, UNICAM_DCS);
+
+ set_field(&val, 2, UNICAM_EDL_MASK);
+ /* Do not wrap at the end of the embedded data buffer */
+ set_field(&val, 0, UNICAM_DBOB);
+
+ reg_write(dev, UNICAM_DCS, val);
+}
+
+static void unicam_start_rx(struct unicam_device *dev, dma_addr_t *addr)
+{
+ int line_int_freq = dev->node[IMAGE_NODE].v_fmt.fmt.pix.height >> 2;
+ unsigned int size, i;
+ u32 val;
+
+ if (line_int_freq < 128)
+ line_int_freq = 128;
+
+ /* Enable lane clocks */
+ val = 1;
+ for (i = 0; i < dev->active_data_lanes; i++)
+ val = val << 2 | 1;
+ clk_write(dev, val);
+
+ /* Basic init */
+ reg_write(dev, UNICAM_CTRL, UNICAM_MEM);
+
+ /* Enable analogue control, and leave in reset. */
+ val = UNICAM_AR;
+ set_field(&val, 7, UNICAM_CTATADJ_MASK);
+ set_field(&val, 7, UNICAM_PTATADJ_MASK);
+ reg_write(dev, UNICAM_ANA, val);
+ usleep_range(1000, 2000);
+
+ /* Come out of reset */
+ reg_write_field(dev, UNICAM_ANA, 0, UNICAM_AR);
+
+ /* Peripheral reset */
+ reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_CPR);
+ reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPR);
+
+ reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPE);
+
+ /* Enable Rx control. */
+ val = reg_read(dev, UNICAM_CTRL);
+ if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
+ set_field(&val, UNICAM_CPM_CSI2, UNICAM_CPM_MASK);
+ set_field(&val, UNICAM_DCM_STROBE, UNICAM_DCM_MASK);
+ } else {
+ set_field(&val, UNICAM_CPM_CCP2, UNICAM_CPM_MASK);
+ set_field(&val, dev->bus_flags, UNICAM_DCM_MASK);
+ }
+ /* Packet framer timeout */
+ set_field(&val, 0xf, UNICAM_PFT_MASK);
+ set_field(&val, 128, UNICAM_OET_MASK);
+ reg_write(dev, UNICAM_CTRL, val);
+
+ reg_write(dev, UNICAM_IHWIN, 0);
+ reg_write(dev, UNICAM_IVWIN, 0);
+
+ /* AXI bus access QoS setup */
+ val = reg_read(dev, UNICAM_PRI);
+ set_field(&val, 0, UNICAM_BL_MASK);
+ set_field(&val, 0, UNICAM_BS_MASK);
+ set_field(&val, 0xe, UNICAM_PP_MASK);
+ set_field(&val, 8, UNICAM_NP_MASK);
+ set_field(&val, 2, UNICAM_PT_MASK);
+ set_field(&val, 1, UNICAM_PE);
+ reg_write(dev, UNICAM_PRI, val);
+
+ reg_write_field(dev, UNICAM_ANA, 0, UNICAM_DDL);
+
+ /* Always start in trigger frame capture mode (UNICAM_FCM set) */
+ val = UNICAM_FSIE | UNICAM_FEIE | UNICAM_FCM | UNICAM_IBOB;
+ set_field(&val, line_int_freq, UNICAM_LCIE_MASK);
+ reg_write(dev, UNICAM_ICTL, val);
+ reg_write(dev, UNICAM_STA, UNICAM_STA_MASK_ALL);
+ reg_write(dev, UNICAM_ISTA, UNICAM_ISTA_MASK_ALL);
+
+ /* tclk_term_en */
+ reg_write_field(dev, UNICAM_CLT, 2, UNICAM_CLT1_MASK);
+ /* tclk_settle */
+ reg_write_field(dev, UNICAM_CLT, 6, UNICAM_CLT2_MASK);
+ /* td_term_en */
+ reg_write_field(dev, UNICAM_DLT, 2, UNICAM_DLT1_MASK);
+ /* ths_settle */
+ reg_write_field(dev, UNICAM_DLT, 6, UNICAM_DLT2_MASK);
+ /* trx_enable */
+ reg_write_field(dev, UNICAM_DLT, 0, UNICAM_DLT3_MASK);
+
+ reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_SOE);
+
+ /* Packet compare setup - required to avoid missing frame ends */
+ val = 0;
+ set_field(&val, 1, UNICAM_PCE);
+ set_field(&val, 1, UNICAM_GI);
+ set_field(&val, 1, UNICAM_CPH);
+ set_field(&val, 0, UNICAM_PCVC_MASK);
+ set_field(&val, 1, UNICAM_PCDT_MASK);
+ reg_write(dev, UNICAM_CMP0, val);
+
+ /* Enable clock lane and set up terminations */
+ val = 0;
+ if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
+ /* CSI2 */
+ set_field(&val, 1, UNICAM_CLE);
+ set_field(&val, 1, UNICAM_CLLPE);
+ if (dev->bus_flags & V4L2_MBUS_CSI2_CONTINUOUS_CLOCK) {
+ set_field(&val, 1, UNICAM_CLTRE);
+ set_field(&val, 1, UNICAM_CLHSE);
+ }
+ } else {
+ /* CCP2 */
+ set_field(&val, 1, UNICAM_CLE);
+ set_field(&val, 1, UNICAM_CLHSE);
+ set_field(&val, 1, UNICAM_CLTRE);
+ }
+ reg_write(dev, UNICAM_CLK, val);
+
+ /*
+ * Enable required data lanes with appropriate terminations.
+ * The same value needs to be written to UNICAM_DATn registers for
+ * the active lanes, and 0 for inactive ones.
+ */
+ val = 0;
+ if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
+ /* CSI2 */
+ set_field(&val, 1, UNICAM_DLE);
+ set_field(&val, 1, UNICAM_DLLPE);
+ if (dev->bus_flags & V4L2_MBUS_CSI2_CONTINUOUS_CLOCK) {
+ set_field(&val, 1, UNICAM_DLTRE);
+ set_field(&val, 1, UNICAM_DLHSE);
+ }
+ } else {
+ /* CCP2 */
+ set_field(&val, 1, UNICAM_DLE);
+ set_field(&val, 1, UNICAM_DLHSE);
+ set_field(&val, 1, UNICAM_DLTRE);
+ }
+ reg_write(dev, UNICAM_DAT0, val);
+
+ if (dev->active_data_lanes == 1)
+ val = 0;
+ reg_write(dev, UNICAM_DAT1, val);
+
+ if (dev->max_data_lanes > 2) {
+ /*
+ * Registers UNICAM_DAT2 and UNICAM_DAT3 only valid if the
+ * instance supports more than 2 data lanes.
+ */
+ if (dev->active_data_lanes == 2)
+ val = 0;
+ reg_write(dev, UNICAM_DAT2, val);
+
+ if (dev->active_data_lanes == 3)
+ val = 0;
+ reg_write(dev, UNICAM_DAT3, val);
+ }
+
+ reg_write(dev, UNICAM_IBLS,
+ dev->node[IMAGE_NODE].v_fmt.fmt.pix.bytesperline);
+ size = dev->node[IMAGE_NODE].v_fmt.fmt.pix.sizeimage;
+ unicam_wr_dma_addr(dev, addr[IMAGE_NODE], size, IMAGE_NODE);
+ unicam_set_packing_config(dev);
+ unicam_cfg_image_id(dev);
+
+ val = reg_read(dev, UNICAM_MISC);
+ set_field(&val, 1, UNICAM_FL0);
+ set_field(&val, 1, UNICAM_FL1);
+ reg_write(dev, UNICAM_MISC, val);
+
+ if (dev->node[METADATA_NODE].streaming) {
+ size = dev->node[METADATA_NODE].v_fmt.fmt.meta.buffersize;
+ unicam_enable_ed(dev);
+ unicam_wr_dma_addr(dev, addr[METADATA_NODE], size, METADATA_NODE);
+ }
+
+ /* Enable peripheral */
+ reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_CPE);
+
+ /* Load image pointers */
+ reg_write_field(dev, UNICAM_ICTL, 1, UNICAM_LIP_MASK);
+
+ /* Load embedded data buffer pointers if needed */
+ if (dev->node[METADATA_NODE].streaming)
+ reg_write_field(dev, UNICAM_DCS, 1, UNICAM_LDP);
+
+ /*
+ * Enable trigger only for the first frame to
+ * sync correctly to the FS from the source.
+ */
+ reg_write_field(dev, UNICAM_ICTL, 1, UNICAM_TFC);
+}
+
+static void unicam_disable(struct unicam_device *dev)
+{
+ /* Analogue lane control disable */
+ reg_write_field(dev, UNICAM_ANA, 1, UNICAM_DDL);
+
+ /* Stop the output engine */
+ reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_SOE);
+
+ /* Disable the data lanes. */
+ reg_write(dev, UNICAM_DAT0, 0);
+ reg_write(dev, UNICAM_DAT1, 0);
+
+ if (dev->max_data_lanes > 2) {
+ reg_write(dev, UNICAM_DAT2, 0);
+ reg_write(dev, UNICAM_DAT3, 0);
+ }
+
+ /* Peripheral reset */
+ reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_CPR);
+ usleep_range(50, 100);
+ reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPR);
+
+ /* Disable peripheral */
+ reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPE);
+
+ /* Clear ED setup */
+ reg_write(dev, UNICAM_DCS, 0);
+
+ /* Disable all lane clocks */
+ clk_write(dev, 0);
+}
+
+static void unicam_return_buffers(struct unicam_node *node,
+ enum vb2_buffer_state state)
+{
+ struct unicam_buffer *buf, *tmp;
+ unsigned long flags;
+
+ spin_lock_irqsave(&node->dma_queue_lock, flags);
+ list_for_each_entry_safe(buf, tmp, &node->dma_queue, list) {
+ list_del(&buf->list);
+ vb2_buffer_done(&buf->vb.vb2_buf, state);
+ }
+
+ if (node->cur_frm)
+ vb2_buffer_done(&node->cur_frm->vb.vb2_buf,
+ state);
+ if (node->next_frm && node->cur_frm != node->next_frm)
+ vb2_buffer_done(&node->next_frm->vb.vb2_buf,
+ state);
+
+ node->cur_frm = NULL;
+ node->next_frm = NULL;
+ spin_unlock_irqrestore(&node->dma_queue_lock, flags);
+}
+
+static int unicam_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+ struct unicam_node *node = vb2_get_drv_priv(vq);
+ struct unicam_device *dev = node->dev;
+ dma_addr_t buffer_addr[MAX_NODES] = { 0 };
+ unsigned long flags;
+ unsigned int i;
+ int ret;
+
+ node->streaming = true;
+ if (!(dev->node[IMAGE_NODE].open && dev->node[IMAGE_NODE].streaming &&
+ (!dev->node[METADATA_NODE].open ||
+ dev->node[METADATA_NODE].streaming))) {
+ /*
+ * Metadata pad must be enabled before image pad if it is
+ * wanted.
+ */
+ dev_dbg(dev->dev, "Not all nodes are streaming yet.");
+ return 0;
+ }
+
+ dev->sequence = 0;
+ ret = pm_runtime_get_sync(dev->dev);
+ if (ret < 0) {
+ dev_err(dev->dev, "pm_runtime_get failed\n");
+ goto err_streaming;
+ }
+
+ /*
+ * TODO: Retrieve the number of active data lanes from the connected
+ * subdevice.
+ */
+ dev->active_data_lanes = dev->max_data_lanes;
+
+ ret = clk_set_min_rate(dev->vpu_clock, MIN_VPU_CLOCK_RATE);
+ if (ret) {
+ dev_err(dev->dev, "failed to set up VPU clock\n");
+ goto err_pm_put;
+ }
+
+ ret = clk_prepare_enable(dev->vpu_clock);
+ if (ret) {
+ dev_err(dev->dev, "Failed to enable VPU clock: %d\n", ret);
+ goto err_pm_put;
+ }
+
+ ret = clk_set_rate(dev->clock, 100 * 1000 * 1000);
+ if (ret) {
+ dev_err(dev->dev, "failed to set up CSI clock\n");
+ goto err_vpu_clock;
+ }
+
+ ret = clk_prepare_enable(dev->clock);
+ if (ret) {
+ dev_err(dev->dev, "Failed to enable CSI clock: %d\n", ret);
+ goto err_vpu_clock;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(dev->node); i++) {
+ struct unicam_buffer *buf;
+
+ if (!dev->node[i].streaming)
+ continue;
+
+ spin_lock_irqsave(&dev->node[i].dma_queue_lock, flags);
+ buf = list_first_entry(&dev->node[i].dma_queue,
+ struct unicam_buffer, list);
+ dev->node[i].cur_frm = buf;
+ dev->node[i].next_frm = buf;
+ list_del(&buf->list);
+ spin_unlock_irqrestore(&dev->node[i].dma_queue_lock, flags);
+
+ buffer_addr[i] =
+ vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
+ }
+
+ unicam_start_rx(dev, buffer_addr);
+
+ ret = v4l2_subdev_call(dev->sensor, video, s_stream, 1);
+ if (ret < 0) {
+ dev_err(dev->dev, "stream on failed in subdev\n");
+ goto err_disable_unicam;
+ }
+
+ return 0;
+
+err_disable_unicam:
+ unicam_disable(dev);
+ clk_disable_unprepare(dev->clock);
+err_vpu_clock:
+ ret = clk_set_min_rate(dev->vpu_clock, 0);
+ if (ret)
+ dev_err(dev->dev, "failed to reset the VPU clock\n");
+ clk_disable_unprepare(dev->vpu_clock);
+err_pm_put:
+ pm_runtime_put_sync(dev->dev);
+err_streaming:
+ unicam_return_buffers(node, VB2_BUF_STATE_QUEUED);
+ node->streaming = false;
+
+ return ret;
+}
+
+static void unicam_stop_streaming(struct vb2_queue *vq)
+{
+ struct unicam_node *node = vb2_get_drv_priv(vq);
+ struct unicam_device *dev = node->dev;
+
+ node->streaming = false;
+
+ if (node->node_id == IMAGE_NODE) {
+ int ret;
+
+ /*
+ * Stop streaming the sensor and disable the peripheral.
+ * We cannot continue streaming embedded data with the
+ * image pad disabled.
+ */
+ if (v4l2_subdev_call(dev->sensor, video, s_stream, 0) < 0)
+ dev_err(dev->dev, "stream off failed in subdev\n");
+
+ unicam_disable(dev);
+
+ ret = clk_set_min_rate(dev->vpu_clock, 0);
+ if (ret)
+ dev_err(dev->dev, "failed to reset the min VPU clock\n");
+
+ clk_disable_unprepare(dev->vpu_clock);
+ clk_disable_unprepare(dev->clock);
+ pm_runtime_put_sync(dev->dev);
+
+ } else if (node->node_id == METADATA_NODE) {
+ /*
+ * Allow the hardware to spin in the dummy buffer.
+ * This is only really needed if the embedded data pad is
+ * disabled before the image pad.
+ */
+ unicam_wr_dma_addr(dev, node->dummy_buf_dma_addr,
+ DUMMY_BUF_SIZE, METADATA_NODE);
+ }
+
+ /* Clear all queued buffers for the node */
+ unicam_return_buffers(node, VB2_BUF_STATE_ERROR);
+}
+
+static int unicam_enum_input(struct file *file, void *priv,
+ struct v4l2_input *inp)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ if (inp->index != 0)
+ return -EINVAL;
+
+ inp->type = V4L2_INPUT_TYPE_CAMERA;
+ if (v4l2_subdev_has_op(dev->sensor, video, s_dv_timings)) {
+ inp->capabilities = V4L2_IN_CAP_DV_TIMINGS;
+ inp->std = 0;
+ } else if (v4l2_subdev_has_op(dev->sensor, video, s_std)) {
+ inp->capabilities = V4L2_IN_CAP_STD;
+ if (v4l2_subdev_call(dev->sensor, video, g_tvnorms, &inp->std) < 0)
+ inp->std = V4L2_STD_ALL;
+ } else {
+ inp->capabilities = 0;
+ inp->std = 0;
+ }
+ strscpy(inp->name, "Camera 0", sizeof(inp->name));
+ return 0;
+}
+
+static int unicam_g_input(struct file *file, void *priv, unsigned int *i)
+{
+ *i = 0;
+
+ return 0;
+}
+
+static int unicam_s_input(struct file *file, void *priv, unsigned int i)
+{
+ /*
+ * FIXME: Ideally we would like to be able to query the source
+ * subdevice for information over the input connectors it supports,
+ * and map that through in to a call to video_ops->s_routing.
+ * There is no infrastructure support for defining that within
+ * devicetree at present. Until that is implemented we can't
+ * map a user physical connector number to s_routing input number.
+ */
+ if (i > 0)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int unicam_querystd(struct file *file, void *priv,
+ v4l2_std_id *std)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, video, querystd, std);
+}
+
+static int unicam_g_std(struct file *file, void *priv, v4l2_std_id *std)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, video, g_std, std);
+}
+
+static int unicam_s_std(struct file *file, void *priv, v4l2_std_id std)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ int ret;
+ v4l2_std_id current_std;
+
+ ret = v4l2_subdev_call(dev->sensor, video, g_std, ¤t_std);
+ if (ret)
+ return ret;
+
+ if (std == current_std)
+ return 0;
+
+ if (vb2_is_busy(&node->buffer_queue))
+ return -EBUSY;
+
+ ret = v4l2_subdev_call(dev->sensor, video, s_std, std);
+
+ /* Force recomputation of bytesperline */
+ node->v_fmt.fmt.pix.bytesperline = 0;
+
+ unicam_reset_format(node);
+
+ return ret;
+}
+
+static int unicam_s_edid(struct file *file, void *priv, struct v4l2_edid *edid)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, pad, set_edid, edid);
+}
+
+static int unicam_g_edid(struct file *file, void *priv, struct v4l2_edid *edid)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, pad, get_edid, edid);
+}
+
+static int unicam_s_selection(struct file *file, void *priv,
+ struct v4l2_selection *sel)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ struct v4l2_subdev_selection sdsel = {
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ .target = sel->target,
+ .flags = sel->flags,
+ .r = sel->r,
+ };
+
+ if (sel->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ return v4l2_subdev_call(dev->sensor, pad, set_selection, NULL, &sdsel);
+}
+
+static int unicam_g_selection(struct file *file, void *priv,
+ struct v4l2_selection *sel)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ struct v4l2_subdev_selection sdsel = {
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ .target = sel->target,
+ };
+ int ret;
+
+ if (sel->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ ret = v4l2_subdev_call(dev->sensor, pad, get_selection, NULL, &sdsel);
+ if (!ret)
+ sel->r = sdsel.r;
+
+ return ret;
+}
+
+static int unicam_enum_framesizes(struct file *file, void *priv,
+ struct v4l2_frmsizeenum *fsize)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ const struct unicam_fmt *fmt;
+ struct v4l2_subdev_frame_size_enum fse;
+ int ret;
+
+ /* check for valid format */
+ fmt = find_format_by_pix(dev, fsize->pixel_format);
+ if (!fmt) {
+ dev_err(dev->dev,
+ "Invalid pixel code: %x\n", fsize->pixel_format);
+ return -EINVAL;
+ }
+ fse.code = fmt->code;
+
+ fse.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+ fse.index = fsize->index;
+ fse.pad = subdev_pads[node->node_id];
+
+ ret = v4l2_subdev_call(dev->sensor, pad, enum_frame_size, NULL, &fse);
+ if (ret)
+ return ret;
+
+ dev_dbg(dev->dev, "%s: index: %d code: %x W:[%d,%d] H:[%d,%d]\n",
+ __func__, fse.index, fse.code, fse.min_width, fse.max_width,
+ fse.min_height, fse.max_height);
+
+ fsize->type = V4L2_FRMSIZE_TYPE_DISCRETE;
+ fsize->discrete.width = fse.max_width;
+ fsize->discrete.height = fse.max_height;
+
+ return 0;
+}
+
+static int unicam_enum_frameintervals(struct file *file, void *priv,
+ struct v4l2_frmivalenum *fival)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ const struct unicam_fmt *fmt;
+ struct v4l2_subdev_frame_interval_enum fie = {
+ .index = fival->index,
+ .width = fival->width,
+ .height = fival->height,
+ .which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ };
+ int ret;
+
+ fmt = find_format_by_pix(dev, fival->pixel_format);
+ if (!fmt)
+ return -EINVAL;
+
+ fie.code = fmt->code;
+ ret = v4l2_subdev_call(dev->sensor, pad, enum_frame_interval,
+ NULL, &fie);
+ if (ret)
+ return ret;
+
+ fival->type = V4L2_FRMIVAL_TYPE_DISCRETE;
+ fival->discrete = fie.interval;
+
+ return 0;
+}
+
+static int unicam_g_parm(struct file *file, void *fh, struct v4l2_streamparm *a)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_g_parm_cap(video_devdata(file), dev->sensor, a);
+}
+
+static int unicam_s_parm(struct file *file, void *fh, struct v4l2_streamparm *a)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_s_parm_cap(video_devdata(file), dev->sensor, a);
+}
+
+static int unicam_g_dv_timings(struct file *file, void *priv,
+ struct v4l2_dv_timings *timings)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, video, g_dv_timings, timings);
+}
+
+static int unicam_s_dv_timings(struct file *file, void *priv,
+ struct v4l2_dv_timings *timings)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ struct v4l2_dv_timings current_timings;
+ int ret;
+
+ ret = v4l2_subdev_call(dev->sensor, video, g_dv_timings,
+ ¤t_timings);
+
+ if (ret < 0)
+ return ret;
+
+ if (v4l2_match_dv_timings(timings, ¤t_timings, 0, false))
+ return 0;
+
+ if (vb2_is_busy(&node->buffer_queue))
+ return -EBUSY;
+
+ ret = v4l2_subdev_call(dev->sensor, video, s_dv_timings, timings);
+
+ /* Force recomputation of bytesperline */
+ node->v_fmt.fmt.pix.bytesperline = 0;
+
+ unicam_reset_format(node);
+
+ return ret;
+}
+
+static int unicam_query_dv_timings(struct file *file, void *priv,
+ struct v4l2_dv_timings *timings)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, video, query_dv_timings, timings);
+}
+
+static int unicam_enum_dv_timings(struct file *file, void *priv,
+ struct v4l2_enum_dv_timings *timings)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, pad, enum_dv_timings, timings);
+}
+
+static int unicam_dv_timings_cap(struct file *file, void *priv,
+ struct v4l2_dv_timings_cap *cap)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+
+ return v4l2_subdev_call(dev->sensor, pad, dv_timings_cap, cap);
+}
+
+static int unicam_subscribe_event(struct v4l2_fh *fh,
+ const struct v4l2_event_subscription *sub)
+{
+ switch (sub->type) {
+ case V4L2_EVENT_FRAME_SYNC:
+ return v4l2_event_subscribe(fh, sub, 2, NULL);
+ case V4L2_EVENT_SOURCE_CHANGE:
+ return v4l2_event_subscribe(fh, sub, 4, NULL);
+ }
+
+ return v4l2_ctrl_subscribe_event(fh, sub);
+}
+
+static int unicam_log_status(struct file *file, void *fh)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ u32 reg;
+
+ /* status for sub devices */
+ v4l2_device_call_all(&dev->v4l2_dev, 0, core, log_status);
+
+ dev_info(dev->dev, "-----Receiver status-----\n");
+ dev_info(dev->dev, "V4L2 width/height: %ux%u\n",
+ node->v_fmt.fmt.pix.width, node->v_fmt.fmt.pix.height);
+ dev_info(dev->dev, "Mediabus format: %08x\n", node->fmt->code);
+ dev_info(dev->dev, "V4L2 format: %08x\n",
+ node->v_fmt.fmt.pix.pixelformat);
+ reg = reg_read(dev, UNICAM_IPIPE);
+ dev_info(dev->dev, "Unpacking/packing: %u / %u\n",
+ get_field(reg, UNICAM_PUM_MASK),
+ get_field(reg, UNICAM_PPM_MASK));
+ dev_info(dev->dev, "----Live data----\n");
+ dev_info(dev->dev, "Programmed stride: %4u\n",
+ reg_read(dev, UNICAM_IBLS));
+ dev_info(dev->dev, "Detected resolution: %ux%u\n",
+ reg_read(dev, UNICAM_IHSTA),
+ reg_read(dev, UNICAM_IVSTA));
+ dev_info(dev->dev, "Write pointer: %08x\n",
+ reg_read(dev, UNICAM_IBWP));
+
+ return 0;
+}
+
+static void unicam_notify(struct v4l2_subdev *sd,
+ unsigned int notification, void *arg)
+{
+ struct unicam_device *dev = to_unicam_device(sd->v4l2_dev);
+
+ switch (notification) {
+ case V4L2_DEVICE_NOTIFY_EVENT:
+ v4l2_event_queue(&dev->node[IMAGE_NODE].video_dev, arg);
+ break;
+ default:
+ break;
+ }
+}
+
+static const struct vb2_ops unicam_video_qops = {
+ .wait_prepare = vb2_ops_wait_prepare,
+ .wait_finish = vb2_ops_wait_finish,
+ .queue_setup = unicam_queue_setup,
+ .buf_prepare = unicam_buffer_prepare,
+ .buf_queue = unicam_buffer_queue,
+ .start_streaming = unicam_start_streaming,
+ .stop_streaming = unicam_stop_streaming,
+};
+
+/*
+ * unicam_v4l2_open : This function is based on the v4l2_fh_open helper
+ * function. It has been augmented to handle sensor subdevice power management,
+ */
+static int unicam_v4l2_open(struct file *file)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ int ret;
+
+ mutex_lock(&node->lock);
+
+ ret = v4l2_fh_open(file);
+ if (ret) {
+ dev_err(dev->dev, "v4l2_fh_open failed\n");
+ goto unlock;
+ }
+
+ node->open++;
+
+ if (!v4l2_fh_is_singular_file(file))
+ goto unlock;
+
+ ret = v4l2_subdev_call(dev->sensor, core, s_power, 1);
+ if (ret < 0 && ret != -ENOIOCTLCMD) {
+ v4l2_fh_release(file);
+ node->open--;
+ goto unlock;
+ }
+
+ ret = 0;
+
+unlock:
+ mutex_unlock(&node->lock);
+ return ret;
+}
+
+static int unicam_v4l2_release(struct file *file)
+{
+ struct unicam_node *node = video_drvdata(file);
+ struct unicam_device *dev = node->dev;
+ struct v4l2_subdev *sd = dev->sensor;
+ bool fh_singular;
+ int ret;
+
+ mutex_lock(&node->lock);
+
+ fh_singular = v4l2_fh_is_singular_file(file);
+
+ ret = _vb2_fop_release(file, NULL);
+
+ if (fh_singular)
+ v4l2_subdev_call(sd, core, s_power, 0);
+
+ node->open--;
+ mutex_unlock(&node->lock);
+
+ return ret;
+}
+
+/* unicam capture driver file operations */
+static const struct v4l2_file_operations unicam_fops = {
+ .owner = THIS_MODULE,
+ .open = unicam_v4l2_open,
+ .release = unicam_v4l2_release,
+ .read = vb2_fop_read,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap,
+};
+
+/* unicam capture ioctl operations */
+static const struct v4l2_ioctl_ops unicam_ioctl_ops = {
+ .vidioc_querycap = unicam_querycap,
+ .vidioc_enum_fmt_vid_cap = unicam_enum_fmt_vid_cap,
+ .vidioc_g_fmt_vid_cap = unicam_g_fmt_vid_cap,
+ .vidioc_s_fmt_vid_cap = unicam_s_fmt_vid_cap,
+ .vidioc_try_fmt_vid_cap = unicam_try_fmt_vid_cap,
+
+ .vidioc_enum_fmt_meta_cap = unicam_enum_fmt_meta_cap,
+ .vidioc_g_fmt_meta_cap = unicam_fmt_meta_cap,
+ .vidioc_s_fmt_meta_cap = unicam_fmt_meta_cap,
+ .vidioc_try_fmt_meta_cap = unicam_fmt_meta_cap,
+
+ .vidioc_enum_input = unicam_enum_input,
+ .vidioc_g_input = unicam_g_input,
+ .vidioc_s_input = unicam_s_input,
+
+ .vidioc_querystd = unicam_querystd,
+ .vidioc_s_std = unicam_s_std,
+ .vidioc_g_std = unicam_g_std,
+
+ .vidioc_g_edid = unicam_g_edid,
+ .vidioc_s_edid = unicam_s_edid,
+
+ .vidioc_enum_framesizes = unicam_enum_framesizes,
+ .vidioc_enum_frameintervals = unicam_enum_frameintervals,
+
+ .vidioc_g_selection = unicam_g_selection,
+ .vidioc_s_selection = unicam_s_selection,
+
+ .vidioc_g_parm = unicam_g_parm,
+ .vidioc_s_parm = unicam_s_parm,
+
+ .vidioc_s_dv_timings = unicam_s_dv_timings,
+ .vidioc_g_dv_timings = unicam_g_dv_timings,
+ .vidioc_query_dv_timings = unicam_query_dv_timings,
+ .vidioc_enum_dv_timings = unicam_enum_dv_timings,
+ .vidioc_dv_timings_cap = unicam_dv_timings_cap,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+
+ .vidioc_log_status = unicam_log_status,
+ .vidioc_subscribe_event = unicam_subscribe_event,
+ .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+static int unicam_async_bound(struct v4l2_async_notifier *notifier,
+ struct v4l2_subdev *subdev,
+ struct v4l2_async_subdev *asd)
+{
+ struct unicam_device *unicam = to_unicam_device(notifier->v4l2_dev);
+
+ if (unicam->sensor) {
+ dev_info(unicam->dev, "Rejecting subdev %s (Already set!!)",
+ subdev->name);
+ return 0;
+ }
+
+ unicam->sensor = subdev;
+ dev_dbg(unicam->dev, "Using sensor %s for capture\n", subdev->name);
+
+ return 0;
+}
+
+static void unicam_release(struct v4l2_device *v4l2_dev)
+{
+ struct unicam_device *unicam =
+ container_of(v4l2_dev, struct unicam_device, v4l2_dev);
+
+ v4l2_ctrl_handler_free(&unicam->ctrl_handler);
+ media_device_cleanup(&unicam->mdev);
+
+ if (unicam->sensor_config)
+ v4l2_subdev_free_pad_config(unicam->sensor_config);
+
+ kfree(unicam);
+}
+
+static int register_node(struct unicam_device *unicam, struct unicam_node *node,
+ enum v4l2_buf_type type, int node_id)
+{
+ struct video_device *vdev;
+ struct vb2_queue *q;
+ struct v4l2_mbus_framefmt mbus_fmt = {0};
+ const struct unicam_fmt *fmt;
+ int ret;
+
+ if (node_id == IMAGE_NODE) {
+ ret = __subdev_get_format(unicam, &mbus_fmt, node_id);
+ if (ret) {
+ dev_err(unicam->dev, "Failed to get_format - ret %d\n",
+ ret);
+ return ret;
+ }
+
+ fmt = find_format_by_code(mbus_fmt.code);
+ if (!fmt) {
+ /*
+ * Find the first format that the sensor and unicam both
+ * support
+ */
+ fmt = get_first_supported_format(unicam);
+
+ if (!fmt)
+ /* No compatible formats */
+ return -EINVAL;
+
+ mbus_fmt.code = fmt->code;
+ ret = __subdev_set_format(unicam, &mbus_fmt, node_id);
+ if (ret)
+ return -EINVAL;
+ }
+ if (mbus_fmt.field != V4L2_FIELD_NONE) {
+ /* Interlaced not supported - disable it now. */
+ mbus_fmt.field = V4L2_FIELD_NONE;
+ ret = __subdev_set_format(unicam, &mbus_fmt, node_id);
+ if (ret)
+ return -EINVAL;
+ }
+
+ node->v_fmt.fmt.pix.pixelformat = fmt->fourcc ? fmt->fourcc
+ : fmt->repacked_fourcc;
+ } else {
+ /* Fix this node format as embedded data. */
+ fmt = find_format_by_code(MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA);
+ node->v_fmt.fmt.meta.dataformat = fmt->fourcc;
+ }
+
+ node->dev = unicam;
+ node->node_id = node_id;
+ node->fmt = fmt;
+
+ /* Read current subdev format */
+ unicam_reset_format(node);
+
+ if (v4l2_subdev_has_op(unicam->sensor, video, s_std)) {
+ v4l2_std_id tvnorms;
+
+ if (WARN_ON(!v4l2_subdev_has_op(unicam->sensor, video,
+ g_tvnorms)))
+ /*
+ * Subdevice should not advertise s_std but not
+ * g_tvnorms
+ */
+ return -EINVAL;
+
+ ret = v4l2_subdev_call(unicam->sensor, video,
+ g_tvnorms, &tvnorms);
+ if (WARN_ON(ret))
+ return -EINVAL;
+ node->video_dev.tvnorms |= tvnorms;
+ }
+
+ spin_lock_init(&node->dma_queue_lock);
+ mutex_init(&node->lock);
+
+ vdev = &node->video_dev;
+ if (node_id == IMAGE_NODE) {
+ /* Add controls from the subdevice */
+ ret = v4l2_ctrl_add_handler(&unicam->ctrl_handler,
+ unicam->sensor->ctrl_handler, NULL,
+ true);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * If the sensor subdevice has any controls, associate the node
+ * with the ctrl handler to allow access from userland.
+ */
+ if (!list_empty(&unicam->ctrl_handler.ctrls))
+ vdev->ctrl_handler = &unicam->ctrl_handler;
+ }
+
+ q = &node->buffer_queue;
+ q->type = type;
+ q->io_modes = VB2_MMAP | VB2_DMABUF | VB2_READ;
+ q->drv_priv = node;
+ q->ops = &unicam_video_qops;
+ q->mem_ops = &vb2_dma_contig_memops;
+ q->buf_struct_size = sizeof(struct unicam_buffer);
+ q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+ q->lock = &node->lock;
+ q->min_buffers_needed = 1;
+ q->dev = unicam->dev;
+
+ ret = vb2_queue_init(q);
+ if (ret) {
+ dev_err(unicam->dev, "vb2_queue_init() failed\n");
+ return ret;
+ }
+
+ INIT_LIST_HEAD(&node->dma_queue);
+
+ vdev->release = video_device_release_empty;
+ vdev->fops = &unicam_fops;
+ vdev->ioctl_ops = &unicam_ioctl_ops;
+ vdev->v4l2_dev = &unicam->v4l2_dev;
+ vdev->vfl_dir = VFL_DIR_RX;
+ vdev->queue = q;
+ vdev->lock = &node->lock;
+ vdev->device_caps = (node_id == IMAGE_NODE) ?
+ V4L2_CAP_VIDEO_CAPTURE : V4L2_CAP_META_CAPTURE;
+ vdev->device_caps |= V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
+
+ /* Define the device names */
+ snprintf(vdev->name, sizeof(vdev->name), "%s-%s", UNICAM_MODULE_NAME,
+ node_id == IMAGE_NODE ? "image" : "embedded");
+
+ video_set_drvdata(vdev, node);
+ if (node_id == IMAGE_NODE)
+ vdev->entity.flags |= MEDIA_ENT_FL_DEFAULT;
+ node->pad.flags = MEDIA_PAD_FL_SINK;
+ media_entity_pads_init(&vdev->entity, 1, &node->pad);
+
+ node->dummy_buf_cpu_addr = dma_alloc_coherent(unicam->dev,
+ DUMMY_BUF_SIZE,
+ &node->dummy_buf_dma_addr,
+ GFP_KERNEL);
+ if (!node->dummy_buf_cpu_addr) {
+ dev_err(unicam->dev, "Unable to allocate dummy buffer.\n");
+ return -ENOMEM;
+ }
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, video, s_std)) {
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_STD);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_STD);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_ENUMSTD);
+ }
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, video, querystd))
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_QUERYSTD);
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, video, s_dv_timings)) {
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_EDID);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_EDID);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_DV_TIMINGS_CAP);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_DV_TIMINGS);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_DV_TIMINGS);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_ENUM_DV_TIMINGS);
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_QUERY_DV_TIMINGS);
+ }
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, pad, enum_frame_interval))
+ v4l2_disable_ioctl(&node->video_dev,
+ VIDIOC_ENUM_FRAMEINTERVALS);
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, video, g_frame_interval))
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_PARM);
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, video, s_frame_interval))
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_PARM);
+
+ if (node_id == METADATA_NODE ||
+ !v4l2_subdev_has_op(unicam->sensor, pad, enum_frame_size))
+ v4l2_disable_ioctl(&node->video_dev, VIDIOC_ENUM_FRAMESIZES);
+
+ ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+ if (ret) {
+ dev_err(unicam->dev, "Unable to register video device %s\n",
+ vdev->name);
+ return ret;
+ }
+
+ node->registered = true;
+
+ ret = media_create_pad_link(&unicam->sensor->entity, node_id,
+ &node->video_dev.entity, 0,
+ MEDIA_LNK_FL_ENABLED |
+ MEDIA_LNK_FL_IMMUTABLE);
+ if (ret)
+ dev_err(unicam->dev, "Unable to create link for %s\n",
+ vdev->name);
+
+ return ret;
+}
+
+static void unregister_nodes(struct unicam_device *unicam)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
+ struct unicam_node *node = &unicam->node[i];
+
+ if (node->dummy_buf_cpu_addr) {
+ dma_free_coherent(unicam->dev, DUMMY_BUF_SIZE,
+ node->dummy_buf_cpu_addr,
+ node->dummy_buf_dma_addr);
+ }
+
+ if (node->registered) {
+ node->registered = false;
+ video_unregister_device(&node->video_dev);
+ }
+ }
+}
+
+static int unicam_async_complete(struct v4l2_async_notifier *notifier)
+{
+ struct unicam_device *unicam = to_unicam_device(notifier->v4l2_dev);
+ int ret;
+
+ unicam->v4l2_dev.notify = unicam_notify;
+
+ unicam->sensor_config = v4l2_subdev_alloc_pad_config(unicam->sensor);
+ if (!unicam->sensor_config)
+ return -ENOMEM;
+
+ ret = register_node(unicam, &unicam->node[IMAGE_NODE],
+ V4L2_BUF_TYPE_VIDEO_CAPTURE, IMAGE_NODE);
+ if (ret) {
+ dev_err(unicam->dev, "Unable to register image video device.\n");
+ goto unregister;
+ }
+
+ /*
+ * Conditionally register the medatata video device if sensor provides
+ * an additional pad for embedded data.
+ */
+ if (unicam->sensor->entity.num_pads >= 2) {
+ ret = register_node(unicam, &unicam->node[METADATA_NODE],
+ V4L2_BUF_TYPE_META_CAPTURE, METADATA_NODE);
+ if (ret) {
+ dev_err(unicam->dev,
+ "Unable to register metadata video device.\n");
+ goto unregister;
+ }
+ }
+
+ ret = v4l2_device_register_ro_subdev_nodes(&unicam->v4l2_dev);
+ if (ret) {
+ dev_err(unicam->dev, "Unable to register subdev nodes.\n");
+ goto unregister;
+ }
+ unicam->v4l2_dev.release = unicam_release;
+
+ return 0;
+
+unregister:
+ unregister_nodes(unicam);
+
+ return ret;
+}
+
+static const struct v4l2_async_notifier_operations unicam_async_ops = {
+ .bound = unicam_async_bound,
+ .complete = unicam_async_complete,
+};
+
+static int of_unicam_connect_subdevs(struct unicam_device *dev)
+{
+ struct v4l2_fwnode_endpoint ep = { 0 };
+ struct device_node *ep_node;
+ int ret;
+
+ if (of_property_read_u32(dev->dev->of_node, "brcm,num-data-lanes",
+ &dev->max_data_lanes) < 0) {
+ dev_err(dev->dev, "number of data lanes not set\n");
+ return -EINVAL;
+ }
+
+ /* Get the local endpoint and remote device. */
+ ep_node = of_graph_get_next_endpoint(dev->dev->of_node, NULL);
+ if (!ep_node) {
+ dev_err(dev->dev, "can't get next endpoint\n");
+ return -EINVAL;
+ }
+
+ /* Parse the local endpoint and validate its configuration. */
+ v4l2_fwnode_endpoint_parse(of_fwnode_handle(ep_node), &ep);
+ dev->bus_type = ep.bus_type;
+
+ switch (ep.bus_type) {
+ case V4L2_MBUS_CSI2_DPHY:
+ switch (ep.bus.mipi_csi2.num_data_lanes) {
+ case 1:
+ case 2:
+ case 4:
+ break;
+
+ default:
+ dev_err(dev->dev, "%u data lanes not supported\n",
+ ep.bus.mipi_csi2.num_data_lanes);
+ goto err_ep_config;
+ }
+
+ if (ep.bus.mipi_csi2.num_data_lanes > dev->max_data_lanes) {
+ dev_err(dev->dev,
+ "%u lanes requested when %u are supported\n",
+ ep.bus.mipi_csi2.num_data_lanes,
+ dev->max_data_lanes);
+ goto err_ep_config;
+ }
+
+ dev->max_data_lanes = ep.bus.mipi_csi2.num_data_lanes;
+ dev->bus_flags = ep.bus.mipi_csi2.flags;
+ break;
+
+ case V4L2_MBUS_CCP2:
+ if (ep.bus.mipi_csi1.clock_lane != 0 ||
+ ep.bus.mipi_csi1.data_lane != 1) {
+ dev_err(dev->dev, "unsupported lanes configuration\n");
+ goto err_ep_config;
+ }
+
+ dev->max_data_lanes = 1;
+ dev->bus_flags = ep.bus.mipi_csi1.strobe;
+ break;
+
+ default:
+ /* Unsupported bus type */
+ dev_err(dev->dev, " unsupported bus type %u\n", ep.bus_type);
+ goto err_ep_config;
+ }
+
+ /* Initialize and register the async notifier. */
+ v4l2_async_notifier_init(&dev->notifier);
+ dev->notifier.ops = &unicam_async_ops;
+
+ ret = v4l2_async_notifier_add_fwnode_remote_subdev(&dev->notifier,
+ of_fwnode_handle(ep_node), &dev->asd);
+ of_node_put(ep_node);
+ if (ret) {
+ dev_err(dev->dev, "Error adding subdevice: %d\n", ret);
+ return ret;
+ }
+
+ ret = v4l2_async_notifier_register(&dev->v4l2_dev, &dev->notifier);
+ if (ret) {
+ dev_err(dev->dev, "Error registering async notifier: %d\n",
+ ret);
+ return ret;
+ }
+
+ return 0;
+
+err_ep_config:
+ of_node_put(ep_node);
+ return -EINVAL;
+}
+
+static int unicam_probe(struct platform_device *pdev)
+{
+ struct unicam_device *unicam;
+ int ret;
+
+ unicam = devm_kzalloc(&pdev->dev, sizeof(*unicam), GFP_KERNEL);
+ if (!unicam)
+ return -ENOMEM;
+
+ unicam->dev = &pdev->dev;
+ unicam->base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(unicam->base)) {
+ dev_err(unicam->dev, "Failed to get main io block\n");
+ return PTR_ERR(unicam->base);
+ }
+
+ unicam->clk_gate_base = devm_platform_ioremap_resource(pdev, 1);
+ if (IS_ERR(unicam->clk_gate_base)) {
+ dev_err(unicam->dev, "Failed to get 2nd io block\n");
+ return PTR_ERR(unicam->clk_gate_base);
+ }
+
+ unicam->clock = devm_clk_get(&pdev->dev, "lp");
+ if (IS_ERR(unicam->clock)) {
+ dev_err(unicam->dev, "Failed to get lp clock\n");
+ return PTR_ERR(unicam->clock);
+ }
+
+ unicam->vpu_clock = devm_clk_get(&pdev->dev, "vpu");
+ if (IS_ERR(unicam->vpu_clock)) {
+ dev_err(unicam->dev, "Failed to get vpu clock\n");
+ return PTR_ERR(unicam->vpu_clock);
+ }
+
+ ret = platform_get_irq(pdev, 0);
+ if (ret <= 0) {
+ dev_err(&pdev->dev, "No IRQ resource\n");
+ return -EINVAL;
+ }
+
+ ret = devm_request_irq(&pdev->dev, ret, unicam_isr, 0,
+ "unicam_capture0", unicam);
+ if (ret) {
+ dev_err(&pdev->dev, "Unable to request interrupt\n");
+ return -EINVAL;
+ }
+
+ unicam->mdev.dev = &pdev->dev;
+ strscpy(unicam->mdev.model, UNICAM_MODULE_NAME,
+ sizeof(unicam->mdev.model));
+ strscpy(unicam->mdev.serial, "", sizeof(unicam->mdev.serial));
+ snprintf(unicam->mdev.bus_info, sizeof(unicam->mdev.bus_info),
+ "platform:%s", dev_name(&pdev->dev));
+ unicam->mdev.hw_revision = 0;
+
+ media_device_init(&unicam->mdev);
+
+ unicam->v4l2_dev.mdev = &unicam->mdev;
+
+ ret = v4l2_device_register(&pdev->dev, &unicam->v4l2_dev);
+ if (ret) {
+ dev_err(unicam->dev, "Unable to register v4l2 device.\n");
+ return ret;
+ }
+
+ ret = media_device_register(&unicam->mdev);
+ if (ret < 0) {
+ dev_err(unicam->dev, "Unable to register media device.\n");
+ goto err_v4l2_unregister;
+ }
+
+ /* Reserve space for the controls */
+ ret = v4l2_ctrl_handler_init(&unicam->ctrl_handler, 16);
+ if (ret < 0)
+ goto err_media_unregister;
+
+ /* set the driver data in platform device */
+ platform_set_drvdata(pdev, unicam);
+
+ ret = of_unicam_connect_subdevs(unicam);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to connect subdevs\n");
+ goto err_media_unregister;
+ }
+
+ pm_runtime_enable(&pdev->dev);
+
+ return 0;
+
+err_media_unregister:
+ media_device_unregister(&unicam->mdev);
+err_v4l2_unregister:
+ v4l2_device_unregister(&unicam->v4l2_dev);
+
+ return ret;
+}
+
+static int unicam_remove(struct platform_device *pdev)
+{
+ struct unicam_device *unicam = platform_get_drvdata(pdev);
+
+ v4l2_async_notifier_unregister(&unicam->notifier);
+ v4l2_device_unregister(&unicam->v4l2_dev);
+ media_device_unregister(&unicam->mdev);
+ unregister_nodes(unicam);
+
+ pm_runtime_disable(&pdev->dev);
+
+ return 0;
+}
+
+static const struct of_device_id unicam_of_match[] = {
+ { .compatible = "brcm,bcm2835-unicam", },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, unicam_of_match);
+
+static struct platform_driver unicam_driver = {
+ .probe = unicam_probe,
+ .remove = unicam_remove,
+ .driver = {
+ .name = UNICAM_MODULE_NAME,
+ .of_match_table = of_match_ptr(unicam_of_match),
+ },
+};
+
+module_platform_driver(unicam_driver);
+
+MODULE_AUTHOR("Dave Stevenson <dave.stevenson@raspberrypi.com>");
+MODULE_DESCRIPTION("BCM2835 Unicam driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION(UNICAM_VERSION);
diff --git a/drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h b/drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
new file mode 100644
index 0000000000000..ae059a171d0fe
--- /dev/null
+++ b/drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
@@ -0,0 +1,253 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+/*
+ * Copyright (C) 2017-2020 Raspberry Pi Trading.
+ * Dave Stevenson <dave.stevenson@raspberrypi.com>
+ */
+
+#ifndef VC4_REGS_UNICAM_H
+#define VC4_REGS_UNICAM_H
+
+/*
+ * The following values are taken from files found within the code drop
+ * made by Broadcom for the BCM21553 Graphics Driver, predominantly in
+ * brcm_usrlib/dag/vmcsx/vcinclude/hardware_vc4.h.
+ * They have been modified to be only the register offset.
+ */
+#define UNICAM_CTRL 0x000
+#define UNICAM_STA 0x004
+#define UNICAM_ANA 0x008
+#define UNICAM_PRI 0x00c
+#define UNICAM_CLK 0x010
+#define UNICAM_CLT 0x014
+#define UNICAM_DAT0 0x018
+#define UNICAM_DAT1 0x01c
+#define UNICAM_DAT2 0x020
+#define UNICAM_DAT3 0x024
+#define UNICAM_DLT 0x028
+#define UNICAM_CMP0 0x02c
+#define UNICAM_CMP1 0x030
+#define UNICAM_CAP0 0x034
+#define UNICAM_CAP1 0x038
+#define UNICAM_ICTL 0x100
+#define UNICAM_ISTA 0x104
+#define UNICAM_IDI0 0x108
+#define UNICAM_IPIPE 0x10c
+#define UNICAM_IBSA0 0x110
+#define UNICAM_IBEA0 0x114
+#define UNICAM_IBLS 0x118
+#define UNICAM_IBWP 0x11c
+#define UNICAM_IHWIN 0x120
+#define UNICAM_IHSTA 0x124
+#define UNICAM_IVWIN 0x128
+#define UNICAM_IVSTA 0x12c
+#define UNICAM_ICC 0x130
+#define UNICAM_ICS 0x134
+#define UNICAM_IDC 0x138
+#define UNICAM_IDPO 0x13c
+#define UNICAM_IDCA 0x140
+#define UNICAM_IDCD 0x144
+#define UNICAM_IDS 0x148
+#define UNICAM_DCS 0x200
+#define UNICAM_DBSA0 0x204
+#define UNICAM_DBEA0 0x208
+#define UNICAM_DBWP 0x20c
+#define UNICAM_DBCTL 0x300
+#define UNICAM_IBSA1 0x304
+#define UNICAM_IBEA1 0x308
+#define UNICAM_IDI1 0x30c
+#define UNICAM_DBSA1 0x310
+#define UNICAM_DBEA1 0x314
+#define UNICAM_MISC 0x400
+
+/*
+ * The following bitmasks are from the kernel released by Broadcom
+ * for Android - https://android.googlesource.com/kernel/bcm/
+ * The Rhea, Hawaii, and Java chips all contain the same VideoCore4
+ * Unicam block as BCM2835, as defined in eg
+ * arch/arm/mach-rhea/include/mach/rdb_A0/brcm_rdb_cam.h and similar.
+ * Values reworked to use the kernel BIT and GENMASK macros.
+ *
+ * Some of the bit mnenomics have been amended to match the datasheet.
+ */
+/* UNICAM_CTRL Register */
+#define UNICAM_CPE BIT(0)
+#define UNICAM_MEM BIT(1)
+#define UNICAM_CPR BIT(2)
+#define UNICAM_CPM_MASK GENMASK(3, 3)
+#define UNICAM_CPM_CSI2 0
+#define UNICAM_CPM_CCP2 1
+#define UNICAM_SOE BIT(4)
+#define UNICAM_DCM_MASK GENMASK(5, 5)
+#define UNICAM_DCM_STROBE 0
+#define UNICAM_DCM_DATA 1
+#define UNICAM_SLS BIT(6)
+#define UNICAM_PFT_MASK GENMASK(11, 8)
+#define UNICAM_OET_MASK GENMASK(20, 12)
+
+/* UNICAM_STA Register */
+#define UNICAM_SYN BIT(0)
+#define UNICAM_CS BIT(1)
+#define UNICAM_SBE BIT(2)
+#define UNICAM_PBE BIT(3)
+#define UNICAM_HOE BIT(4)
+#define UNICAM_PLE BIT(5)
+#define UNICAM_SSC BIT(6)
+#define UNICAM_CRCE BIT(7)
+#define UNICAM_OES BIT(8)
+#define UNICAM_IFO BIT(9)
+#define UNICAM_OFO BIT(10)
+#define UNICAM_BFO BIT(11)
+#define UNICAM_DL BIT(12)
+#define UNICAM_PS BIT(13)
+#define UNICAM_IS BIT(14)
+#define UNICAM_PI0 BIT(15)
+#define UNICAM_PI1 BIT(16)
+#define UNICAM_FSI_S BIT(17)
+#define UNICAM_FEI_S BIT(18)
+#define UNICAM_LCI_S BIT(19)
+#define UNICAM_BUF0_RDY BIT(20)
+#define UNICAM_BUF0_NO BIT(21)
+#define UNICAM_BUF1_RDY BIT(22)
+#define UNICAM_BUF1_NO BIT(23)
+#define UNICAM_DI BIT(24)
+
+#define UNICAM_STA_MASK_ALL \
+ (UNICAM_DL + \
+ UNICAM_SBE + \
+ UNICAM_PBE + \
+ UNICAM_HOE + \
+ UNICAM_PLE + \
+ UNICAM_SSC + \
+ UNICAM_CRCE + \
+ UNICAM_IFO + \
+ UNICAM_OFO + \
+ UNICAM_PS + \
+ UNICAM_PI0 + \
+ UNICAM_PI1)
+
+/* UNICAM_ANA Register */
+#define UNICAM_APD BIT(0)
+#define UNICAM_BPD BIT(1)
+#define UNICAM_AR BIT(2)
+#define UNICAM_DDL BIT(3)
+#define UNICAM_CTATADJ_MASK GENMASK(7, 4)
+#define UNICAM_PTATADJ_MASK GENMASK(11, 8)
+
+/* UNICAM_PRI Register */
+#define UNICAM_PE BIT(0)
+#define UNICAM_PT_MASK GENMASK(2, 1)
+#define UNICAM_NP_MASK GENMASK(7, 4)
+#define UNICAM_PP_MASK GENMASK(11, 8)
+#define UNICAM_BS_MASK GENMASK(15, 12)
+#define UNICAM_BL_MASK GENMASK(17, 16)
+
+/* UNICAM_CLK Register */
+#define UNICAM_CLE BIT(0)
+#define UNICAM_CLPD BIT(1)
+#define UNICAM_CLLPE BIT(2)
+#define UNICAM_CLHSE BIT(3)
+#define UNICAM_CLTRE BIT(4)
+#define UNICAM_CLAC_MASK GENMASK(8, 5)
+#define UNICAM_CLSTE BIT(29)
+
+/* UNICAM_CLT Register */
+#define UNICAM_CLT1_MASK GENMASK(7, 0)
+#define UNICAM_CLT2_MASK GENMASK(15, 8)
+
+/* UNICAM_DATn Registers */
+#define UNICAM_DLE BIT(0)
+#define UNICAM_DLPD BIT(1)
+#define UNICAM_DLLPE BIT(2)
+#define UNICAM_DLHSE BIT(3)
+#define UNICAM_DLTRE BIT(4)
+#define UNICAM_DLSM BIT(5)
+#define UNICAM_DLFO BIT(28)
+#define UNICAM_DLSTE BIT(29)
+
+#define UNICAM_DAT_MASK_ALL (UNICAM_DLSTE + UNICAM_DLFO)
+
+/* UNICAM_DLT Register */
+#define UNICAM_DLT1_MASK GENMASK(7, 0)
+#define UNICAM_DLT2_MASK GENMASK(15, 8)
+#define UNICAM_DLT3_MASK GENMASK(23, 16)
+
+/* UNICAM_ICTL Register */
+#define UNICAM_FSIE BIT(0)
+#define UNICAM_FEIE BIT(1)
+#define UNICAM_IBOB BIT(2)
+#define UNICAM_FCM BIT(3)
+#define UNICAM_TFC BIT(4)
+#define UNICAM_LIP_MASK GENMASK(6, 5)
+#define UNICAM_LCIE_MASK GENMASK(28, 16)
+
+/* UNICAM_IDI0/1 Register */
+#define UNICAM_ID0_MASK GENMASK(7, 0)
+#define UNICAM_ID1_MASK GENMASK(15, 8)
+#define UNICAM_ID2_MASK GENMASK(23, 16)
+#define UNICAM_ID3_MASK GENMASK(31, 24)
+
+/* UNICAM_ISTA Register */
+#define UNICAM_FSI BIT(0)
+#define UNICAM_FEI BIT(1)
+#define UNICAM_LCI BIT(2)
+
+#define UNICAM_ISTA_MASK_ALL (UNICAM_FSI + UNICAM_FEI + UNICAM_LCI)
+
+/* UNICAM_IPIPE Register */
+#define UNICAM_PUM_MASK GENMASK(2, 0)
+ /* Unpacking modes */
+ #define UNICAM_PUM_NONE 0
+ #define UNICAM_PUM_UNPACK6 1
+ #define UNICAM_PUM_UNPACK7 2
+ #define UNICAM_PUM_UNPACK8 3
+ #define UNICAM_PUM_UNPACK10 4
+ #define UNICAM_PUM_UNPACK12 5
+ #define UNICAM_PUM_UNPACK14 6
+ #define UNICAM_PUM_UNPACK16 7
+#define UNICAM_DDM_MASK GENMASK(6, 3)
+#define UNICAM_PPM_MASK GENMASK(9, 7)
+ /* Packing modes */
+ #define UNICAM_PPM_NONE 0
+ #define UNICAM_PPM_PACK8 1
+ #define UNICAM_PPM_PACK10 2
+ #define UNICAM_PPM_PACK12 3
+ #define UNICAM_PPM_PACK14 4
+ #define UNICAM_PPM_PACK16 5
+#define UNICAM_DEM_MASK GENMASK(11, 10)
+#define UNICAM_DEBL_MASK GENMASK(14, 12)
+#define UNICAM_ICM_MASK GENMASK(16, 15)
+#define UNICAM_IDM_MASK GENMASK(17, 17)
+
+/* UNICAM_ICC Register */
+#define UNICAM_ICFL_MASK GENMASK(4, 0)
+#define UNICAM_ICFH_MASK GENMASK(9, 5)
+#define UNICAM_ICST_MASK GENMASK(12, 10)
+#define UNICAM_ICLT_MASK GENMASK(15, 13)
+#define UNICAM_ICLL_MASK GENMASK(31, 16)
+
+/* UNICAM_DCS Register */
+#define UNICAM_DIE BIT(0)
+#define UNICAM_DIM BIT(1)
+#define UNICAM_DBOB BIT(3)
+#define UNICAM_FDE BIT(4)
+#define UNICAM_LDP BIT(5)
+#define UNICAM_EDL_MASK GENMASK(15, 8)
+
+/* UNICAM_DBCTL Register */
+#define UNICAM_DBEN BIT(0)
+#define UNICAM_BUF0_IE BIT(1)
+#define UNICAM_BUF1_IE BIT(2)
+
+/* UNICAM_CMP[0,1] register */
+#define UNICAM_PCE BIT(31)
+#define UNICAM_GI BIT(9)
+#define UNICAM_CPH BIT(8)
+#define UNICAM_PCVC_MASK GENMASK(7, 6)
+#define UNICAM_PCDT_MASK GENMASK(5, 0)
+
+/* UNICAM_MISC register */
+#define UNICAM_FL0 BIT(6)
+#define UNICAM_FL1 BIT(9)
+
+#endif
--
2.29.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v4 4/5] ARM: dts: bcm2711: Add Unicam DT nodes
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
` (2 preceding siblings ...)
2020-11-10 17:40 ` [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface Jacopo Mondi
@ 2020-11-10 17:40 ` Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file Jacopo Mondi
2020-12-01 9:26 ` [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
5 siblings, 0 replies; 31+ messages in thread
From: Jacopo Mondi @ 2020-11-10 17:40 UTC (permalink / raw)
To: linux-media
Cc: Jacopo Mondi, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
sakari.ailus, nsaenzjulienne, mchehab+huawei
From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
The Unicam IP cores are CSI-2/CCP2 receivers. They are part of VC4, but
can be controlled directly from the operating system without going
through the VPU firmware. Exposes them as DT nodes.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
---
arch/arm/boot/dts/bcm2711.dtsi | 41 ++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/arch/arm/boot/dts/bcm2711.dtsi b/arch/arm/boot/dts/bcm2711.dtsi
index 4847dd305317a..68ac76e7c772d 100644
--- a/arch/arm/boot/dts/bcm2711.dtsi
+++ b/arch/arm/boot/dts/bcm2711.dtsi
@@ -2,6 +2,7 @@
#include "bcm283x.dtsi"
#include <dt-bindings/interrupt-controller/arm-gic.h>
+#include <dt-bindings/power/raspberrypi-power.h>
#include <dt-bindings/soc/bcm2835-pm.h>
/ {
@@ -293,6 +294,46 @@ hvs: hvs@7e400000 {
interrupts = <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>;
};
+ csi0: csi@7e800000 {
+ compatible = "brcm,bcm2835-unicam";
+ reg = <0x7e800000 0x800>,
+ <0x7e802000 0x4>;
+ interrupts = <GIC_SPI 102 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&clocks BCM2835_CLOCK_CAM0>,
+ <&firmware_clocks 4>;
+ clock-names = "lp", "vpu";
+ power-domains = <&power RPI_POWER_DOMAIN_UNICAM0>;
+ brcm,num-data-lanes = <2>;
+ #clock-cells = <1>;
+ status = "disabled";
+
+ port {
+ endpoint {
+ data-lanes = <1 2>;
+ };
+ };
+ };
+
+ csi1: csi@7e801000 {
+ compatible = "brcm,bcm2835-unicam";
+ reg = <0x7e801000 0x800>,
+ <0x7e802004 0x4>;
+ interrupts = <GIC_SPI 103 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&clocks BCM2835_CLOCK_CAM1>,
+ <&firmware_clocks 4>;
+ clock-names = "lp", "vpu";
+ power-domains = <&power RPI_POWER_DOMAIN_UNICAM1>;
+ brcm,num-data-lanes = <4>;
+ #clock-cells = <1>;
+ status = "disabled";
+
+ port {
+ endpoint {
+ data-lanes = <1 2 3 4>;
+ };
+ };
+ };
+
pixelvalve3: pixelvalve@7ec12000 {
compatible = "brcm,bcm2711-pixelvalve3";
reg = <0x7ec12000 0x100>;
--
2.29.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
` (3 preceding siblings ...)
2020-11-10 17:40 ` [PATCH v4 4/5] ARM: dts: bcm2711: Add Unicam DT nodes Jacopo Mondi
@ 2020-11-10 17:40 ` Jacopo Mondi
2020-12-02 22:06 ` Sakari Ailus
2020-12-01 9:26 ` [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
5 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-11-10 17:40 UTC (permalink / raw)
To: linux-media
Cc: Jacopo Mondi, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
sakari.ailus, nsaenzjulienne, mchehab+huawei
The bcm2835-unicam driver is currently in staging mainly for
two reasons:
- Handling of CSI-2 embedded data
- Usage of both media controller API and subdev kAPI
Provide a more detailed description of the currently on-going design
discussions in the associated TODO file.
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
---
drivers/staging/media/bcm2835-unicam/TODO | 37 +++++++++++++++++++++++
1 file changed, 37 insertions(+)
create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
diff --git a/drivers/staging/media/bcm2835-unicam/TODO b/drivers/staging/media/bcm2835-unicam/TODO
new file mode 100644
index 0000000000000..c7840872eea4c
--- /dev/null
+++ b/drivers/staging/media/bcm2835-unicam/TODO
@@ -0,0 +1,37 @@
+BCM2835 Unicam driver TODO list
+===============================
+
+The unicam driver could be considered ready to be moved out of the staging
+directory in terms of code quality and expected functionalities.
+
+However there currently are two design issues that suggest the driver is
+better kept in staging for the time being.
+
+CSI-2 Embedded data support:
+----------------------------
+
+The RaspberryPi camera stack and camera applications rely on the availability of
+the sensor produced CSI-2 embedded data, whose support is currently not
+finalized in mainline Linux.
+
+The driver conditionally registers an additional video device node
+'unicam-embedded' with a single sink pad which connects to the sensor
+sub-device source pad #1 to expose ancillary data.
+
+Currently none of the mainline sensor drivers register more than a single pad,
+and consequentially no embedded data from the sensor are exposed to userspace.
+
+The current implementation is then subject to changes depending on how support
+for CSI-2 embedded data gets finalized in Linux.
+
+Media controller support:
+-------------------------
+
+Due to compatibility reasons with the existing RaspberryPi software ecosystem
+the unicam driver implements the media controller interface to allow the
+enumeration of the connected entities but handles the configuration of the
+sensor sub-device using the v4l2-subdev kAPI instead of delegating that to
+user-space.
+
+Discussions are on-going on how this should be better handled (driver option,
+KConfig option etc etc).
--
2.29.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
` (4 preceding siblings ...)
2020-11-10 17:40 ` [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file Jacopo Mondi
@ 2020-12-01 9:26 ` Jacopo Mondi
2020-12-01 12:57 ` Sakari Ailus
5 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-01 9:26 UTC (permalink / raw)
To: linux-media, hverkuil, sakari.ailus
Cc: naush, dave.stevenson, laurent.pinchart, kieran.bingham,
niklas.soderlund, dafna.hirschfeld, nsaenzjulienne,
mchehab+huawei
Hi Hans, Sakari,
On Tue, Nov 10, 2020 at 06:40:31PM +0100, Jacopo Mondi wrote:
> Hello,
> new iteration following
> v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
>
> Major changelog:
> - Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
> - Address Hans' comments on queue setup and metadata format handling function
> - s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
> Hans and rebase on Dafna's fixed metadata format patch
> - Add a TODO file to explain why the driver is in staging.
> - Conditionally register the unicam-embedded video device node on the presence
> of the sensor's metadata source pad.
>
> The media graph, when connected to a sensor that does not report metadata
> looks like:
>
> Media controller API version 5.10.0
>
> Media device information
> ------------------------
> driver unicam
> model unicam
> serial
> bus info platform:fe801000.csi
> hw revision 0x0
> driver version 5.10.0
>
> Device topology
> - entity 1: ov5647 10-0036 (1 pad, 1 link)
> type V4L2 subdev subtype Sensor flags 0
> device node name /dev/v4l-subdev0
> pad0: Source
> [fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
> crop.bounds:(16,16)/2592x1944
> crop:(32,16)/2560x1920]
> -> "unicam-image":0 [ENABLED,IMMUTABLE]
>
> - entity 3: unicam-image (1 pad, 1 link)
> type Node subtype V4L flags 1
> device node name /dev/video0
> pad0: Sink
> <- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
>
>
> If the sensor reports an additional metadata pad:
>
> Media controller API version 5.10.0
>
> Media device information
> ------------------------
> driver unicam
> model unicam
> serial
> bus info platform:fe801000.csi
> hw revision 0x0
> driver version 5.10.0
>
> Device topology
> - entity 1: imx219 10-0010 (2 pads, 2 links)
> type V4L2 subdev subtype Sensor flags 0
> device node name /dev/v4l-subdev0
> pad0: Source
> [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> crop:(0,0)/3280x2464]
> -> "unicam-image":0 [ENABLED,IMMUTABLE]
> pad1: Source
> [fmt:unknown/16384x1 field:none
> crop:(0,0)/3280x2464]
> -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
>
> - entity 4: unicam-image (1 pad, 1 link)
> type Node subtype V4L flags 1
> device node name /dev/video0
> pad0: Sink
> <- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
>
> - entity 10: unicam-embedded (1 pad, 1 link)
> type Node subtype V4L flags 0
> device node name /dev/video1
> pad0: Sink
> <- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
>
> Conditionally registering the metadata video node allows to simplify the
> code in the driver as well, removing the 'sensor_embedded_data' flag.
>
> An additional note: this version will break the libcamera pipeline handler
> which assume the unicam-embedded video device node to always be there.
>
> From Dave's reply to Dafna's comments I get instead that for the existing
> applications ecosystem, having the metadata node not registered if the sensor
> does not support it is not an issue.
I think I've closed comments received on v3.
Do you think this series is ready for being collected ?
Thanks
j
>
> The v4l2-compliance output is reported below, with one test failing, again
> on the selection API, and again I'm not sure about the root cause. Seems like
> the compliance tool complains because a list of discrete formats is reported and
> the selection APIs are supported at the same time. Seems unlikely though.
>
> -------------------------------------------------------------------------------
> v4l2-compliance 1.21.0-4679, 32 bits, 32-bit time_t
> v4l2-compliance SHA: 225c6c2a17be 2020-10-30 15:13:07
>
> Compliance test for unicam device /dev/media0:
>
> Media Driver Info:
> Driver name : unicam
> Model : unicam
> Serial :
> Bus info : platform:fe801000.csi
> Media version : 5.10.0
> Hardware revision: 0x00000000 (0)
> Driver version : 5.10.0
>
> Required ioctls:
> test MEDIA_IOC_DEVICE_INFO: OK
>
> Allow for multiple opens:
> test second /dev/media0 open: OK
> test MEDIA_IOC_DEVICE_INFO: OK
> test for unlimited opens: OK
>
> Media Controller ioctls:
> test MEDIA_IOC_G_TOPOLOGY: OK
> Entities: 2 Interfaces: 2 Pads: 2 Links: 3
> test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
> test MEDIA_IOC_SETUP_LINK: OK
> test invalid ioctls: OK
>
> Total for unicam device /dev/media0: 8, Succeeded: 8, Failed: 0, Warnings: 0
> --------------------------------------------------------------------------------
> Compliance test for unicam device /dev/video0:
>
> Driver Info:
> Driver name : unicam
> Card type : unicam
> Bus info : platform:fe801000.csi
> Driver version : 5.10.0
> Capabilities : 0x85a00001
> Video Capture
> Metadata Capture
> Read/Write
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x05200001
> Video Capture
> Read/Write
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : unicam
> Model : unicam
> Serial :
> Bus info : platform:fe801000.csi
> Media version : 5.10.0
> Hardware revision: 0x00000000 (0)
> Driver version : 5.10.0
> Interface Info:
> ID : 0x03000005
> Type : V4L Video
> Entity Info:
> ID : 0x00000003 (3)
> Name : unicam-image
> Function : V4L2 I/O
> Flags : default
> Pad 0x01000004 : 0: Sink
> Link 0x02000007: from remote pad 0x1000002 of entity 'ov5647 10-0036': Data, Enabled, Immutable
>
> Required ioctls:
> test MC information (see 'Media Driver Info' above): OK
> test VIDIOC_QUERYCAP: OK
>
> Allow for multiple opens:
> test second /dev/video0 open: OK
> test VIDIOC_QUERYCAP: OK
> test VIDIOC_G/S_PRIORITY: OK
> test for unlimited opens: OK
>
> test invalid ioctls: OK
> Debug ioctls:
> test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> test VIDIOC_LOG_STATUS: OK
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 1 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls (Input 0):
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
> test VIDIOC_QUERYCTRL: OK
> test VIDIOC_G/S_CTRL: OK
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 12 Private Controls: 0
>
> Format ioctls (Input 0):
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK
> test VIDIOC_TRY_FMT: OK
> test VIDIOC_S_FMT: OK
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> fail: v4l2-test-formats.cpp(1505): node->frmsizes_count[pixfmt] > 1
> test Cropping: FAIL
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls (Input 0):
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls (Input 0):
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> test VIDIOC_EXPBUF: OK
> test Requests: OK (Not Supported)
>
> Test input 0:
>
> Streaming ioctls:
> test read/write: OK
> test blocking wait: OK
> test MMAP (no poll): OK
> test MMAP (select): OK
> test MMAP (epoll): OK
> test USERPTR (no poll): OK (Not Supported)
> test USERPTR (select): OK (Not Supported)
> test DMABUF: Cannot test, specify --expbuf-device
>
> Stream using all formats:
> test MMAP for Format pBAA, Frame Size 2592x1944:
> Crop 2592x1944@16x16, Stride 3264, Field None: OK
> Crop 2592x1944@16x16, Stride 3328, Field None: OK
> test MMAP for Format pBAA, Frame Size 1920x1080:
> Crop 1928x1080@364x450, Stride 2400, Field None: OK
> Crop 1928x1080@364x450, Stride 2464, Field None: OK
> test MMAP for Format pBAA, Frame Size 1296x972:
> Crop 2592x1944@16x16, Stride 1632, Field None: OK
> Crop 2592x1944@16x16, Stride 1696, Field None: OK
> test MMAP for Format pBAA, Frame Size 640x480:
> Crop 2560x1920@32x16, Stride 800, Field None: OK
> Crop 2560x1920@32x16, Stride 864, Field None: OK
> test MMAP for Format BG10, Frame Size 2592x1944:
> Crop 2592x1944@16x16, Stride 5184, Field None: OK
> Crop 2592x1944@16x16, Stride 5248, Field None: OK
> test MMAP for Format BG10, Frame Size 1920x1080:
> Crop 1928x1080@364x450, Stride 3840, Field None: OK
> Crop 1928x1080@364x450, Stride 3904, Field None: OK
> test MMAP for Format BG10, Frame Size 1296x972:
> Crop 2592x1944@16x16, Stride 2592, Field None: OK
> Crop 2592x1944@16x16, Stride 2656, Field None: OK
> test MMAP for Format BG10, Frame Size 640x480:
> Crop 2560x1920@32x16, Stride 1280, Field None: OK
> Crop 2560x1920@32x16, Stride 1344, Field None: OK
> Total for unicam device /dev/video0: 69, Succeeded: 68, Failed: 1, Warnings: 0
> --------------------------------------------------------------------------------
> Compliance test for device /dev/v4l-subdev0:
>
>
> Required ioctls:
>
> Allow for multiple opens:
> test second /dev/v4l-subdev0 open: OK
> test for unlimited opens: OK
>
> test invalid ioctls: OK
> Debug ioctls:
> test VIDIOC_LOG_STATUS: OK (Not Supported)
>
> Input ioctls:
> test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> test VIDIOC_ENUMAUDIO: OK (Not Supported)
> test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDIO: OK (Not Supported)
> Inputs: 0 Audio Inputs: 0 Tuners: 0
>
> Output ioctls:
> test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> Outputs: 0 Audio Outputs: 0 Modulators: 0
>
> Input/Output configuration ioctls:
> test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> test VIDIOC_G/S_EDID: OK (Not Supported)
>
> Control ioctls:
> test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
> test VIDIOC_QUERYCTRL: OK
> test VIDIOC_G/S_CTRL: OK
> test VIDIOC_G/S/TRY_EXT_CTRLS: OK
> test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
> test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> Standard Controls: 12 Private Controls: 0
>
> Format ioctls:
> test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
> test VIDIOC_G/S_PARM: OK (Not Supported)
> test VIDIOC_G_FBUF: OK (Not Supported)
> test VIDIOC_G_FMT: OK (Not Supported)
> test VIDIOC_TRY_FMT: OK (Not Supported)
> test VIDIOC_S_FMT: OK (Not Supported)
> test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> test Cropping: OK (Not Supported)
> test Composing: OK (Not Supported)
> test Scaling: OK (Not Supported)
>
> Codec ioctls:
> test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
>
> Buffer ioctls:
> test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
> test VIDIOC_EXPBUF: OK (Not Supported)
> test Requests: OK (Not Supported)
>
> Total for device /dev/v4l-subdev0: 41, Succeeded: 41, Failed: 0, Warnings: 0
>
> Grand Total for unicam device /dev/media0: 118, Succeeded: 117, Failed: 1, Warnings: 0
> -------------------------------------------------------------------------------
>
> Tested by capturing frames from RPi Camera Module v1 (ov5647) and v2 (imx219)
> in 10-bit formats and 8-bit formats (imx219 only).
>
> Image are captured in a format I cannot inspect (raw2rgbpnm and 7yuv do not
> support the sensor's Bayern pattern), but I can spot shapes when converting to
> a format with a different pattern permutation, so I assume data is there.
>
> Thanks
> j
>
> Dave Stevenson (1):
> dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
>
> Jacopo Mondi (1):
> media: bcm2835-unicam: Add TODO file
>
> Laurent Pinchart (1):
> ARM: dts: bcm2711: Add Unicam DT nodes
>
> Naushir Patuck (2):
> media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
> media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
>
> .../bindings/media/brcm,bcm2835-unicam.yaml | 155 +
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../media/v4l/pixfmt-meta-sensor-data.rst | 24 +
> .../media/v4l/subdev-formats.rst | 29 +
> MAINTAINERS | 7 +
> arch/arm/boot/dts/bcm2711.dtsi | 41 +
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> drivers/staging/media/Kconfig | 2 +
> drivers/staging/media/Makefile | 1 +
> drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
> drivers/staging/media/bcm2835-unicam/Makefile | 3 +
> drivers/staging/media/bcm2835-unicam/TODO | 37 +
> .../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
> .../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
> include/uapi/linux/media-bus-format.h | 3 +
> include/uapi/linux/videodev2.h | 1 +
> 16 files changed, 3329 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
> create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
> create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
> create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
> create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
>
> --
> 2.29.1
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
2020-12-01 9:26 ` [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
@ 2020-12-01 12:57 ` Sakari Ailus
2020-12-01 13:18 ` Laurent Pinchart
2020-12-01 13:24 ` Jacopo Mondi
0 siblings, 2 replies; 31+ messages in thread
From: Sakari Ailus @ 2020-12-01 12:57 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Tue, Dec 01, 2020 at 10:26:05AM +0100, Jacopo Mondi wrote:
> Hi Hans, Sakari,
>
> On Tue, Nov 10, 2020 at 06:40:31PM +0100, Jacopo Mondi wrote:
> > Hello,
> > new iteration following
> > v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
> >
> > Major changelog:
> > - Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
> > - Address Hans' comments on queue setup and metadata format handling function
> > - s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
> > Hans and rebase on Dafna's fixed metadata format patch
> > - Add a TODO file to explain why the driver is in staging.
> > - Conditionally register the unicam-embedded video device node on the presence
> > of the sensor's metadata source pad.
> >
> > The media graph, when connected to a sensor that does not report metadata
> > looks like:
> >
> > Media controller API version 5.10.0
> >
> > Media device information
> > ------------------------
> > driver unicam
> > model unicam
> > serial
> > bus info platform:fe801000.csi
> > hw revision 0x0
> > driver version 5.10.0
> >
> > Device topology
> > - entity 1: ov5647 10-0036 (1 pad, 1 link)
> > type V4L2 subdev subtype Sensor flags 0
> > device node name /dev/v4l-subdev0
> > pad0: Source
> > [fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
> > crop.bounds:(16,16)/2592x1944
> > crop:(32,16)/2560x1920]
> > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> >
> > - entity 3: unicam-image (1 pad, 1 link)
> > type Node subtype V4L flags 1
> > device node name /dev/video0
> > pad0: Sink
> > <- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
> >
> >
> > If the sensor reports an additional metadata pad:
> >
> > Media controller API version 5.10.0
> >
> > Media device information
> > ------------------------
> > driver unicam
> > model unicam
> > serial
> > bus info platform:fe801000.csi
> > hw revision 0x0
> > driver version 5.10.0
> >
> > Device topology
> > - entity 1: imx219 10-0010 (2 pads, 2 links)
> > type V4L2 subdev subtype Sensor flags 0
> > device node name /dev/v4l-subdev0
> > pad0: Source
> > [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> > crop:(0,0)/3280x2464]
> > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > pad1: Source
> > [fmt:unknown/16384x1 field:none
> > crop:(0,0)/3280x2464]
> > -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> >
> > - entity 4: unicam-image (1 pad, 1 link)
> > type Node subtype V4L flags 1
> > device node name /dev/video0
> > pad0: Sink
> > <- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
> >
> > - entity 10: unicam-embedded (1 pad, 1 link)
> > type Node subtype V4L flags 0
> > device node name /dev/video1
> > pad0: Sink
> > <- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
> >
> > Conditionally registering the metadata video node allows to simplify the
> > code in the driver as well, removing the 'sensor_embedded_data' flag.
> >
> > An additional note: this version will break the libcamera pipeline handler
> > which assume the unicam-embedded video device node to always be there.
> >
> > From Dave's reply to Dafna's comments I get instead that for the existing
> > applications ecosystem, having the metadata node not registered if the sensor
> > does not support it is not an issue.
>
> I think I've closed comments received on v3.
>
> Do you think this series is ready for being collected ?
I'll try to look into this later today / this week.
The problem with the approach appears, based on a quick glance, to be that
it creates an additional way (to the more generic approach in VC support
patchset) to support embedded data, including duplicated sensor driver
support. That appears as a dead end. But let me look into the details
first.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
2020-12-01 12:57 ` Sakari Ailus
@ 2020-12-01 13:18 ` Laurent Pinchart
2020-12-02 21:55 ` Sakari Ailus
2020-12-01 13:24 ` Jacopo Mondi
1 sibling, 1 reply; 31+ messages in thread
From: Laurent Pinchart @ 2020-12-01 13:18 UTC (permalink / raw)
To: Sakari Ailus
Cc: Jacopo Mondi, linux-media, hverkuil, naush, dave.stevenson,
kieran.bingham, niklas.soderlund, dafna.hirschfeld,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Tue, Dec 01, 2020 at 02:57:13PM +0200, Sakari Ailus wrote:
> On Tue, Dec 01, 2020 at 10:26:05AM +0100, Jacopo Mondi wrote:
> > On Tue, Nov 10, 2020 at 06:40:31PM +0100, Jacopo Mondi wrote:
> > > Hello,
> > > new iteration following
> > > v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
> > >
> > > Major changelog:
> > > - Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
> > > - Address Hans' comments on queue setup and metadata format handling function
> > > - s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
> > > Hans and rebase on Dafna's fixed metadata format patch
> > > - Add a TODO file to explain why the driver is in staging.
> > > - Conditionally register the unicam-embedded video device node on the presence
> > > of the sensor's metadata source pad.
> > >
> > > The media graph, when connected to a sensor that does not report metadata
> > > looks like:
> > >
> > > Media controller API version 5.10.0
> > >
> > > Media device information
> > > ------------------------
> > > driver unicam
> > > model unicam
> > > serial
> > > bus info platform:fe801000.csi
> > > hw revision 0x0
> > > driver version 5.10.0
> > >
> > > Device topology
> > > - entity 1: ov5647 10-0036 (1 pad, 1 link)
> > > type V4L2 subdev subtype Sensor flags 0
> > > device node name /dev/v4l-subdev0
> > > pad0: Source
> > > [fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
> > > crop.bounds:(16,16)/2592x1944
> > > crop:(32,16)/2560x1920]
> > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > >
> > > - entity 3: unicam-image (1 pad, 1 link)
> > > type Node subtype V4L flags 1
> > > device node name /dev/video0
> > > pad0: Sink
> > > <- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
> > >
> > >
> > > If the sensor reports an additional metadata pad:
> > >
> > > Media controller API version 5.10.0
> > >
> > > Media device information
> > > ------------------------
> > > driver unicam
> > > model unicam
> > > serial
> > > bus info platform:fe801000.csi
> > > hw revision 0x0
> > > driver version 5.10.0
> > >
> > > Device topology
> > > - entity 1: imx219 10-0010 (2 pads, 2 links)
> > > type V4L2 subdev subtype Sensor flags 0
> > > device node name /dev/v4l-subdev0
> > > pad0: Source
> > > [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> > > crop:(0,0)/3280x2464]
> > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > > pad1: Source
> > > [fmt:unknown/16384x1 field:none
> > > crop:(0,0)/3280x2464]
> > > -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> > >
> > > - entity 4: unicam-image (1 pad, 1 link)
> > > type Node subtype V4L flags 1
> > > device node name /dev/video0
> > > pad0: Sink
> > > <- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
> > >
> > > - entity 10: unicam-embedded (1 pad, 1 link)
> > > type Node subtype V4L flags 0
> > > device node name /dev/video1
> > > pad0: Sink
> > > <- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
> > >
> > > Conditionally registering the metadata video node allows to simplify the
> > > code in the driver as well, removing the 'sensor_embedded_data' flag.
> > >
> > > An additional note: this version will break the libcamera pipeline handler
> > > which assume the unicam-embedded video device node to always be there.
> > >
> > > From Dave's reply to Dafna's comments I get instead that for the existing
> > > applications ecosystem, having the metadata node not registered if the sensor
> > > does not support it is not an issue.
> >
> > I think I've closed comments received on v3.
> >
> > Do you think this series is ready for being collected ?
>
> I'll try to look into this later today / this week.
>
> The problem with the approach appears, based on a quick glance, to be that
> it creates an additional way (to the more generic approach in VC support
> patchset) to support embedded data, including duplicated sensor driver
> support. That appears as a dead end. But let me look into the details
> first.
I don't think there's a big disagreement on that, but merging the driver
in staging may be a good way to get this fixed ? It will make it easier
to collaborate.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
2020-12-01 12:57 ` Sakari Ailus
2020-12-01 13:18 ` Laurent Pinchart
@ 2020-12-01 13:24 ` Jacopo Mondi
2020-12-02 22:00 ` Sakari Ailus
1 sibling, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-01 13:24 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Tue, Dec 01, 2020 at 02:57:13PM +0200, Sakari Ailus wrote:
> Hi Jacopo,
>
> On Tue, Dec 01, 2020 at 10:26:05AM +0100, Jacopo Mondi wrote:
> > Hi Hans, Sakari,
> >
> > On Tue, Nov 10, 2020 at 06:40:31PM +0100, Jacopo Mondi wrote:
> > > Hello,
> > > new iteration following
> > > v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
> > >
> > > Major changelog:
> > > - Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
> > > - Address Hans' comments on queue setup and metadata format handling function
> > > - s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
> > > Hans and rebase on Dafna's fixed metadata format patch
> > > - Add a TODO file to explain why the driver is in staging.
> > > - Conditionally register the unicam-embedded video device node on the presence
> > > of the sensor's metadata source pad.
> > >
> > > The media graph, when connected to a sensor that does not report metadata
> > > looks like:
> > >
> > > Media controller API version 5.10.0
> > >
> > > Media device information
> > > ------------------------
> > > driver unicam
> > > model unicam
> > > serial
> > > bus info platform:fe801000.csi
> > > hw revision 0x0
> > > driver version 5.10.0
> > >
> > > Device topology
> > > - entity 1: ov5647 10-0036 (1 pad, 1 link)
> > > type V4L2 subdev subtype Sensor flags 0
> > > device node name /dev/v4l-subdev0
> > > pad0: Source
> > > [fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
> > > crop.bounds:(16,16)/2592x1944
> > > crop:(32,16)/2560x1920]
> > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > >
> > > - entity 3: unicam-image (1 pad, 1 link)
> > > type Node subtype V4L flags 1
> > > device node name /dev/video0
> > > pad0: Sink
> > > <- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
> > >
> > >
> > > If the sensor reports an additional metadata pad:
> > >
> > > Media controller API version 5.10.0
> > >
> > > Media device information
> > > ------------------------
> > > driver unicam
> > > model unicam
> > > serial
> > > bus info platform:fe801000.csi
> > > hw revision 0x0
> > > driver version 5.10.0
> > >
> > > Device topology
> > > - entity 1: imx219 10-0010 (2 pads, 2 links)
> > > type V4L2 subdev subtype Sensor flags 0
> > > device node name /dev/v4l-subdev0
> > > pad0: Source
> > > [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> > > crop:(0,0)/3280x2464]
> > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > > pad1: Source
> > > [fmt:unknown/16384x1 field:none
> > > crop:(0,0)/3280x2464]
> > > -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> > >
> > > - entity 4: unicam-image (1 pad, 1 link)
> > > type Node subtype V4L flags 1
> > > device node name /dev/video0
> > > pad0: Sink
> > > <- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
> > >
> > > - entity 10: unicam-embedded (1 pad, 1 link)
> > > type Node subtype V4L flags 0
> > > device node name /dev/video1
> > > pad0: Sink
> > > <- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
> > >
> > > Conditionally registering the metadata video node allows to simplify the
> > > code in the driver as well, removing the 'sensor_embedded_data' flag.
> > >
> > > An additional note: this version will break the libcamera pipeline handler
> > > which assume the unicam-embedded video device node to always be there.
> > >
> > > From Dave's reply to Dafna's comments I get instead that for the existing
> > > applications ecosystem, having the metadata node not registered if the sensor
> > > does not support it is not an issue.
> >
> > I think I've closed comments received on v3.
> >
> > Do you think this series is ready for being collected ?
>
> I'll try to look into this later today / this week.
>
Thanks!
> The problem with the approach appears, based on a quick glance, to be that
> it creates an additional way (to the more generic approach in VC support
> patchset) to support embedded data, including duplicated sensor driver
> support. That appears as a dead end. But let me look into the details
> first.
I know :(
That's why the driver has been moved to staging, and that's
why this last version only conditionally registers the additional video
device for metadata based on the presence of the additional sensor's
source pad. 'Regular' sensor driver will work as usual with this last
version (ie. no additional 'unicam-embedded' video node gets
registered to userspace).
We could either wait for support for VC to be finalized, but I'm not
that hopeful this will happen any time soon, or alternatively we can
merge a version without any support for metadata and have vendors
maintain patches to re-add it. I fear this will limit the adoption of
this driver as they will probably keep using whatever they have in
their BSP (which kind of defeat the purpose of upstreaming it).
There is one more controversial point: the MC/subdev kAPI duality.
I've tried to outline both issues in a TODO file at the end of this
series.
Thanks
j
>
> --
> Kind regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
2020-11-10 17:40 ` [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA Jacopo Mondi
@ 2020-12-02 21:16 ` Sakari Ailus
2020-12-03 17:05 ` Jacopo Mondi
0 siblings, 1 reply; 31+ messages in thread
From: Sakari Ailus @ 2020-12-02 21:16 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
Thank you for the patchset.
On Tue, Nov 10, 2020 at 06:40:32PM +0100, Jacopo Mondi wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add V4L2_META_FMT_CUSTOM_SENSOR_DATA format.
>
> This new format will be used to return camera sensor embedded data.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> ---
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../media/v4l/pixfmt-meta-sensor-data.rst | 24 +++++++++++++++
> .../media/v4l/subdev-formats.rst | 29 +++++++++++++++++++
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> include/uapi/linux/media-bus-format.h | 3 ++
> include/uapi/linux/videodev2.h | 1 +
> 6 files changed, 59 insertions(+)
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
>
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index fff25357fe860..b2201d1524eb6 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -15,6 +15,7 @@ These formats are used for the :ref:`metadata` interface only.
> pixfmt-meta-d4xx
> pixfmt-meta-intel-ipu3
> pixfmt-meta-rkisp1
> + pixfmt-meta-sensor-data
> pixfmt-meta-uvc
> pixfmt-meta-vsp1-hgo
> pixfmt-meta-vsp1-hgt
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
> new file mode 100644
> index 0000000000000..cc3929f595389
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
> @@ -0,0 +1,24 @@
> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> +
> +.. _v4l2-meta-fmt-sensor-data:
> +
> +***********************************
> +V4L2_META_FMT_SENSOR_DATA ('SENS')
> +***********************************
> +
> +Sensor Ancillary Metadata
"Sensor Embedded Data"
> +
> +Description
> +===========
> +
> +This format describes ancillary data generated by a camera sensor and
> +transmitted over a stream on the camera bus. Most sensor vendors have their
> +own custom format for this ancillary data. Some vendors follow a generic
> +CSI-2/SMIA embedded data format as described in the `CSI-2 specification.
> +<https://mipi.org/specifications/csi-2>`_
CSI-2 or SMIA? SMIA does define how embedded data is formatted, but this
format is specific to SMIA and may be used for other sensors that are
somewhat aligned with SMIA (or CCS) even if not fully compliant.
SMIA defines both the format of embedded data as well as its packing into
pixels at different bit depths. I have to say I have little knowledge on
how this works on non-compliant sensors. Do other sensors use the same
packing? Given that this isn't standardised, I could imagine there are
other kinds of solutions.
In order to interpret this from the user space (as well as eventually
transferring it to system memory), I think we'd need at least the packing
and bits per pixel information even if the data format itself remained
opaque at that level.
What are the documentation requirements for embedded data that is not
SMIA/CCS, and therefore likely undocumented in publicly available
documentation? Driver documentation?
One alternative would be to define only SMIA/CCS compliant embedded data
formats.
> +
> +The size of the embedded buffer is defined as a single line with a pixel width
> +specified in bytes and obtained by a call to the :c:type:`VIDIOC_SUBDEV_G_FMT`
> +ioctl on the sensor sub-device. Note that this size is fixed and cannot be
> +modified with a call to :c:type:`VIDIOC_SUBDEV_S_FMT`.
> +
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index 7f16cbe46e5c2..99e270bbdd223 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -7926,3 +7926,32 @@ The following table lists the existing metadata formats.
> both sides of the link and the bus format is a fixed
> metadata format that is not configurable from userspace.
> Width and height will be set to 0 for this format.
> +
> +.. _v4l2-mbus-sensor-data:
> +
> +Sensor Ancillary Metadata Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +This section lists ancillary data generated by a camera sensor and
> +transmitted over a stream on the camera bus.
> +
> +The following table lists the existing sensor ancillary metadata formats:
> +
> +.. _v4l2-mbus-pixelcode-sensor-metadata:
> +
> +.. tabularcolumns:: |p{8.0cm}|p{1.4cm}|p{7.7cm}|
> +
> +.. flat-table:: Sensor ancillary metadata formats
> + :header-rows: 1
> + :stub-columns: 0
> +
> + * - Identifier
> + - Code
> + - Comments
> + * .. _MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA:
> +
> + - MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
> + - 0x7002
> + - Sensor vendor specific ancillary metadata. Some vendors follow a generic
> + CSI-2/SMIA embedded data format as described in the `CSI-2 specification.
> + <https://mipi.org/specifications/csi-2>`_
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index f0f6906a879de..83288a00f28e4 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1403,6 +1403,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> case V4L2_META_FMT_UVC: descr = "UVC Payload Header Metadata"; break;
> case V4L2_META_FMT_D4XX: descr = "Intel D4xx UVC Metadata"; break;
> case V4L2_META_FMT_VIVID: descr = "Vivid Metadata"; break;
> + case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
>
> default:
> /* Compressed formats */
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 2ce3d891d3447..726557120fc03 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -164,4 +164,7 @@
> */
> #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001
>
> +/* Sensor ancillary metadata formats - next is 0x7003 */
> +#define MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA 0x7002
> +
> #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 927075fa9099e..d1ef094090a3b 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -767,6 +767,7 @@ struct v4l2_pix_format {
> #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
> #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
> #define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
> +#define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
>
> /* priv field value to indicates that subsequent fields are valid. */
> #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-11-10 17:40 ` [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver Jacopo Mondi
@ 2020-12-02 21:20 ` Sakari Ailus
2020-12-03 16:44 ` Jacopo Mondi
0 siblings, 1 reply; 31+ messages in thread
From: Sakari Ailus @ 2020-12-02 21:20 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Tue, Nov 10, 2020 at 06:40:33PM +0100, Jacopo Mondi wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
> Unicam) on BCM283x SoCs.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> ---
> .../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
> 1 file changed, 155 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
>
> diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> new file mode 100644
> index 0000000000000..6ffc900e8ae8f
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> @@ -0,0 +1,155 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Broadcom BCM283x Camera Interface (Unicam)
> +
> +maintainers:
> + - Dave Stevenson <dave.stevenson@raspberrypi.com>
> + - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
> +
> +description:
> + The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
> + data from image sensors or similar devices.
> +
> + The main platform using this SoC is the Raspberry Pi family of boards. On the
> + Pi the VideoCore firmware can also control this hardware block, and driving
> + it from two different processors will cause issues. To avoid this, the
> + firmware checks the device tree configuration during boot. If it finds device
> + tree nodes whose name starts with "csi" then it will stop the firmware
> + accessing the block, and it can then safely be used via the device tree
> + binding.
> +
> +properties:
> + compatible:
> + const: brcm,bcm2835-unicam
The compatible string doesn't quite match with the title. Which SoCs are
supported?
> +
> + reg:
> + items:
> + - description: Main registers block
> + - description: Clock registers block
> +
> + interrupts:
> + maxItems: 1
> +
> + clocks:
> + minItems: 1
> + maxItems: 2
> +
> + clock-names:
> + minItems: 1
> + items:
> + - const: lp
> + - const: core
> +
> + power-domains:
> + maxItems: 1
> +
> + brcm,num-data-lanes:
> + description:
> + The number of data lanes supported by this Unicam instance. It may be
> + larger than the number of data lanes routed on the board, as described by
> + the data-lanes property of the endpoint.
> + allOf:
> + - $ref: "/schemas/types.yaml#/definitions/uint32"
> + - enum: [1, 2, 4]
Do you need this information besides how many lanes are connected?
Does the number of lanes change even within the same model of SoC? Could
you use the compatible string to differentiate between them?
> +
> + port:
> + type: object
> + description:
> + Input port node, as described in video-interfaces.txt.
> +
> + properties:
> + endpoint:
> + type: object
> +
> + properties:
> + clock-lanes:
> + items:
> + - const: 0
Please drop. There's nothing to tell software here.
> +
> + data-lanes:
> + description:
> + Lane reordering is not supported, items shall be in order,
> + starting at 1.
> + allOf:
> + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> + - maxItems: 1
> + items:
> + minItems: 1
> + maxItems: 4
> + items:
> + - const: 1
> + - const: 2
> + - const: 3
> + - const: 4
No need to specify items.
> +
> + lane-polarities:
> + description:
> + Lane inversion is not supported. If the property is specified, it
> + shall contain all 0's.
Ditto.
> + allOf:
> + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> + - maxItems: 1
> + items:
> + minItems: 2
> + maxItems: 5
> + items:
> + - const: 0
> + - const: 0
> + - const: 0
> + - const: 0
> + - const: 0
> +
> + remote-endpoint: true
> +
> + required:
> + - data-lanes
> + - remote-endpoint
> +
> + additionalProperties: false
> +
> + required:
> + - endpoint
> +
> + additionalProperties: false
> +
> +required:
> + - compatible
> + - reg
> + - interrupts
> + - clocks
> + - clock-names
> + - power-domains
> + - brcm,num-data-lanes
> + - port
> +
> +additionalProperties: false
> +
> +examples:
> + - |
> + #include <dt-bindings/clock/bcm2835.h>
> + #include <dt-bindings/interrupt-controller/arm-gic.h>
> + #include <dt-bindings/power/raspberrypi-power.h>
> +
> + csi@7e801000 {
> + compatible = "brcm,bcm2835-unicam";
> + reg = <0x7e801000 0x800>,
> + <0x7e802004 0x4>;
> + interrupts = <2 7>;
> + clocks = <&clocks BCM2835_CLOCK_CAM1>;
> + clock-names = "lp";
> + power-domains = <&power RPI_POWER_DOMAIN_UNICAM1>;
> + brcm,num-data-lanes = <4>;
> +
> + port {
> + csi1_ep: endpoint {
> + remote-endpoint = <&imx219_0>;
> + data-lanes = <1 2>;
> + };
> + };
> + };
> +...
--
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
2020-11-10 17:40 ` [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface Jacopo Mondi
@ 2020-12-02 21:53 ` Sakari Ailus
2020-12-03 16:30 ` Jacopo Mondi
2020-12-02 22:35 ` Kieran Bingham
1 sibling, 1 reply; 31+ messages in thread
From: Sakari Ailus @ 2020-12-02 21:53 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Tue, Nov 10, 2020 at 06:40:34PM +0100, Jacopo Mondi wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add a driver for the Unicam camera receiver block on BCM283x processors.
> Compared to the bcm2835-camera driver present in staging, this driver
> handles the Unicam block only (CSI-2 receiver), and doesn't depend on
> the VC4 firmware running on the VPU.
>
> The commit is made up of a series of changes cherry-picked from the
> rpi-5.9.y branch of https://github.com/raspberrypi/linux/ at revision
> commit be371a8ffc012
> ("media: bcm2835-unicam: change minimum number of vb2_queue buffers to 1")
> with additional enhancements, forward-ported to the mainline kernel.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> ---
> MAINTAINERS | 7 +
> drivers/staging/media/Kconfig | 2 +
> drivers/staging/media/Makefile | 1 +
> drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
> drivers/staging/media/bcm2835-unicam/Makefile | 3 +
> .../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
> .../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
> 7 files changed, 3037 insertions(+)
> create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
> create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
> create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 811db1d3ca33b..69d55ed67e1cf 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3411,6 +3411,13 @@ N: bcm113*
> N: bcm216*
> N: kona
>
> +BROADCOM BCM2835 CAMERA DRIVER
> +M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
> +L: linux-media@vger.kernel.org
> +S: Maintained
> +F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> +F: drivers/staging/media/bcm2835/
> +
> BROADCOM BCM47XX MIPS ARCHITECTURE
> M: Hauke Mehrtens <hauke@hauke-m.de>
> M: Rafał Miłecki <zajec5@gmail.com>
> diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
> index 747c6cf1d795e..b734e882f63c9 100644
> --- a/drivers/staging/media/Kconfig
> +++ b/drivers/staging/media/Kconfig
> @@ -46,4 +46,6 @@ source "drivers/staging/media/ipu3/Kconfig"
>
> source "drivers/staging/media/rkisp1/Kconfig"
>
> +source "drivers/staging/media/bcm2835-unicam/Kconfig"
> +
> endif
> diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
> index b59571826ba69..93d819ef243f1 100644
> --- a/drivers/staging/media/Makefile
> +++ b/drivers/staging/media/Makefile
> @@ -1,6 +1,7 @@
> # SPDX-License-Identifier: GPL-2.0
> obj-$(CONFIG_VIDEO_ALLEGRO_DVT) += allegro-dvt/
> obj-$(CONFIG_INTEL_ATOMISP) += atomisp/
> +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam/
> obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx/
> obj-$(CONFIG_VIDEO_MESON_VDEC) += meson/vdec/
> obj-$(CONFIG_VIDEO_OMAP4) += omap4iss/
> diff --git a/drivers/staging/media/bcm2835-unicam/Kconfig b/drivers/staging/media/bcm2835-unicam/Kconfig
> new file mode 100644
> index 0000000000000..bd13701996509
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/Kconfig
> @@ -0,0 +1,21 @@
> +# Broadcom VideoCore4 V4L2 camera support
> +
> +config VIDEO_BCM2835_UNICAM
> + tristate "Broadcom BCM283x/BCM271x Unicam video capture driver"
> + depends on VIDEO_V4L2
> + depends on ARCH_BCM2835 || COMPILE_TEST
> + select VIDEO_V4L2_SUBDEV_API
> + select MEDIA_CONTROLLER
> + select VIDEOBUF2_DMA_CONTIG
> + select V4L2_FWNODE
> + help
> + Say Y here to enable support for the BCM283x/BCM271x CSI-2 receiver.
> + This is a V4L2 driver that controls the CSI-2 receiver directly,
> + independently from the VC4 firmware.
> + This driver is mutually exclusive with the use of bcm2835-camera. The
> + firmware will disable all access to the peripheral from within the
> + firmware if it finds a DT node using it, and bcm2835-camera will
> + therefore fail to probe.
> +
> + To compile this driver as a module, choose M here. The module will be
> + called bcm2835-unicam.
> diff --git a/drivers/staging/media/bcm2835-unicam/Makefile b/drivers/staging/media/bcm2835-unicam/Makefile
> new file mode 100644
> index 0000000000000..a98aba03598ab
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/Makefile
> @@ -0,0 +1,3 @@
> +# Makefile for BCM2835 Unicam driver
> +
> +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam.o
> diff --git a/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> new file mode 100644
> index 0000000000000..3e365baa8197f
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> @@ -0,0 +1,2750 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * BCM283x / BCM271x Unicam Capture Driver
> + *
> + * Copyright (C) 2017-2020 - Raspberry Pi (Trading) Ltd.
> + *
> + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> + *
> + * Based on TI am437x driver by
> + * Benoit Parrot <bparrot@ti.com>
> + * Lad, Prabhakar <prabhakar.csengg@gmail.com>
> + *
> + * and TI CAL camera interface driver by
> + * Benoit Parrot <bparrot@ti.com>
> + *
> + *
> + * There are two camera drivers in the kernel for BCM283x - this one
> + * and bcm2835-camera (currently in staging).
> + *
> + * This driver directly controls the Unicam peripheral - there is no
> + * involvement with the VideoCore firmware. Unicam receives CSI-2 or
> + * CCP2 data and writes it into SDRAM.
> + * The only potential processing options are to repack Bayer data into an
> + * alternate format, and applying windowing.
> + * The repacking does not shift the data, so can repack V4L2_PIX_FMT_Sxxxx10P
> + * to V4L2_PIX_FMT_Sxxxx10, or V4L2_PIX_FMT_Sxxxx12P to V4L2_PIX_FMT_Sxxxx12,
> + * but not generically up to V4L2_PIX_FMT_Sxxxx16. The driver will add both
> + * formats where the relevant formats are defined, and will automatically
> + * configure the repacking as required.
> + * Support for windowing may be added later.
> + *
> + * It should be possible to connect this driver to any sensor with a
> + * suitable output interface and V4L2 subdevice driver.
> + *
> + * bcm2835-camera uses the VideoCore firmware to control the sensor,
> + * Unicam, ISP, and all tuner control loops. Fully processed frames are
> + * delivered to the driver by the firmware. It only has sensor drivers
> + * for Omnivision OV5647, and Sony IMX219 sensors.
> + *
> + * The two drivers are mutually exclusive for the same Unicam instance.
> + * The VideoCore firmware checks the device tree configuration during boot.
> + * If it finds device tree nodes called csi0 or csi1 it will block the
> + * firmware from accessing the peripheral, and bcm2835-camera will
> + * not be able to stream data.
> + */
> +
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/module.h>
> +#include <linux/of_device.h>
> +#include <linux/of_graph.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/slab.h>
> +#include <linux/uaccess.h>
> +#include <linux/videodev2.h>
> +
> +#include <media/v4l2-common.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-dv-timings.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-fwnode.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "vc4-regs-unicam.h"
> +
> +#define UNICAM_MODULE_NAME "unicam"
> +#define UNICAM_VERSION "0.1.0"
> +
> +/*
> + * Unicam must request a minimum of 250Mhz from the VPU clock.
> + * Otherwise the input FIFOs overrun and cause image corruption.
> + */
> +#define MIN_VPU_CLOCK_RATE (250 * 1000 * 1000)
> +/*
> + * To protect against a dodgy sensor driver never returning an error from
> + * enum_mbus_code, set a maximum index value to be used.
> + */
> +#define MAX_ENUM_MBUS_CODE 128
Please remove, and instead fix the buggy sensor drivers.
> +
> +/*
> + * Stride is a 16 bit register, but also has to be a multiple of 32.
> + */
> +#define BPL_ALIGNMENT 32
> +#define MAX_BYTESPERLINE ((1 << 16) - BPL_ALIGNMENT)
> +/*
> + * Max width is therefore determined by the max stride divided by
> + * the number of bits per pixel. Take 32bpp as a
> + * worst case.
> + * No imposed limit on the height, so adopt a square image for want
> + * of anything better.
> + */
> +#define MAX_WIDTH (MAX_BYTESPERLINE / 4)
> +#define MAX_HEIGHT MAX_WIDTH
> +/* Define a nominal minimum image size */
> +#define MIN_WIDTH 16
> +#define MIN_HEIGHT 16
> +
> +/*
> + * Size of the dummy buffer. Can be any size really, but the DMA
> + * allocation works in units of page sizes.
> + */
> +#define DUMMY_BUF_SIZE (PAGE_SIZE)
> +
> +/* Enumeration of the video device node identifiers. */
> +enum unicam_node_ids {
> + IMAGE_NODE,
> + METADATA_NODE,
> + MAX_NODES
> +};
> +
> +/*
> + * Associate video device nodes with the subdevice's source pad ids.
> + * The image video device is connected to the subdevice pad #0;
> + * The metadata video device is connected to the subdevice pad #1;
> + */
> +unsigned int subdev_pads[2] = {
> + [IMAGE_NODE] = 0,
> + [METADATA_NODE] = 1,
> +};
I recognise this and the above enum are used for different purpose, but
this is also entirely useless. At the very least make it static.
> +
> +/*
> + * struct unicam_fmt - Unicam media bus format information
> + * @pixelformat: V4L2 pixel format FCC identifier. 0 if n/a.
> + * @repacked_fourcc: V4L2 pixel format FCC identifier if the data is expanded
> + * out to 16bpp. 0 if n/a.
> + * @code: V4L2 media bus format code.
> + * @depth: Bits per pixel as delivered from the source.
> + * @csi_dt: CSI data type.
> + * @check_variants: Flag to denote that there are multiple mediabus formats
> + * still in the list that could match this V4L2 format.
> + */
> +struct unicam_fmt {
> + u32 fourcc;
> + u32 repacked_fourcc;
> + u32 code;
> + u8 depth;
> + u8 csi_dt;
> + u8 check_variants;
> +};
> +
> +static const struct unicam_fmt formats[] = {
> + /* YUV Formats */
> + {
> + .fourcc = V4L2_PIX_FMT_YUYV,
> + .code = MEDIA_BUS_FMT_YUYV8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_UYVY,
> + .code = MEDIA_BUS_FMT_UYVY8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_YVYU,
> + .code = MEDIA_BUS_FMT_YVYU8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_VYUY,
> + .code = MEDIA_BUS_FMT_VYUY8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_YUYV,
> + .code = MEDIA_BUS_FMT_YUYV8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + .fourcc = V4L2_PIX_FMT_UYVY,
> + .code = MEDIA_BUS_FMT_UYVY8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + .fourcc = V4L2_PIX_FMT_YVYU,
> + .code = MEDIA_BUS_FMT_YVYU8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + .fourcc = V4L2_PIX_FMT_VYUY,
> + .code = MEDIA_BUS_FMT_VYUY8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + /* RGB Formats */
> + .fourcc = V4L2_PIX_FMT_RGB565, /* gggbbbbb rrrrrggg */
> + .code = MEDIA_BUS_FMT_RGB565_2X8_LE,
> + .depth = 16,
> + .csi_dt = 0x22,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB565X, /* rrrrrggg gggbbbbb */
> + .code = MEDIA_BUS_FMT_RGB565_2X8_BE,
> + .depth = 16,
> + .csi_dt = 0x22
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB555, /* gggbbbbb arrrrrgg */
> + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE,
> + .depth = 16,
> + .csi_dt = 0x21,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB555X, /* arrrrrgg gggbbbbb */
> + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE,
> + .depth = 16,
> + .csi_dt = 0x21,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB24, /* rgb */
> + .code = MEDIA_BUS_FMT_RGB888_1X24,
> + .depth = 24,
> + .csi_dt = 0x24,
> + }, {
> + .fourcc = V4L2_PIX_FMT_BGR24, /* bgr */
> + .code = MEDIA_BUS_FMT_BGR888_1X24,
> + .depth = 24,
> + .csi_dt = 0x24,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB32, /* argb */
> + .code = MEDIA_BUS_FMT_ARGB8888_1X32,
> + .depth = 32,
> + .csi_dt = 0x0,
> + }, {
> + /* Bayer Formats */
> + .fourcc = V4L2_PIX_FMT_SBGGR8,
> + .code = MEDIA_BUS_FMT_SBGGR8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG8,
> + .code = MEDIA_BUS_FMT_SGBRG8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG8,
> + .code = MEDIA_BUS_FMT_SGRBG8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB8,
> + .code = MEDIA_BUS_FMT_SRGGB8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SBGGR10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SBGGR10,
> + .code = MEDIA_BUS_FMT_SBGGR10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGBRG10,
> + .code = MEDIA_BUS_FMT_SGBRG10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGRBG10,
> + .code = MEDIA_BUS_FMT_SGRBG10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SRGGB10,
> + .code = MEDIA_BUS_FMT_SRGGB10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SBGGR12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SBGGR12,
> + .code = MEDIA_BUS_FMT_SBGGR12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGBRG12,
> + .code = MEDIA_BUS_FMT_SGBRG12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGRBG12,
> + .code = MEDIA_BUS_FMT_SGRBG12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SRGGB12,
> + .code = MEDIA_BUS_FMT_SRGGB12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SBGGR14P,
> + .code = MEDIA_BUS_FMT_SBGGR14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG14P,
> + .code = MEDIA_BUS_FMT_SGBRG14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG14P,
> + .code = MEDIA_BUS_FMT_SGRBG14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB14P,
> + .code = MEDIA_BUS_FMT_SRGGB14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + /*
> + * 16 bit Bayer formats could be supported, but there is no CSI2
> + * data_type defined for raw 16, and no sensors that produce it at
> + * present.
> + */
> +
> + /* Greyscale formats */
> + .fourcc = V4L2_PIX_FMT_GREY,
> + .code = MEDIA_BUS_FMT_Y8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_Y10P,
> + .repacked_fourcc = V4L2_PIX_FMT_Y10,
> + .code = MEDIA_BUS_FMT_Y10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + /* NB There is no packed V4L2 fourcc for this format. */
> + .repacked_fourcc = V4L2_PIX_FMT_Y12,
> + .code = MEDIA_BUS_FMT_Y12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + },
> + /* Embedded data format */
> + {
> + .fourcc = V4L2_META_FMT_SENSOR_DATA,
> + .code = MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA,
> + .depth = 8,
> + }
> +};
> +
> +struct unicam_buffer {
> + struct vb2_v4l2_buffer vb;
> + struct list_head list;
> +};
> +
> +static inline struct unicam_buffer *to_unicam_buffer(struct vb2_buffer *vb)
> +{
> + return container_of(vb, struct unicam_buffer, vb.vb2_buf);
> +}
> +
> +struct unicam_node {
> + bool registered;
> + int open;
> + bool streaming;
> + unsigned int node_id;
> + /* Pointer pointing to current v4l2_buffer */
> + struct unicam_buffer *cur_frm;
> + /* Pointer pointing to next v4l2_buffer */
> + struct unicam_buffer *next_frm;
> + /* video capture */
> + const struct unicam_fmt *fmt;
> + /* Used to store current pixel format */
> + struct v4l2_format v_fmt;
> + /* Used to store current mbus frame format */
> + struct v4l2_mbus_framefmt m_fmt;
> + /* Buffer queue used in video-buf */
> + struct vb2_queue buffer_queue;
> + /* Queue of filled frames */
> + struct list_head dma_queue;
> + /* IRQ lock for DMA queue */
> + spinlock_t dma_queue_lock;
> + /* lock used to access this structure */
> + struct mutex lock;
> + /* Identifies video device for this channel */
> + struct video_device video_dev;
> + /* Pointer to the parent handle */
> + struct unicam_device *dev;
> + struct media_pad pad;
> + /*
> + * Dummy buffer intended to be used by unicam
> + * if we have no other queued buffers to swap to.
> + */
> + void *dummy_buf_cpu_addr;
> + dma_addr_t dummy_buf_dma_addr;
> +};
> +
> +struct unicam_device {
> + struct device *dev;
> +
> + /* V4l2 specific parameters */
> + struct v4l2_async_subdev asd;
> +
> + /* peripheral base address */
> + void __iomem *base;
> + /* clock gating base address */
> + void __iomem *clk_gate_base;
> + /* lp clock handle */
> + struct clk *clock;
> + /* vpu clock handle */
> + struct clk *vpu_clock;
> + /* V4l2 device */
> + struct v4l2_device v4l2_dev;
> + struct media_device mdev;
> +
> + /* subdevice async Notifier */
> + struct v4l2_async_notifier notifier;
> + unsigned int sequence;
> +
> + /* ptr to sub device */
> + struct v4l2_subdev *sensor;
> + /* Pad config for the sensor */
> + struct v4l2_subdev_pad_config *sensor_config;
> +
> + enum v4l2_mbus_type bus_type;
> + /*
> + * Stores bus.mipi_csi2.flags for CSI2 sensors, or
> + * bus.mipi_csi1.strobe for CCP2.
> + */
> + unsigned int bus_flags;
> + unsigned int max_data_lanes;
> + unsigned int active_data_lanes;
> +
> + struct unicam_node node[MAX_NODES];
> + struct v4l2_ctrl_handler ctrl_handler;
> +};
> +
> +static inline struct unicam_device *
> +to_unicam_device(struct v4l2_device *v4l2_dev)
> +{
> + return container_of(v4l2_dev, struct unicam_device, v4l2_dev);
> +}
> +
> +static char *print_fourcc(u32 fmt)
> +{
> + static char code[16];
> +
> + code[0] = (unsigned char)(fmt & 0xff);
> + code[1] = (unsigned char)((fmt >> 8) & 0xff);
> + code[2] = (unsigned char)((fmt >> 16) & 0xff);
> + code[3] = (unsigned char)((fmt >> 24) & 0xff);
> + snprintf(&code[4], 12, "=0x%8x", fmt);
Ugh. Hopefully we'll have %p4cc soon. It's not yet merged at least. :-\
The rest looks fairly normal, apart from the obvious lack of MC support
from a driver that supports embedded data (also a TODO item).
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
2020-12-01 13:18 ` Laurent Pinchart
@ 2020-12-02 21:55 ` Sakari Ailus
0 siblings, 0 replies; 31+ messages in thread
From: Sakari Ailus @ 2020-12-02 21:55 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Jacopo Mondi, linux-media, hverkuil, naush, dave.stevenson,
kieran.bingham, niklas.soderlund, dafna.hirschfeld,
nsaenzjulienne, mchehab+huawei
Hi Laurent,
On Tue, Dec 01, 2020 at 03:18:04PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
>
> On Tue, Dec 01, 2020 at 02:57:13PM +0200, Sakari Ailus wrote:
> > On Tue, Dec 01, 2020 at 10:26:05AM +0100, Jacopo Mondi wrote:
> > > On Tue, Nov 10, 2020 at 06:40:31PM +0100, Jacopo Mondi wrote:
> > > > Hello,
> > > > new iteration following
> > > > v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
> > > >
> > > > Major changelog:
> > > > - Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
> > > > - Address Hans' comments on queue setup and metadata format handling function
> > > > - s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
> > > > Hans and rebase on Dafna's fixed metadata format patch
> > > > - Add a TODO file to explain why the driver is in staging.
> > > > - Conditionally register the unicam-embedded video device node on the presence
> > > > of the sensor's metadata source pad.
> > > >
> > > > The media graph, when connected to a sensor that does not report metadata
> > > > looks like:
> > > >
> > > > Media controller API version 5.10.0
> > > >
> > > > Media device information
> > > > ------------------------
> > > > driver unicam
> > > > model unicam
> > > > serial
> > > > bus info platform:fe801000.csi
> > > > hw revision 0x0
> > > > driver version 5.10.0
> > > >
> > > > Device topology
> > > > - entity 1: ov5647 10-0036 (1 pad, 1 link)
> > > > type V4L2 subdev subtype Sensor flags 0
> > > > device node name /dev/v4l-subdev0
> > > > pad0: Source
> > > > [fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
> > > > crop.bounds:(16,16)/2592x1944
> > > > crop:(32,16)/2560x1920]
> > > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 3: unicam-image (1 pad, 1 link)
> > > > type Node subtype V4L flags 1
> > > > device node name /dev/video0
> > > > pad0: Sink
> > > > <- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
> > > >
> > > >
> > > > If the sensor reports an additional metadata pad:
> > > >
> > > > Media controller API version 5.10.0
> > > >
> > > > Media device information
> > > > ------------------------
> > > > driver unicam
> > > > model unicam
> > > > serial
> > > > bus info platform:fe801000.csi
> > > > hw revision 0x0
> > > > driver version 5.10.0
> > > >
> > > > Device topology
> > > > - entity 1: imx219 10-0010 (2 pads, 2 links)
> > > > type V4L2 subdev subtype Sensor flags 0
> > > > device node name /dev/v4l-subdev0
> > > > pad0: Source
> > > > [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> > > > crop:(0,0)/3280x2464]
> > > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > > > pad1: Source
> > > > [fmt:unknown/16384x1 field:none
> > > > crop:(0,0)/3280x2464]
> > > > -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 4: unicam-image (1 pad, 1 link)
> > > > type Node subtype V4L flags 1
> > > > device node name /dev/video0
> > > > pad0: Sink
> > > > <- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 10: unicam-embedded (1 pad, 1 link)
> > > > type Node subtype V4L flags 0
> > > > device node name /dev/video1
> > > > pad0: Sink
> > > > <- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
> > > >
> > > > Conditionally registering the metadata video node allows to simplify the
> > > > code in the driver as well, removing the 'sensor_embedded_data' flag.
> > > >
> > > > An additional note: this version will break the libcamera pipeline handler
> > > > which assume the unicam-embedded video device node to always be there.
> > > >
> > > > From Dave's reply to Dafna's comments I get instead that for the existing
> > > > applications ecosystem, having the metadata node not registered if the sensor
> > > > does not support it is not an issue.
> > >
> > > I think I've closed comments received on v3.
> > >
> > > Do you think this series is ready for being collected ?
> >
> > I'll try to look into this later today / this week.
> >
> > The problem with the approach appears, based on a quick glance, to be that
> > it creates an additional way (to the more generic approach in VC support
> > patchset) to support embedded data, including duplicated sensor driver
> > support. That appears as a dead end. But let me look into the details
> > first.
>
> I don't think there's a big disagreement on that, but merging the driver
> in staging may be a good way to get this fixed ? It will make it easier
> to collaborate.
I have no objections to that. I'd be happy to see a Unicam driver in
upstream, finally. The staging tree could indeed be a way to get there.
The DT bindings still need to be final but that appears as a relatively
small matter.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver
2020-12-01 13:24 ` Jacopo Mondi
@ 2020-12-02 22:00 ` Sakari Ailus
0 siblings, 0 replies; 31+ messages in thread
From: Sakari Ailus @ 2020-12-02 22:00 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Tue, Dec 01, 2020 at 02:24:45PM +0100, Jacopo Mondi wrote:
> Hi Sakari,
>
> On Tue, Dec 01, 2020 at 02:57:13PM +0200, Sakari Ailus wrote:
> > Hi Jacopo,
> >
> > On Tue, Dec 01, 2020 at 10:26:05AM +0100, Jacopo Mondi wrote:
> > > Hi Hans, Sakari,
> > >
> > > On Tue, Nov 10, 2020 at 06:40:31PM +0100, Jacopo Mondi wrote:
> > > > Hello,
> > > > new iteration following
> > > > v3: https://patchwork.linuxtv.org/project/linux-media/list/?series=3768
> > > >
> > > > Major changelog:
> > > > - Use v4l2_dev.release and drop manual ref-counting as suggested by Dafna
> > > > - Address Hans' comments on queue setup and metadata format handling function
> > > > - s/MEDIA_BUS_FMT_SENSOR_DATA/MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA as suggested by
> > > > Hans and rebase on Dafna's fixed metadata format patch
> > > > - Add a TODO file to explain why the driver is in staging.
> > > > - Conditionally register the unicam-embedded video device node on the presence
> > > > of the sensor's metadata source pad.
> > > >
> > > > The media graph, when connected to a sensor that does not report metadata
> > > > looks like:
> > > >
> > > > Media controller API version 5.10.0
> > > >
> > > > Media device information
> > > > ------------------------
> > > > driver unicam
> > > > model unicam
> > > > serial
> > > > bus info platform:fe801000.csi
> > > > hw revision 0x0
> > > > driver version 5.10.0
> > > >
> > > > Device topology
> > > > - entity 1: ov5647 10-0036 (1 pad, 1 link)
> > > > type V4L2 subdev subtype Sensor flags 0
> > > > device node name /dev/v4l-subdev0
> > > > pad0: Source
> > > > [fmt:SBGGR10_1X10/640x480 field:none colorspace:srgb
> > > > crop.bounds:(16,16)/2592x1944
> > > > crop:(32,16)/2560x1920]
> > > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 3: unicam-image (1 pad, 1 link)
> > > > type Node subtype V4L flags 1
> > > > device node name /dev/video0
> > > > pad0: Sink
> > > > <- "ov5647 10-0036":0 [ENABLED,IMMUTABLE]
> > > >
> > > >
> > > > If the sensor reports an additional metadata pad:
> > > >
> > > > Media controller API version 5.10.0
> > > >
> > > > Media device information
> > > > ------------------------
> > > > driver unicam
> > > > model unicam
> > > > serial
> > > > bus info platform:fe801000.csi
> > > > hw revision 0x0
> > > > driver version 5.10.0
> > > >
> > > > Device topology
> > > > - entity 1: imx219 10-0010 (2 pads, 2 links)
> > > > type V4L2 subdev subtype Sensor flags 0
> > > > device node name /dev/v4l-subdev0
> > > > pad0: Source
> > > > [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> > > > crop:(0,0)/3280x2464]
> > > > -> "unicam-image":0 [ENABLED,IMMUTABLE]
> > > > pad1: Source
> > > > [fmt:unknown/16384x1 field:none
> > > > crop:(0,0)/3280x2464]
> > > > -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 4: unicam-image (1 pad, 1 link)
> > > > type Node subtype V4L flags 1
> > > > device node name /dev/video0
> > > > pad0: Sink
> > > > <- "imx219 10-0010":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 10: unicam-embedded (1 pad, 1 link)
> > > > type Node subtype V4L flags 0
> > > > device node name /dev/video1
> > > > pad0: Sink
> > > > <- "imx219 10-0010":1 [ENABLED,IMMUTABLE]
> > > >
> > > > Conditionally registering the metadata video node allows to simplify the
> > > > code in the driver as well, removing the 'sensor_embedded_data' flag.
> > > >
> > > > An additional note: this version will break the libcamera pipeline handler
> > > > which assume the unicam-embedded video device node to always be there.
> > > >
> > > > From Dave's reply to Dafna's comments I get instead that for the existing
> > > > applications ecosystem, having the metadata node not registered if the sensor
> > > > does not support it is not an issue.
> > >
> > > I think I've closed comments received on v3.
> > >
> > > Do you think this series is ready for being collected ?
> >
> > I'll try to look into this later today / this week.
> >
>
> Thanks!
>
> > The problem with the approach appears, based on a quick glance, to be that
> > it creates an additional way (to the more generic approach in VC support
> > patchset) to support embedded data, including duplicated sensor driver
> > support. That appears as a dead end. But let me look into the details
> > first.
>
> I know :(
>
> That's why the driver has been moved to staging, and that's
> why this last version only conditionally registers the additional video
> device for metadata based on the presence of the additional sensor's
> source pad. 'Regular' sensor driver will work as usual with this last
> version (ie. no additional 'unicam-embedded' video node gets
> registered to userspace).
>
> We could either wait for support for VC to be finalized, but I'm not
> that hopeful this will happen any time soon, or alternatively we can
Not in matter of weeks at least, but I believe there's more and more
interest to get this merged. As I wrote the original set, over the years it
became harder and harder to test the patches on the original hardware setup
I had due to too many dependencies to unsupported stuff.
> merge a version without any support for metadata and have vendors
> maintain patches to re-add it. I fear this will limit the adoption of
> this driver as they will probably keep using whatever they have in
> their BSP (which kind of defeat the purpose of upstreaming it).
>
> There is one more controversial point: the MC/subdev kAPI duality.
> I've tried to outline both issues in a TODO file at the end of this
> series.
I'll comment on that separately but I think we already do share
understanding of the areas that need work.
--
Regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file
2020-11-10 17:40 ` [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file Jacopo Mondi
@ 2020-12-02 22:06 ` Sakari Ailus
2020-12-03 11:42 ` Dave Stevenson
0 siblings, 1 reply; 31+ messages in thread
From: Sakari Ailus @ 2020-12-02 22:06 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Tue, Nov 10, 2020 at 06:40:36PM +0100, Jacopo Mondi wrote:
> The bcm2835-unicam driver is currently in staging mainly for
> two reasons:
> - Handling of CSI-2 embedded data
> - Usage of both media controller API and subdev kAPI
>
> Provide a more detailed description of the currently on-going design
> discussions in the associated TODO file.
>
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> ---
> drivers/staging/media/bcm2835-unicam/TODO | 37 +++++++++++++++++++++++
> 1 file changed, 37 insertions(+)
> create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
>
> diff --git a/drivers/staging/media/bcm2835-unicam/TODO b/drivers/staging/media/bcm2835-unicam/TODO
> new file mode 100644
> index 0000000000000..c7840872eea4c
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/TODO
> @@ -0,0 +1,37 @@
> +BCM2835 Unicam driver TODO list
> +===============================
> +
> +The unicam driver could be considered ready to be moved out of the staging
> +directory in terms of code quality and expected functionalities.
> +
> +However there currently are two design issues that suggest the driver is
> +better kept in staging for the time being.
> +
> +CSI-2 Embedded data support:
> +----------------------------
> +
> +The RaspberryPi camera stack and camera applications rely on the availability of
> +the sensor produced CSI-2 embedded data, whose support is currently not
> +finalized in mainline Linux.
> +
> +The driver conditionally registers an additional video device node
> +'unicam-embedded' with a single sink pad which connects to the sensor
> +sub-device source pad #1 to expose ancillary data.
> +
> +Currently none of the mainline sensor drivers register more than a single pad,
> +and consequentially no embedded data from the sensor are exposed to userspace.
> +
> +The current implementation is then subject to changes depending on how support
> +for CSI-2 embedded data gets finalized in Linux.
Are you looking to use out-of-tree sensor drivers that have two pads? I'd
rather see sensor drivers merged to mainline proper.
But as noted in the other e-mail, we need VC / stream support so this needs
to be addressed for reasons not related to Unicam.
> +
> +Media controller support:
> +-------------------------
> +
> +Due to compatibility reasons with the existing RaspberryPi software ecosystem
> +the unicam driver implements the media controller interface to allow the
> +enumeration of the connected entities but handles the configuration of the
> +sensor sub-device using the v4l2-subdev kAPI instead of delegating that to
> +user-space.
> +
> +Discussions are on-going on how this should be better handled (driver option,
> +KConfig option etc etc).
That's a fair use case. There are two ways to handle it, either in the
kernel where the choice affects how the user space looks like. You have an
option of module parameter or Kconfig option there, and both are true
annoyances.
Another option is to work around it in the user space, wrapping the IOCTL
calls. This way no kernel build or module load time parameters would be
needed to switch between the two sets of user space programs.
We probably can't decide it now, but could an MC-only driver + user space
compatibility layer be an option here?
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
2020-11-10 17:40 ` [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface Jacopo Mondi
2020-12-02 21:53 ` Sakari Ailus
@ 2020-12-02 22:35 ` Kieran Bingham
1 sibling, 0 replies; 31+ messages in thread
From: Kieran Bingham @ 2020-12-02 22:35 UTC (permalink / raw)
To: Jacopo Mondi, linux-media
Cc: naush, dave.stevenson, laurent.pinchart, niklas.soderlund,
dafna.hirschfeld, hverkuil, sakari.ailus, nsaenzjulienne,
mchehab+huawei
Hi Jacopo,
On 10/11/2020 17:40, Jacopo Mondi wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add a driver for the Unicam camera receiver block on BCM283x processors.
> Compared to the bcm2835-camera driver present in staging, this driver
> handles the Unicam block only (CSI-2 receiver), and doesn't depend on
> the VC4 firmware running on the VPU.
>
> The commit is made up of a series of changes cherry-picked from the
> rpi-5.9.y branch of https://github.com/raspberrypi/linux/ at revision
> commit be371a8ffc012
> ("media: bcm2835-unicam: change minimum number of vb2_queue buffers to 1")
> with additional enhancements, forward-ported to the mainline kernel.
There are a few bug-fixes that hit todays RPi-5.4.y tree from an issue
we hit, and Naush has already repaired. (Thanks)
Please integrate those into this on your next version as well.
> f4b5d0de4ffa media: bcm2835-unicam: Clear clock state when stopping streaming
> 52e76d977892 media: bcm2835-unicam: Return early from stop_streaming() if stopped
> ab9fa5a37eef media: bcm2835-unicam: Correctly handle error propagation for stream on
--
Kieran
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> ---
> MAINTAINERS | 7 +
> drivers/staging/media/Kconfig | 2 +
> drivers/staging/media/Makefile | 1 +
> drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
> drivers/staging/media/bcm2835-unicam/Makefile | 3 +
> .../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
> .../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
> 7 files changed, 3037 insertions(+)
> create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
> create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
> create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 811db1d3ca33b..69d55ed67e1cf 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3411,6 +3411,13 @@ N: bcm113*
> N: bcm216*
> N: kona
>
> +BROADCOM BCM2835 CAMERA DRIVER
> +M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
> +L: linux-media@vger.kernel.org
> +S: Maintained
> +F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> +F: drivers/staging/media/bcm2835/
> +
> BROADCOM BCM47XX MIPS ARCHITECTURE
> M: Hauke Mehrtens <hauke@hauke-m.de>
> M: Rafał Miłecki <zajec5@gmail.com>
> diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
> index 747c6cf1d795e..b734e882f63c9 100644
> --- a/drivers/staging/media/Kconfig
> +++ b/drivers/staging/media/Kconfig
> @@ -46,4 +46,6 @@ source "drivers/staging/media/ipu3/Kconfig"
>
> source "drivers/staging/media/rkisp1/Kconfig"
>
> +source "drivers/staging/media/bcm2835-unicam/Kconfig"
> +
> endif
> diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
> index b59571826ba69..93d819ef243f1 100644
> --- a/drivers/staging/media/Makefile
> +++ b/drivers/staging/media/Makefile
> @@ -1,6 +1,7 @@
> # SPDX-License-Identifier: GPL-2.0
> obj-$(CONFIG_VIDEO_ALLEGRO_DVT) += allegro-dvt/
> obj-$(CONFIG_INTEL_ATOMISP) += atomisp/
> +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam/
> obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx/
> obj-$(CONFIG_VIDEO_MESON_VDEC) += meson/vdec/
> obj-$(CONFIG_VIDEO_OMAP4) += omap4iss/
> diff --git a/drivers/staging/media/bcm2835-unicam/Kconfig b/drivers/staging/media/bcm2835-unicam/Kconfig
> new file mode 100644
> index 0000000000000..bd13701996509
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/Kconfig
> @@ -0,0 +1,21 @@
> +# Broadcom VideoCore4 V4L2 camera support
> +
> +config VIDEO_BCM2835_UNICAM
> + tristate "Broadcom BCM283x/BCM271x Unicam video capture driver"
> + depends on VIDEO_V4L2
> + depends on ARCH_BCM2835 || COMPILE_TEST
> + select VIDEO_V4L2_SUBDEV_API
> + select MEDIA_CONTROLLER
> + select VIDEOBUF2_DMA_CONTIG
> + select V4L2_FWNODE
> + help
> + Say Y here to enable support for the BCM283x/BCM271x CSI-2 receiver.
> + This is a V4L2 driver that controls the CSI-2 receiver directly,
> + independently from the VC4 firmware.
> + This driver is mutually exclusive with the use of bcm2835-camera. The
> + firmware will disable all access to the peripheral from within the
> + firmware if it finds a DT node using it, and bcm2835-camera will
> + therefore fail to probe.
> +
> + To compile this driver as a module, choose M here. The module will be
> + called bcm2835-unicam.
> diff --git a/drivers/staging/media/bcm2835-unicam/Makefile b/drivers/staging/media/bcm2835-unicam/Makefile
> new file mode 100644
> index 0000000000000..a98aba03598ab
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/Makefile
> @@ -0,0 +1,3 @@
> +# Makefile for BCM2835 Unicam driver
> +
> +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam.o
> diff --git a/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> new file mode 100644
> index 0000000000000..3e365baa8197f
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> @@ -0,0 +1,2750 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * BCM283x / BCM271x Unicam Capture Driver
> + *
> + * Copyright (C) 2017-2020 - Raspberry Pi (Trading) Ltd.
> + *
> + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> + *
> + * Based on TI am437x driver by
> + * Benoit Parrot <bparrot@ti.com>
> + * Lad, Prabhakar <prabhakar.csengg@gmail.com>
> + *
> + * and TI CAL camera interface driver by
> + * Benoit Parrot <bparrot@ti.com>
> + *
> + *
> + * There are two camera drivers in the kernel for BCM283x - this one
> + * and bcm2835-camera (currently in staging).
> + *
> + * This driver directly controls the Unicam peripheral - there is no
> + * involvement with the VideoCore firmware. Unicam receives CSI-2 or
> + * CCP2 data and writes it into SDRAM.
> + * The only potential processing options are to repack Bayer data into an
> + * alternate format, and applying windowing.
> + * The repacking does not shift the data, so can repack V4L2_PIX_FMT_Sxxxx10P
> + * to V4L2_PIX_FMT_Sxxxx10, or V4L2_PIX_FMT_Sxxxx12P to V4L2_PIX_FMT_Sxxxx12,
> + * but not generically up to V4L2_PIX_FMT_Sxxxx16. The driver will add both
> + * formats where the relevant formats are defined, and will automatically
> + * configure the repacking as required.
> + * Support for windowing may be added later.
> + *
> + * It should be possible to connect this driver to any sensor with a
> + * suitable output interface and V4L2 subdevice driver.
> + *
> + * bcm2835-camera uses the VideoCore firmware to control the sensor,
> + * Unicam, ISP, and all tuner control loops. Fully processed frames are
> + * delivered to the driver by the firmware. It only has sensor drivers
> + * for Omnivision OV5647, and Sony IMX219 sensors.
> + *
> + * The two drivers are mutually exclusive for the same Unicam instance.
> + * The VideoCore firmware checks the device tree configuration during boot.
> + * If it finds device tree nodes called csi0 or csi1 it will block the
> + * firmware from accessing the peripheral, and bcm2835-camera will
> + * not be able to stream data.
> + */
> +
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/module.h>
> +#include <linux/of_device.h>
> +#include <linux/of_graph.h>
> +#include <linux/pinctrl/consumer.h>
> +#include <linux/platform_device.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/slab.h>
> +#include <linux/uaccess.h>
> +#include <linux/videodev2.h>
> +
> +#include <media/v4l2-common.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-dev.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-dv-timings.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-fwnode.h>
> +#include <media/videobuf2-dma-contig.h>
> +
> +#include "vc4-regs-unicam.h"
> +
> +#define UNICAM_MODULE_NAME "unicam"
> +#define UNICAM_VERSION "0.1.0"
> +
> +/*
> + * Unicam must request a minimum of 250Mhz from the VPU clock.
> + * Otherwise the input FIFOs overrun and cause image corruption.
> + */
> +#define MIN_VPU_CLOCK_RATE (250 * 1000 * 1000)
> +/*
> + * To protect against a dodgy sensor driver never returning an error from
> + * enum_mbus_code, set a maximum index value to be used.
> + */
> +#define MAX_ENUM_MBUS_CODE 128
> +
> +/*
> + * Stride is a 16 bit register, but also has to be a multiple of 32.
> + */
> +#define BPL_ALIGNMENT 32
> +#define MAX_BYTESPERLINE ((1 << 16) - BPL_ALIGNMENT)
> +/*
> + * Max width is therefore determined by the max stride divided by
> + * the number of bits per pixel. Take 32bpp as a
> + * worst case.
> + * No imposed limit on the height, so adopt a square image for want
> + * of anything better.
> + */
> +#define MAX_WIDTH (MAX_BYTESPERLINE / 4)
> +#define MAX_HEIGHT MAX_WIDTH
> +/* Define a nominal minimum image size */
> +#define MIN_WIDTH 16
> +#define MIN_HEIGHT 16
> +
> +/*
> + * Size of the dummy buffer. Can be any size really, but the DMA
> + * allocation works in units of page sizes.
> + */
> +#define DUMMY_BUF_SIZE (PAGE_SIZE)
> +
> +/* Enumeration of the video device node identifiers. */
> +enum unicam_node_ids {
> + IMAGE_NODE,
> + METADATA_NODE,
> + MAX_NODES
> +};
> +
> +/*
> + * Associate video device nodes with the subdevice's source pad ids.
> + * The image video device is connected to the subdevice pad #0;
> + * The metadata video device is connected to the subdevice pad #1;
> + */
> +unsigned int subdev_pads[2] = {
> + [IMAGE_NODE] = 0,
> + [METADATA_NODE] = 1,
> +};
> +
> +/*
> + * struct unicam_fmt - Unicam media bus format information
> + * @pixelformat: V4L2 pixel format FCC identifier. 0 if n/a.
> + * @repacked_fourcc: V4L2 pixel format FCC identifier if the data is expanded
> + * out to 16bpp. 0 if n/a.
> + * @code: V4L2 media bus format code.
> + * @depth: Bits per pixel as delivered from the source.
> + * @csi_dt: CSI data type.
> + * @check_variants: Flag to denote that there are multiple mediabus formats
> + * still in the list that could match this V4L2 format.
> + */
> +struct unicam_fmt {
> + u32 fourcc;
> + u32 repacked_fourcc;
> + u32 code;
> + u8 depth;
> + u8 csi_dt;
> + u8 check_variants;
> +};
> +
> +static const struct unicam_fmt formats[] = {
> + /* YUV Formats */
> + {
> + .fourcc = V4L2_PIX_FMT_YUYV,
> + .code = MEDIA_BUS_FMT_YUYV8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_UYVY,
> + .code = MEDIA_BUS_FMT_UYVY8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_YVYU,
> + .code = MEDIA_BUS_FMT_YVYU8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_VYUY,
> + .code = MEDIA_BUS_FMT_VYUY8_2X8,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + .check_variants = 1,
> + }, {
> + .fourcc = V4L2_PIX_FMT_YUYV,
> + .code = MEDIA_BUS_FMT_YUYV8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + .fourcc = V4L2_PIX_FMT_UYVY,
> + .code = MEDIA_BUS_FMT_UYVY8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + .fourcc = V4L2_PIX_FMT_YVYU,
> + .code = MEDIA_BUS_FMT_YVYU8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + .fourcc = V4L2_PIX_FMT_VYUY,
> + .code = MEDIA_BUS_FMT_VYUY8_1X16,
> + .depth = 16,
> + .csi_dt = 0x1e,
> + }, {
> + /* RGB Formats */
> + .fourcc = V4L2_PIX_FMT_RGB565, /* gggbbbbb rrrrrggg */
> + .code = MEDIA_BUS_FMT_RGB565_2X8_LE,
> + .depth = 16,
> + .csi_dt = 0x22,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB565X, /* rrrrrggg gggbbbbb */
> + .code = MEDIA_BUS_FMT_RGB565_2X8_BE,
> + .depth = 16,
> + .csi_dt = 0x22
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB555, /* gggbbbbb arrrrrgg */
> + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE,
> + .depth = 16,
> + .csi_dt = 0x21,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB555X, /* arrrrrgg gggbbbbb */
> + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE,
> + .depth = 16,
> + .csi_dt = 0x21,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB24, /* rgb */
> + .code = MEDIA_BUS_FMT_RGB888_1X24,
> + .depth = 24,
> + .csi_dt = 0x24,
> + }, {
> + .fourcc = V4L2_PIX_FMT_BGR24, /* bgr */
> + .code = MEDIA_BUS_FMT_BGR888_1X24,
> + .depth = 24,
> + .csi_dt = 0x24,
> + }, {
> + .fourcc = V4L2_PIX_FMT_RGB32, /* argb */
> + .code = MEDIA_BUS_FMT_ARGB8888_1X32,
> + .depth = 32,
> + .csi_dt = 0x0,
> + }, {
> + /* Bayer Formats */
> + .fourcc = V4L2_PIX_FMT_SBGGR8,
> + .code = MEDIA_BUS_FMT_SBGGR8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG8,
> + .code = MEDIA_BUS_FMT_SGBRG8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG8,
> + .code = MEDIA_BUS_FMT_SGRBG8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB8,
> + .code = MEDIA_BUS_FMT_SRGGB8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SBGGR10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SBGGR10,
> + .code = MEDIA_BUS_FMT_SBGGR10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGBRG10,
> + .code = MEDIA_BUS_FMT_SGBRG10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGRBG10,
> + .code = MEDIA_BUS_FMT_SGRBG10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB10P,
> + .repacked_fourcc = V4L2_PIX_FMT_SRGGB10,
> + .code = MEDIA_BUS_FMT_SRGGB10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SBGGR12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SBGGR12,
> + .code = MEDIA_BUS_FMT_SBGGR12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGBRG12,
> + .code = MEDIA_BUS_FMT_SGBRG12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SGRBG12,
> + .code = MEDIA_BUS_FMT_SGRBG12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB12P,
> + .repacked_fourcc = V4L2_PIX_FMT_SRGGB12,
> + .code = MEDIA_BUS_FMT_SRGGB12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SBGGR14P,
> + .code = MEDIA_BUS_FMT_SBGGR14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGBRG14P,
> + .code = MEDIA_BUS_FMT_SGBRG14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SGRBG14P,
> + .code = MEDIA_BUS_FMT_SGRBG14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + .fourcc = V4L2_PIX_FMT_SRGGB14P,
> + .code = MEDIA_BUS_FMT_SRGGB14_1X14,
> + .depth = 14,
> + .csi_dt = 0x2d,
> + }, {
> + /*
> + * 16 bit Bayer formats could be supported, but there is no CSI2
> + * data_type defined for raw 16, and no sensors that produce it at
> + * present.
> + */
> +
> + /* Greyscale formats */
> + .fourcc = V4L2_PIX_FMT_GREY,
> + .code = MEDIA_BUS_FMT_Y8_1X8,
> + .depth = 8,
> + .csi_dt = 0x2a,
> + }, {
> + .fourcc = V4L2_PIX_FMT_Y10P,
> + .repacked_fourcc = V4L2_PIX_FMT_Y10,
> + .code = MEDIA_BUS_FMT_Y10_1X10,
> + .depth = 10,
> + .csi_dt = 0x2b,
> + }, {
> + /* NB There is no packed V4L2 fourcc for this format. */
> + .repacked_fourcc = V4L2_PIX_FMT_Y12,
> + .code = MEDIA_BUS_FMT_Y12_1X12,
> + .depth = 12,
> + .csi_dt = 0x2c,
> + },
> + /* Embedded data format */
> + {
> + .fourcc = V4L2_META_FMT_SENSOR_DATA,
> + .code = MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA,
> + .depth = 8,
> + }
> +};
> +
> +struct unicam_buffer {
> + struct vb2_v4l2_buffer vb;
> + struct list_head list;
> +};
> +
> +static inline struct unicam_buffer *to_unicam_buffer(struct vb2_buffer *vb)
> +{
> + return container_of(vb, struct unicam_buffer, vb.vb2_buf);
> +}
> +
> +struct unicam_node {
> + bool registered;
> + int open;
> + bool streaming;
> + unsigned int node_id;
> + /* Pointer pointing to current v4l2_buffer */
> + struct unicam_buffer *cur_frm;
> + /* Pointer pointing to next v4l2_buffer */
> + struct unicam_buffer *next_frm;
> + /* video capture */
> + const struct unicam_fmt *fmt;
> + /* Used to store current pixel format */
> + struct v4l2_format v_fmt;
> + /* Used to store current mbus frame format */
> + struct v4l2_mbus_framefmt m_fmt;
> + /* Buffer queue used in video-buf */
> + struct vb2_queue buffer_queue;
> + /* Queue of filled frames */
> + struct list_head dma_queue;
> + /* IRQ lock for DMA queue */
> + spinlock_t dma_queue_lock;
> + /* lock used to access this structure */
> + struct mutex lock;
> + /* Identifies video device for this channel */
> + struct video_device video_dev;
> + /* Pointer to the parent handle */
> + struct unicam_device *dev;
> + struct media_pad pad;
> + /*
> + * Dummy buffer intended to be used by unicam
> + * if we have no other queued buffers to swap to.
> + */
> + void *dummy_buf_cpu_addr;
> + dma_addr_t dummy_buf_dma_addr;
> +};
> +
> +struct unicam_device {
> + struct device *dev;
> +
> + /* V4l2 specific parameters */
> + struct v4l2_async_subdev asd;
> +
> + /* peripheral base address */
> + void __iomem *base;
> + /* clock gating base address */
> + void __iomem *clk_gate_base;
> + /* lp clock handle */
> + struct clk *clock;
> + /* vpu clock handle */
> + struct clk *vpu_clock;
> + /* V4l2 device */
> + struct v4l2_device v4l2_dev;
> + struct media_device mdev;
> +
> + /* subdevice async Notifier */
> + struct v4l2_async_notifier notifier;
> + unsigned int sequence;
> +
> + /* ptr to sub device */
> + struct v4l2_subdev *sensor;
> + /* Pad config for the sensor */
> + struct v4l2_subdev_pad_config *sensor_config;
> +
> + enum v4l2_mbus_type bus_type;
> + /*
> + * Stores bus.mipi_csi2.flags for CSI2 sensors, or
> + * bus.mipi_csi1.strobe for CCP2.
> + */
> + unsigned int bus_flags;
> + unsigned int max_data_lanes;
> + unsigned int active_data_lanes;
> +
> + struct unicam_node node[MAX_NODES];
> + struct v4l2_ctrl_handler ctrl_handler;
> +};
> +
> +static inline struct unicam_device *
> +to_unicam_device(struct v4l2_device *v4l2_dev)
> +{
> + return container_of(v4l2_dev, struct unicam_device, v4l2_dev);
> +}
> +
> +static char *print_fourcc(u32 fmt)
> +{
> + static char code[16];
> +
> + code[0] = (unsigned char)(fmt & 0xff);
> + code[1] = (unsigned char)((fmt >> 8) & 0xff);
> + code[2] = (unsigned char)((fmt >> 16) & 0xff);
> + code[3] = (unsigned char)((fmt >> 24) & 0xff);
> + snprintf(&code[4], 12, "=0x%8x", fmt);
> +
> + return code;
> +}
> +
> +/* Hardware access */
> +static inline void clk_write(struct unicam_device *dev, u32 val)
> +{
> + writel(val | 0x5a000000, dev->clk_gate_base);
> +}
> +
> +static inline u32 reg_read(struct unicam_device *dev, u32 offset)
> +{
> + return readl(dev->base + offset);
> +}
> +
> +static inline void reg_write(struct unicam_device *dev, u32 offset, u32 val)
> +{
> + writel(val, dev->base + offset);
> +}
> +
> +static inline int get_field(u32 value, u32 mask)
> +{
> + return (value & mask) >> __ffs(mask);
> +}
> +
> +static inline void set_field(u32 *valp, u32 field, u32 mask)
> +{
> + u32 val = *valp;
> +
> + val &= ~mask;
> + val |= (field << __ffs(mask)) & mask;
> + *valp = val;
> +}
> +
> +static inline u32 reg_read_field(struct unicam_device *dev, u32 offset,
> + u32 mask)
> +{
> + return get_field(reg_read(dev, offset), mask);
> +}
> +
> +static inline void reg_write_field(struct unicam_device *dev, u32 offset,
> + u32 field, u32 mask)
> +{
> + u32 val = reg_read(dev, offset);
> +
> + set_field(&val, field, mask);
> + reg_write(dev, offset, val);
> +}
> +
> +/* Format setup functions */
> +static const struct unicam_fmt *find_format_by_code(u32 code)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < ARRAY_SIZE(formats); i++) {
> + if (formats[i].code == code)
> + return &formats[i];
> + }
> +
> + return NULL;
> +}
> +
> +static int check_mbus_format(struct unicam_device *dev,
> + const struct unicam_fmt *format)
> +{
> + unsigned int i;
> + int ret = 0;
> +
> + for (i = 0; !ret && i < MAX_ENUM_MBUS_CODE; i++) {
> + struct v4l2_subdev_mbus_code_enum mbus_code = {
> + .index = i,
> + .pad = subdev_pads[IMAGE_NODE],
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + };
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code,
> + NULL, &mbus_code);
> +
> + if (!ret && mbus_code.code == format->code)
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +static const struct unicam_fmt *find_format_by_pix(struct unicam_device *dev,
> + u32 pixelformat)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < ARRAY_SIZE(formats); i++) {
> + if (formats[i].fourcc == pixelformat ||
> + formats[i].repacked_fourcc == pixelformat) {
> + if (formats[i].check_variants &&
> + !check_mbus_format(dev, &formats[i]))
> + continue;
> + return &formats[i];
> + }
> + }
> +
> + return NULL;
> +}
> +
> +static unsigned int bytes_per_line(u32 width, const struct unicam_fmt *fmt,
> + u32 v4l2_fourcc)
> +{
> + if (v4l2_fourcc == fmt->repacked_fourcc)
> + /* Repacking always goes to 16bpp */
> + return ALIGN(width << 1, BPL_ALIGNMENT);
> + else
> + return ALIGN((width * fmt->depth) >> 3, BPL_ALIGNMENT);
> +}
> +
> +static int __subdev_get_format(struct unicam_device *dev,
> + struct v4l2_mbus_framefmt *fmt, int node_id)
> +{
> + struct v4l2_subdev_format sd_fmt = {
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + .pad = subdev_pads[node_id],
> + };
> + int ret;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, get_fmt, dev->sensor_config,
> + &sd_fmt);
> + if (ret < 0)
> + return ret;
> +
> + *fmt = sd_fmt.format;
> +
> + dev_dbg(dev->dev, "%s %dx%d code:%04x\n", __func__,
> + fmt->width, fmt->height, fmt->code);
> +
> + return 0;
> +}
> +
> +static int __subdev_set_format(struct unicam_device *dev,
> + struct v4l2_mbus_framefmt *fmt, int node_id)
> +{
> + struct v4l2_subdev_format sd_fmt = {
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + .pad = subdev_pads[node_id],
> + };
> + int ret;
> +
> + sd_fmt.format = *fmt;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_config,
> + &sd_fmt);
> + if (ret < 0)
> + return ret;
> +
> + *fmt = sd_fmt.format;
> +
> + if (node_id == IMAGE_NODE)
> + dev_dbg(dev->dev, "%s %dx%d code:%04x\n", __func__, fmt->width,
> + fmt->height, fmt->code);
> + else
> + dev_dbg(dev->dev, "%s Embedded data code:%04x\n", __func__,
> + sd_fmt.format.code);
> +
> + return 0;
> +}
> +
> +static int unicam_calc_format_size_bpl(struct unicam_device *dev,
> + const struct unicam_fmt *fmt,
> + struct v4l2_format *f)
> +{
> + unsigned int min_bytesperline;
> +
> + v4l_bound_align_image(&f->fmt.pix.width, MIN_WIDTH, MAX_WIDTH, 2,
> + &f->fmt.pix.height, MIN_HEIGHT, MAX_HEIGHT, 0,
> + 0);
> +
> + min_bytesperline = bytes_per_line(f->fmt.pix.width, fmt,
> + f->fmt.pix.pixelformat);
> +
> + if (f->fmt.pix.bytesperline > min_bytesperline &&
> + f->fmt.pix.bytesperline <= MAX_BYTESPERLINE)
> + f->fmt.pix.bytesperline = ALIGN(f->fmt.pix.bytesperline,
> + BPL_ALIGNMENT);
> + else
> + f->fmt.pix.bytesperline = min_bytesperline;
> +
> + f->fmt.pix.sizeimage = f->fmt.pix.height * f->fmt.pix.bytesperline;
> +
> + dev_dbg(dev->dev, "%s: fourcc: %s size: %dx%d bpl:%d img_size:%d\n",
> + __func__, print_fourcc(f->fmt.pix.pixelformat),
> + f->fmt.pix.width, f->fmt.pix.height, f->fmt.pix.bytesperline,
> + f->fmt.pix.sizeimage);
> +
> + return 0;
> +}
> +
> +static int unicam_reset_format(struct unicam_node *node)
> +{
> + struct unicam_device *dev = node->dev;
> + struct v4l2_mbus_framefmt mbus_fmt;
> + int ret;
> +
> + ret = __subdev_get_format(dev, &mbus_fmt, node->node_id);
> + if (ret) {
> + dev_err(dev->dev, "Failed to get_format - ret %d\n", ret);
> + return ret;
> + }
> +
> + if (mbus_fmt.code != node->fmt->code) {
> + dev_err(dev->dev,
> + "code mismatch - fmt->code %08x, mbus_fmt.code %08x\n",
> + node->fmt->code, mbus_fmt.code);
> + return ret;
> + }
> +
> + if (node->node_id == IMAGE_NODE) {
> + v4l2_fill_pix_format(&node->v_fmt.fmt.pix, &mbus_fmt);
> + node->v_fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> + unicam_calc_format_size_bpl(dev, node->fmt, &node->v_fmt);
> + } else {
> + node->v_fmt.type = V4L2_BUF_TYPE_META_CAPTURE;
> + node->v_fmt.fmt.meta.dataformat = V4L2_META_FMT_SENSOR_DATA;
> + node->v_fmt.fmt.meta.buffersize = mbus_fmt.width
> + * mbus_fmt.height;
> + }
> +
> + node->m_fmt = mbus_fmt;
> + return 0;
> +}
> +
> +static void unicam_wr_dma_addr(struct unicam_device *dev, dma_addr_t dmaaddr,
> + unsigned int buffer_size, int node_id)
> +{
> + dma_addr_t endaddr = dmaaddr + buffer_size;
> +
> + if (node_id == IMAGE_NODE) {
> + reg_write(dev, UNICAM_IBSA0, dmaaddr);
> + reg_write(dev, UNICAM_IBEA0, endaddr);
> + } else {
> + reg_write(dev, UNICAM_DBSA0, dmaaddr);
> + reg_write(dev, UNICAM_DBEA0, endaddr);
> + }
> +}
> +
> +static unsigned int unicam_get_lines_done(struct unicam_device *dev)
> +{
> + dma_addr_t start_addr, cur_addr;
> + unsigned int stride = dev->node[IMAGE_NODE].v_fmt.fmt.pix.bytesperline;
> + struct unicam_buffer *frm = dev->node[IMAGE_NODE].cur_frm;
> +
> + if (!frm)
> + return 0;
> +
> + start_addr = vb2_dma_contig_plane_dma_addr(&frm->vb.vb2_buf, 0);
> + cur_addr = reg_read(dev, UNICAM_IBWP);
> + return (unsigned int)(cur_addr - start_addr) / stride;
> +}
> +
> +static void unicam_schedule_next_buffer(struct unicam_node *node)
> +{
> + struct unicam_device *dev = node->dev;
> + struct unicam_buffer *buf;
> + unsigned int size;
> + dma_addr_t addr;
> +
> + buf = list_first_entry(&node->dma_queue, struct unicam_buffer, list);
> + node->next_frm = buf;
> + list_del(&buf->list);
> +
> + addr = vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
> + size = (node->node_id == IMAGE_NODE) ?
> + node->v_fmt.fmt.pix.sizeimage :
> + node->v_fmt.fmt.meta.buffersize;
> +
> + unicam_wr_dma_addr(dev, addr, size, node->node_id);
> +}
> +
> +static void unicam_schedule_dummy_buffer(struct unicam_node *node)
> +{
> + struct unicam_device *dev = node->dev;
> +
> + dev_dbg(dev->dev, "Scheduling dummy buffer for node %d\n",
> + node->node_id);
> +
> + unicam_wr_dma_addr(dev, node->dummy_buf_dma_addr, DUMMY_BUF_SIZE,
> + node->node_id);
> + node->next_frm = NULL;
> +}
> +
> +static void unicam_process_buffer_complete(struct unicam_node *node,
> + unsigned int sequence)
> +{
> + node->cur_frm->vb.field = node->m_fmt.field;
> + node->cur_frm->vb.sequence = sequence;
> +
> + vb2_buffer_done(&node->cur_frm->vb.vb2_buf, VB2_BUF_STATE_DONE);
> +}
> +
> +static void unicam_queue_event_sof(struct unicam_device *unicam)
> +{
> + struct v4l2_event event = {
> + .type = V4L2_EVENT_FRAME_SYNC,
> + .u.frame_sync.frame_sequence = unicam->sequence,
> + };
> +
> + v4l2_event_queue(&unicam->node[IMAGE_NODE].video_dev, &event);
> +}
> +
> +/*
> + * unicam_isr : ISR handler for unicam capture
> + * @irq: irq number
> + * @dev_id: dev_id ptr
> + *
> + * It changes status of the captured buffer, takes next buffer from the queue
> + * and sets its address in unicam registers
> + */
> +static irqreturn_t unicam_isr(int irq, void *dev)
> +{
> + struct unicam_device *unicam = dev;
> + unsigned int lines_done = unicam_get_lines_done(dev);
> + unsigned int sequence = unicam->sequence;
> + unsigned int i;
> + u32 ista, sta;
> + u64 ts;
> +
> + sta = reg_read(unicam, UNICAM_STA);
> + /* Write value back to clear the interrupts */
> + reg_write(unicam, UNICAM_STA, sta);
> +
> + ista = reg_read(unicam, UNICAM_ISTA);
> + /* Write value back to clear the interrupts */
> + reg_write(unicam, UNICAM_ISTA, ista);
> +
> + dev_dbg(unicam->dev,
> + "ISR: ISTA: 0x%X, STA: 0x%X, sequence %d, lines done %d",
> + ista, sta, sequence, lines_done);
> +
> + if (!(sta & (UNICAM_IS | UNICAM_PI0)))
> + return IRQ_HANDLED;
> +
> + /*
> + * We must run the frame end handler first. If we have a valid next_frm
> + * and we get a simultaneout FE + FS interrupt, running the FS handler
> + * first would null out the next_frm ptr and we would have lost the
> + * buffer forever.
> + */
> + if (ista & UNICAM_FEI || sta & UNICAM_PI0) {
> + /*
> + * Ensure we have swapped buffers already as we can't
> + * stop the peripheral. If no buffer is available, use a
> + * dummy buffer to dump out frames until we get a new buffer
> + * to use.
> + */
> + for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
> + if (!unicam->node[i].streaming)
> + continue;
> +
> + if (unicam->node[i].cur_frm)
> + unicam_process_buffer_complete(&unicam->node[i],
> + sequence);
> + unicam->node[i].cur_frm = unicam->node[i].next_frm;
> + }
> + unicam->sequence++;
> + }
> +
> + if (ista & UNICAM_FSI) {
> + /*
> + * Timestamp is to be when the first data byte was captured,
> + * aka frame start.
> + */
> + ts = ktime_get_ns();
> + for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
> + if (!unicam->node[i].streaming)
> + continue;
> +
> + if (unicam->node[i].cur_frm)
> + unicam->node[i].cur_frm->vb.vb2_buf.timestamp =
> + ts;
> + /*
> + * Set the next frame output to go to a dummy frame
> + * if we have not managed to obtain another frame
> + * from the queue.
> + */
> + unicam_schedule_dummy_buffer(&unicam->node[i]);
> + }
> +
> + unicam_queue_event_sof(unicam);
> + }
> +
> + /*
> + * Cannot swap buffer at frame end, there may be a race condition
> + * where the HW does not actually swap it if the new frame has
> + * already started.
> + */
> + if (ista & (UNICAM_FSI | UNICAM_LCI) && !(ista & UNICAM_FEI)) {
> + for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
> + if (!unicam->node[i].streaming)
> + continue;
> +
> + spin_lock(&unicam->node[i].dma_queue_lock);
> + if (!list_empty(&unicam->node[i].dma_queue) &&
> + !unicam->node[i].next_frm)
> + unicam_schedule_next_buffer(&unicam->node[i]);
> + spin_unlock(&unicam->node[i].dma_queue_lock);
> + }
> + }
> +
> + if (reg_read(unicam, UNICAM_ICTL) & UNICAM_FCM) {
> + /* Switch out of trigger mode if selected */
> + reg_write_field(unicam, UNICAM_ICTL, 1, UNICAM_TFC);
> + reg_write_field(unicam, UNICAM_ICTL, 0, UNICAM_FCM);
> + }
> + return IRQ_HANDLED;
> +}
> +
> +static int unicam_querycap(struct file *file, void *priv,
> + struct v4l2_capability *cap)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + strscpy(cap->driver, UNICAM_MODULE_NAME, sizeof(cap->driver));
> + strscpy(cap->card, UNICAM_MODULE_NAME, sizeof(cap->card));
> +
> + snprintf(cap->bus_info, sizeof(cap->bus_info),
> + "platform:%s", dev_name(dev->dev));
> +
> + cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_VIDEO_CAPTURE |
> + V4L2_CAP_META_CAPTURE | V4L2_CAP_READWRITE |
> + V4L2_CAP_STREAMING;
> + return 0;
> +}
> +
> +static int unicam_enum_fmt_vid_cap(struct file *file, void *priv,
> + struct v4l2_fmtdesc *f)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + unsigned int index = 0;
> + unsigned int i;
> + int ret = 0;
> +
> + if (node->node_id != IMAGE_NODE)
> + return -EINVAL;
> +
> + for (i = 0; !ret && i < MAX_ENUM_MBUS_CODE; i++) {
> + struct v4l2_subdev_mbus_code_enum mbus_code = {
> + .index = i,
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + };
> + const struct unicam_fmt *fmt;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code,
> + NULL, &mbus_code);
> + if (ret < 0) {
> + dev_dbg(dev->dev,
> + "subdev->enum_mbus_code idx %d = %d\n",
> + i, ret);
> + return -EINVAL;
> + }
> +
> + fmt = find_format_by_code(mbus_code.code);
> + if (fmt) {
> + if (fmt->fourcc) {
> + if (index == f->index) {
> + f->pixelformat = fmt->fourcc;
> + break;
> + }
> + index++;
> + }
> + if (fmt->repacked_fourcc) {
> + if (index == f->index) {
> + f->pixelformat = fmt->repacked_fourcc;
> + break;
> + }
> + index++;
> + }
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int unicam_g_fmt_vid_cap(struct file *file, void *priv,
> + struct v4l2_format *f)
> +{
> + struct v4l2_mbus_framefmt mbus_fmt = {0};
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + const struct unicam_fmt *fmt = NULL;
> + int ret;
> +
> + if (node->node_id != IMAGE_NODE)
> + return -EINVAL;
> +
> + /*
> + * If a flip has occurred in the sensor, the fmt code might have
> + * changed. So we will need to re-fetch the format from the subdevice.
> + */
> + ret = __subdev_get_format(dev, &mbus_fmt, node->node_id);
> + if (ret)
> + return -EINVAL;
> +
> + /* Find the V4L2 format from mbus code. We must match a known format. */
> + fmt = find_format_by_code(mbus_fmt.code);
> + if (!fmt)
> + return -EINVAL;
> +
> + if (node->fmt != fmt) {
> + /*
> + * The sensor format has changed so the pixelformat needs to
> + * be updated. Try and retain the packed/unpacked choice if
> + * at all possible.
> + */
> + if (node->fmt->repacked_fourcc ==
> + node->v_fmt.fmt.pix.pixelformat)
> + /* Using the repacked format */
> + node->v_fmt.fmt.pix.pixelformat = fmt->repacked_fourcc;
> + else
> + /* Using the native format */
> + node->v_fmt.fmt.pix.pixelformat = fmt->fourcc;
> +
> + node->fmt = fmt;
> + }
> +
> + *f = node->v_fmt;
> +
> + return 0;
> +}
> +
> +static const struct unicam_fmt *
> +get_first_supported_format(struct unicam_device *dev)
> +{
> + struct v4l2_subdev_mbus_code_enum mbus_code;
> + const struct unicam_fmt *fmt = NULL;
> + unsigned int i;
> + int ret = 0;
> +
> + for (i = 0; ret != -EINVAL && ret != -ENOIOCTLCMD; ++i) {
> + memset(&mbus_code, 0, sizeof(mbus_code));
> + mbus_code.index = i;
> + mbus_code.pad = subdev_pads[IMAGE_NODE];
> + mbus_code.which = V4L2_SUBDEV_FORMAT_ACTIVE;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code, NULL,
> + &mbus_code);
> + if (ret < 0) {
> + dev_dbg(dev->dev,
> + "subdev->enum_mbus_code idx %u = %d - continue\n",
> + i, ret);
> + continue;
> + }
> +
> + dev_dbg(dev->dev, "subdev %s: code: 0x%08x idx: %u\n",
> + dev->sensor->name, mbus_code.code, i);
> +
> + fmt = find_format_by_code(mbus_code.code);
> + dev_dbg(dev->dev,
> + "fmt 0x%08x returned as %p, V4L2 FOURCC %s, csi_dt 0x%02x\n",
> + mbus_code.code, fmt,
> + print_fourcc(fmt ? fmt->fourcc : 0),
> + fmt ? fmt->csi_dt : 0);
> + if (fmt)
> + return fmt;
> + }
> +
> + return NULL;
> +}
> +
> +static int unicam_try_fmt_vid_cap(struct file *file, void *priv,
> + struct v4l2_format *f)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + struct v4l2_subdev_format sd_fmt = {
> + .which = V4L2_SUBDEV_FORMAT_TRY,
> + .pad = subdev_pads[IMAGE_NODE],
> + };
> + struct v4l2_mbus_framefmt *mbus_fmt = &sd_fmt.format;
> + const struct unicam_fmt *fmt;
> + int ret;
> +
> + if (node->node_id != IMAGE_NODE)
> + return -EINVAL;
> +
> + fmt = find_format_by_pix(dev, f->fmt.pix.pixelformat);
> + if (!fmt) {
> + /*
> + * Pixel format not supported by unicam. Choose the first
> + * supported format, and let the sensor choose something else.
> + */
> + dev_dbg(dev->dev, "Fourcc %s not found. Use first one.\n",
> + print_fourcc(f->fmt.pix.pixelformat));
> +
> + fmt = &formats[0];
> + f->fmt.pix.pixelformat = fmt->fourcc;
> + }
> +
> + v4l2_fill_mbus_format(mbus_fmt, &f->fmt.pix, fmt->code);
> + /*
> + * No support for receiving interlaced video, so never
> + * request it from the sensor subdev.
> + */
> + mbus_fmt->field = V4L2_FIELD_NONE;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_config,
> + &sd_fmt);
> + if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV)
> + return ret;
> +
> + if (mbus_fmt->field != V4L2_FIELD_NONE)
> + dev_info(dev->dev,
> + "Sensor trying to send interlaced video - results may be unpredictable\n");
> +
> + v4l2_fill_pix_format(&f->fmt.pix, &sd_fmt.format);
> + if (mbus_fmt->code != fmt->code) {
> + /* Sensor has returned an alternate format */
> + fmt = find_format_by_code(mbus_fmt->code);
> + if (!fmt) {
> + /*
> + * The alternate format is one unicam can't support.
> + * Find the first format that is supported by both, and
> + * then set that.
> + */
> + fmt = get_first_supported_format(dev);
> + mbus_fmt->code = fmt->code;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, set_fmt,
> + dev->sensor_config, &sd_fmt);
> + if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV)
> + return ret;
> +
> + if (mbus_fmt->field != V4L2_FIELD_NONE)
> + dev_info(dev->dev,
> + "Sensor trying to send interlaced video - results may be unpredictable\n");
> +
> + v4l2_fill_pix_format(&f->fmt.pix, &sd_fmt.format);
> +
> + if (mbus_fmt->code != fmt->code) {
> + /*
> + * We've set a format that the sensor reports
> + * as being supported, but it refuses to set it.
> + * Not much else we can do.
> + * Assume that the sensor driver may accept the
> + * format when it is set (rather than tried).
> + */
> + dev_err(dev->dev,
> + "Sensor won't accept default format, and Unicam can't support sensor default\n");
> + }
> + }
> +
> + if (fmt->fourcc)
> + f->fmt.pix.pixelformat = fmt->fourcc;
> + else
> + f->fmt.pix.pixelformat = fmt->repacked_fourcc;
> + }
> +
> + return unicam_calc_format_size_bpl(dev, fmt, f);
> +}
> +
> +static int unicam_s_fmt_vid_cap(struct file *file, void *priv,
> + struct v4l2_format *f)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + struct vb2_queue *q = &node->buffer_queue;
> + struct v4l2_mbus_framefmt mbus_fmt = {0};
> + const struct unicam_fmt *fmt;
> + int ret;
> +
> + if (vb2_is_busy(q))
> + return -EBUSY;
> +
> + ret = unicam_try_fmt_vid_cap(file, priv, f);
> + if (ret < 0)
> + return ret;
> +
> + fmt = find_format_by_pix(dev, f->fmt.pix.pixelformat);
> + if (!fmt) {
> + /*
> + * Unknown pixel format - adopt a default.
> + * This shouldn't happen as try_fmt should have resolved any
> + * issues first.
> + */
> + fmt = get_first_supported_format(dev);
> + if (!fmt)
> + /*
> + * It shouldn't be possible to get here with no
> + * supported formats
> + */
> + return -EINVAL;
> + f->fmt.pix.pixelformat = fmt->fourcc;
> + return -EINVAL;
> + }
> +
> + v4l2_fill_mbus_format(&mbus_fmt, &f->fmt.pix, fmt->code);
> +
> + ret = __subdev_set_format(dev, &mbus_fmt, node->node_id);
> + if (ret) {
> + dev_err(dev->dev, "%s set_format failed %d\n", __func__, ret);
> + return ret;
> + }
> +
> + /* Just double check nothing has gone wrong */
> + if (mbus_fmt.code != fmt->code) {
> + dev_dbg(dev->dev,
> + "%s subdev changed format, this should not happen\n",
> + __func__);
> + return -EINVAL;
> + }
> +
> + node->fmt = fmt;
> + node->v_fmt.fmt.pix.pixelformat = f->fmt.pix.pixelformat;
> + node->v_fmt.fmt.pix.bytesperline = f->fmt.pix.bytesperline;
> + unicam_reset_format(node);
> +
> + dev_dbg(dev->dev,
> + "%s %dx%d, mbus_fmt 0x%08X, V4L2 pix %s.\n",
> + __func__, node->v_fmt.fmt.pix.width,
> + node->v_fmt.fmt.pix.height, mbus_fmt.code,
> + print_fourcc(node->v_fmt.fmt.pix.pixelformat));
> +
> + *f = node->v_fmt;
> +
> + return 0;
> +}
> +
> +static int unicam_enum_fmt_meta_cap(struct file *file, void *priv,
> + struct v4l2_fmtdesc *f)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + const struct unicam_fmt *fmt;
> + u32 code;
> + int ret;
> +
> + struct v4l2_subdev_mbus_code_enum mbus_code = {
> + .index = f->index,
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + .pad = subdev_pads[METADATA_NODE],
> + };
> +
> + if (node->node_id != METADATA_NODE || f->index != 0)
> + return -EINVAL;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, enum_mbus_code, NULL,
> + &mbus_code);
> + if (ret < 0) {
> + dev_err(dev->dev,
> + "subdev->enum_mbus_code idx 0 = %d - invalid\n", ret);
> + return -EINVAL;
> + }
> +
> + code = mbus_code.code;
> +
> + fmt = find_format_by_code(code);
> + if (fmt)
> + f->pixelformat = fmt->fourcc;
> +
> + return 0;
> +}
> +
> +static int unicam_fmt_meta_cap(struct file *file, void *priv,
> + struct v4l2_format *f)
> +{
> + struct unicam_node *node = video_drvdata(file);
> +
> + if (node->node_id != METADATA_NODE)
> + return -EINVAL;
> +
> + *f = node->v_fmt;
> +
> + return 0;
> +}
> +
> +static int unicam_queue_setup(struct vb2_queue *vq,
> + unsigned int *nbuffers,
> + unsigned int *nplanes,
> + unsigned int sizes[],
> + struct device *alloc_devs[])
> +{
> + struct unicam_node *node = vb2_get_drv_priv(vq);
> + unsigned int size = node->node_id == IMAGE_NODE ?
> + node->v_fmt.fmt.pix.sizeimage :
> + node->v_fmt.fmt.meta.buffersize;
> +
> + if (vq->num_buffers + *nbuffers < 3)
> + *nbuffers = 3 - vq->num_buffers;
> +
> + if (*nplanes)
> + return sizes[0] < size ? -EINVAL : 0;
> +
> + *nplanes = 1;
> + sizes[0] = size;
> +
> + return 0;
> +}
> +
> +static int unicam_buffer_prepare(struct vb2_buffer *vb)
> +{
> + struct unicam_node *node = vb2_get_drv_priv(vb->vb2_queue);
> + struct unicam_device *dev = node->dev;
> + struct unicam_buffer *buf = to_unicam_buffer(vb);
> + unsigned long size;
> +
> + if (WARN_ON(!node->fmt))
> + return -EINVAL;
> +
> + size = node->node_id == IMAGE_NODE ? node->v_fmt.fmt.pix.sizeimage :
> + node->v_fmt.fmt.meta.buffersize;
> + if (vb2_plane_size(vb, 0) < size) {
> + dev_err(dev->dev, "data will not fit into plane (%lu < %lu)\n",
> + vb2_plane_size(vb, 0), size);
> + return -EINVAL;
> + }
> +
> + vb2_set_plane_payload(&buf->vb.vb2_buf, 0, size);
> + return 0;
> +}
> +
> +static void unicam_buffer_queue(struct vb2_buffer *vb)
> +{
> + struct unicam_node *node = vb2_get_drv_priv(vb->vb2_queue);
> + struct unicam_buffer *buf = to_unicam_buffer(vb);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&node->dma_queue_lock, flags);
> + list_add_tail(&buf->list, &node->dma_queue);
> + spin_unlock_irqrestore(&node->dma_queue_lock, flags);
> +}
> +
> +static void unicam_set_packing_config(struct unicam_device *dev)
> +{
> + u32 pack, unpack;
> + u32 val;
> +
> + if (dev->node[IMAGE_NODE].v_fmt.fmt.pix.pixelformat ==
> + dev->node[IMAGE_NODE].fmt->fourcc) {
> + unpack = UNICAM_PUM_NONE;
> + pack = UNICAM_PPM_NONE;
> + } else {
> + switch (dev->node[IMAGE_NODE].fmt->depth) {
> + case 8:
> + unpack = UNICAM_PUM_UNPACK8;
> + break;
> + case 10:
> + unpack = UNICAM_PUM_UNPACK10;
> + break;
> + case 12:
> + unpack = UNICAM_PUM_UNPACK12;
> + break;
> + case 14:
> + unpack = UNICAM_PUM_UNPACK14;
> + break;
> + case 16:
> + unpack = UNICAM_PUM_UNPACK16;
> + break;
> + default:
> + unpack = UNICAM_PUM_NONE;
> + break;
> + }
> +
> + /* Repacking is always to 16bpp */
> + pack = UNICAM_PPM_PACK16;
> + }
> +
> + val = 0;
> + set_field(&val, unpack, UNICAM_PUM_MASK);
> + set_field(&val, pack, UNICAM_PPM_MASK);
> + reg_write(dev, UNICAM_IPIPE, val);
> +}
> +
> +static void unicam_cfg_image_id(struct unicam_device *dev)
> +{
> + if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
> + /* CSI2 mode, hardcode VC 0 for now. */
> + reg_write(dev, UNICAM_IDI0,
> + (0 << 6) | dev->node[IMAGE_NODE].fmt->csi_dt);
> + } else {
> + /* CCP2 mode */
> + reg_write(dev, UNICAM_IDI0,
> + 0x80 | dev->node[IMAGE_NODE].fmt->csi_dt);
> + }
> +}
> +
> +static void unicam_enable_ed(struct unicam_device *dev)
> +{
> + u32 val = reg_read(dev, UNICAM_DCS);
> +
> + set_field(&val, 2, UNICAM_EDL_MASK);
> + /* Do not wrap at the end of the embedded data buffer */
> + set_field(&val, 0, UNICAM_DBOB);
> +
> + reg_write(dev, UNICAM_DCS, val);
> +}
> +
> +static void unicam_start_rx(struct unicam_device *dev, dma_addr_t *addr)
> +{
> + int line_int_freq = dev->node[IMAGE_NODE].v_fmt.fmt.pix.height >> 2;
> + unsigned int size, i;
> + u32 val;
> +
> + if (line_int_freq < 128)
> + line_int_freq = 128;
> +
> + /* Enable lane clocks */
> + val = 1;
> + for (i = 0; i < dev->active_data_lanes; i++)
> + val = val << 2 | 1;
> + clk_write(dev, val);
> +
> + /* Basic init */
> + reg_write(dev, UNICAM_CTRL, UNICAM_MEM);
> +
> + /* Enable analogue control, and leave in reset. */
> + val = UNICAM_AR;
> + set_field(&val, 7, UNICAM_CTATADJ_MASK);
> + set_field(&val, 7, UNICAM_PTATADJ_MASK);
> + reg_write(dev, UNICAM_ANA, val);
> + usleep_range(1000, 2000);
> +
> + /* Come out of reset */
> + reg_write_field(dev, UNICAM_ANA, 0, UNICAM_AR);
> +
> + /* Peripheral reset */
> + reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_CPR);
> + reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPR);
> +
> + reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPE);
> +
> + /* Enable Rx control. */
> + val = reg_read(dev, UNICAM_CTRL);
> + if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
> + set_field(&val, UNICAM_CPM_CSI2, UNICAM_CPM_MASK);
> + set_field(&val, UNICAM_DCM_STROBE, UNICAM_DCM_MASK);
> + } else {
> + set_field(&val, UNICAM_CPM_CCP2, UNICAM_CPM_MASK);
> + set_field(&val, dev->bus_flags, UNICAM_DCM_MASK);
> + }
> + /* Packet framer timeout */
> + set_field(&val, 0xf, UNICAM_PFT_MASK);
> + set_field(&val, 128, UNICAM_OET_MASK);
> + reg_write(dev, UNICAM_CTRL, val);
> +
> + reg_write(dev, UNICAM_IHWIN, 0);
> + reg_write(dev, UNICAM_IVWIN, 0);
> +
> + /* AXI bus access QoS setup */
> + val = reg_read(dev, UNICAM_PRI);
> + set_field(&val, 0, UNICAM_BL_MASK);
> + set_field(&val, 0, UNICAM_BS_MASK);
> + set_field(&val, 0xe, UNICAM_PP_MASK);
> + set_field(&val, 8, UNICAM_NP_MASK);
> + set_field(&val, 2, UNICAM_PT_MASK);
> + set_field(&val, 1, UNICAM_PE);
> + reg_write(dev, UNICAM_PRI, val);
> +
> + reg_write_field(dev, UNICAM_ANA, 0, UNICAM_DDL);
> +
> + /* Always start in trigger frame capture mode (UNICAM_FCM set) */
> + val = UNICAM_FSIE | UNICAM_FEIE | UNICAM_FCM | UNICAM_IBOB;
> + set_field(&val, line_int_freq, UNICAM_LCIE_MASK);
> + reg_write(dev, UNICAM_ICTL, val);
> + reg_write(dev, UNICAM_STA, UNICAM_STA_MASK_ALL);
> + reg_write(dev, UNICAM_ISTA, UNICAM_ISTA_MASK_ALL);
> +
> + /* tclk_term_en */
> + reg_write_field(dev, UNICAM_CLT, 2, UNICAM_CLT1_MASK);
> + /* tclk_settle */
> + reg_write_field(dev, UNICAM_CLT, 6, UNICAM_CLT2_MASK);
> + /* td_term_en */
> + reg_write_field(dev, UNICAM_DLT, 2, UNICAM_DLT1_MASK);
> + /* ths_settle */
> + reg_write_field(dev, UNICAM_DLT, 6, UNICAM_DLT2_MASK);
> + /* trx_enable */
> + reg_write_field(dev, UNICAM_DLT, 0, UNICAM_DLT3_MASK);
> +
> + reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_SOE);
> +
> + /* Packet compare setup - required to avoid missing frame ends */
> + val = 0;
> + set_field(&val, 1, UNICAM_PCE);
> + set_field(&val, 1, UNICAM_GI);
> + set_field(&val, 1, UNICAM_CPH);
> + set_field(&val, 0, UNICAM_PCVC_MASK);
> + set_field(&val, 1, UNICAM_PCDT_MASK);
> + reg_write(dev, UNICAM_CMP0, val);
> +
> + /* Enable clock lane and set up terminations */
> + val = 0;
> + if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
> + /* CSI2 */
> + set_field(&val, 1, UNICAM_CLE);
> + set_field(&val, 1, UNICAM_CLLPE);
> + if (dev->bus_flags & V4L2_MBUS_CSI2_CONTINUOUS_CLOCK) {
> + set_field(&val, 1, UNICAM_CLTRE);
> + set_field(&val, 1, UNICAM_CLHSE);
> + }
> + } else {
> + /* CCP2 */
> + set_field(&val, 1, UNICAM_CLE);
> + set_field(&val, 1, UNICAM_CLHSE);
> + set_field(&val, 1, UNICAM_CLTRE);
> + }
> + reg_write(dev, UNICAM_CLK, val);
> +
> + /*
> + * Enable required data lanes with appropriate terminations.
> + * The same value needs to be written to UNICAM_DATn registers for
> + * the active lanes, and 0 for inactive ones.
> + */
> + val = 0;
> + if (dev->bus_type == V4L2_MBUS_CSI2_DPHY) {
> + /* CSI2 */
> + set_field(&val, 1, UNICAM_DLE);
> + set_field(&val, 1, UNICAM_DLLPE);
> + if (dev->bus_flags & V4L2_MBUS_CSI2_CONTINUOUS_CLOCK) {
> + set_field(&val, 1, UNICAM_DLTRE);
> + set_field(&val, 1, UNICAM_DLHSE);
> + }
> + } else {
> + /* CCP2 */
> + set_field(&val, 1, UNICAM_DLE);
> + set_field(&val, 1, UNICAM_DLHSE);
> + set_field(&val, 1, UNICAM_DLTRE);
> + }
> + reg_write(dev, UNICAM_DAT0, val);
> +
> + if (dev->active_data_lanes == 1)
> + val = 0;
> + reg_write(dev, UNICAM_DAT1, val);
> +
> + if (dev->max_data_lanes > 2) {
> + /*
> + * Registers UNICAM_DAT2 and UNICAM_DAT3 only valid if the
> + * instance supports more than 2 data lanes.
> + */
> + if (dev->active_data_lanes == 2)
> + val = 0;
> + reg_write(dev, UNICAM_DAT2, val);
> +
> + if (dev->active_data_lanes == 3)
> + val = 0;
> + reg_write(dev, UNICAM_DAT3, val);
> + }
> +
> + reg_write(dev, UNICAM_IBLS,
> + dev->node[IMAGE_NODE].v_fmt.fmt.pix.bytesperline);
> + size = dev->node[IMAGE_NODE].v_fmt.fmt.pix.sizeimage;
> + unicam_wr_dma_addr(dev, addr[IMAGE_NODE], size, IMAGE_NODE);
> + unicam_set_packing_config(dev);
> + unicam_cfg_image_id(dev);
> +
> + val = reg_read(dev, UNICAM_MISC);
> + set_field(&val, 1, UNICAM_FL0);
> + set_field(&val, 1, UNICAM_FL1);
> + reg_write(dev, UNICAM_MISC, val);
> +
> + if (dev->node[METADATA_NODE].streaming) {
> + size = dev->node[METADATA_NODE].v_fmt.fmt.meta.buffersize;
> + unicam_enable_ed(dev);
> + unicam_wr_dma_addr(dev, addr[METADATA_NODE], size, METADATA_NODE);
> + }
> +
> + /* Enable peripheral */
> + reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_CPE);
> +
> + /* Load image pointers */
> + reg_write_field(dev, UNICAM_ICTL, 1, UNICAM_LIP_MASK);
> +
> + /* Load embedded data buffer pointers if needed */
> + if (dev->node[METADATA_NODE].streaming)
> + reg_write_field(dev, UNICAM_DCS, 1, UNICAM_LDP);
> +
> + /*
> + * Enable trigger only for the first frame to
> + * sync correctly to the FS from the source.
> + */
> + reg_write_field(dev, UNICAM_ICTL, 1, UNICAM_TFC);
> +}
> +
> +static void unicam_disable(struct unicam_device *dev)
> +{
> + /* Analogue lane control disable */
> + reg_write_field(dev, UNICAM_ANA, 1, UNICAM_DDL);
> +
> + /* Stop the output engine */
> + reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_SOE);
> +
> + /* Disable the data lanes. */
> + reg_write(dev, UNICAM_DAT0, 0);
> + reg_write(dev, UNICAM_DAT1, 0);
> +
> + if (dev->max_data_lanes > 2) {
> + reg_write(dev, UNICAM_DAT2, 0);
> + reg_write(dev, UNICAM_DAT3, 0);
> + }
> +
> + /* Peripheral reset */
> + reg_write_field(dev, UNICAM_CTRL, 1, UNICAM_CPR);
> + usleep_range(50, 100);
> + reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPR);
> +
> + /* Disable peripheral */
> + reg_write_field(dev, UNICAM_CTRL, 0, UNICAM_CPE);
> +
> + /* Clear ED setup */
> + reg_write(dev, UNICAM_DCS, 0);
> +
> + /* Disable all lane clocks */
> + clk_write(dev, 0);
> +}
> +
> +static void unicam_return_buffers(struct unicam_node *node,
> + enum vb2_buffer_state state)
> +{
> + struct unicam_buffer *buf, *tmp;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&node->dma_queue_lock, flags);
> + list_for_each_entry_safe(buf, tmp, &node->dma_queue, list) {
> + list_del(&buf->list);
> + vb2_buffer_done(&buf->vb.vb2_buf, state);
> + }
> +
> + if (node->cur_frm)
> + vb2_buffer_done(&node->cur_frm->vb.vb2_buf,
> + state);
> + if (node->next_frm && node->cur_frm != node->next_frm)
> + vb2_buffer_done(&node->next_frm->vb.vb2_buf,
> + state);
> +
> + node->cur_frm = NULL;
> + node->next_frm = NULL;
> + spin_unlock_irqrestore(&node->dma_queue_lock, flags);
> +}
> +
> +static int unicam_start_streaming(struct vb2_queue *vq, unsigned int count)
> +{
> + struct unicam_node *node = vb2_get_drv_priv(vq);
> + struct unicam_device *dev = node->dev;
> + dma_addr_t buffer_addr[MAX_NODES] = { 0 };
> + unsigned long flags;
> + unsigned int i;
> + int ret;
> +
> + node->streaming = true;
> + if (!(dev->node[IMAGE_NODE].open && dev->node[IMAGE_NODE].streaming &&
> + (!dev->node[METADATA_NODE].open ||
> + dev->node[METADATA_NODE].streaming))) {
> + /*
> + * Metadata pad must be enabled before image pad if it is
> + * wanted.
> + */
> + dev_dbg(dev->dev, "Not all nodes are streaming yet.");
> + return 0;
> + }
> +
> + dev->sequence = 0;
> + ret = pm_runtime_get_sync(dev->dev);
> + if (ret < 0) {
> + dev_err(dev->dev, "pm_runtime_get failed\n");
> + goto err_streaming;
> + }
> +
> + /*
> + * TODO: Retrieve the number of active data lanes from the connected
> + * subdevice.
> + */
> + dev->active_data_lanes = dev->max_data_lanes;
> +
> + ret = clk_set_min_rate(dev->vpu_clock, MIN_VPU_CLOCK_RATE);
> + if (ret) {
> + dev_err(dev->dev, "failed to set up VPU clock\n");
> + goto err_pm_put;
> + }
> +
> + ret = clk_prepare_enable(dev->vpu_clock);
> + if (ret) {
> + dev_err(dev->dev, "Failed to enable VPU clock: %d\n", ret);
> + goto err_pm_put;
> + }
> +
> + ret = clk_set_rate(dev->clock, 100 * 1000 * 1000);
> + if (ret) {
> + dev_err(dev->dev, "failed to set up CSI clock\n");
> + goto err_vpu_clock;
> + }
> +
> + ret = clk_prepare_enable(dev->clock);
> + if (ret) {
> + dev_err(dev->dev, "Failed to enable CSI clock: %d\n", ret);
> + goto err_vpu_clock;
> + }
> +
> + for (i = 0; i < ARRAY_SIZE(dev->node); i++) {
> + struct unicam_buffer *buf;
> +
> + if (!dev->node[i].streaming)
> + continue;
> +
> + spin_lock_irqsave(&dev->node[i].dma_queue_lock, flags);
> + buf = list_first_entry(&dev->node[i].dma_queue,
> + struct unicam_buffer, list);
> + dev->node[i].cur_frm = buf;
> + dev->node[i].next_frm = buf;
> + list_del(&buf->list);
> + spin_unlock_irqrestore(&dev->node[i].dma_queue_lock, flags);
> +
> + buffer_addr[i] =
> + vb2_dma_contig_plane_dma_addr(&buf->vb.vb2_buf, 0);
> + }
> +
> + unicam_start_rx(dev, buffer_addr);
> +
> + ret = v4l2_subdev_call(dev->sensor, video, s_stream, 1);
> + if (ret < 0) {
> + dev_err(dev->dev, "stream on failed in subdev\n");
> + goto err_disable_unicam;
> + }
> +
> + return 0;
> +
> +err_disable_unicam:
> + unicam_disable(dev);
> + clk_disable_unprepare(dev->clock);
> +err_vpu_clock:
> + ret = clk_set_min_rate(dev->vpu_clock, 0);
> + if (ret)
> + dev_err(dev->dev, "failed to reset the VPU clock\n");
> + clk_disable_unprepare(dev->vpu_clock);
> +err_pm_put:
> + pm_runtime_put_sync(dev->dev);
> +err_streaming:
> + unicam_return_buffers(node, VB2_BUF_STATE_QUEUED);
> + node->streaming = false;
> +
> + return ret;
> +}
> +
> +static void unicam_stop_streaming(struct vb2_queue *vq)
> +{
> + struct unicam_node *node = vb2_get_drv_priv(vq);
> + struct unicam_device *dev = node->dev;
> +
> + node->streaming = false;
> +
> + if (node->node_id == IMAGE_NODE) {
> + int ret;
> +
> + /*
> + * Stop streaming the sensor and disable the peripheral.
> + * We cannot continue streaming embedded data with the
> + * image pad disabled.
> + */
> + if (v4l2_subdev_call(dev->sensor, video, s_stream, 0) < 0)
> + dev_err(dev->dev, "stream off failed in subdev\n");
> +
> + unicam_disable(dev);
> +
> + ret = clk_set_min_rate(dev->vpu_clock, 0);
> + if (ret)
> + dev_err(dev->dev, "failed to reset the min VPU clock\n");
> +
> + clk_disable_unprepare(dev->vpu_clock);
> + clk_disable_unprepare(dev->clock);
> + pm_runtime_put_sync(dev->dev);
> +
> + } else if (node->node_id == METADATA_NODE) {
> + /*
> + * Allow the hardware to spin in the dummy buffer.
> + * This is only really needed if the embedded data pad is
> + * disabled before the image pad.
> + */
> + unicam_wr_dma_addr(dev, node->dummy_buf_dma_addr,
> + DUMMY_BUF_SIZE, METADATA_NODE);
> + }
> +
> + /* Clear all queued buffers for the node */
> + unicam_return_buffers(node, VB2_BUF_STATE_ERROR);
> +}
> +
> +static int unicam_enum_input(struct file *file, void *priv,
> + struct v4l2_input *inp)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + if (inp->index != 0)
> + return -EINVAL;
> +
> + inp->type = V4L2_INPUT_TYPE_CAMERA;
> + if (v4l2_subdev_has_op(dev->sensor, video, s_dv_timings)) {
> + inp->capabilities = V4L2_IN_CAP_DV_TIMINGS;
> + inp->std = 0;
> + } else if (v4l2_subdev_has_op(dev->sensor, video, s_std)) {
> + inp->capabilities = V4L2_IN_CAP_STD;
> + if (v4l2_subdev_call(dev->sensor, video, g_tvnorms, &inp->std) < 0)
> + inp->std = V4L2_STD_ALL;
> + } else {
> + inp->capabilities = 0;
> + inp->std = 0;
> + }
> + strscpy(inp->name, "Camera 0", sizeof(inp->name));
> + return 0;
> +}
> +
> +static int unicam_g_input(struct file *file, void *priv, unsigned int *i)
> +{
> + *i = 0;
> +
> + return 0;
> +}
> +
> +static int unicam_s_input(struct file *file, void *priv, unsigned int i)
> +{
> + /*
> + * FIXME: Ideally we would like to be able to query the source
> + * subdevice for information over the input connectors it supports,
> + * and map that through in to a call to video_ops->s_routing.
> + * There is no infrastructure support for defining that within
> + * devicetree at present. Until that is implemented we can't
> + * map a user physical connector number to s_routing input number.
> + */
> + if (i > 0)
> + return -EINVAL;
> +
> + return 0;
> +}
> +
> +static int unicam_querystd(struct file *file, void *priv,
> + v4l2_std_id *std)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, video, querystd, std);
> +}
> +
> +static int unicam_g_std(struct file *file, void *priv, v4l2_std_id *std)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, video, g_std, std);
> +}
> +
> +static int unicam_s_std(struct file *file, void *priv, v4l2_std_id std)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + int ret;
> + v4l2_std_id current_std;
> +
> + ret = v4l2_subdev_call(dev->sensor, video, g_std, ¤t_std);
> + if (ret)
> + return ret;
> +
> + if (std == current_std)
> + return 0;
> +
> + if (vb2_is_busy(&node->buffer_queue))
> + return -EBUSY;
> +
> + ret = v4l2_subdev_call(dev->sensor, video, s_std, std);
> +
> + /* Force recomputation of bytesperline */
> + node->v_fmt.fmt.pix.bytesperline = 0;
> +
> + unicam_reset_format(node);
> +
> + return ret;
> +}
> +
> +static int unicam_s_edid(struct file *file, void *priv, struct v4l2_edid *edid)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, pad, set_edid, edid);
> +}
> +
> +static int unicam_g_edid(struct file *file, void *priv, struct v4l2_edid *edid)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, pad, get_edid, edid);
> +}
> +
> +static int unicam_s_selection(struct file *file, void *priv,
> + struct v4l2_selection *sel)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + struct v4l2_subdev_selection sdsel = {
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + .target = sel->target,
> + .flags = sel->flags,
> + .r = sel->r,
> + };
> +
> + if (sel->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> + return -EINVAL;
> +
> + return v4l2_subdev_call(dev->sensor, pad, set_selection, NULL, &sdsel);
> +}
> +
> +static int unicam_g_selection(struct file *file, void *priv,
> + struct v4l2_selection *sel)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + struct v4l2_subdev_selection sdsel = {
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + .target = sel->target,
> + };
> + int ret;
> +
> + if (sel->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> + return -EINVAL;
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, get_selection, NULL, &sdsel);
> + if (!ret)
> + sel->r = sdsel.r;
> +
> + return ret;
> +}
> +
> +static int unicam_enum_framesizes(struct file *file, void *priv,
> + struct v4l2_frmsizeenum *fsize)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + const struct unicam_fmt *fmt;
> + struct v4l2_subdev_frame_size_enum fse;
> + int ret;
> +
> + /* check for valid format */
> + fmt = find_format_by_pix(dev, fsize->pixel_format);
> + if (!fmt) {
> + dev_err(dev->dev,
> + "Invalid pixel code: %x\n", fsize->pixel_format);
> + return -EINVAL;
> + }
> + fse.code = fmt->code;
> +
> + fse.which = V4L2_SUBDEV_FORMAT_ACTIVE;
> + fse.index = fsize->index;
> + fse.pad = subdev_pads[node->node_id];
> +
> + ret = v4l2_subdev_call(dev->sensor, pad, enum_frame_size, NULL, &fse);
> + if (ret)
> + return ret;
> +
> + dev_dbg(dev->dev, "%s: index: %d code: %x W:[%d,%d] H:[%d,%d]\n",
> + __func__, fse.index, fse.code, fse.min_width, fse.max_width,
> + fse.min_height, fse.max_height);
> +
> + fsize->type = V4L2_FRMSIZE_TYPE_DISCRETE;
> + fsize->discrete.width = fse.max_width;
> + fsize->discrete.height = fse.max_height;
> +
> + return 0;
> +}
> +
> +static int unicam_enum_frameintervals(struct file *file, void *priv,
> + struct v4l2_frmivalenum *fival)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + const struct unicam_fmt *fmt;
> + struct v4l2_subdev_frame_interval_enum fie = {
> + .index = fival->index,
> + .width = fival->width,
> + .height = fival->height,
> + .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> + };
> + int ret;
> +
> + fmt = find_format_by_pix(dev, fival->pixel_format);
> + if (!fmt)
> + return -EINVAL;
> +
> + fie.code = fmt->code;
> + ret = v4l2_subdev_call(dev->sensor, pad, enum_frame_interval,
> + NULL, &fie);
> + if (ret)
> + return ret;
> +
> + fival->type = V4L2_FRMIVAL_TYPE_DISCRETE;
> + fival->discrete = fie.interval;
> +
> + return 0;
> +}
> +
> +static int unicam_g_parm(struct file *file, void *fh, struct v4l2_streamparm *a)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_g_parm_cap(video_devdata(file), dev->sensor, a);
> +}
> +
> +static int unicam_s_parm(struct file *file, void *fh, struct v4l2_streamparm *a)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_s_parm_cap(video_devdata(file), dev->sensor, a);
> +}
> +
> +static int unicam_g_dv_timings(struct file *file, void *priv,
> + struct v4l2_dv_timings *timings)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, video, g_dv_timings, timings);
> +}
> +
> +static int unicam_s_dv_timings(struct file *file, void *priv,
> + struct v4l2_dv_timings *timings)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + struct v4l2_dv_timings current_timings;
> + int ret;
> +
> + ret = v4l2_subdev_call(dev->sensor, video, g_dv_timings,
> + ¤t_timings);
> +
> + if (ret < 0)
> + return ret;
> +
> + if (v4l2_match_dv_timings(timings, ¤t_timings, 0, false))
> + return 0;
> +
> + if (vb2_is_busy(&node->buffer_queue))
> + return -EBUSY;
> +
> + ret = v4l2_subdev_call(dev->sensor, video, s_dv_timings, timings);
> +
> + /* Force recomputation of bytesperline */
> + node->v_fmt.fmt.pix.bytesperline = 0;
> +
> + unicam_reset_format(node);
> +
> + return ret;
> +}
> +
> +static int unicam_query_dv_timings(struct file *file, void *priv,
> + struct v4l2_dv_timings *timings)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, video, query_dv_timings, timings);
> +}
> +
> +static int unicam_enum_dv_timings(struct file *file, void *priv,
> + struct v4l2_enum_dv_timings *timings)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, pad, enum_dv_timings, timings);
> +}
> +
> +static int unicam_dv_timings_cap(struct file *file, void *priv,
> + struct v4l2_dv_timings_cap *cap)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> +
> + return v4l2_subdev_call(dev->sensor, pad, dv_timings_cap, cap);
> +}
> +
> +static int unicam_subscribe_event(struct v4l2_fh *fh,
> + const struct v4l2_event_subscription *sub)
> +{
> + switch (sub->type) {
> + case V4L2_EVENT_FRAME_SYNC:
> + return v4l2_event_subscribe(fh, sub, 2, NULL);
> + case V4L2_EVENT_SOURCE_CHANGE:
> + return v4l2_event_subscribe(fh, sub, 4, NULL);
> + }
> +
> + return v4l2_ctrl_subscribe_event(fh, sub);
> +}
> +
> +static int unicam_log_status(struct file *file, void *fh)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + u32 reg;
> +
> + /* status for sub devices */
> + v4l2_device_call_all(&dev->v4l2_dev, 0, core, log_status);
> +
> + dev_info(dev->dev, "-----Receiver status-----\n");
> + dev_info(dev->dev, "V4L2 width/height: %ux%u\n",
> + node->v_fmt.fmt.pix.width, node->v_fmt.fmt.pix.height);
> + dev_info(dev->dev, "Mediabus format: %08x\n", node->fmt->code);
> + dev_info(dev->dev, "V4L2 format: %08x\n",
> + node->v_fmt.fmt.pix.pixelformat);
> + reg = reg_read(dev, UNICAM_IPIPE);
> + dev_info(dev->dev, "Unpacking/packing: %u / %u\n",
> + get_field(reg, UNICAM_PUM_MASK),
> + get_field(reg, UNICAM_PPM_MASK));
> + dev_info(dev->dev, "----Live data----\n");
> + dev_info(dev->dev, "Programmed stride: %4u\n",
> + reg_read(dev, UNICAM_IBLS));
> + dev_info(dev->dev, "Detected resolution: %ux%u\n",
> + reg_read(dev, UNICAM_IHSTA),
> + reg_read(dev, UNICAM_IVSTA));
> + dev_info(dev->dev, "Write pointer: %08x\n",
> + reg_read(dev, UNICAM_IBWP));
> +
> + return 0;
> +}
> +
> +static void unicam_notify(struct v4l2_subdev *sd,
> + unsigned int notification, void *arg)
> +{
> + struct unicam_device *dev = to_unicam_device(sd->v4l2_dev);
> +
> + switch (notification) {
> + case V4L2_DEVICE_NOTIFY_EVENT:
> + v4l2_event_queue(&dev->node[IMAGE_NODE].video_dev, arg);
> + break;
> + default:
> + break;
> + }
> +}
> +
> +static const struct vb2_ops unicam_video_qops = {
> + .wait_prepare = vb2_ops_wait_prepare,
> + .wait_finish = vb2_ops_wait_finish,
> + .queue_setup = unicam_queue_setup,
> + .buf_prepare = unicam_buffer_prepare,
> + .buf_queue = unicam_buffer_queue,
> + .start_streaming = unicam_start_streaming,
> + .stop_streaming = unicam_stop_streaming,
> +};
> +
> +/*
> + * unicam_v4l2_open : This function is based on the v4l2_fh_open helper
> + * function. It has been augmented to handle sensor subdevice power management,
> + */
> +static int unicam_v4l2_open(struct file *file)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + int ret;
> +
> + mutex_lock(&node->lock);
> +
> + ret = v4l2_fh_open(file);
> + if (ret) {
> + dev_err(dev->dev, "v4l2_fh_open failed\n");
> + goto unlock;
> + }
> +
> + node->open++;
> +
> + if (!v4l2_fh_is_singular_file(file))
> + goto unlock;
> +
> + ret = v4l2_subdev_call(dev->sensor, core, s_power, 1);
> + if (ret < 0 && ret != -ENOIOCTLCMD) {
> + v4l2_fh_release(file);
> + node->open--;
> + goto unlock;
> + }
> +
> + ret = 0;
> +
> +unlock:
> + mutex_unlock(&node->lock);
> + return ret;
> +}
> +
> +static int unicam_v4l2_release(struct file *file)
> +{
> + struct unicam_node *node = video_drvdata(file);
> + struct unicam_device *dev = node->dev;
> + struct v4l2_subdev *sd = dev->sensor;
> + bool fh_singular;
> + int ret;
> +
> + mutex_lock(&node->lock);
> +
> + fh_singular = v4l2_fh_is_singular_file(file);
> +
> + ret = _vb2_fop_release(file, NULL);
> +
> + if (fh_singular)
> + v4l2_subdev_call(sd, core, s_power, 0);
> +
> + node->open--;
> + mutex_unlock(&node->lock);
> +
> + return ret;
> +}
> +
> +/* unicam capture driver file operations */
> +static const struct v4l2_file_operations unicam_fops = {
> + .owner = THIS_MODULE,
> + .open = unicam_v4l2_open,
> + .release = unicam_v4l2_release,
> + .read = vb2_fop_read,
> + .poll = vb2_fop_poll,
> + .unlocked_ioctl = video_ioctl2,
> + .mmap = vb2_fop_mmap,
> +};
> +
> +/* unicam capture ioctl operations */
> +static const struct v4l2_ioctl_ops unicam_ioctl_ops = {
> + .vidioc_querycap = unicam_querycap,
> + .vidioc_enum_fmt_vid_cap = unicam_enum_fmt_vid_cap,
> + .vidioc_g_fmt_vid_cap = unicam_g_fmt_vid_cap,
> + .vidioc_s_fmt_vid_cap = unicam_s_fmt_vid_cap,
> + .vidioc_try_fmt_vid_cap = unicam_try_fmt_vid_cap,
> +
> + .vidioc_enum_fmt_meta_cap = unicam_enum_fmt_meta_cap,
> + .vidioc_g_fmt_meta_cap = unicam_fmt_meta_cap,
> + .vidioc_s_fmt_meta_cap = unicam_fmt_meta_cap,
> + .vidioc_try_fmt_meta_cap = unicam_fmt_meta_cap,
> +
> + .vidioc_enum_input = unicam_enum_input,
> + .vidioc_g_input = unicam_g_input,
> + .vidioc_s_input = unicam_s_input,
> +
> + .vidioc_querystd = unicam_querystd,
> + .vidioc_s_std = unicam_s_std,
> + .vidioc_g_std = unicam_g_std,
> +
> + .vidioc_g_edid = unicam_g_edid,
> + .vidioc_s_edid = unicam_s_edid,
> +
> + .vidioc_enum_framesizes = unicam_enum_framesizes,
> + .vidioc_enum_frameintervals = unicam_enum_frameintervals,
> +
> + .vidioc_g_selection = unicam_g_selection,
> + .vidioc_s_selection = unicam_s_selection,
> +
> + .vidioc_g_parm = unicam_g_parm,
> + .vidioc_s_parm = unicam_s_parm,
> +
> + .vidioc_s_dv_timings = unicam_s_dv_timings,
> + .vidioc_g_dv_timings = unicam_g_dv_timings,
> + .vidioc_query_dv_timings = unicam_query_dv_timings,
> + .vidioc_enum_dv_timings = unicam_enum_dv_timings,
> + .vidioc_dv_timings_cap = unicam_dv_timings_cap,
> +
> + .vidioc_reqbufs = vb2_ioctl_reqbufs,
> + .vidioc_create_bufs = vb2_ioctl_create_bufs,
> + .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
> + .vidioc_querybuf = vb2_ioctl_querybuf,
> + .vidioc_qbuf = vb2_ioctl_qbuf,
> + .vidioc_dqbuf = vb2_ioctl_dqbuf,
> + .vidioc_expbuf = vb2_ioctl_expbuf,
> + .vidioc_streamon = vb2_ioctl_streamon,
> + .vidioc_streamoff = vb2_ioctl_streamoff,
> +
> + .vidioc_log_status = unicam_log_status,
> + .vidioc_subscribe_event = unicam_subscribe_event,
> + .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
> +};
> +
> +static int unicam_async_bound(struct v4l2_async_notifier *notifier,
> + struct v4l2_subdev *subdev,
> + struct v4l2_async_subdev *asd)
> +{
> + struct unicam_device *unicam = to_unicam_device(notifier->v4l2_dev);
> +
> + if (unicam->sensor) {
> + dev_info(unicam->dev, "Rejecting subdev %s (Already set!!)",
> + subdev->name);
> + return 0;
> + }
> +
> + unicam->sensor = subdev;
> + dev_dbg(unicam->dev, "Using sensor %s for capture\n", subdev->name);
> +
> + return 0;
> +}
> +
> +static void unicam_release(struct v4l2_device *v4l2_dev)
> +{
> + struct unicam_device *unicam =
> + container_of(v4l2_dev, struct unicam_device, v4l2_dev);
> +
> + v4l2_ctrl_handler_free(&unicam->ctrl_handler);
> + media_device_cleanup(&unicam->mdev);
> +
> + if (unicam->sensor_config)
> + v4l2_subdev_free_pad_config(unicam->sensor_config);
> +
> + kfree(unicam);
> +}
> +
> +static int register_node(struct unicam_device *unicam, struct unicam_node *node,
> + enum v4l2_buf_type type, int node_id)
> +{
> + struct video_device *vdev;
> + struct vb2_queue *q;
> + struct v4l2_mbus_framefmt mbus_fmt = {0};
> + const struct unicam_fmt *fmt;
> + int ret;
> +
> + if (node_id == IMAGE_NODE) {
> + ret = __subdev_get_format(unicam, &mbus_fmt, node_id);
> + if (ret) {
> + dev_err(unicam->dev, "Failed to get_format - ret %d\n",
> + ret);
> + return ret;
> + }
> +
> + fmt = find_format_by_code(mbus_fmt.code);
> + if (!fmt) {
> + /*
> + * Find the first format that the sensor and unicam both
> + * support
> + */
> + fmt = get_first_supported_format(unicam);
> +
> + if (!fmt)
> + /* No compatible formats */
> + return -EINVAL;
> +
> + mbus_fmt.code = fmt->code;
> + ret = __subdev_set_format(unicam, &mbus_fmt, node_id);
> + if (ret)
> + return -EINVAL;
> + }
> + if (mbus_fmt.field != V4L2_FIELD_NONE) {
> + /* Interlaced not supported - disable it now. */
> + mbus_fmt.field = V4L2_FIELD_NONE;
> + ret = __subdev_set_format(unicam, &mbus_fmt, node_id);
> + if (ret)
> + return -EINVAL;
> + }
> +
> + node->v_fmt.fmt.pix.pixelformat = fmt->fourcc ? fmt->fourcc
> + : fmt->repacked_fourcc;
> + } else {
> + /* Fix this node format as embedded data. */
> + fmt = find_format_by_code(MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA);
> + node->v_fmt.fmt.meta.dataformat = fmt->fourcc;
> + }
> +
> + node->dev = unicam;
> + node->node_id = node_id;
> + node->fmt = fmt;
> +
> + /* Read current subdev format */
> + unicam_reset_format(node);
> +
> + if (v4l2_subdev_has_op(unicam->sensor, video, s_std)) {
> + v4l2_std_id tvnorms;
> +
> + if (WARN_ON(!v4l2_subdev_has_op(unicam->sensor, video,
> + g_tvnorms)))
> + /*
> + * Subdevice should not advertise s_std but not
> + * g_tvnorms
> + */
> + return -EINVAL;
> +
> + ret = v4l2_subdev_call(unicam->sensor, video,
> + g_tvnorms, &tvnorms);
> + if (WARN_ON(ret))
> + return -EINVAL;
> + node->video_dev.tvnorms |= tvnorms;
> + }
> +
> + spin_lock_init(&node->dma_queue_lock);
> + mutex_init(&node->lock);
> +
> + vdev = &node->video_dev;
> + if (node_id == IMAGE_NODE) {
> + /* Add controls from the subdevice */
> + ret = v4l2_ctrl_add_handler(&unicam->ctrl_handler,
> + unicam->sensor->ctrl_handler, NULL,
> + true);
> + if (ret < 0)
> + return ret;
> +
> + /*
> + * If the sensor subdevice has any controls, associate the node
> + * with the ctrl handler to allow access from userland.
> + */
> + if (!list_empty(&unicam->ctrl_handler.ctrls))
> + vdev->ctrl_handler = &unicam->ctrl_handler;
> + }
> +
> + q = &node->buffer_queue;
> + q->type = type;
> + q->io_modes = VB2_MMAP | VB2_DMABUF | VB2_READ;
> + q->drv_priv = node;
> + q->ops = &unicam_video_qops;
> + q->mem_ops = &vb2_dma_contig_memops;
> + q->buf_struct_size = sizeof(struct unicam_buffer);
> + q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
> + q->lock = &node->lock;
> + q->min_buffers_needed = 1;
> + q->dev = unicam->dev;
> +
> + ret = vb2_queue_init(q);
> + if (ret) {
> + dev_err(unicam->dev, "vb2_queue_init() failed\n");
> + return ret;
> + }
> +
> + INIT_LIST_HEAD(&node->dma_queue);
> +
> + vdev->release = video_device_release_empty;
> + vdev->fops = &unicam_fops;
> + vdev->ioctl_ops = &unicam_ioctl_ops;
> + vdev->v4l2_dev = &unicam->v4l2_dev;
> + vdev->vfl_dir = VFL_DIR_RX;
> + vdev->queue = q;
> + vdev->lock = &node->lock;
> + vdev->device_caps = (node_id == IMAGE_NODE) ?
> + V4L2_CAP_VIDEO_CAPTURE : V4L2_CAP_META_CAPTURE;
> + vdev->device_caps |= V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
> +
> + /* Define the device names */
> + snprintf(vdev->name, sizeof(vdev->name), "%s-%s", UNICAM_MODULE_NAME,
> + node_id == IMAGE_NODE ? "image" : "embedded");
> +
> + video_set_drvdata(vdev, node);
> + if (node_id == IMAGE_NODE)
> + vdev->entity.flags |= MEDIA_ENT_FL_DEFAULT;
> + node->pad.flags = MEDIA_PAD_FL_SINK;
> + media_entity_pads_init(&vdev->entity, 1, &node->pad);
> +
> + node->dummy_buf_cpu_addr = dma_alloc_coherent(unicam->dev,
> + DUMMY_BUF_SIZE,
> + &node->dummy_buf_dma_addr,
> + GFP_KERNEL);
> + if (!node->dummy_buf_cpu_addr) {
> + dev_err(unicam->dev, "Unable to allocate dummy buffer.\n");
> + return -ENOMEM;
> + }
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, video, s_std)) {
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_STD);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_STD);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_ENUMSTD);
> + }
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, video, querystd))
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_QUERYSTD);
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, video, s_dv_timings)) {
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_EDID);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_EDID);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_DV_TIMINGS_CAP);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_DV_TIMINGS);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_DV_TIMINGS);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_ENUM_DV_TIMINGS);
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_QUERY_DV_TIMINGS);
> + }
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, pad, enum_frame_interval))
> + v4l2_disable_ioctl(&node->video_dev,
> + VIDIOC_ENUM_FRAMEINTERVALS);
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, video, g_frame_interval))
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_G_PARM);
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, video, s_frame_interval))
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_S_PARM);
> +
> + if (node_id == METADATA_NODE ||
> + !v4l2_subdev_has_op(unicam->sensor, pad, enum_frame_size))
> + v4l2_disable_ioctl(&node->video_dev, VIDIOC_ENUM_FRAMESIZES);
> +
> + ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
> + if (ret) {
> + dev_err(unicam->dev, "Unable to register video device %s\n",
> + vdev->name);
> + return ret;
> + }
> +
> + node->registered = true;
> +
> + ret = media_create_pad_link(&unicam->sensor->entity, node_id,
> + &node->video_dev.entity, 0,
> + MEDIA_LNK_FL_ENABLED |
> + MEDIA_LNK_FL_IMMUTABLE);
> + if (ret)
> + dev_err(unicam->dev, "Unable to create link for %s\n",
> + vdev->name);
> +
> + return ret;
> +}
> +
> +static void unregister_nodes(struct unicam_device *unicam)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < ARRAY_SIZE(unicam->node); i++) {
> + struct unicam_node *node = &unicam->node[i];
> +
> + if (node->dummy_buf_cpu_addr) {
> + dma_free_coherent(unicam->dev, DUMMY_BUF_SIZE,
> + node->dummy_buf_cpu_addr,
> + node->dummy_buf_dma_addr);
> + }
> +
> + if (node->registered) {
> + node->registered = false;
> + video_unregister_device(&node->video_dev);
> + }
> + }
> +}
> +
> +static int unicam_async_complete(struct v4l2_async_notifier *notifier)
> +{
> + struct unicam_device *unicam = to_unicam_device(notifier->v4l2_dev);
> + int ret;
> +
> + unicam->v4l2_dev.notify = unicam_notify;
> +
> + unicam->sensor_config = v4l2_subdev_alloc_pad_config(unicam->sensor);
> + if (!unicam->sensor_config)
> + return -ENOMEM;
> +
> + ret = register_node(unicam, &unicam->node[IMAGE_NODE],
> + V4L2_BUF_TYPE_VIDEO_CAPTURE, IMAGE_NODE);
> + if (ret) {
> + dev_err(unicam->dev, "Unable to register image video device.\n");
> + goto unregister;
> + }
> +
> + /*
> + * Conditionally register the medatata video device if sensor provides
> + * an additional pad for embedded data.
> + */
> + if (unicam->sensor->entity.num_pads >= 2) {
> + ret = register_node(unicam, &unicam->node[METADATA_NODE],
> + V4L2_BUF_TYPE_META_CAPTURE, METADATA_NODE);
> + if (ret) {
> + dev_err(unicam->dev,
> + "Unable to register metadata video device.\n");
> + goto unregister;
> + }
> + }
> +
> + ret = v4l2_device_register_ro_subdev_nodes(&unicam->v4l2_dev);
> + if (ret) {
> + dev_err(unicam->dev, "Unable to register subdev nodes.\n");
> + goto unregister;
> + }
> + unicam->v4l2_dev.release = unicam_release;
> +
> + return 0;
> +
> +unregister:
> + unregister_nodes(unicam);
> +
> + return ret;
> +}
> +
> +static const struct v4l2_async_notifier_operations unicam_async_ops = {
> + .bound = unicam_async_bound,
> + .complete = unicam_async_complete,
> +};
> +
> +static int of_unicam_connect_subdevs(struct unicam_device *dev)
> +{
> + struct v4l2_fwnode_endpoint ep = { 0 };
> + struct device_node *ep_node;
> + int ret;
> +
> + if (of_property_read_u32(dev->dev->of_node, "brcm,num-data-lanes",
> + &dev->max_data_lanes) < 0) {
> + dev_err(dev->dev, "number of data lanes not set\n");
> + return -EINVAL;
> + }
> +
> + /* Get the local endpoint and remote device. */
> + ep_node = of_graph_get_next_endpoint(dev->dev->of_node, NULL);
> + if (!ep_node) {
> + dev_err(dev->dev, "can't get next endpoint\n");
> + return -EINVAL;
> + }
> +
> + /* Parse the local endpoint and validate its configuration. */
> + v4l2_fwnode_endpoint_parse(of_fwnode_handle(ep_node), &ep);
> + dev->bus_type = ep.bus_type;
> +
> + switch (ep.bus_type) {
> + case V4L2_MBUS_CSI2_DPHY:
> + switch (ep.bus.mipi_csi2.num_data_lanes) {
> + case 1:
> + case 2:
> + case 4:
> + break;
> +
> + default:
> + dev_err(dev->dev, "%u data lanes not supported\n",
> + ep.bus.mipi_csi2.num_data_lanes);
> + goto err_ep_config;
> + }
> +
> + if (ep.bus.mipi_csi2.num_data_lanes > dev->max_data_lanes) {
> + dev_err(dev->dev,
> + "%u lanes requested when %u are supported\n",
> + ep.bus.mipi_csi2.num_data_lanes,
> + dev->max_data_lanes);
> + goto err_ep_config;
> + }
> +
> + dev->max_data_lanes = ep.bus.mipi_csi2.num_data_lanes;
> + dev->bus_flags = ep.bus.mipi_csi2.flags;
> + break;
> +
> + case V4L2_MBUS_CCP2:
> + if (ep.bus.mipi_csi1.clock_lane != 0 ||
> + ep.bus.mipi_csi1.data_lane != 1) {
> + dev_err(dev->dev, "unsupported lanes configuration\n");
> + goto err_ep_config;
> + }
> +
> + dev->max_data_lanes = 1;
> + dev->bus_flags = ep.bus.mipi_csi1.strobe;
> + break;
> +
> + default:
> + /* Unsupported bus type */
> + dev_err(dev->dev, " unsupported bus type %u\n", ep.bus_type);
> + goto err_ep_config;
> + }
> +
> + /* Initialize and register the async notifier. */
> + v4l2_async_notifier_init(&dev->notifier);
> + dev->notifier.ops = &unicam_async_ops;
> +
> + ret = v4l2_async_notifier_add_fwnode_remote_subdev(&dev->notifier,
> + of_fwnode_handle(ep_node), &dev->asd);
> + of_node_put(ep_node);
> + if (ret) {
> + dev_err(dev->dev, "Error adding subdevice: %d\n", ret);
> + return ret;
> + }
> +
> + ret = v4l2_async_notifier_register(&dev->v4l2_dev, &dev->notifier);
> + if (ret) {
> + dev_err(dev->dev, "Error registering async notifier: %d\n",
> + ret);
> + return ret;
> + }
> +
> + return 0;
> +
> +err_ep_config:
> + of_node_put(ep_node);
> + return -EINVAL;
> +}
> +
> +static int unicam_probe(struct platform_device *pdev)
> +{
> + struct unicam_device *unicam;
> + int ret;
> +
> + unicam = devm_kzalloc(&pdev->dev, sizeof(*unicam), GFP_KERNEL);
> + if (!unicam)
> + return -ENOMEM;
> +
> + unicam->dev = &pdev->dev;
> + unicam->base = devm_platform_ioremap_resource(pdev, 0);
> + if (IS_ERR(unicam->base)) {
> + dev_err(unicam->dev, "Failed to get main io block\n");
> + return PTR_ERR(unicam->base);
> + }
> +
> + unicam->clk_gate_base = devm_platform_ioremap_resource(pdev, 1);
> + if (IS_ERR(unicam->clk_gate_base)) {
> + dev_err(unicam->dev, "Failed to get 2nd io block\n");
> + return PTR_ERR(unicam->clk_gate_base);
> + }
> +
> + unicam->clock = devm_clk_get(&pdev->dev, "lp");
> + if (IS_ERR(unicam->clock)) {
> + dev_err(unicam->dev, "Failed to get lp clock\n");
> + return PTR_ERR(unicam->clock);
> + }
> +
> + unicam->vpu_clock = devm_clk_get(&pdev->dev, "vpu");
> + if (IS_ERR(unicam->vpu_clock)) {
> + dev_err(unicam->dev, "Failed to get vpu clock\n");
> + return PTR_ERR(unicam->vpu_clock);
> + }
> +
> + ret = platform_get_irq(pdev, 0);
> + if (ret <= 0) {
> + dev_err(&pdev->dev, "No IRQ resource\n");
> + return -EINVAL;
> + }
> +
> + ret = devm_request_irq(&pdev->dev, ret, unicam_isr, 0,
> + "unicam_capture0", unicam);
> + if (ret) {
> + dev_err(&pdev->dev, "Unable to request interrupt\n");
> + return -EINVAL;
> + }
> +
> + unicam->mdev.dev = &pdev->dev;
> + strscpy(unicam->mdev.model, UNICAM_MODULE_NAME,
> + sizeof(unicam->mdev.model));
> + strscpy(unicam->mdev.serial, "", sizeof(unicam->mdev.serial));
> + snprintf(unicam->mdev.bus_info, sizeof(unicam->mdev.bus_info),
> + "platform:%s", dev_name(&pdev->dev));
> + unicam->mdev.hw_revision = 0;
> +
> + media_device_init(&unicam->mdev);
> +
> + unicam->v4l2_dev.mdev = &unicam->mdev;
> +
> + ret = v4l2_device_register(&pdev->dev, &unicam->v4l2_dev);
> + if (ret) {
> + dev_err(unicam->dev, "Unable to register v4l2 device.\n");
> + return ret;
> + }
> +
> + ret = media_device_register(&unicam->mdev);
> + if (ret < 0) {
> + dev_err(unicam->dev, "Unable to register media device.\n");
> + goto err_v4l2_unregister;
> + }
> +
> + /* Reserve space for the controls */
> + ret = v4l2_ctrl_handler_init(&unicam->ctrl_handler, 16);
> + if (ret < 0)
> + goto err_media_unregister;
> +
> + /* set the driver data in platform device */
> + platform_set_drvdata(pdev, unicam);
> +
> + ret = of_unicam_connect_subdevs(unicam);
> + if (ret) {
> + dev_err(&pdev->dev, "Failed to connect subdevs\n");
> + goto err_media_unregister;
> + }
> +
> + pm_runtime_enable(&pdev->dev);
> +
> + return 0;
> +
> +err_media_unregister:
> + media_device_unregister(&unicam->mdev);
> +err_v4l2_unregister:
> + v4l2_device_unregister(&unicam->v4l2_dev);
> +
> + return ret;
> +}
> +
> +static int unicam_remove(struct platform_device *pdev)
> +{
> + struct unicam_device *unicam = platform_get_drvdata(pdev);
> +
> + v4l2_async_notifier_unregister(&unicam->notifier);
> + v4l2_device_unregister(&unicam->v4l2_dev);
> + media_device_unregister(&unicam->mdev);
> + unregister_nodes(unicam);
> +
> + pm_runtime_disable(&pdev->dev);
> +
> + return 0;
> +}
> +
> +static const struct of_device_id unicam_of_match[] = {
> + { .compatible = "brcm,bcm2835-unicam", },
> + { /* sentinel */ },
> +};
> +MODULE_DEVICE_TABLE(of, unicam_of_match);
> +
> +static struct platform_driver unicam_driver = {
> + .probe = unicam_probe,
> + .remove = unicam_remove,
> + .driver = {
> + .name = UNICAM_MODULE_NAME,
> + .of_match_table = of_match_ptr(unicam_of_match),
> + },
> +};
> +
> +module_platform_driver(unicam_driver);
> +
> +MODULE_AUTHOR("Dave Stevenson <dave.stevenson@raspberrypi.com>");
> +MODULE_DESCRIPTION("BCM2835 Unicam driver");
> +MODULE_LICENSE("GPL");
> +MODULE_VERSION(UNICAM_VERSION);
> diff --git a/drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h b/drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
> new file mode 100644
> index 0000000000000..ae059a171d0fe
> --- /dev/null
> +++ b/drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
> @@ -0,0 +1,253 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +
> +/*
> + * Copyright (C) 2017-2020 Raspberry Pi Trading.
> + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> + */
> +
> +#ifndef VC4_REGS_UNICAM_H
> +#define VC4_REGS_UNICAM_H
> +
> +/*
> + * The following values are taken from files found within the code drop
> + * made by Broadcom for the BCM21553 Graphics Driver, predominantly in
> + * brcm_usrlib/dag/vmcsx/vcinclude/hardware_vc4.h.
> + * They have been modified to be only the register offset.
> + */
> +#define UNICAM_CTRL 0x000
> +#define UNICAM_STA 0x004
> +#define UNICAM_ANA 0x008
> +#define UNICAM_PRI 0x00c
> +#define UNICAM_CLK 0x010
> +#define UNICAM_CLT 0x014
> +#define UNICAM_DAT0 0x018
> +#define UNICAM_DAT1 0x01c
> +#define UNICAM_DAT2 0x020
> +#define UNICAM_DAT3 0x024
> +#define UNICAM_DLT 0x028
> +#define UNICAM_CMP0 0x02c
> +#define UNICAM_CMP1 0x030
> +#define UNICAM_CAP0 0x034
> +#define UNICAM_CAP1 0x038
> +#define UNICAM_ICTL 0x100
> +#define UNICAM_ISTA 0x104
> +#define UNICAM_IDI0 0x108
> +#define UNICAM_IPIPE 0x10c
> +#define UNICAM_IBSA0 0x110
> +#define UNICAM_IBEA0 0x114
> +#define UNICAM_IBLS 0x118
> +#define UNICAM_IBWP 0x11c
> +#define UNICAM_IHWIN 0x120
> +#define UNICAM_IHSTA 0x124
> +#define UNICAM_IVWIN 0x128
> +#define UNICAM_IVSTA 0x12c
> +#define UNICAM_ICC 0x130
> +#define UNICAM_ICS 0x134
> +#define UNICAM_IDC 0x138
> +#define UNICAM_IDPO 0x13c
> +#define UNICAM_IDCA 0x140
> +#define UNICAM_IDCD 0x144
> +#define UNICAM_IDS 0x148
> +#define UNICAM_DCS 0x200
> +#define UNICAM_DBSA0 0x204
> +#define UNICAM_DBEA0 0x208
> +#define UNICAM_DBWP 0x20c
> +#define UNICAM_DBCTL 0x300
> +#define UNICAM_IBSA1 0x304
> +#define UNICAM_IBEA1 0x308
> +#define UNICAM_IDI1 0x30c
> +#define UNICAM_DBSA1 0x310
> +#define UNICAM_DBEA1 0x314
> +#define UNICAM_MISC 0x400
> +
> +/*
> + * The following bitmasks are from the kernel released by Broadcom
> + * for Android - https://android.googlesource.com/kernel/bcm/
> + * The Rhea, Hawaii, and Java chips all contain the same VideoCore4
> + * Unicam block as BCM2835, as defined in eg
> + * arch/arm/mach-rhea/include/mach/rdb_A0/brcm_rdb_cam.h and similar.
> + * Values reworked to use the kernel BIT and GENMASK macros.
> + *
> + * Some of the bit mnenomics have been amended to match the datasheet.
> + */
> +/* UNICAM_CTRL Register */
> +#define UNICAM_CPE BIT(0)
> +#define UNICAM_MEM BIT(1)
> +#define UNICAM_CPR BIT(2)
> +#define UNICAM_CPM_MASK GENMASK(3, 3)
> +#define UNICAM_CPM_CSI2 0
> +#define UNICAM_CPM_CCP2 1
> +#define UNICAM_SOE BIT(4)
> +#define UNICAM_DCM_MASK GENMASK(5, 5)
> +#define UNICAM_DCM_STROBE 0
> +#define UNICAM_DCM_DATA 1
> +#define UNICAM_SLS BIT(6)
> +#define UNICAM_PFT_MASK GENMASK(11, 8)
> +#define UNICAM_OET_MASK GENMASK(20, 12)
> +
> +/* UNICAM_STA Register */
> +#define UNICAM_SYN BIT(0)
> +#define UNICAM_CS BIT(1)
> +#define UNICAM_SBE BIT(2)
> +#define UNICAM_PBE BIT(3)
> +#define UNICAM_HOE BIT(4)
> +#define UNICAM_PLE BIT(5)
> +#define UNICAM_SSC BIT(6)
> +#define UNICAM_CRCE BIT(7)
> +#define UNICAM_OES BIT(8)
> +#define UNICAM_IFO BIT(9)
> +#define UNICAM_OFO BIT(10)
> +#define UNICAM_BFO BIT(11)
> +#define UNICAM_DL BIT(12)
> +#define UNICAM_PS BIT(13)
> +#define UNICAM_IS BIT(14)
> +#define UNICAM_PI0 BIT(15)
> +#define UNICAM_PI1 BIT(16)
> +#define UNICAM_FSI_S BIT(17)
> +#define UNICAM_FEI_S BIT(18)
> +#define UNICAM_LCI_S BIT(19)
> +#define UNICAM_BUF0_RDY BIT(20)
> +#define UNICAM_BUF0_NO BIT(21)
> +#define UNICAM_BUF1_RDY BIT(22)
> +#define UNICAM_BUF1_NO BIT(23)
> +#define UNICAM_DI BIT(24)
> +
> +#define UNICAM_STA_MASK_ALL \
> + (UNICAM_DL + \
> + UNICAM_SBE + \
> + UNICAM_PBE + \
> + UNICAM_HOE + \
> + UNICAM_PLE + \
> + UNICAM_SSC + \
> + UNICAM_CRCE + \
> + UNICAM_IFO + \
> + UNICAM_OFO + \
> + UNICAM_PS + \
> + UNICAM_PI0 + \
> + UNICAM_PI1)
> +
> +/* UNICAM_ANA Register */
> +#define UNICAM_APD BIT(0)
> +#define UNICAM_BPD BIT(1)
> +#define UNICAM_AR BIT(2)
> +#define UNICAM_DDL BIT(3)
> +#define UNICAM_CTATADJ_MASK GENMASK(7, 4)
> +#define UNICAM_PTATADJ_MASK GENMASK(11, 8)
> +
> +/* UNICAM_PRI Register */
> +#define UNICAM_PE BIT(0)
> +#define UNICAM_PT_MASK GENMASK(2, 1)
> +#define UNICAM_NP_MASK GENMASK(7, 4)
> +#define UNICAM_PP_MASK GENMASK(11, 8)
> +#define UNICAM_BS_MASK GENMASK(15, 12)
> +#define UNICAM_BL_MASK GENMASK(17, 16)
> +
> +/* UNICAM_CLK Register */
> +#define UNICAM_CLE BIT(0)
> +#define UNICAM_CLPD BIT(1)
> +#define UNICAM_CLLPE BIT(2)
> +#define UNICAM_CLHSE BIT(3)
> +#define UNICAM_CLTRE BIT(4)
> +#define UNICAM_CLAC_MASK GENMASK(8, 5)
> +#define UNICAM_CLSTE BIT(29)
> +
> +/* UNICAM_CLT Register */
> +#define UNICAM_CLT1_MASK GENMASK(7, 0)
> +#define UNICAM_CLT2_MASK GENMASK(15, 8)
> +
> +/* UNICAM_DATn Registers */
> +#define UNICAM_DLE BIT(0)
> +#define UNICAM_DLPD BIT(1)
> +#define UNICAM_DLLPE BIT(2)
> +#define UNICAM_DLHSE BIT(3)
> +#define UNICAM_DLTRE BIT(4)
> +#define UNICAM_DLSM BIT(5)
> +#define UNICAM_DLFO BIT(28)
> +#define UNICAM_DLSTE BIT(29)
> +
> +#define UNICAM_DAT_MASK_ALL (UNICAM_DLSTE + UNICAM_DLFO)
> +
> +/* UNICAM_DLT Register */
> +#define UNICAM_DLT1_MASK GENMASK(7, 0)
> +#define UNICAM_DLT2_MASK GENMASK(15, 8)
> +#define UNICAM_DLT3_MASK GENMASK(23, 16)
> +
> +/* UNICAM_ICTL Register */
> +#define UNICAM_FSIE BIT(0)
> +#define UNICAM_FEIE BIT(1)
> +#define UNICAM_IBOB BIT(2)
> +#define UNICAM_FCM BIT(3)
> +#define UNICAM_TFC BIT(4)
> +#define UNICAM_LIP_MASK GENMASK(6, 5)
> +#define UNICAM_LCIE_MASK GENMASK(28, 16)
> +
> +/* UNICAM_IDI0/1 Register */
> +#define UNICAM_ID0_MASK GENMASK(7, 0)
> +#define UNICAM_ID1_MASK GENMASK(15, 8)
> +#define UNICAM_ID2_MASK GENMASK(23, 16)
> +#define UNICAM_ID3_MASK GENMASK(31, 24)
> +
> +/* UNICAM_ISTA Register */
> +#define UNICAM_FSI BIT(0)
> +#define UNICAM_FEI BIT(1)
> +#define UNICAM_LCI BIT(2)
> +
> +#define UNICAM_ISTA_MASK_ALL (UNICAM_FSI + UNICAM_FEI + UNICAM_LCI)
> +
> +/* UNICAM_IPIPE Register */
> +#define UNICAM_PUM_MASK GENMASK(2, 0)
> + /* Unpacking modes */
> + #define UNICAM_PUM_NONE 0
> + #define UNICAM_PUM_UNPACK6 1
> + #define UNICAM_PUM_UNPACK7 2
> + #define UNICAM_PUM_UNPACK8 3
> + #define UNICAM_PUM_UNPACK10 4
> + #define UNICAM_PUM_UNPACK12 5
> + #define UNICAM_PUM_UNPACK14 6
> + #define UNICAM_PUM_UNPACK16 7
> +#define UNICAM_DDM_MASK GENMASK(6, 3)
> +#define UNICAM_PPM_MASK GENMASK(9, 7)
> + /* Packing modes */
> + #define UNICAM_PPM_NONE 0
> + #define UNICAM_PPM_PACK8 1
> + #define UNICAM_PPM_PACK10 2
> + #define UNICAM_PPM_PACK12 3
> + #define UNICAM_PPM_PACK14 4
> + #define UNICAM_PPM_PACK16 5
> +#define UNICAM_DEM_MASK GENMASK(11, 10)
> +#define UNICAM_DEBL_MASK GENMASK(14, 12)
> +#define UNICAM_ICM_MASK GENMASK(16, 15)
> +#define UNICAM_IDM_MASK GENMASK(17, 17)
> +
> +/* UNICAM_ICC Register */
> +#define UNICAM_ICFL_MASK GENMASK(4, 0)
> +#define UNICAM_ICFH_MASK GENMASK(9, 5)
> +#define UNICAM_ICST_MASK GENMASK(12, 10)
> +#define UNICAM_ICLT_MASK GENMASK(15, 13)
> +#define UNICAM_ICLL_MASK GENMASK(31, 16)
> +
> +/* UNICAM_DCS Register */
> +#define UNICAM_DIE BIT(0)
> +#define UNICAM_DIM BIT(1)
> +#define UNICAM_DBOB BIT(3)
> +#define UNICAM_FDE BIT(4)
> +#define UNICAM_LDP BIT(5)
> +#define UNICAM_EDL_MASK GENMASK(15, 8)
> +
> +/* UNICAM_DBCTL Register */
> +#define UNICAM_DBEN BIT(0)
> +#define UNICAM_BUF0_IE BIT(1)
> +#define UNICAM_BUF1_IE BIT(2)
> +
> +/* UNICAM_CMP[0,1] register */
> +#define UNICAM_PCE BIT(31)
> +#define UNICAM_GI BIT(9)
> +#define UNICAM_CPH BIT(8)
> +#define UNICAM_PCVC_MASK GENMASK(7, 6)
> +#define UNICAM_PCDT_MASK GENMASK(5, 0)
> +
> +/* UNICAM_MISC register */
> +#define UNICAM_FL0 BIT(6)
> +#define UNICAM_FL1 BIT(9)
> +
> +#endif
>
--
Regards
--
Kieran
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file
2020-12-02 22:06 ` Sakari Ailus
@ 2020-12-03 11:42 ` Dave Stevenson
2021-01-05 13:30 ` Laurent Pinchart
0 siblings, 1 reply; 31+ messages in thread
From: Dave Stevenson @ 2020-12-03 11:42 UTC (permalink / raw)
To: Sakari Ailus
Cc: Jacopo Mondi, Linux Media Mailing List, Naushir Patuck,
Laurent Pinchart, cc: Kieran Bingham, Niklas Söderlund,
Dafna Hirschfeld, Hans Verkuil, Nicolas Saenz Julienne,
mchehab+huawei
Hi Sakari
On Wed, 2 Dec 2020 at 22:07, Sakari Ailus <sakari.ailus@iki.fi> wrote:
>
> Hi Jacopo,
>
> On Tue, Nov 10, 2020 at 06:40:36PM +0100, Jacopo Mondi wrote:
> > The bcm2835-unicam driver is currently in staging mainly for
> > two reasons:
> > - Handling of CSI-2 embedded data
> > - Usage of both media controller API and subdev kAPI
> >
> > Provide a more detailed description of the currently on-going design
> > discussions in the associated TODO file.
> >
> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > ---
> > drivers/staging/media/bcm2835-unicam/TODO | 37 +++++++++++++++++++++++
> > 1 file changed, 37 insertions(+)
> > create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
> >
> > diff --git a/drivers/staging/media/bcm2835-unicam/TODO b/drivers/staging/media/bcm2835-unicam/TODO
> > new file mode 100644
> > index 0000000000000..c7840872eea4c
> > --- /dev/null
> > +++ b/drivers/staging/media/bcm2835-unicam/TODO
> > @@ -0,0 +1,37 @@
> > +BCM2835 Unicam driver TODO list
> > +===============================
> > +
> > +The unicam driver could be considered ready to be moved out of the staging
> > +directory in terms of code quality and expected functionalities.
> > +
> > +However there currently are two design issues that suggest the driver is
> > +better kept in staging for the time being.
> > +
> > +CSI-2 Embedded data support:
> > +----------------------------
> > +
> > +The RaspberryPi camera stack and camera applications rely on the availability of
> > +the sensor produced CSI-2 embedded data, whose support is currently not
> > +finalized in mainline Linux.
> > +
> > +The driver conditionally registers an additional video device node
> > +'unicam-embedded' with a single sink pad which connects to the sensor
> > +sub-device source pad #1 to expose ancillary data.
> > +
> > +Currently none of the mainline sensor drivers register more than a single pad,
> > +and consequentially no embedded data from the sensor are exposed to userspace.
> > +
> > +The current implementation is then subject to changes depending on how support
> > +for CSI-2 embedded data gets finalized in Linux.
>
> Are you looking to use out-of-tree sensor drivers that have two pads? I'd
> rather see sensor drivers merged to mainline proper.
>
> But as noted in the other e-mail, we need VC / stream support so this needs
> to be addressed for reasons not related to Unicam.
There's a downstream patch for imx219 that adds the second pad [1].
The imx477 driver that is currently only out-of-tree also supports it,
and should be upstreamed once this first wave of patches have got
somewhere.
[1] https://github.com/raspberrypi/linux/commit/fa8131cb1399f2c22de3f29e08ec1658db76552b
It's on the rpi-5.10.y branch too, but that is still being frequently
rebased so no stable commit hashes
> > +
> > +Media controller support:
> > +-------------------------
> > +
> > +Due to compatibility reasons with the existing RaspberryPi software ecosystem
> > +the unicam driver implements the media controller interface to allow the
> > +enumeration of the connected entities but handles the configuration of the
> > +sensor sub-device using the v4l2-subdev kAPI instead of delegating that to
> > +user-space.
> > +
> > +Discussions are on-going on how this should be better handled (driver option,
> > +KConfig option etc etc).
>
> That's a fair use case. There are two ways to handle it, either in the
> kernel where the choice affects how the user space looks like. You have an
> option of module parameter or Kconfig option there, and both are true
> annoyances.
>
> Another option is to work around it in the user space, wrapping the IOCTL
> calls. This way no kernel build or module load time parameters would be
> needed to switch between the two sets of user space programs.
>
> We probably can't decide it now, but could an MC-only driver + user space
> compatibility layer be an option here?
Iff the user-space compatibility layer worked with all standard users
(eg v4l2-ctl, FFmpeg and Gstreamer), then it's plausible, but is that
realistic?
The non-MC approach is mainly for things like the TC358743 HDMI->CSI
bridge and ADV728x-M chips where MC adds nothing, and they do just
work with the likes of FFmpeg and Gstreamer.
Image sensors are generally going to be used under libcamera umbrella,
so there MC works.
Kconfig isn't an option for us as one kernel build needs to support
all potential source devices via DT / runtime changes alone.
I haven't looked in detail at what the previous VC / stream patches
proposed as the API for passing the configuration. We only need MC for
the more complex use cases, so if that VC/stream API usage can be
detected at probe then we can switch to MC. It doesn't seem
unreasonable to expect any sensor drivers to be upgraded to correctly
use the new API even if they don't actually produce embedded data.
We also want MC for ADV748x HDMI&analog->CSI bridges that expose
multiple subdevs, but there we can look at the upstream endpoint and
see if it has any sink pads. Sink pads mean MC is needed. (I now have
a board with an ADV7482 on, so when time allows I'm intending to have
an experiment with it).
Use a module parameter as a last resort should the detection fail, or
I guess if you want to override the detected setting for some reason
(a simple sensor being used outside of libcamera for example).
Cheers.
Dave
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
2020-12-02 21:53 ` Sakari Ailus
@ 2020-12-03 16:30 ` Jacopo Mondi
2020-12-03 17:12 ` Dave Stevenson
0 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-03 16:30 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Wed, Dec 02, 2020 at 11:53:20PM +0200, Sakari Ailus wrote:
> Hi Jacopo,
>
> On Tue, Nov 10, 2020 at 06:40:34PM +0100, Jacopo Mondi wrote:
> > From: Naushir Patuck <naush@raspberrypi.com>
> >
> > Add a driver for the Unicam camera receiver block on BCM283x processors.
> > Compared to the bcm2835-camera driver present in staging, this driver
> > handles the Unicam block only (CSI-2 receiver), and doesn't depend on
> > the VC4 firmware running on the VPU.
> >
> > The commit is made up of a series of changes cherry-picked from the
> > rpi-5.9.y branch of https://github.com/raspberrypi/linux/ at revision
> > commit be371a8ffc012
> > ("media: bcm2835-unicam: change minimum number of vb2_queue buffers to 1")
> > with additional enhancements, forward-ported to the mainline kernel.
> >
> > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > ---
> > MAINTAINERS | 7 +
> > drivers/staging/media/Kconfig | 2 +
> > drivers/staging/media/Makefile | 1 +
> > drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
> > drivers/staging/media/bcm2835-unicam/Makefile | 3 +
> > .../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
> > .../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
> > 7 files changed, 3037 insertions(+)
> > create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
> > create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
> > create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> > create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 811db1d3ca33b..69d55ed67e1cf 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -3411,6 +3411,13 @@ N: bcm113*
> > N: bcm216*
> > N: kona
> >
> > +BROADCOM BCM2835 CAMERA DRIVER
> > +M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
> > +L: linux-media@vger.kernel.org
> > +S: Maintained
> > +F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > +F: drivers/staging/media/bcm2835/
> > +
> > BROADCOM BCM47XX MIPS ARCHITECTURE
> > M: Hauke Mehrtens <hauke@hauke-m.de>
> > M: Rafał Miłecki <zajec5@gmail.com>
> > diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
> > index 747c6cf1d795e..b734e882f63c9 100644
> > --- a/drivers/staging/media/Kconfig
> > +++ b/drivers/staging/media/Kconfig
> > @@ -46,4 +46,6 @@ source "drivers/staging/media/ipu3/Kconfig"
> >
> > source "drivers/staging/media/rkisp1/Kconfig"
> >
> > +source "drivers/staging/media/bcm2835-unicam/Kconfig"
> > +
> > endif
> > diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
> > index b59571826ba69..93d819ef243f1 100644
> > --- a/drivers/staging/media/Makefile
> > +++ b/drivers/staging/media/Makefile
> > @@ -1,6 +1,7 @@
> > # SPDX-License-Identifier: GPL-2.0
> > obj-$(CONFIG_VIDEO_ALLEGRO_DVT) += allegro-dvt/
> > obj-$(CONFIG_INTEL_ATOMISP) += atomisp/
> > +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam/
> > obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx/
> > obj-$(CONFIG_VIDEO_MESON_VDEC) += meson/vdec/
> > obj-$(CONFIG_VIDEO_OMAP4) += omap4iss/
> > diff --git a/drivers/staging/media/bcm2835-unicam/Kconfig b/drivers/staging/media/bcm2835-unicam/Kconfig
> > new file mode 100644
> > index 0000000000000..bd13701996509
> > --- /dev/null
> > +++ b/drivers/staging/media/bcm2835-unicam/Kconfig
> > @@ -0,0 +1,21 @@
> > +# Broadcom VideoCore4 V4L2 camera support
> > +
> > +config VIDEO_BCM2835_UNICAM
> > + tristate "Broadcom BCM283x/BCM271x Unicam video capture driver"
> > + depends on VIDEO_V4L2
> > + depends on ARCH_BCM2835 || COMPILE_TEST
> > + select VIDEO_V4L2_SUBDEV_API
> > + select MEDIA_CONTROLLER
> > + select VIDEOBUF2_DMA_CONTIG
> > + select V4L2_FWNODE
> > + help
> > + Say Y here to enable support for the BCM283x/BCM271x CSI-2 receiver.
> > + This is a V4L2 driver that controls the CSI-2 receiver directly,
> > + independently from the VC4 firmware.
> > + This driver is mutually exclusive with the use of bcm2835-camera. The
> > + firmware will disable all access to the peripheral from within the
> > + firmware if it finds a DT node using it, and bcm2835-camera will
> > + therefore fail to probe.
> > +
> > + To compile this driver as a module, choose M here. The module will be
> > + called bcm2835-unicam.
> > diff --git a/drivers/staging/media/bcm2835-unicam/Makefile b/drivers/staging/media/bcm2835-unicam/Makefile
> > new file mode 100644
> > index 0000000000000..a98aba03598ab
> > --- /dev/null
> > +++ b/drivers/staging/media/bcm2835-unicam/Makefile
> > @@ -0,0 +1,3 @@
> > +# Makefile for BCM2835 Unicam driver
> > +
> > +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam.o
> > diff --git a/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> > new file mode 100644
> > index 0000000000000..3e365baa8197f
> > --- /dev/null
> > +++ b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> > @@ -0,0 +1,2750 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * BCM283x / BCM271x Unicam Capture Driver
> > + *
> > + * Copyright (C) 2017-2020 - Raspberry Pi (Trading) Ltd.
> > + *
> > + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> > + *
> > + * Based on TI am437x driver by
> > + * Benoit Parrot <bparrot@ti.com>
> > + * Lad, Prabhakar <prabhakar.csengg@gmail.com>
> > + *
> > + * and TI CAL camera interface driver by
> > + * Benoit Parrot <bparrot@ti.com>
> > + *
> > + *
> > + * There are two camera drivers in the kernel for BCM283x - this one
> > + * and bcm2835-camera (currently in staging).
> > + *
> > + * This driver directly controls the Unicam peripheral - there is no
> > + * involvement with the VideoCore firmware. Unicam receives CSI-2 or
> > + * CCP2 data and writes it into SDRAM.
> > + * The only potential processing options are to repack Bayer data into an
> > + * alternate format, and applying windowing.
> > + * The repacking does not shift the data, so can repack V4L2_PIX_FMT_Sxxxx10P
> > + * to V4L2_PIX_FMT_Sxxxx10, or V4L2_PIX_FMT_Sxxxx12P to V4L2_PIX_FMT_Sxxxx12,
> > + * but not generically up to V4L2_PIX_FMT_Sxxxx16. The driver will add both
> > + * formats where the relevant formats are defined, and will automatically
> > + * configure the repacking as required.
> > + * Support for windowing may be added later.
> > + *
> > + * It should be possible to connect this driver to any sensor with a
> > + * suitable output interface and V4L2 subdevice driver.
> > + *
> > + * bcm2835-camera uses the VideoCore firmware to control the sensor,
> > + * Unicam, ISP, and all tuner control loops. Fully processed frames are
> > + * delivered to the driver by the firmware. It only has sensor drivers
> > + * for Omnivision OV5647, and Sony IMX219 sensors.
> > + *
> > + * The two drivers are mutually exclusive for the same Unicam instance.
> > + * The VideoCore firmware checks the device tree configuration during boot.
> > + * If it finds device tree nodes called csi0 or csi1 it will block the
> > + * firmware from accessing the peripheral, and bcm2835-camera will
> > + * not be able to stream data.
> > + */
> > +
> > +#include <linux/clk.h>
> > +#include <linux/delay.h>
> > +#include <linux/device.h>
> > +#include <linux/dma-mapping.h>
> > +#include <linux/err.h>
> > +#include <linux/init.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/io.h>
> > +#include <linux/module.h>
> > +#include <linux/of_device.h>
> > +#include <linux/of_graph.h>
> > +#include <linux/pinctrl/consumer.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/slab.h>
> > +#include <linux/uaccess.h>
> > +#include <linux/videodev2.h>
> > +
> > +#include <media/v4l2-common.h>
> > +#include <media/v4l2-ctrls.h>
> > +#include <media/v4l2-dev.h>
> > +#include <media/v4l2-device.h>
> > +#include <media/v4l2-dv-timings.h>
> > +#include <media/v4l2-event.h>
> > +#include <media/v4l2-ioctl.h>
> > +#include <media/v4l2-fwnode.h>
> > +#include <media/videobuf2-dma-contig.h>
> > +
> > +#include "vc4-regs-unicam.h"
> > +
> > +#define UNICAM_MODULE_NAME "unicam"
> > +#define UNICAM_VERSION "0.1.0"
> > +
> > +/*
> > + * Unicam must request a minimum of 250Mhz from the VPU clock.
> > + * Otherwise the input FIFOs overrun and cause image corruption.
> > + */
> > +#define MIN_VPU_CLOCK_RATE (250 * 1000 * 1000)
> > +/*
> > + * To protect against a dodgy sensor driver never returning an error from
> > + * enum_mbus_code, set a maximum index value to be used.
> > + */
> > +#define MAX_ENUM_MBUS_CODE 128
>
> Please remove, and instead fix the buggy sensor drivers.
>
Dave has use cases that require this iirc.
I guess this could be made an out-of-tree on-top patch ?
> > +
> > +/*
> > + * Stride is a 16 bit register, but also has to be a multiple of 32.
> > + */
> > +#define BPL_ALIGNMENT 32
> > +#define MAX_BYTESPERLINE ((1 << 16) - BPL_ALIGNMENT)
> > +/*
> > + * Max width is therefore determined by the max stride divided by
> > + * the number of bits per pixel. Take 32bpp as a
> > + * worst case.
> > + * No imposed limit on the height, so adopt a square image for want
> > + * of anything better.
> > + */
> > +#define MAX_WIDTH (MAX_BYTESPERLINE / 4)
> > +#define MAX_HEIGHT MAX_WIDTH
> > +/* Define a nominal minimum image size */
> > +#define MIN_WIDTH 16
> > +#define MIN_HEIGHT 16
> > +
> > +/*
> > + * Size of the dummy buffer. Can be any size really, but the DMA
> > + * allocation works in units of page sizes.
> > + */
> > +#define DUMMY_BUF_SIZE (PAGE_SIZE)
> > +
> > +/* Enumeration of the video device node identifiers. */
> > +enum unicam_node_ids {
> > + IMAGE_NODE,
> > + METADATA_NODE,
> > + MAX_NODES
> > +};
> > +
> > +/*
> > + * Associate video device nodes with the subdevice's source pad ids.
> > + * The image video device is connected to the subdevice pad #0;
> > + * The metadata video device is connected to the subdevice pad #1;
> > + */
> > +unsigned int subdev_pads[2] = {
> > + [IMAGE_NODE] = 0,
> > + [METADATA_NODE] = 1,
> > +};
>
> I recognise this and the above enum are used for different purpose, but
> this is also entirely useless. At the very least make it static.
>
It's a small syntactic sugar that's makes identifying pads a little
more explicit.
I'll make it static const
> > +
> > +/*
> > + * struct unicam_fmt - Unicam media bus format information
> > + * @pixelformat: V4L2 pixel format FCC identifier. 0 if n/a.
> > + * @repacked_fourcc: V4L2 pixel format FCC identifier if the data is expanded
> > + * out to 16bpp. 0 if n/a.
> > + * @code: V4L2 media bus format code.
> > + * @depth: Bits per pixel as delivered from the source.
> > + * @csi_dt: CSI data type.
> > + * @check_variants: Flag to denote that there are multiple mediabus formats
> > + * still in the list that could match this V4L2 format.
> > + */
> > +struct unicam_fmt {
> > + u32 fourcc;
> > + u32 repacked_fourcc;
> > + u32 code;
> > + u8 depth;
> > + u8 csi_dt;
> > + u8 check_variants;
> > +};
> > +
> > +static const struct unicam_fmt formats[] = {
> > + /* YUV Formats */
> > + {
> > + .fourcc = V4L2_PIX_FMT_YUYV,
> > + .code = MEDIA_BUS_FMT_YUYV8_2X8,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + .check_variants = 1,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_UYVY,
> > + .code = MEDIA_BUS_FMT_UYVY8_2X8,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + .check_variants = 1,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_YVYU,
> > + .code = MEDIA_BUS_FMT_YVYU8_2X8,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + .check_variants = 1,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_VYUY,
> > + .code = MEDIA_BUS_FMT_VYUY8_2X8,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + .check_variants = 1,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_YUYV,
> > + .code = MEDIA_BUS_FMT_YUYV8_1X16,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_UYVY,
> > + .code = MEDIA_BUS_FMT_UYVY8_1X16,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_YVYU,
> > + .code = MEDIA_BUS_FMT_YVYU8_1X16,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_VYUY,
> > + .code = MEDIA_BUS_FMT_VYUY8_1X16,
> > + .depth = 16,
> > + .csi_dt = 0x1e,
> > + }, {
> > + /* RGB Formats */
> > + .fourcc = V4L2_PIX_FMT_RGB565, /* gggbbbbb rrrrrggg */
> > + .code = MEDIA_BUS_FMT_RGB565_2X8_LE,
> > + .depth = 16,
> > + .csi_dt = 0x22,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_RGB565X, /* rrrrrggg gggbbbbb */
> > + .code = MEDIA_BUS_FMT_RGB565_2X8_BE,
> > + .depth = 16,
> > + .csi_dt = 0x22
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_RGB555, /* gggbbbbb arrrrrgg */
> > + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE,
> > + .depth = 16,
> > + .csi_dt = 0x21,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_RGB555X, /* arrrrrgg gggbbbbb */
> > + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE,
> > + .depth = 16,
> > + .csi_dt = 0x21,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_RGB24, /* rgb */
> > + .code = MEDIA_BUS_FMT_RGB888_1X24,
> > + .depth = 24,
> > + .csi_dt = 0x24,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_BGR24, /* bgr */
> > + .code = MEDIA_BUS_FMT_BGR888_1X24,
> > + .depth = 24,
> > + .csi_dt = 0x24,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_RGB32, /* argb */
> > + .code = MEDIA_BUS_FMT_ARGB8888_1X32,
> > + .depth = 32,
> > + .csi_dt = 0x0,
> > + }, {
> > + /* Bayer Formats */
> > + .fourcc = V4L2_PIX_FMT_SBGGR8,
> > + .code = MEDIA_BUS_FMT_SBGGR8_1X8,
> > + .depth = 8,
> > + .csi_dt = 0x2a,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGBRG8,
> > + .code = MEDIA_BUS_FMT_SGBRG8_1X8,
> > + .depth = 8,
> > + .csi_dt = 0x2a,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGRBG8,
> > + .code = MEDIA_BUS_FMT_SGRBG8_1X8,
> > + .depth = 8,
> > + .csi_dt = 0x2a,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SRGGB8,
> > + .code = MEDIA_BUS_FMT_SRGGB8_1X8,
> > + .depth = 8,
> > + .csi_dt = 0x2a,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SBGGR10P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SBGGR10,
> > + .code = MEDIA_BUS_FMT_SBGGR10_1X10,
> > + .depth = 10,
> > + .csi_dt = 0x2b,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGBRG10P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SGBRG10,
> > + .code = MEDIA_BUS_FMT_SGBRG10_1X10,
> > + .depth = 10,
> > + .csi_dt = 0x2b,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGRBG10P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SGRBG10,
> > + .code = MEDIA_BUS_FMT_SGRBG10_1X10,
> > + .depth = 10,
> > + .csi_dt = 0x2b,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SRGGB10P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SRGGB10,
> > + .code = MEDIA_BUS_FMT_SRGGB10_1X10,
> > + .depth = 10,
> > + .csi_dt = 0x2b,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SBGGR12P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SBGGR12,
> > + .code = MEDIA_BUS_FMT_SBGGR12_1X12,
> > + .depth = 12,
> > + .csi_dt = 0x2c,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGBRG12P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SGBRG12,
> > + .code = MEDIA_BUS_FMT_SGBRG12_1X12,
> > + .depth = 12,
> > + .csi_dt = 0x2c,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGRBG12P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SGRBG12,
> > + .code = MEDIA_BUS_FMT_SGRBG12_1X12,
> > + .depth = 12,
> > + .csi_dt = 0x2c,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SRGGB12P,
> > + .repacked_fourcc = V4L2_PIX_FMT_SRGGB12,
> > + .code = MEDIA_BUS_FMT_SRGGB12_1X12,
> > + .depth = 12,
> > + .csi_dt = 0x2c,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SBGGR14P,
> > + .code = MEDIA_BUS_FMT_SBGGR14_1X14,
> > + .depth = 14,
> > + .csi_dt = 0x2d,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGBRG14P,
> > + .code = MEDIA_BUS_FMT_SGBRG14_1X14,
> > + .depth = 14,
> > + .csi_dt = 0x2d,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SGRBG14P,
> > + .code = MEDIA_BUS_FMT_SGRBG14_1X14,
> > + .depth = 14,
> > + .csi_dt = 0x2d,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_SRGGB14P,
> > + .code = MEDIA_BUS_FMT_SRGGB14_1X14,
> > + .depth = 14,
> > + .csi_dt = 0x2d,
> > + }, {
> > + /*
> > + * 16 bit Bayer formats could be supported, but there is no CSI2
> > + * data_type defined for raw 16, and no sensors that produce it at
> > + * present.
> > + */
> > +
> > + /* Greyscale formats */
> > + .fourcc = V4L2_PIX_FMT_GREY,
> > + .code = MEDIA_BUS_FMT_Y8_1X8,
> > + .depth = 8,
> > + .csi_dt = 0x2a,
> > + }, {
> > + .fourcc = V4L2_PIX_FMT_Y10P,
> > + .repacked_fourcc = V4L2_PIX_FMT_Y10,
> > + .code = MEDIA_BUS_FMT_Y10_1X10,
> > + .depth = 10,
> > + .csi_dt = 0x2b,
> > + }, {
> > + /* NB There is no packed V4L2 fourcc for this format. */
> > + .repacked_fourcc = V4L2_PIX_FMT_Y12,
> > + .code = MEDIA_BUS_FMT_Y12_1X12,
> > + .depth = 12,
> > + .csi_dt = 0x2c,
> > + },
> > + /* Embedded data format */
> > + {
> > + .fourcc = V4L2_META_FMT_SENSOR_DATA,
> > + .code = MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA,
> > + .depth = 8,
> > + }
> > +};
> > +
> > +struct unicam_buffer {
> > + struct vb2_v4l2_buffer vb;
> > + struct list_head list;
> > +};
> > +
> > +static inline struct unicam_buffer *to_unicam_buffer(struct vb2_buffer *vb)
> > +{
> > + return container_of(vb, struct unicam_buffer, vb.vb2_buf);
> > +}
> > +
> > +struct unicam_node {
> > + bool registered;
> > + int open;
> > + bool streaming;
> > + unsigned int node_id;
> > + /* Pointer pointing to current v4l2_buffer */
> > + struct unicam_buffer *cur_frm;
> > + /* Pointer pointing to next v4l2_buffer */
> > + struct unicam_buffer *next_frm;
> > + /* video capture */
> > + const struct unicam_fmt *fmt;
> > + /* Used to store current pixel format */
> > + struct v4l2_format v_fmt;
> > + /* Used to store current mbus frame format */
> > + struct v4l2_mbus_framefmt m_fmt;
> > + /* Buffer queue used in video-buf */
> > + struct vb2_queue buffer_queue;
> > + /* Queue of filled frames */
> > + struct list_head dma_queue;
> > + /* IRQ lock for DMA queue */
> > + spinlock_t dma_queue_lock;
> > + /* lock used to access this structure */
> > + struct mutex lock;
> > + /* Identifies video device for this channel */
> > + struct video_device video_dev;
> > + /* Pointer to the parent handle */
> > + struct unicam_device *dev;
> > + struct media_pad pad;
> > + /*
> > + * Dummy buffer intended to be used by unicam
> > + * if we have no other queued buffers to swap to.
> > + */
> > + void *dummy_buf_cpu_addr;
> > + dma_addr_t dummy_buf_dma_addr;
> > +};
> > +
> > +struct unicam_device {
> > + struct device *dev;
> > +
> > + /* V4l2 specific parameters */
> > + struct v4l2_async_subdev asd;
> > +
> > + /* peripheral base address */
> > + void __iomem *base;
> > + /* clock gating base address */
> > + void __iomem *clk_gate_base;
> > + /* lp clock handle */
> > + struct clk *clock;
> > + /* vpu clock handle */
> > + struct clk *vpu_clock;
> > + /* V4l2 device */
> > + struct v4l2_device v4l2_dev;
> > + struct media_device mdev;
> > +
> > + /* subdevice async Notifier */
> > + struct v4l2_async_notifier notifier;
> > + unsigned int sequence;
> > +
> > + /* ptr to sub device */
> > + struct v4l2_subdev *sensor;
> > + /* Pad config for the sensor */
> > + struct v4l2_subdev_pad_config *sensor_config;
> > +
> > + enum v4l2_mbus_type bus_type;
> > + /*
> > + * Stores bus.mipi_csi2.flags for CSI2 sensors, or
> > + * bus.mipi_csi1.strobe for CCP2.
> > + */
> > + unsigned int bus_flags;
> > + unsigned int max_data_lanes;
> > + unsigned int active_data_lanes;
> > +
> > + struct unicam_node node[MAX_NODES];
> > + struct v4l2_ctrl_handler ctrl_handler;
> > +};
> > +
> > +static inline struct unicam_device *
> > +to_unicam_device(struct v4l2_device *v4l2_dev)
> > +{
> > + return container_of(v4l2_dev, struct unicam_device, v4l2_dev);
> > +}
> > +
> > +static char *print_fourcc(u32 fmt)
> > +{
> > + static char code[16];
> > +
> > + code[0] = (unsigned char)(fmt & 0xff);
> > + code[1] = (unsigned char)((fmt >> 8) & 0xff);
> > + code[2] = (unsigned char)((fmt >> 16) & 0xff);
> > + code[3] = (unsigned char)((fmt >> 24) & 0xff);
> > + snprintf(&code[4], 12, "=0x%8x", fmt);
>
> Ugh. Hopefully we'll have %p4cc soon. It's not yet merged at least. :-\
>
> The rest looks fairly normal, apart from the obvious lack of MC support
> from a driver that supports embedded data (also a TODO item).
Isn't it recorded in the TODO I've added at the end of the series ?
Thanks
j
>
> --
> Kind regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-02 21:20 ` Sakari Ailus
@ 2020-12-03 16:44 ` Jacopo Mondi
2020-12-03 17:25 ` Sakari Ailus
0 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-03 16:44 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Wed, Dec 02, 2020 at 11:20:31PM +0200, Sakari Ailus wrote:
> Hi Jacopo,
>
> On Tue, Nov 10, 2020 at 06:40:33PM +0100, Jacopo Mondi wrote:
> > From: Dave Stevenson <dave.stevenson@raspberrypi.com>
> >
> > Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
> > Unicam) on BCM283x SoCs.
> >
> > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > ---
> > .../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
> > 1 file changed, 155 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > new file mode 100644
> > index 0000000000000..6ffc900e8ae8f
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > @@ -0,0 +1,155 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
> > +%YAML 1.2
> > +---
> > +$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
> > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > +
> > +title: Broadcom BCM283x Camera Interface (Unicam)
> > +
> > +maintainers:
> > + - Dave Stevenson <dave.stevenson@raspberrypi.com>
> > + - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
> > +
> > +description:
> > + The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
> > + data from image sensors or similar devices.
> > +
> > + The main platform using this SoC is the Raspberry Pi family of boards. On the
> > + Pi the VideoCore firmware can also control this hardware block, and driving
> > + it from two different processors will cause issues. To avoid this, the
> > + firmware checks the device tree configuration during boot. If it finds device
> > + tree nodes whose name starts with "csi" then it will stop the firmware
> > + accessing the block, and it can then safely be used via the device tree
> > + binding.
> > +
> > +properties:
> > + compatible:
> > + const: brcm,bcm2835-unicam
>
> The compatible string doesn't quite match with the title. Which SoCs are
> supported?
>
Afaict bcm2835 and bcm2711. There might be more I'm not aware of.
There should be no distinction between then, but Dave knows more.
Would a per-soc compatible work better or is a single brcm,unicam
good enough to start with and we can add per-soc compatibles if
there's any per-SoC difference that will need to be handled in future ?
> > +
> > + reg:
> > + items:
> > + - description: Main registers block
> > + - description: Clock registers block
> > +
> > + interrupts:
> > + maxItems: 1
> > +
> > + clocks:
> > + minItems: 1
> > + maxItems: 2
> > +
> > + clock-names:
> > + minItems: 1
> > + items:
> > + - const: lp
> > + - const: core
> > +
> > + power-domains:
> > + maxItems: 1
> > +
> > + brcm,num-data-lanes:
> > + description:
> > + The number of data lanes supported by this Unicam instance. It may be
> > + larger than the number of data lanes routed on the board, as described by
> > + the data-lanes property of the endpoint.
> > + allOf:
> > + - $ref: "/schemas/types.yaml#/definitions/uint32"
> > + - enum: [1, 2, 4]
>
> Do you need this information besides how many lanes are connected?
>
> Does the number of lanes change even within the same model of SoC? Could
> you use the compatible string to differentiate between them?
>
I had a similar question:
https://patchwork.linuxtv.org/project/linux-media/patch/20200504092611.9798-5-laurent.pinchart@ideasonboard.com/#121079
I think the discussion ended with the fact that not all the available
lanes in the IP might be routed to the connector, and that depends on
the board.
> > +
> > + port:
> > + type: object
> > + description:
> > + Input port node, as described in video-interfaces.txt.
> > +
> > + properties:
> > + endpoint:
> > + type: object
> > +
> > + properties:
> > + clock-lanes:
> > + items:
> > + - const: 0
>
> Please drop. There's nothing to tell software here.
>
Ack
> > +
> > + data-lanes:
> > + description:
> > + Lane reordering is not supported, items shall be in order,
> > + starting at 1.
> > + allOf:
> > + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> > + - maxItems: 1
> > + items:
> > + minItems: 1
> > + maxItems: 4
> > + items:
> > + - const: 1
> > + - const: 2
> > + - const: 3
> > + - const: 4
>
> No need to specify items.
>
Actually here we might need to support [1, 2] and [1, 2, 3, 4].
I'll rebase on the converted video-interfaces to see it that helps
> > +
> > + lane-polarities:
> > + description:
> > + Lane inversion is not supported. If the property is specified, it
> > + shall contain all 0's.
>
> Ditto.
>
Ack
> > + allOf:
> > + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> > + - maxItems: 1
> > + items:
> > + minItems: 2
> > + maxItems: 5
> > + items:
> > + - const: 0
> > + - const: 0
> > + - const: 0
> > + - const: 0
> > + - const: 0
> > +
> > + remote-endpoint: true
> > +
> > + required:
> > + - data-lanes
> > + - remote-endpoint
> > +
> > + additionalProperties: false
> > +
> > + required:
> > + - endpoint
> > +
> > + additionalProperties: false
> > +
> > +required:
> > + - compatible
> > + - reg
> > + - interrupts
> > + - clocks
> > + - clock-names
> > + - power-domains
> > + - brcm,num-data-lanes
> > + - port
> > +
> > +additionalProperties: false
> > +
> > +examples:
> > + - |
> > + #include <dt-bindings/clock/bcm2835.h>
> > + #include <dt-bindings/interrupt-controller/arm-gic.h>
> > + #include <dt-bindings/power/raspberrypi-power.h>
> > +
> > + csi@7e801000 {
> > + compatible = "brcm,bcm2835-unicam";
> > + reg = <0x7e801000 0x800>,
> > + <0x7e802004 0x4>;
> > + interrupts = <2 7>;
> > + clocks = <&clocks BCM2835_CLOCK_CAM1>;
> > + clock-names = "lp";
> > + power-domains = <&power RPI_POWER_DOMAIN_UNICAM1>;
> > + brcm,num-data-lanes = <4>;
> > +
> > + port {
> > + csi1_ep: endpoint {
> > + remote-endpoint = <&imx219_0>;
> > + data-lanes = <1 2>;
> > + };
> > + };
> > + };
> > +...
>
> --
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
2020-12-02 21:16 ` Sakari Ailus
@ 2020-12-03 17:05 ` Jacopo Mondi
0 siblings, 0 replies; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-03 17:05 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Wed, Dec 02, 2020 at 11:16:48PM +0200, Sakari Ailus wrote:
> Hi Jacopo,
>
> Thank you for the patchset.
>
> On Tue, Nov 10, 2020 at 06:40:32PM +0100, Jacopo Mondi wrote:
> > From: Naushir Patuck <naush@raspberrypi.com>
> >
> > Add V4L2_META_FMT_CUSTOM_SENSOR_DATA format.
> >
> > This new format will be used to return camera sensor embedded data.
> >
> > Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > ---
> > .../userspace-api/media/v4l/meta-formats.rst | 1 +
> > .../media/v4l/pixfmt-meta-sensor-data.rst | 24 +++++++++++++++
> > .../media/v4l/subdev-formats.rst | 29 +++++++++++++++++++
> > drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> > include/uapi/linux/media-bus-format.h | 3 ++
> > include/uapi/linux/videodev2.h | 1 +
> > 6 files changed, 59 insertions(+)
> > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
> >
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index fff25357fe860..b2201d1524eb6 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -15,6 +15,7 @@ These formats are used for the :ref:`metadata` interface only.
> > pixfmt-meta-d4xx
> > pixfmt-meta-intel-ipu3
> > pixfmt-meta-rkisp1
> > + pixfmt-meta-sensor-data
> > pixfmt-meta-uvc
> > pixfmt-meta-vsp1-hgo
> > pixfmt-meta-vsp1-hgt
> > diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
> > new file mode 100644
> > index 0000000000000..cc3929f595389
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-sensor-data.rst
> > @@ -0,0 +1,24 @@
> > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> > +
> > +.. _v4l2-meta-fmt-sensor-data:
> > +
> > +***********************************
> > +V4L2_META_FMT_SENSOR_DATA ('SENS')
> > +***********************************
> > +
> > +Sensor Ancillary Metadata
>
> "Sensor Embedded Data"
>
> > +
> > +Description
> > +===========
> > +
> > +This format describes ancillary data generated by a camera sensor and
> > +transmitted over a stream on the camera bus. Most sensor vendors have their
> > +own custom format for this ancillary data. Some vendors follow a generic
> > +CSI-2/SMIA embedded data format as described in the `CSI-2 specification.
> > +<https://mipi.org/specifications/csi-2>`_
>
> CSI-2 or SMIA? SMIA does define how embedded data is formatted, but this
> format is specific to SMIA and may be used for other sensors that are
> somewhat aligned with SMIA (or CCS) even if not fully compliant.
>
> SMIA defines both the format of embedded data as well as its packing into
> pixels at different bit depths. I have to say I have little knowledge on
> how this works on non-compliant sensors. Do other sensors use the same
> packing? Given that this isn't standardised, I could imagine there are
> other kinds of solutions.
Even for SMIA compliant formats, the point Dave had about the format
changing when the image format change, still holds and we don't have
a model that defines how metadata streams are exposed to userspace at the
moment, so I would say we don't have enough information at this time
to really answer these questions.
>
> In order to interpret this from the user space (as well as eventually
> transferring it to system memory), I think we'd need at least the packing
> and bits per pixel information even if the data format itself remained
> opaque at that level.
I understand about the bit depth
I'm not sure what packaging information we can capture in the format
definition though.
As per my last reply
https://patchwork.linuxtv.org/project/linux-media/patch/20201102165258.408049-2-jacopo@jmondi.org/#123351
at the receiver level no information should be required buffer size
apart (and the current mechanism is not v4l2-compliant, I agree). Do
you agree ?
>
> What are the documentation requirements for embedded data that is not
> SMIA/CCS, and therefore likely undocumented in publicly available
> documentation? Driver documentation?
>
I always assumed userspace being a sensor-specific component that
knows how to interpret those data but I never questioned where that
knowledge come from. I assume from documentation and as much as I like
to have it documented somewhere, I'm not sure this can be done in any
way but best-effort (as if the documentation is public, well,
everybody can read it, if it's confidential and we require it to be
documented in the driver I don't think we'll receive it anyway).
> One alternative would be to define only SMIA/CCS compliant embedded data
> formats.
And use opaque for the others ?
Thanks
j
>
> > +
> > +The size of the embedded buffer is defined as a single line with a pixel width
> > +specified in bytes and obtained by a call to the :c:type:`VIDIOC_SUBDEV_G_FMT`
> > +ioctl on the sensor sub-device. Note that this size is fixed and cannot be
> > +modified with a call to :c:type:`VIDIOC_SUBDEV_S_FMT`.
> > +
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index 7f16cbe46e5c2..99e270bbdd223 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -7926,3 +7926,32 @@ The following table lists the existing metadata formats.
> > both sides of the link and the bus format is a fixed
> > metadata format that is not configurable from userspace.
> > Width and height will be set to 0 for this format.
> > +
> > +.. _v4l2-mbus-sensor-data:
> > +
> > +Sensor Ancillary Metadata Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +This section lists ancillary data generated by a camera sensor and
> > +transmitted over a stream on the camera bus.
> > +
> > +The following table lists the existing sensor ancillary metadata formats:
> > +
> > +.. _v4l2-mbus-pixelcode-sensor-metadata:
> > +
> > +.. tabularcolumns:: |p{8.0cm}|p{1.4cm}|p{7.7cm}|
> > +
> > +.. flat-table:: Sensor ancillary metadata formats
> > + :header-rows: 1
> > + :stub-columns: 0
> > +
> > + * - Identifier
> > + - Code
> > + - Comments
> > + * .. _MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA:
> > +
> > + - MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA
> > + - 0x7002
> > + - Sensor vendor specific ancillary metadata. Some vendors follow a generic
> > + CSI-2/SMIA embedded data format as described in the `CSI-2 specification.
> > + <https://mipi.org/specifications/csi-2>`_
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index f0f6906a879de..83288a00f28e4 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1403,6 +1403,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > case V4L2_META_FMT_UVC: descr = "UVC Payload Header Metadata"; break;
> > case V4L2_META_FMT_D4XX: descr = "Intel D4xx UVC Metadata"; break;
> > case V4L2_META_FMT_VIVID: descr = "Vivid Metadata"; break;
> > + case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
> >
> > default:
> > /* Compressed formats */
> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index 2ce3d891d3447..726557120fc03 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -164,4 +164,7 @@
> > */
> > #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001
> >
> > +/* Sensor ancillary metadata formats - next is 0x7003 */
> > +#define MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA 0x7002
> > +
> > #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 927075fa9099e..d1ef094090a3b 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -767,6 +767,7 @@ struct v4l2_pix_format {
> > #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
> > #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
> > #define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
> > +#define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
> >
> > /* priv field value to indicates that subsequent fields are valid. */
> > #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
>
> --
> Kind regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface
2020-12-03 16:30 ` Jacopo Mondi
@ 2020-12-03 17:12 ` Dave Stevenson
0 siblings, 0 replies; 31+ messages in thread
From: Dave Stevenson @ 2020-12-03 17:12 UTC (permalink / raw)
To: Jacopo Mondi
Cc: Sakari Ailus, Linux Media Mailing List, Naushir Patuck,
Laurent Pinchart, cc: Kieran Bingham, Niklas Söderlund,
Dafna Hirschfeld, Hans Verkuil, Nicolas Saenz Julienne,
mchehab+huawei
Hi Jacopo and Sakari
On Thu, 3 Dec 2020 at 16:30, Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Hi Sakari,
>
> On Wed, Dec 02, 2020 at 11:53:20PM +0200, Sakari Ailus wrote:
> > Hi Jacopo,
> >
> > On Tue, Nov 10, 2020 at 06:40:34PM +0100, Jacopo Mondi wrote:
> > > From: Naushir Patuck <naush@raspberrypi.com>
> > >
> > > Add a driver for the Unicam camera receiver block on BCM283x processors.
> > > Compared to the bcm2835-camera driver present in staging, this driver
> > > handles the Unicam block only (CSI-2 receiver), and doesn't depend on
> > > the VC4 firmware running on the VPU.
> > >
> > > The commit is made up of a series of changes cherry-picked from the
> > > rpi-5.9.y branch of https://github.com/raspberrypi/linux/ at revision
> > > commit be371a8ffc012
> > > ("media: bcm2835-unicam: change minimum number of vb2_queue buffers to 1")
> > > with additional enhancements, forward-ported to the mainline kernel.
> > >
> > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > ---
> > > MAINTAINERS | 7 +
> > > drivers/staging/media/Kconfig | 2 +
> > > drivers/staging/media/Makefile | 1 +
> > > drivers/staging/media/bcm2835-unicam/Kconfig | 21 +
> > > drivers/staging/media/bcm2835-unicam/Makefile | 3 +
> > > .../media/bcm2835-unicam/bcm2835-unicam.c | 2750 +++++++++++++++++
> > > .../media/bcm2835-unicam/vc4-regs-unicam.h | 253 ++
> > > 7 files changed, 3037 insertions(+)
> > > create mode 100644 drivers/staging/media/bcm2835-unicam/Kconfig
> > > create mode 100644 drivers/staging/media/bcm2835-unicam/Makefile
> > > create mode 100644 drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> > > create mode 100644 drivers/staging/media/bcm2835-unicam/vc4-regs-unicam.h
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > index 811db1d3ca33b..69d55ed67e1cf 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -3411,6 +3411,13 @@ N: bcm113*
> > > N: bcm216*
> > > N: kona
> > >
> > > +BROADCOM BCM2835 CAMERA DRIVER
> > > +M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
> > > +L: linux-media@vger.kernel.org
> > > +S: Maintained
> > > +F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > +F: drivers/staging/media/bcm2835/
> > > +
> > > BROADCOM BCM47XX MIPS ARCHITECTURE
> > > M: Hauke Mehrtens <hauke@hauke-m.de>
> > > M: Rafał Miłecki <zajec5@gmail.com>
> > > diff --git a/drivers/staging/media/Kconfig b/drivers/staging/media/Kconfig
> > > index 747c6cf1d795e..b734e882f63c9 100644
> > > --- a/drivers/staging/media/Kconfig
> > > +++ b/drivers/staging/media/Kconfig
> > > @@ -46,4 +46,6 @@ source "drivers/staging/media/ipu3/Kconfig"
> > >
> > > source "drivers/staging/media/rkisp1/Kconfig"
> > >
> > > +source "drivers/staging/media/bcm2835-unicam/Kconfig"
> > > +
> > > endif
> > > diff --git a/drivers/staging/media/Makefile b/drivers/staging/media/Makefile
> > > index b59571826ba69..93d819ef243f1 100644
> > > --- a/drivers/staging/media/Makefile
> > > +++ b/drivers/staging/media/Makefile
> > > @@ -1,6 +1,7 @@
> > > # SPDX-License-Identifier: GPL-2.0
> > > obj-$(CONFIG_VIDEO_ALLEGRO_DVT) += allegro-dvt/
> > > obj-$(CONFIG_INTEL_ATOMISP) += atomisp/
> > > +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam/
> > > obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx/
> > > obj-$(CONFIG_VIDEO_MESON_VDEC) += meson/vdec/
> > > obj-$(CONFIG_VIDEO_OMAP4) += omap4iss/
> > > diff --git a/drivers/staging/media/bcm2835-unicam/Kconfig b/drivers/staging/media/bcm2835-unicam/Kconfig
> > > new file mode 100644
> > > index 0000000000000..bd13701996509
> > > --- /dev/null
> > > +++ b/drivers/staging/media/bcm2835-unicam/Kconfig
> > > @@ -0,0 +1,21 @@
> > > +# Broadcom VideoCore4 V4L2 camera support
> > > +
> > > +config VIDEO_BCM2835_UNICAM
> > > + tristate "Broadcom BCM283x/BCM271x Unicam video capture driver"
> > > + depends on VIDEO_V4L2
> > > + depends on ARCH_BCM2835 || COMPILE_TEST
> > > + select VIDEO_V4L2_SUBDEV_API
> > > + select MEDIA_CONTROLLER
> > > + select VIDEOBUF2_DMA_CONTIG
> > > + select V4L2_FWNODE
> > > + help
> > > + Say Y here to enable support for the BCM283x/BCM271x CSI-2 receiver.
> > > + This is a V4L2 driver that controls the CSI-2 receiver directly,
> > > + independently from the VC4 firmware.
> > > + This driver is mutually exclusive with the use of bcm2835-camera. The
> > > + firmware will disable all access to the peripheral from within the
> > > + firmware if it finds a DT node using it, and bcm2835-camera will
> > > + therefore fail to probe.
> > > +
> > > + To compile this driver as a module, choose M here. The module will be
> > > + called bcm2835-unicam.
> > > diff --git a/drivers/staging/media/bcm2835-unicam/Makefile b/drivers/staging/media/bcm2835-unicam/Makefile
> > > new file mode 100644
> > > index 0000000000000..a98aba03598ab
> > > --- /dev/null
> > > +++ b/drivers/staging/media/bcm2835-unicam/Makefile
> > > @@ -0,0 +1,3 @@
> > > +# Makefile for BCM2835 Unicam driver
> > > +
> > > +obj-$(CONFIG_VIDEO_BCM2835_UNICAM) += bcm2835-unicam.o
> > > diff --git a/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> > > new file mode 100644
> > > index 0000000000000..3e365baa8197f
> > > --- /dev/null
> > > +++ b/drivers/staging/media/bcm2835-unicam/bcm2835-unicam.c
> > > @@ -0,0 +1,2750 @@
> > > +// SPDX-License-Identifier: GPL-2.0-only
> > > +/*
> > > + * BCM283x / BCM271x Unicam Capture Driver
> > > + *
> > > + * Copyright (C) 2017-2020 - Raspberry Pi (Trading) Ltd.
> > > + *
> > > + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > + *
> > > + * Based on TI am437x driver by
> > > + * Benoit Parrot <bparrot@ti.com>
> > > + * Lad, Prabhakar <prabhakar.csengg@gmail.com>
> > > + *
> > > + * and TI CAL camera interface driver by
> > > + * Benoit Parrot <bparrot@ti.com>
> > > + *
> > > + *
> > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > + * and bcm2835-camera (currently in staging).
> > > + *
> > > + * This driver directly controls the Unicam peripheral - there is no
> > > + * involvement with the VideoCore firmware. Unicam receives CSI-2 or
> > > + * CCP2 data and writes it into SDRAM.
> > > + * The only potential processing options are to repack Bayer data into an
> > > + * alternate format, and applying windowing.
> > > + * The repacking does not shift the data, so can repack V4L2_PIX_FMT_Sxxxx10P
> > > + * to V4L2_PIX_FMT_Sxxxx10, or V4L2_PIX_FMT_Sxxxx12P to V4L2_PIX_FMT_Sxxxx12,
> > > + * but not generically up to V4L2_PIX_FMT_Sxxxx16. The driver will add both
> > > + * formats where the relevant formats are defined, and will automatically
> > > + * configure the repacking as required.
> > > + * Support for windowing may be added later.
> > > + *
> > > + * It should be possible to connect this driver to any sensor with a
> > > + * suitable output interface and V4L2 subdevice driver.
> > > + *
> > > + * bcm2835-camera uses the VideoCore firmware to control the sensor,
> > > + * Unicam, ISP, and all tuner control loops. Fully processed frames are
> > > + * delivered to the driver by the firmware. It only has sensor drivers
> > > + * for Omnivision OV5647, and Sony IMX219 sensors.
> > > + *
> > > + * The two drivers are mutually exclusive for the same Unicam instance.
> > > + * The VideoCore firmware checks the device tree configuration during boot.
> > > + * If it finds device tree nodes called csi0 or csi1 it will block the
> > > + * firmware from accessing the peripheral, and bcm2835-camera will
> > > + * not be able to stream data.
> > > + */
> > > +
> > > +#include <linux/clk.h>
> > > +#include <linux/delay.h>
> > > +#include <linux/device.h>
> > > +#include <linux/dma-mapping.h>
> > > +#include <linux/err.h>
> > > +#include <linux/init.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/io.h>
> > > +#include <linux/module.h>
> > > +#include <linux/of_device.h>
> > > +#include <linux/of_graph.h>
> > > +#include <linux/pinctrl/consumer.h>
> > > +#include <linux/platform_device.h>
> > > +#include <linux/pm_runtime.h>
> > > +#include <linux/slab.h>
> > > +#include <linux/uaccess.h>
> > > +#include <linux/videodev2.h>
> > > +
> > > +#include <media/v4l2-common.h>
> > > +#include <media/v4l2-ctrls.h>
> > > +#include <media/v4l2-dev.h>
> > > +#include <media/v4l2-device.h>
> > > +#include <media/v4l2-dv-timings.h>
> > > +#include <media/v4l2-event.h>
> > > +#include <media/v4l2-ioctl.h>
> > > +#include <media/v4l2-fwnode.h>
> > > +#include <media/videobuf2-dma-contig.h>
> > > +
> > > +#include "vc4-regs-unicam.h"
> > > +
> > > +#define UNICAM_MODULE_NAME "unicam"
> > > +#define UNICAM_VERSION "0.1.0"
> > > +
> > > +/*
> > > + * Unicam must request a minimum of 250Mhz from the VPU clock.
> > > + * Otherwise the input FIFOs overrun and cause image corruption.
> > > + */
> > > +#define MIN_VPU_CLOCK_RATE (250 * 1000 * 1000)
> > > +/*
> > > + * To protect against a dodgy sensor driver never returning an error from
> > > + * enum_mbus_code, set a maximum index value to be used.
> > > + */
> > > +#define MAX_ENUM_MBUS_CODE 128
> >
> > Please remove, and instead fix the buggy sensor drivers.
> >
>
> Dave has use cases that require this iirc.
> I guess this could be made an out-of-tree on-top patch ?
I responded to exactly this question on the last revision
https://patchwork.linuxtv.org/project/linux-media/patch/20200504092611.9798-5-laurent.pinchart@ideasonboard.com/#121496
Personally I don't like getting stuck in loops within the kernel and
prefer defensive programming, but accept that others think
differently.
Dave
> > > +
> > > +/*
> > > + * Stride is a 16 bit register, but also has to be a multiple of 32.
> > > + */
> > > +#define BPL_ALIGNMENT 32
> > > +#define MAX_BYTESPERLINE ((1 << 16) - BPL_ALIGNMENT)
> > > +/*
> > > + * Max width is therefore determined by the max stride divided by
> > > + * the number of bits per pixel. Take 32bpp as a
> > > + * worst case.
> > > + * No imposed limit on the height, so adopt a square image for want
> > > + * of anything better.
> > > + */
> > > +#define MAX_WIDTH (MAX_BYTESPERLINE / 4)
> > > +#define MAX_HEIGHT MAX_WIDTH
> > > +/* Define a nominal minimum image size */
> > > +#define MIN_WIDTH 16
> > > +#define MIN_HEIGHT 16
> > > +
> > > +/*
> > > + * Size of the dummy buffer. Can be any size really, but the DMA
> > > + * allocation works in units of page sizes.
> > > + */
> > > +#define DUMMY_BUF_SIZE (PAGE_SIZE)
> > > +
> > > +/* Enumeration of the video device node identifiers. */
> > > +enum unicam_node_ids {
> > > + IMAGE_NODE,
> > > + METADATA_NODE,
> > > + MAX_NODES
> > > +};
> > > +
> > > +/*
> > > + * Associate video device nodes with the subdevice's source pad ids.
> > > + * The image video device is connected to the subdevice pad #0;
> > > + * The metadata video device is connected to the subdevice pad #1;
> > > + */
> > > +unsigned int subdev_pads[2] = {
> > > + [IMAGE_NODE] = 0,
> > > + [METADATA_NODE] = 1,
> > > +};
> >
> > I recognise this and the above enum are used for different purpose, but
> > this is also entirely useless. At the very least make it static.
> >
>
> It's a small syntactic sugar that's makes identifying pads a little
> more explicit.
>
> I'll make it static const
>
> > > +
> > > +/*
> > > + * struct unicam_fmt - Unicam media bus format information
> > > + * @pixelformat: V4L2 pixel format FCC identifier. 0 if n/a.
> > > + * @repacked_fourcc: V4L2 pixel format FCC identifier if the data is expanded
> > > + * out to 16bpp. 0 if n/a.
> > > + * @code: V4L2 media bus format code.
> > > + * @depth: Bits per pixel as delivered from the source.
> > > + * @csi_dt: CSI data type.
> > > + * @check_variants: Flag to denote that there are multiple mediabus formats
> > > + * still in the list that could match this V4L2 format.
> > > + */
> > > +struct unicam_fmt {
> > > + u32 fourcc;
> > > + u32 repacked_fourcc;
> > > + u32 code;
> > > + u8 depth;
> > > + u8 csi_dt;
> > > + u8 check_variants;
> > > +};
> > > +
> > > +static const struct unicam_fmt formats[] = {
> > > + /* YUV Formats */
> > > + {
> > > + .fourcc = V4L2_PIX_FMT_YUYV,
> > > + .code = MEDIA_BUS_FMT_YUYV8_2X8,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + .check_variants = 1,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_UYVY,
> > > + .code = MEDIA_BUS_FMT_UYVY8_2X8,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + .check_variants = 1,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_YVYU,
> > > + .code = MEDIA_BUS_FMT_YVYU8_2X8,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + .check_variants = 1,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_VYUY,
> > > + .code = MEDIA_BUS_FMT_VYUY8_2X8,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + .check_variants = 1,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_YUYV,
> > > + .code = MEDIA_BUS_FMT_YUYV8_1X16,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_UYVY,
> > > + .code = MEDIA_BUS_FMT_UYVY8_1X16,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_YVYU,
> > > + .code = MEDIA_BUS_FMT_YVYU8_1X16,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_VYUY,
> > > + .code = MEDIA_BUS_FMT_VYUY8_1X16,
> > > + .depth = 16,
> > > + .csi_dt = 0x1e,
> > > + }, {
> > > + /* RGB Formats */
> > > + .fourcc = V4L2_PIX_FMT_RGB565, /* gggbbbbb rrrrrggg */
> > > + .code = MEDIA_BUS_FMT_RGB565_2X8_LE,
> > > + .depth = 16,
> > > + .csi_dt = 0x22,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_RGB565X, /* rrrrrggg gggbbbbb */
> > > + .code = MEDIA_BUS_FMT_RGB565_2X8_BE,
> > > + .depth = 16,
> > > + .csi_dt = 0x22
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_RGB555, /* gggbbbbb arrrrrgg */
> > > + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE,
> > > + .depth = 16,
> > > + .csi_dt = 0x21,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_RGB555X, /* arrrrrgg gggbbbbb */
> > > + .code = MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE,
> > > + .depth = 16,
> > > + .csi_dt = 0x21,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_RGB24, /* rgb */
> > > + .code = MEDIA_BUS_FMT_RGB888_1X24,
> > > + .depth = 24,
> > > + .csi_dt = 0x24,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_BGR24, /* bgr */
> > > + .code = MEDIA_BUS_FMT_BGR888_1X24,
> > > + .depth = 24,
> > > + .csi_dt = 0x24,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_RGB32, /* argb */
> > > + .code = MEDIA_BUS_FMT_ARGB8888_1X32,
> > > + .depth = 32,
> > > + .csi_dt = 0x0,
> > > + }, {
> > > + /* Bayer Formats */
> > > + .fourcc = V4L2_PIX_FMT_SBGGR8,
> > > + .code = MEDIA_BUS_FMT_SBGGR8_1X8,
> > > + .depth = 8,
> > > + .csi_dt = 0x2a,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGBRG8,
> > > + .code = MEDIA_BUS_FMT_SGBRG8_1X8,
> > > + .depth = 8,
> > > + .csi_dt = 0x2a,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGRBG8,
> > > + .code = MEDIA_BUS_FMT_SGRBG8_1X8,
> > > + .depth = 8,
> > > + .csi_dt = 0x2a,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SRGGB8,
> > > + .code = MEDIA_BUS_FMT_SRGGB8_1X8,
> > > + .depth = 8,
> > > + .csi_dt = 0x2a,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SBGGR10P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SBGGR10,
> > > + .code = MEDIA_BUS_FMT_SBGGR10_1X10,
> > > + .depth = 10,
> > > + .csi_dt = 0x2b,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGBRG10P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SGBRG10,
> > > + .code = MEDIA_BUS_FMT_SGBRG10_1X10,
> > > + .depth = 10,
> > > + .csi_dt = 0x2b,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGRBG10P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SGRBG10,
> > > + .code = MEDIA_BUS_FMT_SGRBG10_1X10,
> > > + .depth = 10,
> > > + .csi_dt = 0x2b,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SRGGB10P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SRGGB10,
> > > + .code = MEDIA_BUS_FMT_SRGGB10_1X10,
> > > + .depth = 10,
> > > + .csi_dt = 0x2b,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SBGGR12P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SBGGR12,
> > > + .code = MEDIA_BUS_FMT_SBGGR12_1X12,
> > > + .depth = 12,
> > > + .csi_dt = 0x2c,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGBRG12P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SGBRG12,
> > > + .code = MEDIA_BUS_FMT_SGBRG12_1X12,
> > > + .depth = 12,
> > > + .csi_dt = 0x2c,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGRBG12P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SGRBG12,
> > > + .code = MEDIA_BUS_FMT_SGRBG12_1X12,
> > > + .depth = 12,
> > > + .csi_dt = 0x2c,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SRGGB12P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_SRGGB12,
> > > + .code = MEDIA_BUS_FMT_SRGGB12_1X12,
> > > + .depth = 12,
> > > + .csi_dt = 0x2c,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SBGGR14P,
> > > + .code = MEDIA_BUS_FMT_SBGGR14_1X14,
> > > + .depth = 14,
> > > + .csi_dt = 0x2d,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGBRG14P,
> > > + .code = MEDIA_BUS_FMT_SGBRG14_1X14,
> > > + .depth = 14,
> > > + .csi_dt = 0x2d,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SGRBG14P,
> > > + .code = MEDIA_BUS_FMT_SGRBG14_1X14,
> > > + .depth = 14,
> > > + .csi_dt = 0x2d,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_SRGGB14P,
> > > + .code = MEDIA_BUS_FMT_SRGGB14_1X14,
> > > + .depth = 14,
> > > + .csi_dt = 0x2d,
> > > + }, {
> > > + /*
> > > + * 16 bit Bayer formats could be supported, but there is no CSI2
> > > + * data_type defined for raw 16, and no sensors that produce it at
> > > + * present.
> > > + */
> > > +
> > > + /* Greyscale formats */
> > > + .fourcc = V4L2_PIX_FMT_GREY,
> > > + .code = MEDIA_BUS_FMT_Y8_1X8,
> > > + .depth = 8,
> > > + .csi_dt = 0x2a,
> > > + }, {
> > > + .fourcc = V4L2_PIX_FMT_Y10P,
> > > + .repacked_fourcc = V4L2_PIX_FMT_Y10,
> > > + .code = MEDIA_BUS_FMT_Y10_1X10,
> > > + .depth = 10,
> > > + .csi_dt = 0x2b,
> > > + }, {
> > > + /* NB There is no packed V4L2 fourcc for this format. */
> > > + .repacked_fourcc = V4L2_PIX_FMT_Y12,
> > > + .code = MEDIA_BUS_FMT_Y12_1X12,
> > > + .depth = 12,
> > > + .csi_dt = 0x2c,
> > > + },
> > > + /* Embedded data format */
> > > + {
> > > + .fourcc = V4L2_META_FMT_SENSOR_DATA,
> > > + .code = MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA,
> > > + .depth = 8,
> > > + }
> > > +};
> > > +
> > > +struct unicam_buffer {
> > > + struct vb2_v4l2_buffer vb;
> > > + struct list_head list;
> > > +};
> > > +
> > > +static inline struct unicam_buffer *to_unicam_buffer(struct vb2_buffer *vb)
> > > +{
> > > + return container_of(vb, struct unicam_buffer, vb.vb2_buf);
> > > +}
> > > +
> > > +struct unicam_node {
> > > + bool registered;
> > > + int open;
> > > + bool streaming;
> > > + unsigned int node_id;
> > > + /* Pointer pointing to current v4l2_buffer */
> > > + struct unicam_buffer *cur_frm;
> > > + /* Pointer pointing to next v4l2_buffer */
> > > + struct unicam_buffer *next_frm;
> > > + /* video capture */
> > > + const struct unicam_fmt *fmt;
> > > + /* Used to store current pixel format */
> > > + struct v4l2_format v_fmt;
> > > + /* Used to store current mbus frame format */
> > > + struct v4l2_mbus_framefmt m_fmt;
> > > + /* Buffer queue used in video-buf */
> > > + struct vb2_queue buffer_queue;
> > > + /* Queue of filled frames */
> > > + struct list_head dma_queue;
> > > + /* IRQ lock for DMA queue */
> > > + spinlock_t dma_queue_lock;
> > > + /* lock used to access this structure */
> > > + struct mutex lock;
> > > + /* Identifies video device for this channel */
> > > + struct video_device video_dev;
> > > + /* Pointer to the parent handle */
> > > + struct unicam_device *dev;
> > > + struct media_pad pad;
> > > + /*
> > > + * Dummy buffer intended to be used by unicam
> > > + * if we have no other queued buffers to swap to.
> > > + */
> > > + void *dummy_buf_cpu_addr;
> > > + dma_addr_t dummy_buf_dma_addr;
> > > +};
> > > +
> > > +struct unicam_device {
> > > + struct device *dev;
> > > +
> > > + /* V4l2 specific parameters */
> > > + struct v4l2_async_subdev asd;
> > > +
> > > + /* peripheral base address */
> > > + void __iomem *base;
> > > + /* clock gating base address */
> > > + void __iomem *clk_gate_base;
> > > + /* lp clock handle */
> > > + struct clk *clock;
> > > + /* vpu clock handle */
> > > + struct clk *vpu_clock;
> > > + /* V4l2 device */
> > > + struct v4l2_device v4l2_dev;
> > > + struct media_device mdev;
> > > +
> > > + /* subdevice async Notifier */
> > > + struct v4l2_async_notifier notifier;
> > > + unsigned int sequence;
> > > +
> > > + /* ptr to sub device */
> > > + struct v4l2_subdev *sensor;
> > > + /* Pad config for the sensor */
> > > + struct v4l2_subdev_pad_config *sensor_config;
> > > +
> > > + enum v4l2_mbus_type bus_type;
> > > + /*
> > > + * Stores bus.mipi_csi2.flags for CSI2 sensors, or
> > > + * bus.mipi_csi1.strobe for CCP2.
> > > + */
> > > + unsigned int bus_flags;
> > > + unsigned int max_data_lanes;
> > > + unsigned int active_data_lanes;
> > > +
> > > + struct unicam_node node[MAX_NODES];
> > > + struct v4l2_ctrl_handler ctrl_handler;
> > > +};
> > > +
> > > +static inline struct unicam_device *
> > > +to_unicam_device(struct v4l2_device *v4l2_dev)
> > > +{
> > > + return container_of(v4l2_dev, struct unicam_device, v4l2_dev);
> > > +}
> > > +
> > > +static char *print_fourcc(u32 fmt)
> > > +{
> > > + static char code[16];
> > > +
> > > + code[0] = (unsigned char)(fmt & 0xff);
> > > + code[1] = (unsigned char)((fmt >> 8) & 0xff);
> > > + code[2] = (unsigned char)((fmt >> 16) & 0xff);
> > > + code[3] = (unsigned char)((fmt >> 24) & 0xff);
> > > + snprintf(&code[4], 12, "=0x%8x", fmt);
> >
> > Ugh. Hopefully we'll have %p4cc soon. It's not yet merged at least. :-\
> >
> > The rest looks fairly normal, apart from the obvious lack of MC support
> > from a driver that supports embedded data (also a TODO item).
>
> Isn't it recorded in the TODO I've added at the end of the series ?
>
> Thanks
> j
>
> >
> > --
> > Kind regards,
> >
> > Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-03 16:44 ` Jacopo Mondi
@ 2020-12-03 17:25 ` Sakari Ailus
2020-12-03 17:34 ` Chen-Yu Tsai
2020-12-03 17:37 ` Jacopo Mondi
0 siblings, 2 replies; 31+ messages in thread
From: Sakari Ailus @ 2020-12-03 17:25 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Thu, Dec 03, 2020 at 05:44:17PM +0100, Jacopo Mondi wrote:
> Hi Sakari,
>
> On Wed, Dec 02, 2020 at 11:20:31PM +0200, Sakari Ailus wrote:
> > Hi Jacopo,
> >
> > On Tue, Nov 10, 2020 at 06:40:33PM +0100, Jacopo Mondi wrote:
> > > From: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > >
> > > Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
> > > Unicam) on BCM283x SoCs.
> > >
> > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > ---
> > > .../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
> > > 1 file changed, 155 insertions(+)
> > > create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > >
> > > diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > new file mode 100644
> > > index 0000000000000..6ffc900e8ae8f
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > @@ -0,0 +1,155 @@
> > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > +# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
> > > +%YAML 1.2
> > > +---
> > > +$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
> > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > +
> > > +title: Broadcom BCM283x Camera Interface (Unicam)
> > > +
> > > +maintainers:
> > > + - Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > + - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
> > > +
> > > +description:
> > > + The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
> > > + data from image sensors or similar devices.
> > > +
> > > + The main platform using this SoC is the Raspberry Pi family of boards. On the
> > > + Pi the VideoCore firmware can also control this hardware block, and driving
> > > + it from two different processors will cause issues. To avoid this, the
> > > + firmware checks the device tree configuration during boot. If it finds device
> > > + tree nodes whose name starts with "csi" then it will stop the firmware
> > > + accessing the block, and it can then safely be used via the device tree
> > > + binding.
> > > +
> > > +properties:
> > > + compatible:
> > > + const: brcm,bcm2835-unicam
> >
> > The compatible string doesn't quite match with the title. Which SoCs are
> > supported?
> >
>
> Afaict bcm2835 and bcm2711. There might be more I'm not aware of.
> There should be no distinction between then, but Dave knows more.
>
> Would a per-soc compatible work better or is a single brcm,unicam
> good enough to start with and we can add per-soc compatibles if
> there's any per-SoC difference that will need to be handled in future ?
If there are any differences between them, that's the way to take them into
account (assuming they can't told apart at runtime). In that case you could
just do the check in the driver. I think some Allwinner CSI-2 / parallel
receivers use that.
>
> > > +
> > > + reg:
> > > + items:
> > > + - description: Main registers block
> > > + - description: Clock registers block
> > > +
> > > + interrupts:
> > > + maxItems: 1
> > > +
> > > + clocks:
> > > + minItems: 1
> > > + maxItems: 2
> > > +
> > > + clock-names:
> > > + minItems: 1
> > > + items:
> > > + - const: lp
> > > + - const: core
> > > +
> > > + power-domains:
> > > + maxItems: 1
> > > +
> > > + brcm,num-data-lanes:
> > > + description:
> > > + The number of data lanes supported by this Unicam instance. It may be
> > > + larger than the number of data lanes routed on the board, as described by
> > > + the data-lanes property of the endpoint.
> > > + allOf:
> > > + - $ref: "/schemas/types.yaml#/definitions/uint32"
> > > + - enum: [1, 2, 4]
> >
> > Do you need this information besides how many lanes are connected?
> >
> > Does the number of lanes change even within the same model of SoC? Could
> > you use the compatible string to differentiate between them?
> >
>
> I had a similar question:
> https://patchwork.linuxtv.org/project/linux-media/patch/20200504092611.9798-5-laurent.pinchart@ideasonboard.com/#121079
>
> I think the discussion ended with the fact that not all the available
> lanes in the IP might be routed to the connector, and that depends on
> the board.
Right, and this is what this property describes.
Going back to my earlier question, doesn't the number of lanes available in
the SoC depend on the SoC where this block is integrated, which could be
indicated by a compatible string?
I don't really have a strong opinion on either, but knowing the exact piece
of hardware might be helpful for other reasons, too (I'm not commenting on
Unicam in particular here).
>
> > > +
> > > + port:
> > > + type: object
> > > + description:
> > > + Input port node, as described in video-interfaces.txt.
> > > +
> > > + properties:
> > > + endpoint:
> > > + type: object
> > > +
> > > + properties:
> > > + clock-lanes:
> > > + items:
> > > + - const: 0
> >
> > Please drop. There's nothing to tell software here.
> >
>
> Ack
>
> > > +
> > > + data-lanes:
> > > + description:
> > > + Lane reordering is not supported, items shall be in order,
> > > + starting at 1.
> > > + allOf:
> > > + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> > > + - maxItems: 1
> > > + items:
> > > + minItems: 1
> > > + maxItems: 4
> > > + items:
> > > + - const: 1
> > > + - const: 2
> > > + - const: 3
> > > + - const: 4
> >
> > No need to specify items.
> >
>
> Actually here we might need to support [1, 2] and [1, 2, 3, 4].
> I'll rebase on the converted video-interfaces to see it that helps
If you do not support lane reordering, then just the number of values
matters. Ideally, indeed, it'd be monotonically incrementing integers from
1.
See e.g. Documentation/devicetree/bindings/media/i2c/mipi-ccs.yaml for an
example. It got merged earlier today.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-03 17:25 ` Sakari Ailus
@ 2020-12-03 17:34 ` Chen-Yu Tsai
2020-12-03 17:37 ` Jacopo Mondi
1 sibling, 0 replies; 31+ messages in thread
From: Chen-Yu Tsai @ 2020-12-03 17:34 UTC (permalink / raw)
To: Sakari Ailus, Jacopo Mondi
Cc: Linux Media Mailing List, naush, Dave Stevenson,
Laurent Pinchart, kieran.bingham, niklas.soderlund,
dafna.hirschfeld, Hans Verkuil, Nicolas Saenz Julienne,
mchehab+huawei
Hi,
On Fri, Dec 4, 2020 at 1:27 AM Sakari Ailus <sakari.ailus@iki.fi> wrote:
>
> Hi Jacopo,
>
> On Thu, Dec 03, 2020 at 05:44:17PM +0100, Jacopo Mondi wrote:
> > Hi Sakari,
> >
> > On Wed, Dec 02, 2020 at 11:20:31PM +0200, Sakari Ailus wrote:
> > > Hi Jacopo,
> > >
> > > On Tue, Nov 10, 2020 at 06:40:33PM +0100, Jacopo Mondi wrote:
> > > > From: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > >
> > > > Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
> > > > Unicam) on BCM283x SoCs.
> > > >
> > > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > > ---
> > > > .../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
> > > > 1 file changed, 155 insertions(+)
> > > > create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > >
> > > > diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > new file mode 100644
> > > > index 0000000000000..6ffc900e8ae8f
> > > > --- /dev/null
> > > > +++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > @@ -0,0 +1,155 @@
> > > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > > +# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
> > > > +%YAML 1.2
> > > > +---
> > > > +$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
> > > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > > +
> > > > +title: Broadcom BCM283x Camera Interface (Unicam)
> > > > +
> > > > +maintainers:
> > > > + - Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > + - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
> > > > +
> > > > +description:
> > > > + The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
> > > > + data from image sensors or similar devices.
> > > > +
> > > > + The main platform using this SoC is the Raspberry Pi family of boards. On the
> > > > + Pi the VideoCore firmware can also control this hardware block, and driving
> > > > + it from two different processors will cause issues. To avoid this, the
> > > > + firmware checks the device tree configuration during boot. If it finds device
> > > > + tree nodes whose name starts with "csi" then it will stop the firmware
> > > > + accessing the block, and it can then safely be used via the device tree
> > > > + binding.
> > > > +
> > > > +properties:
> > > > + compatible:
> > > > + const: brcm,bcm2835-unicam
> > >
> > > The compatible string doesn't quite match with the title. Which SoCs are
> > > supported?
> > >
> >
> > Afaict bcm2835 and bcm2711. There might be more I'm not aware of.
> > There should be no distinction between then, but Dave knows more.
> >
> > Would a per-soc compatible work better or is a single brcm,unicam
> > good enough to start with and we can add per-soc compatibles if
> > there's any per-SoC difference that will need to be handled in future ?
>
> If there are any differences between them, that's the way to take them into
> account (assuming they can't told apart at runtime). In that case you could
> just do the check in the driver. I think some Allwinner CSI-2 / parallel
> receivers use that.
What we do is we use the first SoC that introduced the hardware block
in the compatible string. For subsequent SoCs, if there is any doubt
that the hardware might be different, then a SoC-specific compatible
string is added in front. If the hardware ends up being the same, then
the new compatible doesn't get used. If the hardware ends up different,
then the new compatible string gets added to the driver to deal with
the difference.
If the hardware ends up being *incompatible*, then you have to remove
the old compatible string, and only keep the SoC-specific one.
So in your case you would have:
brcm,bcm2835-unicam
and
brcm,bcm2711-unicam + brcm,bcm2835-unicam
assuming the hardware is the same or at least backward-compatible.
If you know a priori that the hardware is 100% the same, then you
can choose whatever compatible you like, subject to approval from
the DT binding maintainers I suppose.
Regards
ChenYu
> > > > +
> > > > + reg:
> > > > + items:
> > > > + - description: Main registers block
> > > > + - description: Clock registers block
> > > > +
> > > > + interrupts:
> > > > + maxItems: 1
> > > > +
> > > > + clocks:
> > > > + minItems: 1
> > > > + maxItems: 2
> > > > +
> > > > + clock-names:
> > > > + minItems: 1
> > > > + items:
> > > > + - const: lp
> > > > + - const: core
> > > > +
> > > > + power-domains:
> > > > + maxItems: 1
> > > > +
> > > > + brcm,num-data-lanes:
> > > > + description:
> > > > + The number of data lanes supported by this Unicam instance. It may be
> > > > + larger than the number of data lanes routed on the board, as described by
> > > > + the data-lanes property of the endpoint.
> > > > + allOf:
> > > > + - $ref: "/schemas/types.yaml#/definitions/uint32"
> > > > + - enum: [1, 2, 4]
> > >
> > > Do you need this information besides how many lanes are connected?
> > >
> > > Does the number of lanes change even within the same model of SoC? Could
> > > you use the compatible string to differentiate between them?
> > >
> >
> > I had a similar question:
> > https://patchwork.linuxtv.org/project/linux-media/patch/20200504092611.9798-5-laurent.pinchart@ideasonboard.com/#121079
> >
> > I think the discussion ended with the fact that not all the available
> > lanes in the IP might be routed to the connector, and that depends on
> > the board.
>
> Right, and this is what this property describes.
>
> Going back to my earlier question, doesn't the number of lanes available in
> the SoC depend on the SoC where this block is integrated, which could be
> indicated by a compatible string?
>
> I don't really have a strong opinion on either, but knowing the exact piece
> of hardware might be helpful for other reasons, too (I'm not commenting on
> Unicam in particular here).
>
> >
> > > > +
> > > > + port:
> > > > + type: object
> > > > + description:
> > > > + Input port node, as described in video-interfaces.txt.
> > > > +
> > > > + properties:
> > > > + endpoint:
> > > > + type: object
> > > > +
> > > > + properties:
> > > > + clock-lanes:
> > > > + items:
> > > > + - const: 0
> > >
> > > Please drop. There's nothing to tell software here.
> > >
> >
> > Ack
> >
> > > > +
> > > > + data-lanes:
> > > > + description:
> > > > + Lane reordering is not supported, items shall be in order,
> > > > + starting at 1.
> > > > + allOf:
> > > > + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> > > > + - maxItems: 1
> > > > + items:
> > > > + minItems: 1
> > > > + maxItems: 4
> > > > + items:
> > > > + - const: 1
> > > > + - const: 2
> > > > + - const: 3
> > > > + - const: 4
> > >
> > > No need to specify items.
> > >
> >
> > Actually here we might need to support [1, 2] and [1, 2, 3, 4].
> > I'll rebase on the converted video-interfaces to see it that helps
>
> If you do not support lane reordering, then just the number of values
> matters. Ideally, indeed, it'd be monotonically incrementing integers from
> 1.
>
> See e.g. Documentation/devicetree/bindings/media/i2c/mipi-ccs.yaml for an
> example. It got merged earlier today.
>
> --
> Kind regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-03 17:25 ` Sakari Ailus
2020-12-03 17:34 ` Chen-Yu Tsai
@ 2020-12-03 17:37 ` Jacopo Mondi
2020-12-03 17:41 ` Sakari Ailus
1 sibling, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-03 17:37 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Thu, Dec 03, 2020 at 07:25:19PM +0200, Sakari Ailus wrote:
> Hi Jacopo,
>
> On Thu, Dec 03, 2020 at 05:44:17PM +0100, Jacopo Mondi wrote:
> > Hi Sakari,
> >
> > On Wed, Dec 02, 2020 at 11:20:31PM +0200, Sakari Ailus wrote:
> > > Hi Jacopo,
> > >
> > > On Tue, Nov 10, 2020 at 06:40:33PM +0100, Jacopo Mondi wrote:
> > > > From: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > >
> > > > Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
> > > > Unicam) on BCM283x SoCs.
> > > >
> > > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > > ---
> > > > .../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
> > > > 1 file changed, 155 insertions(+)
> > > > create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > >
> > > > diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > new file mode 100644
> > > > index 0000000000000..6ffc900e8ae8f
> > > > --- /dev/null
> > > > +++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > @@ -0,0 +1,155 @@
> > > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > > +# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
> > > > +%YAML 1.2
> > > > +---
> > > > +$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
> > > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > > +
> > > > +title: Broadcom BCM283x Camera Interface (Unicam)
> > > > +
> > > > +maintainers:
> > > > + - Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > + - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
> > > > +
> > > > +description:
> > > > + The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
> > > > + data from image sensors or similar devices.
> > > > +
> > > > + The main platform using this SoC is the Raspberry Pi family of boards. On the
> > > > + Pi the VideoCore firmware can also control this hardware block, and driving
> > > > + it from two different processors will cause issues. To avoid this, the
> > > > + firmware checks the device tree configuration during boot. If it finds device
> > > > + tree nodes whose name starts with "csi" then it will stop the firmware
> > > > + accessing the block, and it can then safely be used via the device tree
> > > > + binding.
> > > > +
> > > > +properties:
> > > > + compatible:
> > > > + const: brcm,bcm2835-unicam
> > >
> > > The compatible string doesn't quite match with the title. Which SoCs are
> > > supported?
> > >
> >
> > Afaict bcm2835 and bcm2711. There might be more I'm not aware of.
> > There should be no distinction between then, but Dave knows more.
> >
> > Would a per-soc compatible work better or is a single brcm,unicam
> > good enough to start with and we can add per-soc compatibles if
> > there's any per-SoC difference that will need to be handled in future ?
>
> If there are any differences between them, that's the way to take them into
> account (assuming they can't told apart at runtime). In that case you could
> just do the check in the driver. I think some Allwinner CSI-2 / parallel
> receivers use that.
>
There might be difference, I don't know, but as of now the driver does
not need to tell them apart. I agree if we ever find a SoC-specific
different then supporting old DTBS with a single compatible might get
tricky.
I'll defer this to Dave, he knows how many SoC uses this IP and if
they will ever have to be told apart.
> >
> > > > +
> > > > + reg:
> > > > + items:
> > > > + - description: Main registers block
> > > > + - description: Clock registers block
> > > > +
> > > > + interrupts:
> > > > + maxItems: 1
> > > > +
> > > > + clocks:
> > > > + minItems: 1
> > > > + maxItems: 2
> > > > +
> > > > + clock-names:
> > > > + minItems: 1
> > > > + items:
> > > > + - const: lp
> > > > + - const: core
> > > > +
> > > > + power-domains:
> > > > + maxItems: 1
> > > > +
> > > > + brcm,num-data-lanes:
> > > > + description:
> > > > + The number of data lanes supported by this Unicam instance. It may be
> > > > + larger than the number of data lanes routed on the board, as described by
> > > > + the data-lanes property of the endpoint.
> > > > + allOf:
> > > > + - $ref: "/schemas/types.yaml#/definitions/uint32"
> > > > + - enum: [1, 2, 4]
> > >
> > > Do you need this information besides how many lanes are connected?
> > >
> > > Does the number of lanes change even within the same model of SoC? Could
> > > you use the compatible string to differentiate between them?
> > >
> >
> > I had a similar question:
> > https://patchwork.linuxtv.org/project/linux-media/patch/20200504092611.9798-5-laurent.pinchart@ideasonboard.com/#121079
> >
> > I think the discussion ended with the fact that not all the available
> > lanes in the IP might be routed to the connector, and that depends on
> > the board.
>
> Right, and this is what this property describes.
>
Ack to keep it then ?
> Going back to my earlier question, doesn't the number of lanes available in
> the SoC depend on the SoC where this block is integrated, which could be
> indicated by a compatible string?
>
> I don't really have a strong opinion on either, but knowing the exact piece
> of hardware might be helpful for other reasons, too (I'm not commenting on
> Unicam in particular here).
>
> >
> > > > +
> > > > + port:
> > > > + type: object
> > > > + description:
> > > > + Input port node, as described in video-interfaces.txt.
> > > > +
> > > > + properties:
> > > > + endpoint:
> > > > + type: object
> > > > +
> > > > + properties:
> > > > + clock-lanes:
> > > > + items:
> > > > + - const: 0
> > >
> > > Please drop. There's nothing to tell software here.
> > >
> >
> > Ack
> >
> > > > +
> > > > + data-lanes:
> > > > + description:
> > > > + Lane reordering is not supported, items shall be in order,
> > > > + starting at 1.
> > > > + allOf:
> > > > + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> > > > + - maxItems: 1
> > > > + items:
> > > > + minItems: 1
> > > > + maxItems: 4
> > > > + items:
> > > > + - const: 1
> > > > + - const: 2
> > > > + - const: 3
> > > > + - const: 4
> > >
> > > No need to specify items.
> > >
> >
> > Actually here we might need to support [1, 2] and [1, 2, 3, 4].
> > I'll rebase on the converted video-interfaces to see it that helps
>
> If you do not support lane reordering, then just the number of values
> matters. Ideally, indeed, it'd be monotonically incrementing integers from
> 1.
>
> See e.g. Documentation/devicetree/bindings/media/i2c/mipi-ccs.yaml for an
> example. It got merged earlier today.
Ah! maximum: 8. I was about to reply to Rob's video-interfaces
conversion and this might need to be pointed out.
I was also about to ask it it's an issue if the bindings validation
does not catch arrays as: [1, 3] as the constraint of being
monotonically increasing values is expressed in words only. I guess
it's fine, the alternative syntax is awful:
anyOf:
- items:
- const: 1
- const: 2
- items:
- const: 1
- const: 2
- const: 3
- const: 4
>
> --
> Kind regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-03 17:37 ` Jacopo Mondi
@ 2020-12-03 17:41 ` Sakari Ailus
2020-12-03 17:50 ` Jacopo Mondi
0 siblings, 1 reply; 31+ messages in thread
From: Sakari Ailus @ 2020-12-03 17:41 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Jacopo,
On Thu, Dec 03, 2020 at 06:37:33PM +0100, Jacopo Mondi wrote:
> Hi Sakari,
>
> On Thu, Dec 03, 2020 at 07:25:19PM +0200, Sakari Ailus wrote:
> > Hi Jacopo,
> >
> > On Thu, Dec 03, 2020 at 05:44:17PM +0100, Jacopo Mondi wrote:
> > > Hi Sakari,
> > >
> > > On Wed, Dec 02, 2020 at 11:20:31PM +0200, Sakari Ailus wrote:
> > > > Hi Jacopo,
> > > >
> > > > On Tue, Nov 10, 2020 at 06:40:33PM +0100, Jacopo Mondi wrote:
> > > > > From: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > >
> > > > > Document the DT bindings for the CSI2/CCP2 receiver peripheral (known as
> > > > > Unicam) on BCM283x SoCs.
> > > > >
> > > > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > > > ---
> > > > > .../bindings/media/brcm,bcm2835-unicam.yaml | 155 ++++++++++++++++++
> > > > > 1 file changed, 155 insertions(+)
> > > > > create mode 100644 Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > >
> > > > > diff --git a/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > > new file mode 100644
> > > > > index 0000000000000..6ffc900e8ae8f
> > > > > --- /dev/null
> > > > > +++ b/Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
> > > > > @@ -0,0 +1,155 @@
> > > > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > > > +# Copyright (C) 2020 Raspberry Pi (Trading) Ltd.
> > > > > +%YAML 1.2
> > > > > +---
> > > > > +$id: http://devicetree.org/schemas/media/brcm,bcm2835-unicam.yaml#
> > > > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > > > +
> > > > > +title: Broadcom BCM283x Camera Interface (Unicam)
> > > > > +
> > > > > +maintainers:
> > > > > + - Dave Stevenson <dave.stevenson@raspberrypi.com>
> > > > > + - Raspberry Pi kernel list <kernel-list@raspberrypi.com>
> > > > > +
> > > > > +description:
> > > > > + The Unicam block on BCM283x SoCs is the receiver for either CSI-2 or CCP2
> > > > > + data from image sensors or similar devices.
> > > > > +
> > > > > + The main platform using this SoC is the Raspberry Pi family of boards. On the
> > > > > + Pi the VideoCore firmware can also control this hardware block, and driving
> > > > > + it from two different processors will cause issues. To avoid this, the
> > > > > + firmware checks the device tree configuration during boot. If it finds device
> > > > > + tree nodes whose name starts with "csi" then it will stop the firmware
> > > > > + accessing the block, and it can then safely be used via the device tree
> > > > > + binding.
> > > > > +
> > > > > +properties:
> > > > > + compatible:
> > > > > + const: brcm,bcm2835-unicam
> > > >
> > > > The compatible string doesn't quite match with the title. Which SoCs are
> > > > supported?
> > > >
> > >
> > > Afaict bcm2835 and bcm2711. There might be more I'm not aware of.
> > > There should be no distinction between then, but Dave knows more.
> > >
> > > Would a per-soc compatible work better or is a single brcm,unicam
> > > good enough to start with and we can add per-soc compatibles if
> > > there's any per-SoC difference that will need to be handled in future ?
> >
> > If there are any differences between them, that's the way to take them into
> > account (assuming they can't told apart at runtime). In that case you could
> > just do the check in the driver. I think some Allwinner CSI-2 / parallel
> > receivers use that.
> >
>
> There might be difference, I don't know, but as of now the driver does
> not need to tell them apart. I agree if we ever find a SoC-specific
> different then supporting old DTBS with a single compatible might get
> tricky.
>
> I'll defer this to Dave, he knows how many SoC uses this IP and if
> they will ever have to be told apart.
>
> > >
> > > > > +
> > > > > + reg:
> > > > > + items:
> > > > > + - description: Main registers block
> > > > > + - description: Clock registers block
> > > > > +
> > > > > + interrupts:
> > > > > + maxItems: 1
> > > > > +
> > > > > + clocks:
> > > > > + minItems: 1
> > > > > + maxItems: 2
> > > > > +
> > > > > + clock-names:
> > > > > + minItems: 1
> > > > > + items:
> > > > > + - const: lp
> > > > > + - const: core
> > > > > +
> > > > > + power-domains:
> > > > > + maxItems: 1
> > > > > +
> > > > > + brcm,num-data-lanes:
> > > > > + description:
> > > > > + The number of data lanes supported by this Unicam instance. It may be
> > > > > + larger than the number of data lanes routed on the board, as described by
> > > > > + the data-lanes property of the endpoint.
> > > > > + allOf:
> > > > > + - $ref: "/schemas/types.yaml#/definitions/uint32"
> > > > > + - enum: [1, 2, 4]
> > > >
> > > > Do you need this information besides how many lanes are connected?
> > > >
> > > > Does the number of lanes change even within the same model of SoC? Could
> > > > you use the compatible string to differentiate between them?
> > > >
> > >
> > > I had a similar question:
> > > https://patchwork.linuxtv.org/project/linux-media/patch/20200504092611.9798-5-laurent.pinchart@ideasonboard.com/#121079
> > >
> > > I think the discussion ended with the fact that not all the available
> > > lanes in the IP might be routed to the connector, and that depends on
> > > the board.
> >
> > Right, and this is what this property describes.
> >
>
> Ack to keep it then ?
I mean that the data-lanes property describes the board.
>
> > Going back to my earlier question, doesn't the number of lanes available in
> > the SoC depend on the SoC where this block is integrated, which could be
> > indicated by a compatible string?
> >
> > I don't really have a strong opinion on either, but knowing the exact piece
> > of hardware might be helpful for other reasons, too (I'm not commenting on
> > Unicam in particular here).
> >
> > >
> > > > > +
> > > > > + port:
> > > > > + type: object
> > > > > + description:
> > > > > + Input port node, as described in video-interfaces.txt.
> > > > > +
> > > > > + properties:
> > > > > + endpoint:
> > > > > + type: object
> > > > > +
> > > > > + properties:
> > > > > + clock-lanes:
> > > > > + items:
> > > > > + - const: 0
> > > >
> > > > Please drop. There's nothing to tell software here.
> > > >
> > >
> > > Ack
> > >
> > > > > +
> > > > > + data-lanes:
> > > > > + description:
> > > > > + Lane reordering is not supported, items shall be in order,
> > > > > + starting at 1.
> > > > > + allOf:
> > > > > + - $ref: "/schemas/types.yaml#/definitions/uint32-array"
> > > > > + - maxItems: 1
> > > > > + items:
> > > > > + minItems: 1
> > > > > + maxItems: 4
> > > > > + items:
> > > > > + - const: 1
> > > > > + - const: 2
> > > > > + - const: 3
> > > > > + - const: 4
> > > >
> > > > No need to specify items.
> > > >
> > >
> > > Actually here we might need to support [1, 2] and [1, 2, 3, 4].
> > > I'll rebase on the converted video-interfaces to see it that helps
> >
> > If you do not support lane reordering, then just the number of values
> > matters. Ideally, indeed, it'd be monotonically incrementing integers from
> > 1.
> >
> > See e.g. Documentation/devicetree/bindings/media/i2c/mipi-ccs.yaml for an
> > example. It got merged earlier today.
>
> Ah! maximum: 8. I was about to reply to Rob's video-interfaces
> conversion and this might need to be pointed out.
> I was also about to ask it it's an issue if the bindings validation
> does not catch arrays as: [1, 3] as the constraint of being
> monotonically increasing values is expressed in words only. I guess
> it's fine, the alternative syntax is awful:
>
> anyOf:
> - items:
> - const: 1
> - const: 2
> - items:
> - const: 1
> - const: 2
> - const: 3
> - const: 4
The v4l2-fwnode framework doesn't do that, what it does check for, though,
is that the same lane isn't used in more than a single position. After that
it's up to the driver to make use of the configuration provided. If the
driver doesn't support lane reordering, then it just ignores that
information (well, devices where lane allocation to PHYs isn't static need
some more than this).
--
Regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-03 17:41 ` Sakari Ailus
@ 2020-12-03 17:50 ` Jacopo Mondi
2020-12-03 17:50 ` Sakari Ailus
0 siblings, 1 reply; 31+ messages in thread
From: Jacopo Mondi @ 2020-12-03 17:50 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
Hi Sakari,
On Thu, Dec 03, 2020 at 07:41:02PM +0200, Sakari Ailus wrote:
[snip]
> >
> > Ah! maximum: 8. I was about to reply to Rob's video-interfaces
> > conversion and this might need to be pointed out.
> > I was also about to ask it it's an issue if the bindings validation
> > does not catch arrays as: [1, 3] as the constraint of being
> > monotonically increasing values is expressed in words only. I guess
> > it's fine, the alternative syntax is awful:
> >
> > anyOf:
> > - items:
> > - const: 1
> > - const: 2
> > - items:
> > - const: 1
> > - const: 2
> > - const: 3
> > - const: 4
>
> The v4l2-fwnode framework doesn't do that, what it does check for, though,
I meant the dt-validation :)
> is that the same lane isn't used in more than a single position. After that
> it's up to the driver to make use of the configuration provided. If the
> driver doesn't support lane reordering, then it just ignores that
> information (well, devices where lane allocation to PHYs isn't static need
> some more than this).
>
> --
> Regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver
2020-12-03 17:50 ` Jacopo Mondi
@ 2020-12-03 17:50 ` Sakari Ailus
0 siblings, 0 replies; 31+ messages in thread
From: Sakari Ailus @ 2020-12-03 17:50 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, naush, dave.stevenson, laurent.pinchart,
kieran.bingham, niklas.soderlund, dafna.hirschfeld, hverkuil,
nsaenzjulienne, mchehab+huawei
On Thu, Dec 03, 2020 at 06:50:27PM +0100, Jacopo Mondi wrote:
> Hi Sakari,
>
> On Thu, Dec 03, 2020 at 07:41:02PM +0200, Sakari Ailus wrote:
>
> [snip]
>
> > >
> > > Ah! maximum: 8. I was about to reply to Rob's video-interfaces
> > > conversion and this might need to be pointed out.
> > > I was also about to ask it it's an issue if the bindings validation
> > > does not catch arrays as: [1, 3] as the constraint of being
> > > monotonically increasing values is expressed in words only. I guess
> > > it's fine, the alternative syntax is awful:
> > >
> > > anyOf:
> > > - items:
> > > - const: 1
> > > - const: 2
> > > - items:
> > > - const: 1
> > > - const: 2
> > > - const: 3
> > > - const: 4
> >
> > The v4l2-fwnode framework doesn't do that, what it does check for, though,
>
> I meant the dt-validation :)
I think it'd be very nice to have that check there, too. :-)
--
Sakari Ailus
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file
2020-12-03 11:42 ` Dave Stevenson
@ 2021-01-05 13:30 ` Laurent Pinchart
2021-01-11 15:12 ` Dave Stevenson
0 siblings, 1 reply; 31+ messages in thread
From: Laurent Pinchart @ 2021-01-05 13:30 UTC (permalink / raw)
To: Dave Stevenson
Cc: Sakari Ailus, Jacopo Mondi, Linux Media Mailing List,
Naushir Patuck, cc: Kieran Bingham, Niklas Söderlund,
Dafna Hirschfeld, Hans Verkuil, Nicolas Saenz Julienne,
mchehab+huawei
Hi Dave and Sakari,
On Thu, Dec 03, 2020 at 11:42:38AM +0000, Dave Stevenson wrote:
> On Wed, 2 Dec 2020 at 22:07, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > On Tue, Nov 10, 2020 at 06:40:36PM +0100, Jacopo Mondi wrote:
> > > The bcm2835-unicam driver is currently in staging mainly for
> > > two reasons:
> > > - Handling of CSI-2 embedded data
> > > - Usage of both media controller API and subdev kAPI
> > >
> > > Provide a more detailed description of the currently on-going design
> > > discussions in the associated TODO file.
> > >
> > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > ---
> > > drivers/staging/media/bcm2835-unicam/TODO | 37 +++++++++++++++++++++++
> > > 1 file changed, 37 insertions(+)
> > > create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
> > >
> > > diff --git a/drivers/staging/media/bcm2835-unicam/TODO b/drivers/staging/media/bcm2835-unicam/TODO
> > > new file mode 100644
> > > index 0000000000000..c7840872eea4c
> > > --- /dev/null
> > > +++ b/drivers/staging/media/bcm2835-unicam/TODO
> > > @@ -0,0 +1,37 @@
> > > +BCM2835 Unicam driver TODO list
> > > +===============================
> > > +
> > > +The unicam driver could be considered ready to be moved out of the staging
> > > +directory in terms of code quality and expected functionalities.
> > > +
> > > +However there currently are two design issues that suggest the driver is
> > > +better kept in staging for the time being.
> > > +
> > > +CSI-2 Embedded data support:
> > > +----------------------------
> > > +
> > > +The RaspberryPi camera stack and camera applications rely on the availability of
> > > +the sensor produced CSI-2 embedded data, whose support is currently not
> > > +finalized in mainline Linux.
> > > +
> > > +The driver conditionally registers an additional video device node
> > > +'unicam-embedded' with a single sink pad which connects to the sensor
> > > +sub-device source pad #1 to expose ancillary data.
> > > +
> > > +Currently none of the mainline sensor drivers register more than a single pad,
> > > +and consequentially no embedded data from the sensor are exposed to userspace.
> > > +
> > > +The current implementation is then subject to changes depending on how support
> > > +for CSI-2 embedded data gets finalized in Linux.
> >
> > Are you looking to use out-of-tree sensor drivers that have two pads? I'd
> > rather see sensor drivers merged to mainline proper.
> >
> > But as noted in the other e-mail, we need VC / stream support so this needs
> > to be addressed for reasons not related to Unicam.
>
> There's a downstream patch for imx219 that adds the second pad [1].
> The imx477 driver that is currently only out-of-tree also supports it,
> and should be upstreamed once this first wave of patches have got
> somewhere.
>
> [1] https://github.com/raspberrypi/linux/commit/fa8131cb1399f2c22de3f29e08ec1658db76552b
> It's on the rpi-5.10.y branch too, but that is still being frequently
> rebased so no stable commit hashes
>
> > > +
> > > +Media controller support:
> > > +-------------------------
> > > +
> > > +Due to compatibility reasons with the existing RaspberryPi software ecosystem
> > > +the unicam driver implements the media controller interface to allow the
> > > +enumeration of the connected entities but handles the configuration of the
> > > +sensor sub-device using the v4l2-subdev kAPI instead of delegating that to
> > > +user-space.
> > > +
> > > +Discussions are on-going on how this should be better handled (driver option,
> > > +KConfig option etc etc).
> >
> > That's a fair use case. There are two ways to handle it, either in the
> > kernel where the choice affects how the user space looks like. You have an
> > option of module parameter or Kconfig option there, and both are true
> > annoyances.
> >
> > Another option is to work around it in the user space, wrapping the IOCTL
> > calls. This way no kernel build or module load time parameters would be
> > needed to switch between the two sets of user space programs.
> >
> > We probably can't decide it now, but could an MC-only driver + user space
> > compatibility layer be an option here?
>
> Iff the user-space compatibility layer worked with all standard users
> (eg v4l2-ctl, FFmpeg and Gstreamer), then it's plausible, but is that
> realistic?
We have a compatibility layer in libcamera, implemented with a
best-effort approach (as the gap between a V4L2 video node and a complex
camera is larger to bridge than between a video node and an MC-based
device with a simple linear capture pipeline). It works, but has its
share of issues as LD_PRELOAD'ing a library isn't always a practical
option.
I believe the wrapper that Sakari envisions is feasible to implement as
the problem space is smaller (in a way the code already exists in kernel
space in the unicam driver, it bridges between the video node and the
input subdev). The tricky part will be to make it nice to use.
> The non-MC approach is mainly for things like the TC358743 HDMI->CSI
> bridge and ADV728x-M chips where MC adds nothing, and they do just
> work with the likes of FFmpeg and Gstreamer.
There are of devices that won't benefit much from an MC-based approach,
but note that in the non-camera use cases, you could have chips such as
the ADV748x which has one HDMI input, 8 analog inputs and two CSI-2
outputs with configurable routing in-between.
> Image sensors are generally going to be used under libcamera umbrella,
> so there MC works.
>
> Kconfig isn't an option for us as one kernel build needs to support
> all potential source devices via DT / runtime changes alone.
>
> I haven't looked in detail at what the previous VC / stream patches
> proposed as the API for passing the configuration. We only need MC for
> the more complex use cases, so if that VC/stream API usage can be
> detected at probe then we can switch to MC. It doesn't seem
> unreasonable to expect any sensor drivers to be upgraded to correctly
> use the new API even if they don't actually produce embedded data.
I'm not sure that would be the case, support for multiplexed streams
would be on top of the V4L2 subdev in-kernel API, and I don't expect
changes in camera sensors that don't support multiple streams.
> We also want MC for ADV748x HDMI&analog->CSI bridges that expose
There we go :-)
> multiple subdevs, but there we can look at the upstream endpoint and
> see if it has any sink pads. Sink pads mean MC is needed. (I now have
> a board with an ADV7482 on, so when time allows I'm intending to have
> an experiment with it).
Note that we will likely, in the future, introduce media entities for
connectors, which means that a device such as the TC358743 would have a
sink pad connected to an HDMI connector entity.
> Use a module parameter as a last resort should the detection fail, or
> I guess if you want to override the detected setting for some reason
> (a simple sensor being used outside of libcamera for example).
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file
2021-01-05 13:30 ` Laurent Pinchart
@ 2021-01-11 15:12 ` Dave Stevenson
2021-01-11 15:33 ` Laurent Pinchart
0 siblings, 1 reply; 31+ messages in thread
From: Dave Stevenson @ 2021-01-11 15:12 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Sakari Ailus, Jacopo Mondi, Linux Media Mailing List,
Naushir Patuck, cc: Kieran Bingham, Niklas Söderlund,
Dafna Hirschfeld, Hans Verkuil, Nicolas Saenz Julienne,
mchehab+huawei
Hi Laurent
On Tue, 5 Jan 2021 at 13:30, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dave and Sakari,
>
> On Thu, Dec 03, 2020 at 11:42:38AM +0000, Dave Stevenson wrote:
> > On Wed, 2 Dec 2020 at 22:07, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > On Tue, Nov 10, 2020 at 06:40:36PM +0100, Jacopo Mondi wrote:
> > > > The bcm2835-unicam driver is currently in staging mainly for
> > > > two reasons:
> > > > - Handling of CSI-2 embedded data
> > > > - Usage of both media controller API and subdev kAPI
> > > >
> > > > Provide a more detailed description of the currently on-going design
> > > > discussions in the associated TODO file.
> > > >
> > > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > > ---
> > > > drivers/staging/media/bcm2835-unicam/TODO | 37 +++++++++++++++++++++++
> > > > 1 file changed, 37 insertions(+)
> > > > create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
> > > >
> > > > diff --git a/drivers/staging/media/bcm2835-unicam/TODO b/drivers/staging/media/bcm2835-unicam/TODO
> > > > new file mode 100644
> > > > index 0000000000000..c7840872eea4c
> > > > --- /dev/null
> > > > +++ b/drivers/staging/media/bcm2835-unicam/TODO
> > > > @@ -0,0 +1,37 @@
> > > > +BCM2835 Unicam driver TODO list
> > > > +===============================
> > > > +
> > > > +The unicam driver could be considered ready to be moved out of the staging
> > > > +directory in terms of code quality and expected functionalities.
> > > > +
> > > > +However there currently are two design issues that suggest the driver is
> > > > +better kept in staging for the time being.
> > > > +
> > > > +CSI-2 Embedded data support:
> > > > +----------------------------
> > > > +
> > > > +The RaspberryPi camera stack and camera applications rely on the availability of
> > > > +the sensor produced CSI-2 embedded data, whose support is currently not
> > > > +finalized in mainline Linux.
> > > > +
> > > > +The driver conditionally registers an additional video device node
> > > > +'unicam-embedded' with a single sink pad which connects to the sensor
> > > > +sub-device source pad #1 to expose ancillary data.
> > > > +
> > > > +Currently none of the mainline sensor drivers register more than a single pad,
> > > > +and consequentially no embedded data from the sensor are exposed to userspace.
> > > > +
> > > > +The current implementation is then subject to changes depending on how support
> > > > +for CSI-2 embedded data gets finalized in Linux.
> > >
> > > Are you looking to use out-of-tree sensor drivers that have two pads? I'd
> > > rather see sensor drivers merged to mainline proper.
> > >
> > > But as noted in the other e-mail, we need VC / stream support so this needs
> > > to be addressed for reasons not related to Unicam.
> >
> > There's a downstream patch for imx219 that adds the second pad [1].
> > The imx477 driver that is currently only out-of-tree also supports it,
> > and should be upstreamed once this first wave of patches have got
> > somewhere.
> >
> > [1] https://github.com/raspberrypi/linux/commit/fa8131cb1399f2c22de3f29e08ec1658db76552b
> > It's on the rpi-5.10.y branch too, but that is still being frequently
> > rebased so no stable commit hashes
> >
> > > > +
> > > > +Media controller support:
> > > > +-------------------------
> > > > +
> > > > +Due to compatibility reasons with the existing RaspberryPi software ecosystem
> > > > +the unicam driver implements the media controller interface to allow the
> > > > +enumeration of the connected entities but handles the configuration of the
> > > > +sensor sub-device using the v4l2-subdev kAPI instead of delegating that to
> > > > +user-space.
> > > > +
> > > > +Discussions are on-going on how this should be better handled (driver option,
> > > > +KConfig option etc etc).
> > >
> > > That's a fair use case. There are two ways to handle it, either in the
> > > kernel where the choice affects how the user space looks like. You have an
> > > option of module parameter or Kconfig option there, and both are true
> > > annoyances.
> > >
> > > Another option is to work around it in the user space, wrapping the IOCTL
> > > calls. This way no kernel build or module load time parameters would be
> > > needed to switch between the two sets of user space programs.
> > >
> > > We probably can't decide it now, but could an MC-only driver + user space
> > > compatibility layer be an option here?
> >
> > Iff the user-space compatibility layer worked with all standard users
> > (eg v4l2-ctl, FFmpeg and Gstreamer), then it's plausible, but is that
> > realistic?
>
> We have a compatibility layer in libcamera, implemented with a
> best-effort approach (as the gap between a V4L2 video node and a complex
> camera is larger to bridge than between a video node and an MC-based
> device with a simple linear capture pipeline). It works, but has its
> share of issues as LD_PRELOAD'ing a library isn't always a practical
> option.
>
> I believe the wrapper that Sakari envisions is feasible to implement as
> the problem space is smaller (in a way the code already exists in kernel
> space in the unicam driver, it bridges between the video node and the
> input subdev). The tricky part will be to make it nice to use.
>
> > The non-MC approach is mainly for things like the TC358743 HDMI->CSI
> > bridge and ADV728x-M chips where MC adds nothing, and they do just
> > work with the likes of FFmpeg and Gstreamer.
>
> There are of devices that won't benefit much from an MC-based approach,
> but note that in the non-camera use cases, you could have chips such as
> the ADV748x which has one HDMI input, 8 analog inputs and two CSI-2
> outputs with configurable routing in-between.
The main difference there is that no one has ever been able to use
ADV748x without MC.
Running a mono image sensor, ADV728x-M or TC358743 is all perfectly
possible in a non-MC world. I don't want the headache of all the user
complaints should that get removed, even if able to workaround it with
a LD_PRELOAD library.
Should a wrapper appear then that would potentially remove the need
for the kernel to support a non-MC mode, but currently it looks like a
driver supporting both modes is the only answer.
> > Image sensors are generally going to be used under libcamera umbrella,
> > so there MC works.
> >
> > Kconfig isn't an option for us as one kernel build needs to support
> > all potential source devices via DT / runtime changes alone.
> >
> > I haven't looked in detail at what the previous VC / stream patches
> > proposed as the API for passing the configuration. We only need MC for
> > the more complex use cases, so if that VC/stream API usage can be
> > detected at probe then we can switch to MC. It doesn't seem
> > unreasonable to expect any sensor drivers to be upgraded to correctly
> > use the new API even if they don't actually produce embedded data.
>
> I'm not sure that would be the case, support for multiplexed streams
> would be on top of the V4L2 subdev in-kernel API, and I don't expect
> changes in camera sensors that don't support multiple streams.
>
> > We also want MC for ADV748x HDMI&analog->CSI bridges that expose
>
> There we go :-)
New use case, therefore no existing user expectations on how it would work.
> > multiple subdevs, but there we can look at the upstream endpoint and
> > see if it has any sink pads. Sink pads mean MC is needed. (I now have
> > a board with an ADV7482 on, so when time allows I'm intending to have
> > an experiment with it).
>
> Note that we will likely, in the future, introduce media entities for
> connectors, which means that a device such as the TC358743 would have a
> sink pad connected to an HDMI connector entity.
Would that count as an improvement?
I thought the golden rule was not to break userspace, but you'd be
enforcing extra userspace steps in configuring the device if it gains
more MC entities. Not my call though.
I'm tied up with other stuff at present, but will be looking at adding
MC rather than video node based mode of operation when time allows.
Dave
> > Use a module parameter as a last resort should the detection fail, or
> > I guess if you want to override the detected setting for some reason
> > (a simple sensor being used outside of libcamera for example).
>
> --
> Regards,
>
> Laurent Pinchart
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file
2021-01-11 15:12 ` Dave Stevenson
@ 2021-01-11 15:33 ` Laurent Pinchart
0 siblings, 0 replies; 31+ messages in thread
From: Laurent Pinchart @ 2021-01-11 15:33 UTC (permalink / raw)
To: Dave Stevenson
Cc: Sakari Ailus, Jacopo Mondi, Linux Media Mailing List,
Naushir Patuck, cc: Kieran Bingham, Niklas Söderlund,
Dafna Hirschfeld, Hans Verkuil, Nicolas Saenz Julienne,
mchehab+huawei
Hi Dave,
On Mon, Jan 11, 2021 at 03:12:10PM +0000, Dave Stevenson wrote:
> On Tue, 5 Jan 2021 at 13:30, Laurent Pinchart wrote:
> > On Thu, Dec 03, 2020 at 11:42:38AM +0000, Dave Stevenson wrote:
> > > On Wed, 2 Dec 2020 at 22:07, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > > On Tue, Nov 10, 2020 at 06:40:36PM +0100, Jacopo Mondi wrote:
> > > > > The bcm2835-unicam driver is currently in staging mainly for
> > > > > two reasons:
> > > > > - Handling of CSI-2 embedded data
> > > > > - Usage of both media controller API and subdev kAPI
> > > > >
> > > > > Provide a more detailed description of the currently on-going design
> > > > > discussions in the associated TODO file.
> > > > >
> > > > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > > > > ---
> > > > > drivers/staging/media/bcm2835-unicam/TODO | 37 +++++++++++++++++++++++
> > > > > 1 file changed, 37 insertions(+)
> > > > > create mode 100644 drivers/staging/media/bcm2835-unicam/TODO
> > > > >
> > > > > diff --git a/drivers/staging/media/bcm2835-unicam/TODO b/drivers/staging/media/bcm2835-unicam/TODO
> > > > > new file mode 100644
> > > > > index 0000000000000..c7840872eea4c
> > > > > --- /dev/null
> > > > > +++ b/drivers/staging/media/bcm2835-unicam/TODO
> > > > > @@ -0,0 +1,37 @@
> > > > > +BCM2835 Unicam driver TODO list
> > > > > +===============================
> > > > > +
> > > > > +The unicam driver could be considered ready to be moved out of the staging
> > > > > +directory in terms of code quality and expected functionalities.
> > > > > +
> > > > > +However there currently are two design issues that suggest the driver is
> > > > > +better kept in staging for the time being.
> > > > > +
> > > > > +CSI-2 Embedded data support:
> > > > > +----------------------------
> > > > > +
> > > > > +The RaspberryPi camera stack and camera applications rely on the availability of
> > > > > +the sensor produced CSI-2 embedded data, whose support is currently not
> > > > > +finalized in mainline Linux.
> > > > > +
> > > > > +The driver conditionally registers an additional video device node
> > > > > +'unicam-embedded' with a single sink pad which connects to the sensor
> > > > > +sub-device source pad #1 to expose ancillary data.
> > > > > +
> > > > > +Currently none of the mainline sensor drivers register more than a single pad,
> > > > > +and consequentially no embedded data from the sensor are exposed to userspace.
> > > > > +
> > > > > +The current implementation is then subject to changes depending on how support
> > > > > +for CSI-2 embedded data gets finalized in Linux.
> > > >
> > > > Are you looking to use out-of-tree sensor drivers that have two pads? I'd
> > > > rather see sensor drivers merged to mainline proper.
> > > >
> > > > But as noted in the other e-mail, we need VC / stream support so this needs
> > > > to be addressed for reasons not related to Unicam.
> > >
> > > There's a downstream patch for imx219 that adds the second pad [1].
> > > The imx477 driver that is currently only out-of-tree also supports it,
> > > and should be upstreamed once this first wave of patches have got
> > > somewhere.
> > >
> > > [1] https://github.com/raspberrypi/linux/commit/fa8131cb1399f2c22de3f29e08ec1658db76552b
> > > It's on the rpi-5.10.y branch too, but that is still being frequently
> > > rebased so no stable commit hashes
> > >
> > > > > +
> > > > > +Media controller support:
> > > > > +-------------------------
> > > > > +
> > > > > +Due to compatibility reasons with the existing RaspberryPi software ecosystem
> > > > > +the unicam driver implements the media controller interface to allow the
> > > > > +enumeration of the connected entities but handles the configuration of the
> > > > > +sensor sub-device using the v4l2-subdev kAPI instead of delegating that to
> > > > > +user-space.
> > > > > +
> > > > > +Discussions are on-going on how this should be better handled (driver option,
> > > > > +KConfig option etc etc).
> > > >
> > > > That's a fair use case. There are two ways to handle it, either in the
> > > > kernel where the choice affects how the user space looks like. You have an
> > > > option of module parameter or Kconfig option there, and both are true
> > > > annoyances.
> > > >
> > > > Another option is to work around it in the user space, wrapping the IOCTL
> > > > calls. This way no kernel build or module load time parameters would be
> > > > needed to switch between the two sets of user space programs.
> > > >
> > > > We probably can't decide it now, but could an MC-only driver + user space
> > > > compatibility layer be an option here?
> > >
> > > Iff the user-space compatibility layer worked with all standard users
> > > (eg v4l2-ctl, FFmpeg and Gstreamer), then it's plausible, but is that
> > > realistic?
> >
> > We have a compatibility layer in libcamera, implemented with a
> > best-effort approach (as the gap between a V4L2 video node and a complex
> > camera is larger to bridge than between a video node and an MC-based
> > device with a simple linear capture pipeline). It works, but has its
> > share of issues as LD_PRELOAD'ing a library isn't always a practical
> > option.
> >
> > I believe the wrapper that Sakari envisions is feasible to implement as
> > the problem space is smaller (in a way the code already exists in kernel
> > space in the unicam driver, it bridges between the video node and the
> > input subdev). The tricky part will be to make it nice to use.
> >
> > > The non-MC approach is mainly for things like the TC358743 HDMI->CSI
> > > bridge and ADV728x-M chips where MC adds nothing, and they do just
> > > work with the likes of FFmpeg and Gstreamer.
> >
> > There are of devices that won't benefit much from an MC-based approach,
> > but note that in the non-camera use cases, you could have chips such as
> > the ADV748x which has one HDMI input, 8 analog inputs and two CSI-2
> > outputs with configurable routing in-between.
>
> The main difference there is that no one has ever been able to use
> ADV748x without MC.
>
> Running a mono image sensor, ADV728x-M or TC358743 is all perfectly
> possible in a non-MC world. I don't want the headache of all the user
> complaints should that get removed, even if able to workaround it with
> a LD_PRELOAD library.
I agree, an LD_PRELOAD wrapper doesn't bring a great user experience :-(
> Should a wrapper appear then that would potentially remove the need
> for the kernel to support a non-MC mode, but currently it looks like a
> driver supporting both modes is the only answer.
From a backward compatibility point of view, yes, that's the only
solution we have now that will be 100% transparent and compatible.
> > > Image sensors are generally going to be used under libcamera umbrella,
> > > so there MC works.
> > >
> > > Kconfig isn't an option for us as one kernel build needs to support
> > > all potential source devices via DT / runtime changes alone.
> > >
> > > I haven't looked in detail at what the previous VC / stream patches
> > > proposed as the API for passing the configuration. We only need MC for
> > > the more complex use cases, so if that VC/stream API usage can be
> > > detected at probe then we can switch to MC. It doesn't seem
> > > unreasonable to expect any sensor drivers to be upgraded to correctly
> > > use the new API even if they don't actually produce embedded data.
> >
> > I'm not sure that would be the case, support for multiplexed streams
> > would be on top of the V4L2 subdev in-kernel API, and I don't expect
> > changes in camera sensors that don't support multiple streams.
> >
> > > We also want MC for ADV748x HDMI&analog->CSI bridges that expose
> >
> > There we go :-)
>
> New use case, therefore no existing user expectations on how it would work.
Sure, that was only outlining that we need MC support, not that we have
to drop everything else.
> > > multiple subdevs, but there we can look at the upstream endpoint and
> > > see if it has any sink pads. Sink pads mean MC is needed. (I now have
> > > a board with an ADV7482 on, so when time allows I'm intending to have
> > > an experiment with it).
> >
> > Note that we will likely, in the future, introduce media entities for
> > connectors, which means that a device such as the TC358743 would have a
> > sink pad connected to an HDMI connector entity.
>
> Would that count as an improvement?
> I thought the golden rule was not to break userspace, but you'd be
> enforcing extra userspace steps in configuring the device if it gains
> more MC entities. Not my call though.
Just to clarify, this change will not be applied to existing drivers if
it risks breaking userspace. Furthermore, there will likely be no need
for userspace to configure the connector entity. My point was that logic
in the kernel that looks at the source to see if it has more than just
one output port or pad may not be future-proof, but that's a kernel-side
problem, it shouldn't affect userspace. I'm sure we'll be able to find a
good enough heuristic on the kernel side.
> I'm tied up with other stuff at present, but will be looking at adding
> MC rather than video node based mode of operation when time allows.
That's something I can look at too. The question is how to handle
backward compatibility, my preference would be for the driver core to be
MC-based, and for the compatibility layer to be based on top of that,
isolated from the rest of the code as much as possible. I think it's
doable.
> > > Use a module parameter as a last resort should the detection fail, or
> > > I guess if you want to override the detected setting for some reason
> > > (a simple sensor being used outside of libcamera for example).
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2021-01-11 15:34 UTC | newest]
Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-10 17:40 [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 1/5] media: uapi: Add MEDIA_BUS_FMT_CUSTOM_SENSOR_DATA Jacopo Mondi
2020-12-02 21:16 ` Sakari Ailus
2020-12-03 17:05 ` Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 2/5] dt-bindings: media: Document BCM283x CSI2/CCP2 receiver Jacopo Mondi
2020-12-02 21:20 ` Sakari Ailus
2020-12-03 16:44 ` Jacopo Mondi
2020-12-03 17:25 ` Sakari Ailus
2020-12-03 17:34 ` Chen-Yu Tsai
2020-12-03 17:37 ` Jacopo Mondi
2020-12-03 17:41 ` Sakari Ailus
2020-12-03 17:50 ` Jacopo Mondi
2020-12-03 17:50 ` Sakari Ailus
2020-11-10 17:40 ` [PATCH v4 3/5] media: bcm2835-unicam: Driver for CCP2/CSI2 camera interface Jacopo Mondi
2020-12-02 21:53 ` Sakari Ailus
2020-12-03 16:30 ` Jacopo Mondi
2020-12-03 17:12 ` Dave Stevenson
2020-12-02 22:35 ` Kieran Bingham
2020-11-10 17:40 ` [PATCH v4 4/5] ARM: dts: bcm2711: Add Unicam DT nodes Jacopo Mondi
2020-11-10 17:40 ` [PATCH v4 5/5] media: bcm2835-unicam: Add TODO file Jacopo Mondi
2020-12-02 22:06 ` Sakari Ailus
2020-12-03 11:42 ` Dave Stevenson
2021-01-05 13:30 ` Laurent Pinchart
2021-01-11 15:12 ` Dave Stevenson
2021-01-11 15:33 ` Laurent Pinchart
2020-12-01 9:26 ` [PATCH v4 0/5] media: staging: Add bcm2835-unicam driver Jacopo Mondi
2020-12-01 12:57 ` Sakari Ailus
2020-12-01 13:18 ` Laurent Pinchart
2020-12-02 21:55 ` Sakari Ailus
2020-12-01 13:24 ` Jacopo Mondi
2020-12-02 22:00 ` Sakari Ailus
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.