Re: [PATCH v2 00/34] Drivers for the BCM283x CSI-2/CCP2 receiver and ISP

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hi Nicolas,

On Mon, May 04, 2020 at 05:15:42PM +0200, Nicolas Saenz Julienne wrote:
> On Mon, 2020-05-04 at 12:25 +0300, Laurent Pinchart wrote:
> > Hello,
> 
> Hi Laurent, thanks for the series!
> Would you mind CCing on further iterations, I was lucky enough to find this
> referenced in twitter, and I'll be more than happy to review/test it. I'm the
> BCM2711/BCM2835 arm architecture & vc04_services maintainer :).
> 
> An please also CC:
> bcm-kernel-feedback-list@broadcom.com
> linux-rpi-kernel@lists.infradead.org
> linux-arm-kernel@lists.infradead.org

Sure. I was so focussed on the userspace part and on getting everything
ready for today's release that I completely forgot to run the patches
through get_maintainers.pl and CC the appropriate people and lists.
Sorry about the oversight, I'll make sure to fix it next time.

Speaking of BCM2711/BCM2835 maintenance... :-) This series isn't
complete as it's missing full DT integration of camera sensor modules.
This is handled through DT overlays in the Raspberry Pi downstream
kernel, which are applied to the base DT by the boot loader (a neat
solution in my opinion). I wonder what we should do for mainline.
Ideally we should be able to at least use the DT overlays from the
downstream Raspberry Pi kernel on mainline, but that's not possible
today due to incompatibilities in the base DT (in particular related to
the I2C port to which the camera sensors are connected). Dave, have you
given that a thought by any chance ?

Another more general question for Nicolas, is there a plan to try and
get vc04_services out of staging ?

> > This patch series adds drivers for the CSI-2/CCP2 receiver and ISP found
> > in the Broadcom BCM283x and compatible SoCs (namely the BCM2711). The
> > CSI-2/CCP2 receiver IP core is known as Unicam. The most well known
> > platforms representative of these SoCs are the Raspberry Pi. The
> > previous version of the series was titled "[PATCH 0/5] Driver for the
> > BCM283x CSI-2/CCP2 receiver" and contained the CSI-2/CCP2 receiver only.
> > 
> > A driver already exists in drivers/staging/vc04_services/bcm2835-camera/
> > to support cameras on Raspberry Pi platforms. The driver relies on the
> > firmware running on the VC4 VPU to control the camera, and is thus
> > limited to the camera sensors supported in the firmware. These drivers,
> > on the other hand, have limited dependencies on the firmware:
> > 
> > - The Unicam peripheral driver doesn't rely on the firmware at all and
> >   controls the Unicam hardware directly. It supports any camera sensor
> >   compatible with those interfaces.
> > 
> > - The ISP driver relies on the VC4 VPU firmware to communicate with the
> >   ISP hardware, but, unlike with the existing bcm2835-camera driver, the
> >   firmware exposes the ISP instead of controlling it internally.
> > 
> > The code is upported from the Raspberry Pi Linux kernel repository
> > available at https://github.com/raspberrypi/linux. The patches are based
> > on top of v5.7-rc2 with Jacopo's "[PATCH v4 0/5] media: Register
> > read-only sub-dev devnode" series applied.
> > 
> > Patches 01/34 to 05/34 are related to the bcm2835-unicam driver. Patches
> > 01/34 and 02/34 add new a pixel format and media bus code for sensor
> > embedded data. Patch 03/34 then adds DT bindings for the driver, and
> > patch 04/34 adds the driver itself. Patch 05/34 adds the Unicam
> > peripheral instances to the BCM2711 device tree.
> > 
> > The bcm2835-unicam driver supports capturing both image data and
> > embedded data. Support for embedded data is currently implemented
> > through two sink pads, requiring two source pads on the sensor driver
> > side. Work is ongoing to try and replace this with usage of a
> > multiplexed streams API.
> > 
> > The remaining patches are related to the bcm2835-isp driver. As the
> > driver needs to communicate with the VC4 VPU firmware, whose support is
> > currently in staging, the new driver is added to staging too.
> > 
> > Patch 06/34 adds a new driver named vc-sm-cma to handle memory sharing
> > with the VC4 VPU. This will likely be reworked in the future to use
> > dmabuf heaps. Patch 07/34 then breaks the VC4 VPU multimedia access
> > layer code (named vchiq-mmal) out of the existing bcm2835-camera driver
> > to a new directory, to be shared with the bcm2835-isp driver. Patches
> > 08/34 to 24/34 then upport various improvements to the MMAL code.
> > Patches 25/34 to 31/34 follow with an upport of various improvement to
> > the VCHIQ code, which handles the communication with the VC4 VPU (and is
> > used by the MMAL code).
> > 
> > At patch 32/34 we move to the media side, with a small extension to
> > videobuf2. Patch 33/34 adds the bcm2835-isp driver, along with
> > documentation of the related metadata format. Patch 34/34 then wires
> > this up with the vchiq-mmal driver.
> > 
> > The two drivers will likely be split into separate patch series for v3.
> > I however wanted to bundle them here to emphasize that they're related,
> > and that together they support full control of the Raspberry Pi cameras
> > through libcamera without any closed-source software. The corresponding
> > libcamera code is available from
> > 
> > 	git://linuxtv.org/libcamera.git raspberrypi
> > 
> > The 'raspberrypi' branch is temporary until the code gets integrated in
> > the master branch after public review.
> > 
> > I would like to thank Dave Stevenson, Naushir Patuk and David Plowman
> > from Raspberry Pi (Trading) Ltd. for their hard work that made this all
> > possible, as well as Jacopo Mondi, Kieran Bingham and Niklas Söderlund
> > from the libcamera team for all their efforts on both the kernel and
> > libcamera side. This is, I'm sure, the beginning of a new journey for
> > computational camera support in Linux.
> > 
> > And now, the customary v4l2-compliance report. There are three new
> > failures with bcm2835-unicam compared to the previous version, and they
> > will addressed in v3.

[snip]

-- 
Regards,

Laurent Pinchart