Re: (EXT) Re: Sony IMX290 link frequency

[Dave Stevenson <dave.stevenson@raspberrypi.com>]

On Tue, 21 Sept 2021 at 07:49, Stein, Alexander
<Alexander.Stein@tq-group.com> wrote:
>
> Hello Dave,
>
> thank you for your verbose explanation. I think I understand a lot more
> now about those sensors. I'm using an IMX327 as well, but there doesn't
> seem to be a huge difference.

IMX290 adds a 1080p 100 or 120fps mode, but otherwise seems identical.
The driver doesn't support that extra mode, so will drive IMX327 and
IMX290 equally.

> On Montag, 20 Sept 2021 at 12:27 +0100, Dave Stevenson
> > It has two clock paths - one driving the pixel array, and one driving
> > the MIPI core. There is a FIFO between the two, so they can run at
> > different rates.
> >
> > My conclusion is that the pixel array always runs at the same pixel
> > rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that
> > is
> > the result needed for vblank and hblank controls to work correctly in
> > computing frame rate. The datasheet again contains fixed register
> > settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX
> > (register 0x301c/d), but they are all just linearly scaled values of
> > each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed
> > for
> > 1080p and 720p modes, so maps to V4L2_CID_VBLANK.
>
> Given that there are fixed numbers for 1080p/720p for VMAX and for
> 25,30,50,60 fps for HMAX shouldn't those controls be read-only?
> What's the benefit of being writable in [1]?

The datasheet lists fixed numbers, but any numbers work following the
normal mode of operation for these controls.

eg for the 1080p mode, VMAX is specified as 0x465 (1125 decimal).
HMAX is then
25fps - 0x14a0 (5280). 5280*1125*25 = 148.5M
30fps - 0x1130 (4400). 4400*1125*30 = 148.5M
50fps - 0x0a50 (2640). 2640*1125*50 = 148.5M
60fps - 0x0898 (2200). 2200*1125*60 = 148.5M
All values of HMAX inbetween those work fine, hence V4L2_CID_VBLANK
can be a control with range from the minimum permitted (2200-height
for 60fps) to the max the sensor will take.

VMAX mapping to V4L2_CID_VBLANK could be a read only control, but it's
functional in the sensor over the full range, so why not expose it.
Frame rate control within libcamera typically varies V4L2_CID_VBLANK
too, so that means it fits into the normal frameworks better.

> > The MIPI PHY then runs at a link frequency sufficient to convey the
> > desired pixels:
> > - 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp
> > - 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped
> > modes up to 60fps, 10 or 12bpp.
> > - 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or
> > 12bpp
> > This is controlled predominantly by INCKSEL1 & 2 (registers
> > 0x305c/d).
> > I verified the actual link frequencies used on a scope, and they are
> > as described.
> > 1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp
> > just increases the per line blanking period on the CSI2 interface.
>
> Is there actual blanking on CSI2 interface, as in inserting "dummy"
> pixels in a row and dummy lines? Given that the spec is not available
> freely it's hard to understand what's actually happening on the
> interface.
> Given that the link frequency is fixed (in this case depending on
> window mode, input clock and number of lanes) I was wondering what's
> happening when the payload length changes (e.g. switching bpp or
> cropping).

Whilst the spec is restricted, there are a few introductions that give
a basic understanding, eg [1]

MIPI D-PHY has mutliple lane states, but the main ones are low power
mode (LP-11) and high speed mode (HS).
For power saving it is very common for a sensor to switch to HS mode,
send the line start short packet, line data long packet, line end
short packet, and then drop back to LP-11 between lines. For the
beginning and end of frames there are frame start and frame end short
packets, but during the idle time the lanes will drop back to LP-11.
See IMX219 datasheet [2] pages 47 & 48 for an example.

When sending less data due to cropping or blanking periods, the data
lanes are just in LP-11 for a longer period.

Clock lanes and data lanes may do slightly different things. It is not
uncommon for the clock lane to remain in HS mode at all times, but the
data lanes will almost always drop to LP-11. Device tree has a
clock-noncontinuous property to denote that the clock lane drops to
LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK.
There is no mention in the datasheet as to whether IMX290/327 produces
a continuous clock signal or not, and I can't say that I've checked.
Some receivers can get confused if the clock is continuous and they
miss the initial state change from LP to HS.

[1] https://download.tek.com/document/61W_25772_0_HR.pdf
[2] https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Raspberry-Pi-V2-CMOS/master/RASPBERRY%20PI%20CAMERA%20V2%20DATASHEET%20IMX219PQH5_7.0.0_Datasheet_XXX.PDF

> > AFAICT you could just always run at the 1782 Mbit/s rate with
> > slightly
> > increased idle time on the CSI2 bus for the 720p mode, but that isn't
> > the way Sony have specified it.
> >
> > There is a further register to halve the link frequency again for max
> > 25/30fps modes (0x3405), although doing so has limited benefit (it'd
> > increase rolling shutter effects as it would increase the temporal
> > difference between each line).
> >
> > I hope that makes things a little clearer. Indeed the current driver
> > is slightly wrong, but only in relation to pixel rate, not link
> > frequency.
>
> Thanks for confirmation. I meanwhile found the table providing the link
> frequencies.
> I assume now that my current problem regarding settle time is somewhere
> else. I can currently only assume the escape clock is not correct, but
> there is pretty much no documentation at all on that topic.

What platform are you working with?
There are registers in the IMX290/327 which configure the MIPI timings
- 0x3445-0x3455. The values do differ based on the link frequency, but
checking our driver to the datasheet they appear to be all correct.

Hope that helps.
  Dave

> Thanks and best regards
> Alexander
>
> [1]
> https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c
>
> --
> Mit freundlichen Grüßen
>
> i.A. Alexander Stein
> Entwicklung Standort Chemnitz
> Tel. +49 371 433151-0, Fax +49 371 433151-22
> Zwickauer Straße 173, 09116 Chemnitz
> mailto: Alexander.Stein@tq-group.com
>
> TQ-Systems GmbH
> Mühlstraße 2, Gut Delling, 82229 Seefeld
> Amtsgericht München, HRB 105018
> Sitz der Gesellschaft: Seefeld
> Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
> www.tq-group.com
>