Sony IMX290 link frequency

Sony IMX290 link frequency

[Stein, Alexander]

Hello,

the imx290 driver specifies 2 tables of link frequencies, depending on
number of lanes and the vertical resolution.
These numbers vary from 148.5 MHz up to 445.5 MHz.
I'm wondering where these number come from, it's not clear for me. Also
shouldn't those numbers also depend in bits per pixel and the frames
per second?
I suspect that the returned link frequency is not correct which breaks
things further down using v4l2_get_link_freq().

Best regards
Alexander
-- 
i.A. Alexander Stein
Entwicklung Standort Chemnitz
Tel. +49 371 433151-0, Fax +49 371 433151-22
Zwickauer Straße 173, 09116 Chemnitz
mailto: Alexander.Stein@tq-group.com
 
TQ-Systems GmbH
Mühlstraße 2, Gut Delling, 82229 Seefeld
Amtsgericht München, HRB 105018
Sitz der Gesellschaft: Seefeld
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
www.tq-group.com

Re: Sony IMX290 link frequency

[Dave Stevenson]

Hi Alexander

On Mon, 20 Sept 2021 at 08:15, Stein, Alexander
<Alexander.Stein@tq-group.com> wrote:
>
> Hello,
>
> the imx290 driver specifies 2 tables of link frequencies, depending on
> number of lanes and the vertical resolution.
> These numbers vary from 148.5 MHz up to 445.5 MHz.
> I'm wondering where these number come from, it's not clear for me. Also
> shouldn't those numbers also depend in bits per pixel and the frames
> per second?
> I suspect that the returned link frequency is not correct which breaks
> things further down using v4l2_get_link_freq().

I looked at imx290 and little brother imx327 a few months back to get
it running with libcamera on the Pi. I have a couple of patches that
ought to be upstreamed when I get a chance [1].

The Sony datasheet provides register configurations for specific modes
(1080p, window cropping, and 720p) and frame rates, rather than a full
description of what the registers do.

It has two clock paths - one driving the pixel array, and one driving
the MIPI core. There is a FIFO between the two, so they can run at
different rates.

My conclusion is that the pixel array always runs at the same pixel
rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that is
the result needed for vblank and hblank controls to work correctly in
computing frame rate. The datasheet again contains fixed register
settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX
(register 0x301c/d), but they are all just linearly scaled values of
each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed for
1080p and 720p modes, so maps to V4L2_CID_VBLANK.

The MIPI PHY then runs at a link frequency sufficient to convey the
desired pixels:
- 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp
- 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped
modes up to 60fps, 10 or 12bpp.
- 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or 12bpp
This is controlled predominantly by INCKSEL1 & 2 (registers 0x305c/d).
I verified the actual link frequencies used on a scope, and they are
as described.
1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp
just increases the per line blanking period on the CSI2 interface.

AFAICT you could just always run at the 1782 Mbit/s rate with slightly
increased idle time on the CSI2 bus for the 720p mode, but that isn't
the way Sony have specified it.

There is a further register to halve the link frequency again for max
25/30fps modes (0x3405), although doing so has limited benefit (it'd
increase rolling shutter effects as it would increase the temporal
difference between each line).

I hope that makes things a little clearer. Indeed the current driver
is slightly wrong, but only in relation to pixel rate, not link
frequency.

  Dave

[1] https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c

> Best regards
> Alexander
> --
> i.A. Alexander Stein
> Entwicklung Standort Chemnitz
> Tel. +49 371 433151-0, Fax +49 371 433151-22
> Zwickauer Straße 173, 09116 Chemnitz
> mailto: Alexander.Stein@tq-group.com
>
> TQ-Systems GmbH
> Mühlstraße 2, Gut Delling, 82229 Seefeld
> Amtsgericht München, HRB 105018
> Sitz der Gesellschaft: Seefeld
> Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
> www.tq-group.com

Re: (EXT) Re: Sony IMX290 link frequency

[Stein, Alexander]

Hello Dave,

thank you for your verbose explanation. I think I understand a lot more
now about those sensors. I'm using an IMX327 as well, but there doesn't
seem to be a huge difference.

On Montag, 20 Sept 2021 at 12:27 +0100, Dave Stevenson
> It has two clock paths - one driving the pixel array, and one driving
> the MIPI core. There is a FIFO between the two, so they can run at
> different rates.
> 
> My conclusion is that the pixel array always runs at the same pixel
> rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that
> is
> the result needed for vblank and hblank controls to work correctly in
> computing frame rate. The datasheet again contains fixed register
> settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX
> (register 0x301c/d), but they are all just linearly scaled values of
> each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed
> for
> 1080p and 720p modes, so maps to V4L2_CID_VBLANK.

Given that there are fixed numbers for 1080p/720p for VMAX and for
25,30,50,60 fps for HMAX shouldn't those controls be read-only?
What's the benefit of being writable in [1]?

> The MIPI PHY then runs at a link frequency sufficient to convey the
> desired pixels:
> - 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp
> - 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped
> modes up to 60fps, 10 or 12bpp.
> - 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or
> 12bpp
> This is controlled predominantly by INCKSEL1 & 2 (registers
> 0x305c/d).
> I verified the actual link frequencies used on a scope, and they are
> as described.
> 1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp
> just increases the per line blanking period on the CSI2 interface.

Is there actual blanking on CSI2 interface, as in inserting "dummy"
pixels in a row and dummy lines? Given that the spec is not available
freely it's hard to understand what's actually happening on the
interface.
Given that the link frequency is fixed (in this case depending on
window mode, input clock and number of lanes) I was wondering what's
happening when the payload length changes (e.g. switching bpp or
cropping).

> AFAICT you could just always run at the 1782 Mbit/s rate with
> slightly
> increased idle time on the CSI2 bus for the 720p mode, but that isn't
> the way Sony have specified it.
> 
> There is a further register to halve the link frequency again for max
> 25/30fps modes (0x3405), although doing so has limited benefit (it'd
> increase rolling shutter effects as it would increase the temporal
> difference between each line).
> 
> I hope that makes things a little clearer. Indeed the current driver
> is slightly wrong, but only in relation to pixel rate, not link
> frequency.

Thanks for confirmation. I meanwhile found the table providing the link
frequencies.
I assume now that my current problem regarding settle time is somewhere
else. I can currently only assume the escape clock is not correct, but
there is pretty much no documentation at all on that topic.

Thanks and best regards
Alexander

[1] 
https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c

-- 
Mit freundlichen Grüßen
 
i.A. Alexander Stein
Entwicklung Standort Chemnitz
Tel. +49 371 433151-0, Fax +49 371 433151-22
Zwickauer Straße 173, 09116 Chemnitz
mailto: Alexander.Stein@tq-group.com
 
TQ-Systems GmbH
Mühlstraße 2, Gut Delling, 82229 Seefeld
Amtsgericht München, HRB 105018
Sitz der Gesellschaft: Seefeld
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
www.tq-group.com

Re: (EXT) Re: Sony IMX290 link frequency

[Dave Stevenson]

On Tue, 21 Sept 2021 at 07:49, Stein, Alexander
<Alexander.Stein@tq-group.com> wrote:
>
> Hello Dave,
>
> thank you for your verbose explanation. I think I understand a lot more
> now about those sensors. I'm using an IMX327 as well, but there doesn't
> seem to be a huge difference.

IMX290 adds a 1080p 100 or 120fps mode, but otherwise seems identical.
The driver doesn't support that extra mode, so will drive IMX327 and
IMX290 equally.

> On Montag, 20 Sept 2021 at 12:27 +0100, Dave Stevenson
> > It has two clock paths - one driving the pixel array, and one driving
> > the MIPI core. There is a FIFO between the two, so they can run at
> > different rates.
> >
> > My conclusion is that the pixel array always runs at the same pixel
> > rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that
> > is
> > the result needed for vblank and hblank controls to work correctly in
> > computing frame rate. The datasheet again contains fixed register
> > settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX
> > (register 0x301c/d), but they are all just linearly scaled values of
> > each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed
> > for
> > 1080p and 720p modes, so maps to V4L2_CID_VBLANK.
>
> Given that there are fixed numbers for 1080p/720p for VMAX and for
> 25,30,50,60 fps for HMAX shouldn't those controls be read-only?
> What's the benefit of being writable in [1]?

The datasheet lists fixed numbers, but any numbers work following the
normal mode of operation for these controls.

eg for the 1080p mode, VMAX is specified as 0x465 (1125 decimal).
HMAX is then
25fps - 0x14a0 (5280). 5280*1125*25 = 148.5M
30fps - 0x1130 (4400). 4400*1125*30 = 148.5M
50fps - 0x0a50 (2640). 2640*1125*50 = 148.5M
60fps - 0x0898 (2200). 2200*1125*60 = 148.5M
All values of HMAX inbetween those work fine, hence V4L2_CID_VBLANK
can be a control with range from the minimum permitted (2200-height
for 60fps) to the max the sensor will take.

VMAX mapping to V4L2_CID_VBLANK could be a read only control, but it's
functional in the sensor over the full range, so why not expose it.
Frame rate control within libcamera typically varies V4L2_CID_VBLANK
too, so that means it fits into the normal frameworks better.

> > The MIPI PHY then runs at a link frequency sufficient to convey the
> > desired pixels:
> > - 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp
> > - 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped
> > modes up to 60fps, 10 or 12bpp.
> > - 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or
> > 12bpp
> > This is controlled predominantly by INCKSEL1 & 2 (registers
> > 0x305c/d).
> > I verified the actual link frequencies used on a scope, and they are
> > as described.
> > 1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp
> > just increases the per line blanking period on the CSI2 interface.
>
> Is there actual blanking on CSI2 interface, as in inserting "dummy"
> pixels in a row and dummy lines? Given that the spec is not available
> freely it's hard to understand what's actually happening on the
> interface.
> Given that the link frequency is fixed (in this case depending on
> window mode, input clock and number of lanes) I was wondering what's
> happening when the payload length changes (e.g. switching bpp or
> cropping).

Whilst the spec is restricted, there are a few introductions that give
a basic understanding, eg [1]

MIPI D-PHY has mutliple lane states, but the main ones are low power
mode (LP-11) and high speed mode (HS).
For power saving it is very common for a sensor to switch to HS mode,
send the line start short packet, line data long packet, line end
short packet, and then drop back to LP-11 between lines. For the
beginning and end of frames there are frame start and frame end short
packets, but during the idle time the lanes will drop back to LP-11.
See IMX219 datasheet [2] pages 47 & 48 for an example.

When sending less data due to cropping or blanking periods, the data
lanes are just in LP-11 for a longer period.

Clock lanes and data lanes may do slightly different things. It is not
uncommon for the clock lane to remain in HS mode at all times, but the
data lanes will almost always drop to LP-11. Device tree has a
clock-noncontinuous property to denote that the clock lane drops to
LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK.
There is no mention in the datasheet as to whether IMX290/327 produces
a continuous clock signal or not, and I can't say that I've checked.
Some receivers can get confused if the clock is continuous and they
miss the initial state change from LP to HS.

[1] https://download.tek.com/document/61W_25772_0_HR.pdf
[2] https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Raspberry-Pi-V2-CMOS/master/RASPBERRY%20PI%20CAMERA%20V2%20DATASHEET%20IMX219PQH5_7.0.0_Datasheet_XXX.PDF

> > AFAICT you could just always run at the 1782 Mbit/s rate with
> > slightly
> > increased idle time on the CSI2 bus for the 720p mode, but that isn't
> > the way Sony have specified it.
> >
> > There is a further register to halve the link frequency again for max
> > 25/30fps modes (0x3405), although doing so has limited benefit (it'd
> > increase rolling shutter effects as it would increase the temporal
> > difference between each line).
> >
> > I hope that makes things a little clearer. Indeed the current driver
> > is slightly wrong, but only in relation to pixel rate, not link
> > frequency.
>
> Thanks for confirmation. I meanwhile found the table providing the link
> frequencies.
> I assume now that my current problem regarding settle time is somewhere
> else. I can currently only assume the escape clock is not correct, but
> there is pretty much no documentation at all on that topic.

What platform are you working with?
There are registers in the IMX290/327 which configure the MIPI timings
- 0x3445-0x3455. The values do differ based on the link frequency, but
checking our driver to the datasheet they appear to be all correct.

Hope that helps.
  Dave

> Thanks and best regards
> Alexander
>
> [1]
> https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c
>
> --
> Mit freundlichen Grüßen
>
> i.A. Alexander Stein
> Entwicklung Standort Chemnitz
> Tel. +49 371 433151-0, Fax +49 371 433151-22
> Zwickauer Straße 173, 09116 Chemnitz
> mailto: Alexander.Stein@tq-group.com
>
> TQ-Systems GmbH
> Mühlstraße 2, Gut Delling, 82229 Seefeld
> Amtsgericht München, HRB 105018
> Sitz der Gesellschaft: Seefeld
> Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
> www.tq-group.com
>

Re: (EXT) Re: (EXT) Re: Sony IMX290 link frequency

[Stein, Alexander]

Hello Dave,

On Tue, 21 Sept 2021 at 11:41 +0100, Dave Stevenson:
> Whilst the spec is restricted, there are a few introductions that
> give
> a basic understanding, eg [1]

Thanks, this helped a bit.

> Clock lanes and data lanes may do slightly different things. It is
> not
> uncommon for the clock lane to remain in HS mode at all times, but
> the
> data lanes will almost always drop to LP-11. Device tree has a
> clock-noncontinuous property to denote that the clock lane drops to
> LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK.
> There is no mention in the datasheet as to whether IMX290/327
> produces
> a continuous clock signal or not, and I can't say that I've checked.
> Some receivers can get confused if the clock is continuous and they
> miss the initial state change from LP to HS.

Who drives the clock lane? The sensor or the host processor? If it is
the sensor how do you support several sensors on the same bus?

> > > > > > I assume now that my current problem regarding settle time is
> > somewhere
else. I can currently only assume the escape clock is not correct,
> > but
there is pretty much no documentation at all on that topic.
> 
> What platform are you working with?

I'm working on an i.MX8QM based board. A public RM is available since
several days [1]. The MIPI-CSI subsystem chapters are a bit lacking.
Also I am aware that the ISI does some strange things with CSI input,
see also [1], but that is a different issue.

As far as I understand different documents, the escape clock is used
during LP-11 and shall be from 10 to 20 MHz.
The device tree configures some clock to 72MHz [2], but I do not know
what's happening internally.

I used the same approach for calculating the settle time as [3]. But I
have to use a value slightly below the maximum rather than the average

> There are registers in the IMX290/327 which configure the MIPI
> timings
> - 0x3445-0x3455. The values do differ based on the link frequency,
> but
> checking our driver to the datasheet they appear to be all correct.

Agreed. 

[1] https://www.nxp.com/docs/en/reference-manual/IMX8QMRM.pdf
[2] 
https://yhbt.net/lore/all/20200226151431.GY5379@paasikivi.fi.intel.com/T/

[3] 
https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8-ss-img.dtsi?h=lf-5.10.y#n516

[4] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/media/imx/imx8mq-mipi-csi2.c#n343

-- 
Mit freundlichen Grüßen
 
i.A. Alexander Stein
Entwicklung Standort
Chemnitz
Tel. +49 371 433151-0, Fax +49 371 433151-22
Zwickauer Straße
173, 09116 Chemnitz
mailto: Alexander.Stein@tq-group.com
 
TQ-Systems GmbH
Mühlstraße 2, Gut Delling, 82229 Seefeld
Amtsgericht München, HRB 105018
Sitz der Gesellschaft: Seefeld
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
www.tq-group.com

Re: (EXT) Re: (EXT) Re: Sony IMX290 link frequency

[Dave Stevenson]

On Thu, 23 Sept 2021 at 14:44, Stein, Alexander
<Alexander.Stein@tq-group.com> wrote:
>
> Hello Dave,
>
> On Tue, 21 Sept 2021 at 11:41 +0100, Dave Stevenson:
> > Whilst the spec is restricted, there are a few introductions that
> > give
> > a basic understanding, eg [1]
>
> Thanks, this helped a bit.
>
> > Clock lanes and data lanes may do slightly different things. It is
> > not
> > uncommon for the clock lane to remain in HS mode at all times, but
> > the
> > data lanes will almost always drop to LP-11. Device tree has a
> > clock-noncontinuous property to denote that the clock lane drops to
> > LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK.
> > There is no mention in the datasheet as to whether IMX290/327
> > produces
> > a continuous clock signal or not, and I can't say that I've checked.
> > Some receivers can get confused if the clock is continuous and they
> > miss the initial state change from LP to HS.
>
> Who drives the clock lane? The sensor or the host processor? If it is
> the sensor how do you support several sensors on the same bus?

The sensor.
With MIPI D-PHY a source (CSI2 or DSI) will always have one clock lane
and one or more data lanes.
Some SoC's allow you to reconfigure how lanes are used, but I've never
known a way to combine multiple sensors into a single CSI-2 receiver.

> > > > > > > I assume now that my current problem regarding settle time is
> > > somewhere
> else. I can currently only assume the escape clock is not correct,
> > > but
> there is pretty much no documentation at all on that topic.
> >
> > What platform are you working with?
>
> I'm working on an i.MX8QM based board. A public RM is available since
> several days [1]. The MIPI-CSI subsystem chapters are a bit lacking.
> Also I am aware that the ISI does some strange things with CSI input,
> see also [1], but that is a different issue.
>
> As far as I understand different documents, the escape clock is used
> during LP-11 and shall be from 10 to 20 MHz.
> The device tree configures some clock to 72MHz [2], but I do not know
> what's happening internally.
>
> I used the same approach for calculating the settle time as [3]. But I
> have to use a value slightly below the maximum rather than the average

I don't know the details of iMX8.
For BCM283x we supply an LP clock of 100MHz, and indeed that clocks
the block state machine for handling LP state transitions. LP mode is
relatively low speed in comparison to that - typically 10-20MHz if
memory serves correctly.

I vaguely remember one of the SoCs being very fussy about when the
sensor is left at LP-11. Looking at history, that appears to be
Renesas, but may apply to iMX8.
The IMX290 datasheet implies that the sensor will exit ULPS (LP-00)
state 20usecs after XCLR goes high, and remain in LP-11 standby mode
is cancelled (the clock lane then enters HS mode).

  Dave

> > There are registers in the IMX290/327 which configure the MIPI
> > timings
> > - 0x3445-0x3455. The values do differ based on the link frequency,
> > but
> > checking our driver to the datasheet they appear to be all correct.
>
> Agreed.
>
> [1] https://www.nxp.com/docs/en/reference-manual/IMX8QMRM.pdf
> [2]
> https://yhbt.net/lore/all/20200226151431.GY5379@paasikivi.fi.intel.com/T/
>
> [3]
> https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8-ss-img.dtsi?h=lf-5.10.y#n516
>
> [4]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/media/imx/imx8mq-mipi-csi2.c#n343
>
> --
> Mit freundlichen Grüßen
>
> i.A. Alexander Stein
> Entwicklung Standort
> Chemnitz
> Tel. +49 371 433151-0, Fax +49 371 433151-22
> Zwickauer Straße
> 173, 09116 Chemnitz
> mailto: Alexander.Stein@tq-group.com
>
> TQ-Systems GmbH
> Mühlstraße 2, Gut Delling, 82229 Seefeld
> Amtsgericht München, HRB 105018
> Sitz der Gesellschaft: Seefeld
> Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
> www.tq-group.com