* Sony IMX290 link frequency @ 2021-09-20 7:08 Stein, Alexander 2021-09-20 11:27 ` Dave Stevenson 0 siblings, 1 reply; 6+ messages in thread From: Stein, Alexander @ 2021-09-20 7:08 UTC (permalink / raw) To: mani, mchehab; +Cc: linux-media Hello, the imx290 driver specifies 2 tables of link frequencies, depending on number of lanes and the vertical resolution. These numbers vary from 148.5 MHz up to 445.5 MHz. I'm wondering where these number come from, it's not clear for me. Also shouldn't those numbers also depend in bits per pixel and the frames per second? I suspect that the returned link frequency is not correct which breaks things further down using v4l2_get_link_freq(). Best regards Alexander -- i.A. Alexander Stein Entwicklung Standort Chemnitz Tel. +49 371 433151-0, Fax +49 371 433151-22 Zwickauer Straße 173, 09116 Chemnitz mailto: Alexander.Stein@tq-group.com TQ-Systems GmbH Mühlstraße 2, Gut Delling, 82229 Seefeld Amtsgericht München, HRB 105018 Sitz der Gesellschaft: Seefeld Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider www.tq-group.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Sony IMX290 link frequency 2021-09-20 7:08 Sony IMX290 link frequency Stein, Alexander @ 2021-09-20 11:27 ` Dave Stevenson 2021-09-21 6:49 ` (EXT) " Stein, Alexander 0 siblings, 1 reply; 6+ messages in thread From: Dave Stevenson @ 2021-09-20 11:27 UTC (permalink / raw) To: Stein, Alexander; +Cc: mani, mchehab, linux-media Hi Alexander On Mon, 20 Sept 2021 at 08:15, Stein, Alexander <Alexander.Stein@tq-group.com> wrote: > > Hello, > > the imx290 driver specifies 2 tables of link frequencies, depending on > number of lanes and the vertical resolution. > These numbers vary from 148.5 MHz up to 445.5 MHz. > I'm wondering where these number come from, it's not clear for me. Also > shouldn't those numbers also depend in bits per pixel and the frames > per second? > I suspect that the returned link frequency is not correct which breaks > things further down using v4l2_get_link_freq(). I looked at imx290 and little brother imx327 a few months back to get it running with libcamera on the Pi. I have a couple of patches that ought to be upstreamed when I get a chance [1]. The Sony datasheet provides register configurations for specific modes (1080p, window cropping, and 720p) and frame rates, rather than a full description of what the registers do. It has two clock paths - one driving the pixel array, and one driving the MIPI core. There is a FIFO between the two, so they can run at different rates. My conclusion is that the pixel array always runs at the same pixel rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that is the result needed for vblank and hblank controls to work correctly in computing frame rate. The datasheet again contains fixed register settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX (register 0x301c/d), but they are all just linearly scaled values of each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed for 1080p and 720p modes, so maps to V4L2_CID_VBLANK. The MIPI PHY then runs at a link frequency sufficient to convey the desired pixels: - 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp - 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped modes up to 60fps, 10 or 12bpp. - 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or 12bpp This is controlled predominantly by INCKSEL1 & 2 (registers 0x305c/d). I verified the actual link frequencies used on a scope, and they are as described. 1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp just increases the per line blanking period on the CSI2 interface. AFAICT you could just always run at the 1782 Mbit/s rate with slightly increased idle time on the CSI2 bus for the 720p mode, but that isn't the way Sony have specified it. There is a further register to halve the link frequency again for max 25/30fps modes (0x3405), although doing so has limited benefit (it'd increase rolling shutter effects as it would increase the temporal difference between each line). I hope that makes things a little clearer. Indeed the current driver is slightly wrong, but only in relation to pixel rate, not link frequency. Dave [1] https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c > Best regards > Alexander > -- > i.A. Alexander Stein > Entwicklung Standort Chemnitz > Tel. +49 371 433151-0, Fax +49 371 433151-22 > Zwickauer Straße 173, 09116 Chemnitz > mailto: Alexander.Stein@tq-group.com > > TQ-Systems GmbH > Mühlstraße 2, Gut Delling, 82229 Seefeld > Amtsgericht München, HRB 105018 > Sitz der Gesellschaft: Seefeld > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > www.tq-group.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: (EXT) Re: Sony IMX290 link frequency 2021-09-20 11:27 ` Dave Stevenson @ 2021-09-21 6:49 ` Stein, Alexander 2021-09-21 10:41 ` Dave Stevenson 0 siblings, 1 reply; 6+ messages in thread From: Stein, Alexander @ 2021-09-21 6:49 UTC (permalink / raw) To: dave.stevenson; +Cc: mani, mchehab, linux-media Hello Dave, thank you for your verbose explanation. I think I understand a lot more now about those sensors. I'm using an IMX327 as well, but there doesn't seem to be a huge difference. On Montag, 20 Sept 2021 at 12:27 +0100, Dave Stevenson > It has two clock paths - one driving the pixel array, and one driving > the MIPI core. There is a FIFO between the two, so they can run at > different rates. > > My conclusion is that the pixel array always runs at the same pixel > rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that > is > the result needed for vblank and hblank controls to work correctly in > computing frame rate. The datasheet again contains fixed register > settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX > (register 0x301c/d), but they are all just linearly scaled values of > each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed > for > 1080p and 720p modes, so maps to V4L2_CID_VBLANK. Given that there are fixed numbers for 1080p/720p for VMAX and for 25,30,50,60 fps for HMAX shouldn't those controls be read-only? What's the benefit of being writable in [1]? > The MIPI PHY then runs at a link frequency sufficient to convey the > desired pixels: > - 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp > - 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped > modes up to 60fps, 10 or 12bpp. > - 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or > 12bpp > This is controlled predominantly by INCKSEL1 & 2 (registers > 0x305c/d). > I verified the actual link frequencies used on a scope, and they are > as described. > 1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp > just increases the per line blanking period on the CSI2 interface. Is there actual blanking on CSI2 interface, as in inserting "dummy" pixels in a row and dummy lines? Given that the spec is not available freely it's hard to understand what's actually happening on the interface. Given that the link frequency is fixed (in this case depending on window mode, input clock and number of lanes) I was wondering what's happening when the payload length changes (e.g. switching bpp or cropping). > AFAICT you could just always run at the 1782 Mbit/s rate with > slightly > increased idle time on the CSI2 bus for the 720p mode, but that isn't > the way Sony have specified it. > > There is a further register to halve the link frequency again for max > 25/30fps modes (0x3405), although doing so has limited benefit (it'd > increase rolling shutter effects as it would increase the temporal > difference between each line). > > I hope that makes things a little clearer. Indeed the current driver > is slightly wrong, but only in relation to pixel rate, not link > frequency. Thanks for confirmation. I meanwhile found the table providing the link frequencies. I assume now that my current problem regarding settle time is somewhere else. I can currently only assume the escape clock is not correct, but there is pretty much no documentation at all on that topic. Thanks and best regards Alexander [1] https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c -- Mit freundlichen Grüßen i.A. Alexander Stein Entwicklung Standort Chemnitz Tel. +49 371 433151-0, Fax +49 371 433151-22 Zwickauer Straße 173, 09116 Chemnitz mailto: Alexander.Stein@tq-group.com TQ-Systems GmbH Mühlstraße 2, Gut Delling, 82229 Seefeld Amtsgericht München, HRB 105018 Sitz der Gesellschaft: Seefeld Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider www.tq-group.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: (EXT) Re: Sony IMX290 link frequency 2021-09-21 6:49 ` (EXT) " Stein, Alexander @ 2021-09-21 10:41 ` Dave Stevenson 2021-09-23 13:44 ` (EXT) " Stein, Alexander 0 siblings, 1 reply; 6+ messages in thread From: Dave Stevenson @ 2021-09-21 10:41 UTC (permalink / raw) To: Stein, Alexander; +Cc: mani, mchehab, linux-media On Tue, 21 Sept 2021 at 07:49, Stein, Alexander <Alexander.Stein@tq-group.com> wrote: > > Hello Dave, > > thank you for your verbose explanation. I think I understand a lot more > now about those sensors. I'm using an IMX327 as well, but there doesn't > seem to be a huge difference. IMX290 adds a 1080p 100 or 120fps mode, but otherwise seems identical. The driver doesn't support that extra mode, so will drive IMX327 and IMX290 equally. > On Montag, 20 Sept 2021 at 12:27 +0100, Dave Stevenson > > It has two clock paths - one driving the pixel array, and one driving > > the MIPI core. There is a FIFO between the two, so they can run at > > different rates. > > > > My conclusion is that the pixel array always runs at the same pixel > > rate, whether 1080p, 720p, or cropped - 148.5MPix/s. Certainly that > > is > > the result needed for vblank and hblank controls to work correctly in > > computing frame rate. The datasheet again contains fixed register > > settings for 25, 30, 50, 60, 100, and 120fps by changing HMAX > > (register 0x301c/d), but they are all just linearly scaled values of > > each other, so it maps cleanly onto V4L2_CID_HBLANK. VMAX is fixed > > for > > 1080p and 720p modes, so maps to V4L2_CID_VBLANK. > > Given that there are fixed numbers for 1080p/720p for VMAX and for > 25,30,50,60 fps for HMAX shouldn't those controls be read-only? > What's the benefit of being writable in [1]? The datasheet lists fixed numbers, but any numbers work following the normal mode of operation for these controls. eg for the 1080p mode, VMAX is specified as 0x465 (1125 decimal). HMAX is then 25fps - 0x14a0 (5280). 5280*1125*25 = 148.5M 30fps - 0x1130 (4400). 4400*1125*30 = 148.5M 50fps - 0x0a50 (2640). 2640*1125*50 = 148.5M 60fps - 0x0898 (2200). 2200*1125*60 = 148.5M All values of HMAX inbetween those work fine, hence V4L2_CID_VBLANK can be a control with range from the minimum permitted (2200-height for 60fps) to the max the sensor will take. VMAX mapping to V4L2_CID_VBLANK could be a read only control, but it's functional in the sensor over the full range, so why not expose it. Frame rate control within libcamera typically varies V4L2_CID_VBLANK too, so that means it fits into the normal frameworks better. > > The MIPI PHY then runs at a link frequency sufficient to convey the > > desired pixels: > > - 3564 Mbit/s across 4 lanes for 1080p100/120 10bpp > > - 1782 Mbit/s split across 2 or 4 lanes for 1080p or window cropped > > modes up to 60fps, 10 or 12bpp. > > - 1188 Mbit/s split across 2 or 4 lanes for 720p up to 60fps, 10 or > > 12bpp > > This is controlled predominantly by INCKSEL1 & 2 (registers > > 0x305c/d). > > I verified the actual link frequencies used on a scope, and they are > > as described. > > 1782Mbit/s is sufficient for 1920x1080@60fps 12bpp. Cropping or 10bpp > > just increases the per line blanking period on the CSI2 interface. > > Is there actual blanking on CSI2 interface, as in inserting "dummy" > pixels in a row and dummy lines? Given that the spec is not available > freely it's hard to understand what's actually happening on the > interface. > Given that the link frequency is fixed (in this case depending on > window mode, input clock and number of lanes) I was wondering what's > happening when the payload length changes (e.g. switching bpp or > cropping). Whilst the spec is restricted, there are a few introductions that give a basic understanding, eg [1] MIPI D-PHY has mutliple lane states, but the main ones are low power mode (LP-11) and high speed mode (HS). For power saving it is very common for a sensor to switch to HS mode, send the line start short packet, line data long packet, line end short packet, and then drop back to LP-11 between lines. For the beginning and end of frames there are frame start and frame end short packets, but during the idle time the lanes will drop back to LP-11. See IMX219 datasheet [2] pages 47 & 48 for an example. When sending less data due to cropping or blanking periods, the data lanes are just in LP-11 for a longer period. Clock lanes and data lanes may do slightly different things. It is not uncommon for the clock lane to remain in HS mode at all times, but the data lanes will almost always drop to LP-11. Device tree has a clock-noncontinuous property to denote that the clock lane drops to LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK. There is no mention in the datasheet as to whether IMX290/327 produces a continuous clock signal or not, and I can't say that I've checked. Some receivers can get confused if the clock is continuous and they miss the initial state change from LP to HS. [1] https://download.tek.com/document/61W_25772_0_HR.pdf [2] https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Raspberry-Pi-V2-CMOS/master/RASPBERRY%20PI%20CAMERA%20V2%20DATASHEET%20IMX219PQH5_7.0.0_Datasheet_XXX.PDF > > AFAICT you could just always run at the 1782 Mbit/s rate with > > slightly > > increased idle time on the CSI2 bus for the 720p mode, but that isn't > > the way Sony have specified it. > > > > There is a further register to halve the link frequency again for max > > 25/30fps modes (0x3405), although doing so has limited benefit (it'd > > increase rolling shutter effects as it would increase the temporal > > difference between each line). > > > > I hope that makes things a little clearer. Indeed the current driver > > is slightly wrong, but only in relation to pixel rate, not link > > frequency. > > Thanks for confirmation. I meanwhile found the table providing the link > frequencies. > I assume now that my current problem regarding settle time is somewhere > else. I can currently only assume the escape clock is not correct, but > there is pretty much no documentation at all on that topic. What platform are you working with? There are registers in the IMX290/327 which configure the MIPI timings - 0x3445-0x3455. The values do differ based on the link frequency, but checking our driver to the datasheet they appear to be all correct. Hope that helps. Dave > Thanks and best regards > Alexander > > [1] > https://github.com/raspberrypi/linux/commits/rpi-5.10.y/drivers/media/i2c/imx290.c > > -- > Mit freundlichen Grüßen > > i.A. Alexander Stein > Entwicklung Standort Chemnitz > Tel. +49 371 433151-0, Fax +49 371 433151-22 > Zwickauer Straße 173, 09116 Chemnitz > mailto: Alexander.Stein@tq-group.com > > TQ-Systems GmbH > Mühlstraße 2, Gut Delling, 82229 Seefeld > Amtsgericht München, HRB 105018 > Sitz der Gesellschaft: Seefeld > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > www.tq-group.com > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: (EXT) Re: (EXT) Re: Sony IMX290 link frequency 2021-09-21 10:41 ` Dave Stevenson @ 2021-09-23 13:44 ` Stein, Alexander 2021-09-27 13:56 ` Dave Stevenson 0 siblings, 1 reply; 6+ messages in thread From: Stein, Alexander @ 2021-09-23 13:44 UTC (permalink / raw) To: dave.stevenson; +Cc: mani, mchehab, linux-media Hello Dave, On Tue, 21 Sept 2021 at 11:41 +0100, Dave Stevenson: > Whilst the spec is restricted, there are a few introductions that > give > a basic understanding, eg [1] Thanks, this helped a bit. > Clock lanes and data lanes may do slightly different things. It is > not > uncommon for the clock lane to remain in HS mode at all times, but > the > data lanes will almost always drop to LP-11. Device tree has a > clock-noncontinuous property to denote that the clock lane drops to > LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK. > There is no mention in the datasheet as to whether IMX290/327 > produces > a continuous clock signal or not, and I can't say that I've checked. > Some receivers can get confused if the clock is continuous and they > miss the initial state change from LP to HS. Who drives the clock lane? The sensor or the host processor? If it is the sensor how do you support several sensors on the same bus? > > > > > > I assume now that my current problem regarding settle time is > > somewhere else. I can currently only assume the escape clock is not correct, > > but there is pretty much no documentation at all on that topic. > > What platform are you working with? I'm working on an i.MX8QM based board. A public RM is available since several days [1]. The MIPI-CSI subsystem chapters are a bit lacking. Also I am aware that the ISI does some strange things with CSI input, see also [1], but that is a different issue. As far as I understand different documents, the escape clock is used during LP-11 and shall be from 10 to 20 MHz. The device tree configures some clock to 72MHz [2], but I do not know what's happening internally. I used the same approach for calculating the settle time as [3]. But I have to use a value slightly below the maximum rather than the average > There are registers in the IMX290/327 which configure the MIPI > timings > - 0x3445-0x3455. The values do differ based on the link frequency, > but > checking our driver to the datasheet they appear to be all correct. Agreed. [1] https://www.nxp.com/docs/en/reference-manual/IMX8QMRM.pdf [2] https://yhbt.net/lore/all/20200226151431.GY5379@paasikivi.fi.intel.com/T/ [3] https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8-ss-img.dtsi?h=lf-5.10.y#n516 [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/media/imx/imx8mq-mipi-csi2.c#n343 -- Mit freundlichen Grüßen i.A. Alexander Stein Entwicklung Standort Chemnitz Tel. +49 371 433151-0, Fax +49 371 433151-22 Zwickauer Straße 173, 09116 Chemnitz mailto: Alexander.Stein@tq-group.com TQ-Systems GmbH Mühlstraße 2, Gut Delling, 82229 Seefeld Amtsgericht München, HRB 105018 Sitz der Gesellschaft: Seefeld Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider www.tq-group.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: (EXT) Re: (EXT) Re: Sony IMX290 link frequency 2021-09-23 13:44 ` (EXT) " Stein, Alexander @ 2021-09-27 13:56 ` Dave Stevenson 0 siblings, 0 replies; 6+ messages in thread From: Dave Stevenson @ 2021-09-27 13:56 UTC (permalink / raw) To: Stein, Alexander; +Cc: mani, mchehab, linux-media On Thu, 23 Sept 2021 at 14:44, Stein, Alexander <Alexander.Stein@tq-group.com> wrote: > > Hello Dave, > > On Tue, 21 Sept 2021 at 11:41 +0100, Dave Stevenson: > > Whilst the spec is restricted, there are a few introductions that > > give > > a basic understanding, eg [1] > > Thanks, this helped a bit. > > > Clock lanes and data lanes may do slightly different things. It is > > not > > uncommon for the clock lane to remain in HS mode at all times, but > > the > > data lanes will almost always drop to LP-11. Device tree has a > > clock-noncontinuous property to denote that the clock lane drops to > > LP, which maps to the V4L2 flag V4L2_MBUS_CSI2_NONCONTINUOUS_CLOCK. > > There is no mention in the datasheet as to whether IMX290/327 > > produces > > a continuous clock signal or not, and I can't say that I've checked. > > Some receivers can get confused if the clock is continuous and they > > miss the initial state change from LP to HS. > > Who drives the clock lane? The sensor or the host processor? If it is > the sensor how do you support several sensors on the same bus? The sensor. With MIPI D-PHY a source (CSI2 or DSI) will always have one clock lane and one or more data lanes. Some SoC's allow you to reconfigure how lanes are used, but I've never known a way to combine multiple sensors into a single CSI-2 receiver. > > > > > > > I assume now that my current problem regarding settle time is > > > somewhere > else. I can currently only assume the escape clock is not correct, > > > but > there is pretty much no documentation at all on that topic. > > > > What platform are you working with? > > I'm working on an i.MX8QM based board. A public RM is available since > several days [1]. The MIPI-CSI subsystem chapters are a bit lacking. > Also I am aware that the ISI does some strange things with CSI input, > see also [1], but that is a different issue. > > As far as I understand different documents, the escape clock is used > during LP-11 and shall be from 10 to 20 MHz. > The device tree configures some clock to 72MHz [2], but I do not know > what's happening internally. > > I used the same approach for calculating the settle time as [3]. But I > have to use a value slightly below the maximum rather than the average I don't know the details of iMX8. For BCM283x we supply an LP clock of 100MHz, and indeed that clocks the block state machine for handling LP state transitions. LP mode is relatively low speed in comparison to that - typically 10-20MHz if memory serves correctly. I vaguely remember one of the SoCs being very fussy about when the sensor is left at LP-11. Looking at history, that appears to be Renesas, but may apply to iMX8. The IMX290 datasheet implies that the sensor will exit ULPS (LP-00) state 20usecs after XCLR goes high, and remain in LP-11 standby mode is cancelled (the clock lane then enters HS mode). Dave > > There are registers in the IMX290/327 which configure the MIPI > > timings > > - 0x3445-0x3455. The values do differ based on the link frequency, > > but > > checking our driver to the datasheet they appear to be all correct. > > Agreed. > > [1] https://www.nxp.com/docs/en/reference-manual/IMX8QMRM.pdf > [2] > https://yhbt.net/lore/all/20200226151431.GY5379@paasikivi.fi.intel.com/T/ > > [3] > https://source.codeaurora.org/external/imx/linux-imx/tree/arch/arm64/boot/dts/freescale/imx8-ss-img.dtsi?h=lf-5.10.y#n516 > > [4] > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/media/imx/imx8mq-mipi-csi2.c#n343 > > -- > Mit freundlichen Grüßen > > i.A. Alexander Stein > Entwicklung Standort > Chemnitz > Tel. +49 371 433151-0, Fax +49 371 433151-22 > Zwickauer Straße > 173, 09116 Chemnitz > mailto: Alexander.Stein@tq-group.com > > TQ-Systems GmbH > Mühlstraße 2, Gut Delling, 82229 Seefeld > Amtsgericht München, HRB 105018 > Sitz der Gesellschaft: Seefeld > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > www.tq-group.com ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-27 13:56 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-20 7:08 Sony IMX290 link frequency Stein, Alexander 2021-09-20 11:27 ` Dave Stevenson 2021-09-21 6:49 ` (EXT) " Stein, Alexander 2021-09-21 10:41 ` Dave Stevenson 2021-09-23 13:44 ` (EXT) " Stein, Alexander 2021-09-27 13:56 ` Dave Stevenson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.