MT9M131 on I.MX6DL CSI color issue

MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hello all,

I have a Phytec VM-009 camera based on MT9M131 connected to CSI0 of a I.MX6DL
based board running mainline 4.13.0 + custom devicetree. Its using the parallel
interface, 8 bit bus width on pins 12 to 19.

Basically it works pretty well apart from the really strange colors. I guess its
some YUV vs. RGB issue or similar. Here [1] is an example generated with the
following command.

gst-launch v4l2src device=/dev/video4 num-buffers=1 ! jpegenc ! filesink
location=capture1.jpeg

Apart from the colors everything is fine.
I'm pretty sure I have not seen such an effect before - what might be wrong here?

The current setup looks like this:

IF=UYVY2X8
GEOM="1280x1024"
media-ctl -l "'mt9m111 2-0048':0 -> 'ipu1_csi0_mux':4[1]"
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"

media-ctl -d /dev/media0 -v -V "'ipu1_csi0':2 [fmt:${IF}/${GEOM} field:none]"
media-ctl -d /dev/media0 -v -V "'ipu1_csi0 capture':0 [fmt:${IF}/${GEOM}
field:none]"
media-ctl -d /dev/media0 -v -V "'ipu1_csi0_mux':4 [fmt:${IF}/${GEOM} field: none]"
media-ctl -d /dev/media0 -v -V "'ipu1_csi0_mux':5 [fmt:${IF}/${GEOM} field: none]"
media-ctl -d /dev/media0 -v -V "'mt9m111 2-0048':0 [fmt:${IF}/${GEOM} field: none]"


Greetings

Florian

[1] http://www.kernelconcepts.de/~florian/capture1.jpeg

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

Re: MT9M131 on I.MX6DL CSI color issue

[Anatolij Gustschin]

On Fri, 12 Jan 2018 01:16:03 +0100
Florian Boor florian.boor@kernelconcepts.de wrote:
...
>Basically it works pretty well apart from the really strange colors. I guess its
>some YUV vs. RGB issue or similar. Here [1] is an example generated with the
>following command.
>
>gst-launch v4l2src device=/dev/video4 num-buffers=1 ! jpegenc ! filesink
>location=capture1.jpeg
>
>Apart from the colors everything is fine.
>I'm pretty sure I have not seen such an effect before - what might be wrong here?

You need conversion to RGB before JPEG encoding. Try with

 gst-launch v4l2src device=/dev/video4 num-buffers=1 ! \
            videoparse format=5 width=1280 height=1024 framerate=25/1 ! \
            jpegenc ! filesink location=capture1.jpeg

For "format" codes see gst-inspect-1.0 videoparse.

HTH,

Anatolij

Re: MT9M131 on I.MX6DL CSI color issue

[Anatolij Gustschin]

On Fri, 12 Jan 2018 10:58:40 +0100
Anatolij Gustschin agust@denx.de wrote:
...
> gst-launch v4l2src device=/dev/video4 num-buffers=1 ! \
>            videoparse format=5 width=1280 height=1024 framerate=25/1 ! \
>            jpegenc ! filesink location=capture1.jpeg

I forgot the videoconvert, sorry. Try

  gst-launch v4l2src device=/dev/video4 num-buffers=1 ! \
             filesink location=frame.raw

  gst-launch filesrc num-buffers=1 location=frame.raw ! \
             videoparse format=5 width=1280 height=1024 framerate=25/1 ! \
             videoconvert ! jpegenc ! filesink location=capture1.jpeg

Anatolij

Re: MT9M131 on I.MX6DL CSI color issue

[Nicolas Dufresne]

Le vendredi 12 janvier 2018 à 10:58 +0100, Anatolij Gustschin a écrit :
> On Fri, 12 Jan 2018 01:16:03 +0100
> Florian Boor florian.boor@kernelconcepts.de wrote:
> ...
> > Basically it works pretty well apart from the really strange
> > colors. I guess its
> > some YUV vs. RGB issue or similar. Here [1] is an example generated
> > with the
> > following command.
> > 
> > gst-launch v4l2src device=/dev/video4 num-buffers=1 ! jpegenc !
> > filesink
> > location=capture1.jpeg
> > 
> > Apart from the colors everything is fine.
> > I'm pretty sure I have not seen such an effect before - what might
> > be wrong here?
> 
> You need conversion to RGB before JPEG encoding. Try with
> 
>  gst-launch v4l2src device=/dev/video4 num-buffers=1 ! \
>             videoparse format=5 width=1280 height=1024 framerate=25/1
> ! \
>             jpegenc ! filesink location=capture1.jpeg
> 
> For "format" codes see gst-inspect-1.0 videoparse.

A properly written driver should never permit this.

Nicolas

Re: MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hello Anatolij,

many thanks for explaining. It changed something at least - see below.

On 12.01.2018 11:06, Anatolij Gustschin wrote:
> On Fri, 12 Jan 2018 10:58:40 +0100
> Anatolij Gustschin agust@denx.de wrote:
> ...
> I forgot the videoconvert, sorry. Try
> 
>   gst-launch v4l2src device=/dev/video4 num-buffers=1 ! \
>              filesink location=frame.raw
> 
>   gst-launch filesrc num-buffers=1 Lotion=frame.raw ! \
>              videoparse format=5 width=1280 height=1024 framerate=25/1 ! \
>              videoconvert ! jpegenc ! filesink location=capture1.jpeg

Capturing like this the colors turn a little bit less psychedelic green and
purple. Looks like this:
http://www.kernelconcepts.de/~florian/capture2.jpeg
The dark area is in fact a very bright one. So maybe the format I read from the
sensor is not exactly what it is supposed to be or similar...

Greetings

Florian

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

Re: MT9M131 on I.MX6DL CSI color issue

[Philipp Zabel]

Hi Florian,

On Fri, 2018-01-12 at 01:16 +0100, Florian Boor wrote:
> Hello all,
> 
> I have a Phytec VM-009 camera based on MT9M131 connected to CSI0 of a I.MX6DL
> based board running mainline 4.13.0 + custom devicetree. Its using the parallel
> interface, 8 bit bus width on pins 12 to 19.
> 
> Basically it works pretty well apart from the really strange colors. I guess its
> some YUV vs. RGB issue or similar. Here [1] is an example generated with the
> following command.
> 
> gst-launch v4l2src device=/dev/video4 num-buffers=1 ! jpegenc ! filesink
> location=capture1.jpeg
> 
> Apart from the colors everything is fine.
> I'm pretty sure I have not seen such an effect before - what might be wrong here?
> 
> The current setup looks like this:
> 
> IF=UYVY2X8
> GEOM="1280x1024"
> media-ctl -l "'mt9m111 2-0048':0 -> 'ipu1_csi0_mux':4[1]"
> media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> 
> media-ctl -d /dev/media0 -v -V "'ipu1_csi0':2 [fmt:${IF}/${GEOM} field:none]"
> media-ctl -d /dev/media0 -v -V "'ipu1_csi0 capture':0 [fmt:${IF}/${GEOM}
> field:none]"
> media-ctl -d /dev/media0 -v -V "'ipu1_csi0_mux':4 [fmt:${IF}/${GEOM} field: none]"
> media-ctl -d /dev/media0 -v -V "'ipu1_csi0_mux':5 [fmt:${IF}/${GEOM} field: none]"
> media-ctl -d /dev/media0 -v -V "'mt9m111 2-0048':0 [fmt:${IF}/${GEOM} field: none]"

media-ctl propagates video formats downstream, can you try reversing the
order?
Also, while the external format is UYVY2X8, internally the IPU only
supports AYUV32, so the last call should be 

media-ctl -d /dev/media0 -v -V "'ipu1_csi0':2 [fmt:AYUV32/${GEOM} field:none]"

not that it should make a difference.
And setting a format on 'ipu1_csi0 capture' is not necessary.

The new picture looks a little like there is 10-bit sensor data and only
the lower 8-bit arrive in memory, given the number of wraparounds.

Can you show the output of "media-ctl -p" (or "media-ctl --get-v4l2" for
each pad in the pipeline)?

media-ctl --get-v4l2 "'mt9m111 2-0048':0"
media-ctl --get-v4l2 "'ipu1_csi0_mux':4"
media-ctl --get-v4l2 "'ipu1_csi0_mux':5"
media-ctl --get-v4l2 "'ipu1_csi0':0"
media-ctl --get-v4l2 "'ipu1_csi0':2"

regards
Philipp

Re: MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hi Philipp,

On 15.01.2018 13:49, Philipp Zabel wrote:
> media-ctl propagates video formats downstream, can you try reversing the
> order?

I did but it does not make a difference.

> Also, while the external format is UYVY2X8, internally the IPU only
> supports AYUV32, so the last call should be 
> 
> media-ctl -d /dev/media0 -v -V "'ipu1_csi0':2 [fmt:AYUV32/${GEOM} field:none]"
> not that it should make a visible difference.
> And setting a format on 'ipu1_csi0 capture' is not necessary.

Changed this as well. What I do now is the following:

SF="UYVY2X8"
IF="AYUV32"
GEOM="1280x1024"

media-ctl -r
media-ctl -l "'mt9m111 2-0048':0 -> 'ipu1_csi0_mux':4[1]"
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"

media-ctl -d /dev/media0 -v -V "'mt9m111 2-0048':0 [fmt:${SF}/${GEOM} field: none]"
media-ctl -d /dev/media0 -v -V "'ipu1_csi0_mux':4 [fmt:${SF}/${GEOM} field: none]"
media-ctl -d /dev/media0 -v -V "'ipu1_csi0_mux':5 [fmt:${SF}/${GEOM} field: none]"
media-ctl -d /dev/media0 -v -V "'ipu1_csi0':2 [fmt:${IF}/${GEOM} field:none]"


> The new picture looks a little like there is 10-bit sensor data and only
> the lower 8-bit arrive in memory, given the number of wraparounds.

I will take a look at the sensor configuration. Maybe there is some issue or a
difference among all th MT9M1x1 semsors the driver does not support.

> Can you show the output of "media-ctl -p" (or "media-ctl --get-v4l2" for
> each pad in the pipeline)?

> media-ctl --get-v4l2 "'mt9m111 2-0048':0"
                [fmt:UYVY2X8/1280x1024 field:none]
                 crop.bounds:(26,8)/1280x1024
                 crop:(26,8)/1280x1024]
> media-ctl --get-v4l2 "'ipu1_csi0_mux':4"
                [fmt:UYVY2X8/1280x1024 field:none]
> media-ctl --get-v4l2 "'ipu1_csi0_mux':5"
                [fmt:UYVY2X8/1280x1024 field:none]
> media-ctl --get-v4l2 "'ipu1_csi0':0"
                [fmt:UYVY2X8/1280x1024 field:none
                 crop.bounds:(0,0)/1280x1024
                 crop:(0,0)/1280x1024
                 compose.bounds:(0,0)/1280x1024
                 compose:(0,0)/1280x1024]
> media-ctl --get-v4l2 "'ipu1_csi0':2"
                [fmt:AYUV32/1280x1024 field:none]

I uploaded the complete topology output from media-ctrl -p as well [1].

Greetings

Florian


[1] http://www.kernelconcepts.de/~florian/media-ctl-topology.txt



-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

Re: MT9M131 on I.MX6DL CSI color issue

[Daniel Glöckner]

Hi,

On Mon, Jan 15, 2018 at 11:40:32AM +0100, Florian Boor wrote:
> Capturing like this the colors turn a little bit less psychedelic green and
> purple. Looks like this:
> http://www.kernelconcepts.de/~florian/capture2.jpeg
> The dark area is in fact a very bright one. So maybe the format I read from the
> sensor is not exactly what it is supposed to be or similar...

May I suggest that you capture something of known colors, by pointing
the camera at a monitor displaying this?:
http://www.avsforum.com/photopost/data/2224298/0/04/04d8edc6_8bit_full_grad_color.png

The colors will of course be off by a bit, but it should still be
possible to guess how the RGB primaries were mangled.

Best regards,

  Daniel

Re: MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hi,

On 17.01.2018 11:31, Daniel Glöckner wrote:
> May I suggest that you capture something of known colors, by pointing
> the camera at a monitor displaying this?:
> http://www.avsforum.com/photopost/data/2224298/0/04/04d8edc6_8bit_full_grad_color.png
> 
> The colors will of course be off by a bit, but it should still be
> possible to guess how the RGB primaries were mangled.

yes that's a good idea. The result with original and camera view side by side
looks like this:

http://www.kernelconcepts.de/~florian/screenshot.png

Its not really obvious for me what is wrong but these wraparounds Philipp
mentioned are really nice to see within the bars.

Greetings

Florian

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

Re: MT9M131 on I.MX6DL CSI color issue

[Daniel Glöckner]

On Wed, Jan 17, 2018 at 01:58:42PM +0100, Florian Boor wrote:
> http://www.kernelconcepts.de/~florian/screenshot.png
> 
> Its not really obvious for me what is wrong but these wraparounds Philipp
> mentioned are really nice to see within the bars.

The vertical lines tell me that videoparse format=5 is wrong. Since the
U and V planes are so similar in the screenshot, it is most likely
format 4 or 19.

But that does not explain the wraparounds. Can you rule out that the
data lines have been connected in the wrong order?

Best regards,

  Daniel

Re: MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hello Daniel,

On 18.01.2018 05:03, Daniel Glöckner wrote:
> The vertical lines tell me that videoparse format=5 is wrong. Since the
> U and V planes are so similar in the screenshot, it is most likely
> format 4 or 19.

okay - it gets more colorful with these at least :-)
See [1] and it looks quite similar with both format 4 and 19.

> But that does not explain the wraparounds. Can you rule out that the
> data lines have been connected in the wrong order?

According to the schematics and camera documentation I have the order of the
data lines is correct. I checked one more time... but I'm not sure if the
configuration of the parallel camera input is perfectly right.

The hardware uses CSI0 data lines 12 to 19 and so I used the configuration from
the SabreLite board:

&ipu1_csi0_from_ipu1_csi0_mux {
	bus-width = <8>;
	data-shift = <12>; /* Lines 19:12 used */
...

So what I did is to change data-shift to 0. This should cause unused lines to be
sampled but it did not change anything.
So even assuming it is correct per default some of the settings might never make
it to the hardware interface configuration.

Greetings

Florian


[1] http://www.kernelconcepts.de/~florian/screenshot2.png

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

Re: MT9M131 on I.MX6DL CSI color issue

[Daniel Glöckner]

Hello Florian,

On Thu, Jan 18, 2018 at 05:31:43PM +0100, Florian Boor wrote:
> > But that does not explain the wraparounds. Can you rule out that the
> > data lines have been connected in the wrong order?
> 
> According to the schematics and camera documentation I have the order of the
> data lines is correct. I checked one more time... but I'm not sure if the
> configuration of the parallel camera input is perfectly right.
> 
> The hardware uses CSI0 data lines 12 to 19 and so I used the configuration
> from the SabreLite board:

The VM-009 has 10 data lines. Do you use a board designed by Phytec?
If not, did you connect the lower or the upper 8 data lines to the i.MX6?
Using the upper 8 data lines is correct.

I'm asking because the raw frames I asked for off list* contain only odd
bytes except for some null bytes. And for all components they exceed the
standard value range (Y 16-235, Cb/Cr 16-240).

Have you ever tried to capture images in one of the RGB formats?

Best regards,

  Daniel

*)
http://www.kernelconcepts.de/~florian/frame.raw
http://www.kernelconcepts.de/~florian/frame2.raw

Re: MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hello Daniel,

On 20.01.2018 16:02, Daniel Glöckner wrote:
> The VM-009 has 10 data lines. Do you use a board designed by Phytec?
> If not, did you connect the lower or the upper 8 data lines to the i.MX6?
> Using the upper 8 data lines is correct.

its a VM-009 but not a Phytec baseboard and luckily I am not responsible for the
hardware design :-)
I guess you found the issue: The lower data lines are connected. I guess the
designer of this hardware did not have the schematic of the camera and assumed
that the numeration of the data lines of the camera module is the same like on
the MT9M131 chip - which does not seem to be the case.

> I'm asking because the raw frames I asked for off list* contain only odd
> bytes except for some null bytes. And for all components they exceed the
> standard value range (Y 16-235, Cb/Cr 16-240).
> 
> Have you ever tried to capture images in one of the RGB formats?
Not so far but I just tried. The JPEG compressor does not seem to like it but I
can import the raw frame as ARGB into i.e. ImageJ and what I get is a RGB image
with quite a lot of overflows and wrong colors. It looks pretty much broken like
I would expect it to look missing the upper bits.

I'll see if I can fix the hardware now :-)

Greetings

Florian



-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.		Fax: +49 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613

Re: MT9M131 on I.MX6DL CSI color issue

[Florian Boor]

Hello all,

On 22.01.2018 15:52, Florian Boor wrote:
>> Have you ever tried to capture images in one of the RGB formats?
> Not so far but I just tried. The JPEG compressor does not seem to like it but I
> can import the raw frame as ARGB into i.e. ImageJ and what I get is a RGB image
> with quite a lot of overflows and wrong colors. It looks pretty much broken like
> I would expect it to look missing the upper bits.

RGB seems to be complicated. I did not find a working setup.

> I'll see if I can fix the hardware now :-)

But fixing the hardware almost got it right. The last remaining issue was the
pixel clock that required inverting. Now the images from the camera look
correct. (I'm currently testing with YUYV2X8 configuration and videoparse
"format=4" setting.)

Many thanks for all the help!

Greetings

Florian

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49(0) 271-771091-15
and the reality of tomorrow.		Fax: +49(0) 271-338857-29
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de
                                        http://www.kernelconcepts.de/en

kernel concepts GmbH
Hauptstraße 16
D-57074 Siegen
Geschäftsführer: Ole Reinhardt
HR Siegen, HR B 9613