imx-media: MT9P031 Capture issues on IMX6

imx-media: MT9P031 Capture issues on IMX6

[Ibtsam Ul-Haq]

Greetings everyone,
I am using Linux 4.14.31 on an IMX6 platform, with an MT9P031 sensor
attached to the ipu1_csi0 (parallel).
My Gstreamer version is 1.14.0 and v4l-utils version is 1.14.2.
The problem is that I am unable to set up a capture pipeline.

Even the simplest capture pipeline such as:

gst-launch-1.0 v4l2src device=/dev/video4 ! fakesink

returns the following error:
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Internal data stream error.
Additional debug info:
../../../../gstreamer-1.14.0/libs/gst/base/gstbasesrc.c(3055):
gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.

I get the same error on any pipeline involving v4l2src.

I have set up the media entity links using:
media-ctl -l "'mt9p031 0-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"

And I configure the pads using:
media-ctl -V "'mt9p031 0-0048':0 [fmt:SGRBG8/640x480 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:SGRBG8/640x480 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:SGRBG8/640x480 field:none]"

And I do not get any errors from these commands.

The output of "v4l2-ctl --all -d /dev/video4" is:

Driver Info (not using libv4l2):
        Driver name   : imx-media-captu
        Card type     : imx-media-capture
        Bus info      : platform:ipu1_csi0
        Driver version: 4.14.31
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Priority: 2
Format Video Capture:
        Width/Height      : 656/486
        Pixel Format      : 'GR16'
        Field             : None
        Bytes per Line    : 1312
        Size Image        : 637632
        Colorspace        : sRGB
        Transfer Function : sRGB
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Flags             : ee8abea8
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0

User Controls

                       exposure 0x00980911 (int)    : min=1
max=1048575 step=1 default=1943 value=1943
                           gain 0x00980913 (int)    : min=8 max=1024
step=1 default=8 value=8
                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0
                       blc_auto 0x00981902 (bool)   : default=1 value=1
               blc_target_level 0x00981903 (int)    : min=0 max=4095
step=1 default=168 value=168
              blc_analog_offset 0x00981904 (int)    : min=-255 max=255
step=1 default=32 value=32
             blc_digital_offset 0x00981905 (int)    : min=-2048
max=2047 step=1 default=40 value=40
                     fim_enable 0x00981990 (bool)   : default=0 value=0
                fim_num_average 0x00981991 (int)    : min=1 max=64
step=1 default=8 value=8
              fim_tolerance_min 0x00981992 (int)    : min=2 max=200
step=1 default=50 value=50
              fim_tolerance_max 0x00981993 (int)    : min=0 max=500
step=1 default=0 value=0
                   fim_num_skip 0x00981994 (int)    : min=0 max=256
step=1 default=2 value=2
         fim_input_capture_edge 0x00981995 (int)    : min=0 max=3
step=1 default=0 value=0
      fim_input_capture_channel 0x00981996 (int)    : min=0 max=1
step=1 default=0 value=0

Image Processing Controls

                     pixel_rate 0x009f0902 (int64)  : min=54000000
max=54000000 step=1 default=54000000 value=54000000 flags=read-only
                   test_pattern 0x009f0903 (menu)   : min=0 max=9
default=0 value=0


What confuses me here is that the Pixel Format shown by v4l2-ctl is
'GR16', which is not what I expected. And it seems like the media-ctl
pad configuration commands are unable to change the Pixel Format, even
though they do not return any errors.

I would really appreciate if anyone could help in debugging this issue.

Thanks and best regards,
Ibtsam Haq

Re: imx-media: MT9P031 Capture issues on IMX6

[Philipp Zabel]

Hi Ibtsam,

On Thu, 2018-04-12 at 16:00 +0200, Ibtsam Ul-Haq wrote:
> Greetings everyone,
> I am using Linux 4.14.31 on an IMX6 platform, with an MT9P031 sensor
> attached to the ipu1_csi0 (parallel).
> My Gstreamer version is 1.14.0 and v4l-utils version is 1.14.2.
> The problem is that I am unable to set up a capture pipeline.
> 
> Even the simplest capture pipeline such as:
> 
> gst-launch-1.0 v4l2src device=/dev/video4 ! fakesink
> 
> returns the following error:
> ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> Internal data stream error.
> Additional debug info:
> ../../../../gstreamer-1.14.0/libs/gst/base/gstbasesrc.c(3055):
> gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> streaming stopped, reason not-negotiated (-4)
> ERROR: pipeline doesn't want to preroll.
> 
> I get the same error on any pipeline involving v4l2src.
> 
> I have set up the media entity links using:
> media-ctl -l "'mt9p031 0-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
>
> And I configure the pads using:
> media-ctl -V "'mt9p031 0-0048':0 [fmt:SGRBG8/640x480 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:SGRBG8/640x480 field:none]"
> media-ctl -V "'ipu1_csi0':2 [fmt:SGRBG8/640x480 field:none]"

What is the actual format that all pads are configured to? I found it
helpful to double check the whole pipeline after configuring pads (or
use media-ctl -v):

media-ctl --get-v4l2 "'mt9p031 0-0048':0"
media-ctl --get-v4l2 "'ipu1_csi0_mux':1"
media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
media-ctl --get-v4l2 "'ipu1_csi0':0"
media-ctl --get-v4l2 "'ipu1_csi0':2"

I assume that because mt9p031 only supports SGRBG12_1X12, that's what
will be propagated down the pipeline to the CSI, which will then expand
it to SGRBG16.

I suppose we should allow, at least for parallel input, to let the CSI
'convert' 12-bit input formats to 8-bit output formats by just ignoring
the LSBs.
Another possibility would be to just allow SGRBG12_1X12 -> SGRBG8_1X8
mbus codes in link_validate. Actually, does your hardware have 12 data
lines connected between sensor and i.MX6, or just 8 ?

> And I do not get any errors from these commands.

That's because of the way the V4L2 API works, unsupported formats are
adjusted by the drivers:

https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/vidioc-subdev-g-fmt.html#description

[...]
> What confuses me here is that the Pixel Format shown by v4l2-ctl is
> 'GR16', which is not what I expected. And it seems like the media-ctl
> pad configuration commands are unable to change the Pixel Format, even
> though they do not return any errors.

Whenever you media-ctl -V on a source pad, it will also try to set the
connected sink pad to the same format. And for subdevices with sink and
source pads, the source pads usually mirror (or somehow depend on) the
format of the sink pad. Due to the way the V4L2 APIs correct your input
to possible values, calling media-ctl -V on all source pads of the
pipeline in downstream direction essentially propagates the sensor
source pad format.

regards
Philipp

Re: imx-media: MT9P031 Capture issues on IMX6

[Ibtsam Ul-Haq]

Hi Philipp,

Thanks a lot for your response.

On Fri, Apr 13, 2018 at 4:18 PM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> Hi Ibtsam,
>
> On Thu, 2018-04-12 at 16:00 +0200, Ibtsam Ul-Haq wrote:
>> Greetings everyone,
>> I am using Linux 4.14.31 on an IMX6 platform, with an MT9P031 sensor
>> attached to the ipu1_csi0 (parallel).
>> My Gstreamer version is 1.14.0 and v4l-utils version is 1.14.2.
>> The problem is that I am unable to set up a capture pipeline.
>>
>> Even the simplest capture pipeline such as:
>>
>> gst-launch-1.0 v4l2src device=/dev/video4 ! fakesink
>>
>> returns the following error:
>> ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
>> Internal data stream error.
>> Additional debug info:
>> ../../../../gstreamer-1.14.0/libs/gst/base/gstbasesrc.c(3055):
>> gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
>> streaming stopped, reason not-negotiated (-4)
>> ERROR: pipeline doesn't want to preroll.
>>
>> I get the same error on any pipeline involving v4l2src.
>>
>> I have set up the media entity links using:
>> media-ctl -l "'mt9p031 0-0048':0 -> 'ipu1_csi0_mux':1[1]"
>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
>>
>> And I configure the pads using:
>> media-ctl -V "'mt9p031 0-0048':0 [fmt:SGRBG8/640x480 field:none]"
>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:SGRBG8/640x480 field:none]"
>> media-ctl -V "'ipu1_csi0':2 [fmt:SGRBG8/640x480 field:none]"
>
> What is the actual format that all pads are configured to? I found it
> helpful to double check the whole pipeline after configuring pads (or
> use media-ctl -v):
>
> media-ctl --get-v4l2 "'mt9p031 0-0048':0"
> media-ctl --get-v4l2 "'ipu1_csi0_mux':1"
> media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
> media-ctl --get-v4l2 "'ipu1_csi0':0"
> media-ctl --get-v4l2 "'ipu1_csi0':2"
>


Here is what I get after I have executed the commands to configure all
the pads to SGRBG8/640x480:

:~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
[fmt:SGRBG12_1X12/648x486 field:none colorspace:srgb

:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':1"
[fmt:SGRBG12_1X12/648x486 field:none colorspace:srgb]

:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
[fmt:SGRBG12_1X12/648x486 field:none colorspace:srgb]

:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
[fmt:SGRBG12_1X12/656x486@1/30 field:none colorspace:srgb xfer:srgb
ycbcr:601 quantization:full-range
crop.bounds:(0,0)/656x486
crop:(0,0)/656x486
compose.bounds:(0,0)/656x486
compose:(0,0)/656x486]

:~# media-ctl --get-v4l2 "'ipu1_csi0':2"
[fmt:SGRBG12_1X12/656x486@1/30 field:none colorspace:srgb xfer:srgb
ycbcr:601 quantization:full-range]


> I assume that because mt9p031 only supports SGRBG12_1X12, that's what
> will be propagated down the pipeline to the CSI, which will then expand
> it to SGRBG16.
>


This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?
To be honest, I had not seen GR16 as FourCC before.
And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
do not know this FourCC:
v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
Unsupported fourcc 0x36315247 GR16

Is there a way we can get by this?


> I suppose we should allow, at least for parallel input, to let the CSI
> 'convert' 12-bit input formats to 8-bit output formats by just ignoring
> the LSBs.
> Another possibility would be to just allow SGRBG12_1X12 -> SGRBG8_1X8
> mbus codes in link_validate. Actually, does your hardware have 12 data
> lines connected between sensor and i.MX6, or just 8 ?
>

Currently we have 8. Although we can populate extra resistors to
connect the remaining 4.
The device tree is set for 8 lines currently.


>> And I do not get any errors from these commands.
>
> That's because of the way the V4L2 API works, unsupported formats are
> adjusted by the drivers:
>
> https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/vidioc-subdev-g-fmt.html#description
>
> [...]
>> What confuses me here is that the Pixel Format shown by v4l2-ctl is
>> 'GR16', which is not what I expected. And it seems like the media-ctl
>> pad configuration commands are unable to change the Pixel Format, even
>> though they do not return any errors.
>
> Whenever you media-ctl -V on a source pad, it will also try to set the
> connected sink pad to the same format. And for subdevices with sink and
> source pads, the source pads usually mirror (or somehow depend on) the
> format of the sink pad. Due to the way the V4L2 APIs correct your input
> to possible values, calling media-ctl -V on all source pads of the
> pipeline in downstream direction essentially propagates the sensor
> source pad format.
>

Thanks for the nice explanation. This is valuable learning for me as I
am just a beginner in this stuff.

> regards
> Philipp

Best regards,
Ibtsam Haq

Re: imx-media: MT9P031 Capture issues on IMX6

[Philipp Zabel]

On Mon, 2018-04-16 at 09:54 +0200, Ibtsam Ul-Haq wrote:
[...]
> This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?

Yes, see Documentation/media/uapi/v4l/pixfmt-srggb16.rst:
https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-srggb16.html

> To be honest, I had not seen GR16 as FourCC before.
> And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
> do not know this FourCC:
> v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
> Unsupported fourcc 0x36315247 GR16

The GStreamer V4L2 elements currently only support 8-bit per component
Bayer formats.

> Is there a way we can get by this?

There's two ways to handle this correctly, IMHO. One would be adding
SGRBG8_1X8 support to the mt9p031 driver. This is the correct way if the
device tree is configured for 8-bit parallel and there are only 8 data
lines connected between camera and SoC. As a quick hack, I think you
could just:

  sed "s/MEDIA_BUS_FMT_SGRBG12_1X12/MEDIA_BUS_FMT_SGRBG8_1X8/" -i drivers/media/i2c/mt9p031.c

The other would be to connect all 12 data lines, configure the device
tree with 12 bit data width, and extend the imx-media CSI subdevice
driver to allow setting SGRBG12_1X12 on the sink pad and SGRBG8_1X8 on
the source pad at the same time (and then just internally configuring
the hardware to 8-bit, ignoring the 4 LSB). That would be a bit more
involved.

Another possiblity would be to replace v4l2_subdev_link_validate() in
drivers/media/v4l2-core/v4l2-subdev.c with a variant that allows
source_fmt->format.code != sink_fmt->format.code in case the source
format can be turned into the sink format by just dropping LSB for one
of the links.

regards
Philipp

Re: imx-media: MT9P031 Capture issues on IMX6

[Ibtsam Ul-Haq]

On Mon, Apr 16, 2018 at 11:30 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> On Mon, 2018-04-16 at 09:54 +0200, Ibtsam Ul-Haq wrote:
> [...]
>> This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?
>
> Yes, see Documentation/media/uapi/v4l/pixfmt-srggb16.rst:
> https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-srggb16.html
>
>> To be honest, I had not seen GR16 as FourCC before.
>> And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
>> do not know this FourCC:
>> v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
>> Unsupported fourcc 0x36315247 GR16
>
> The GStreamer V4L2 elements currently only support 8-bit per component
> Bayer formats.
>
>> Is there a way we can get by this?
>
> There's two ways to handle this correctly, IMHO. One would be adding
> SGRBG8_1X8 support to the mt9p031 driver. This is the correct way if the
> device tree is configured for 8-bit parallel and there are only 8 data
> lines connected between camera and SoC. As a quick hack, I think you
> could just:
>
>   sed "s/MEDIA_BUS_FMT_SGRBG12_1X12/MEDIA_BUS_FMT_SGRBG8_1X8/" -i drivers/media/i2c/mt9p031.c
>


I tried that and it works well for my case. All pads in the pipeline
now report their format as SGRBG8_1X8.

However, now I am getting a Broken Pipe error from STREAMON when I try
to run the pipeline:
v4l2bufferpool gstv4l2bufferpool.c:677:gst_v4l2_buffer_pool_streamon:<v4l2src0:pool:src>
[00m
error with STREAMON 32 (Broken pipe)

I also get a corresponding error on the dmesg:
[ 1398.723524] ipu1_csi0: pipeline start failed with -32

The pipeline I intend to use is:
gst-launch-1.0 v4l2src io-mode=dmabuf device=/dev/video4 ! bayer2rgb !
videoconvert ! v4l2h264enc output-io-mode=dmabuf-import ! rtph264pay !
udpsink


> The other would be to connect all 12 data lines, configure the device
> tree with 12 bit data width, and extend the imx-media CSI subdevice
> driver to allow setting SGRBG12_1X12 on the sink pad and SGRBG8_1X8 on
> the source pad at the same time (and then just internally configuring
> the hardware to 8-bit, ignoring the 4 LSB). That would be a bit more
> involved.
>
> Another possiblity would be to replace v4l2_subdev_link_validate() in
> drivers/media/v4l2-core/v4l2-subdev.c with a variant that allows
> source_fmt->format.code != sink_fmt->format.code in case the source
> format can be turned into the sink format by just dropping LSB for one
> of the links.
>
> regards
> Philipp

Best regards,
Ibtsam Haq

Re: imx-media: MT9P031 Capture issues on IMX6

[Philipp Zabel]

Hi Ibtsam,

On Tue, 2018-04-17 at 09:26 +0200, Ibtsam Ul-Haq wrote:
> On Mon, Apr 16, 2018 at 11:30 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> > On Mon, 2018-04-16 at 09:54 +0200, Ibtsam Ul-Haq wrote:
> > [...]
> > > This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?
> > 
> > Yes, see Documentation/media/uapi/v4l/pixfmt-srggb16.rst:
> > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-srggb16.html
> > 
> > > To be honest, I had not seen GR16 as FourCC before.
> > > And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
> > > do not know this FourCC:
> > > v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
> > > Unsupported fourcc 0x36315247 GR16
> > 
> > The GStreamer V4L2 elements currently only support 8-bit per component
> > Bayer formats.
> > 
> > > Is there a way we can get by this?
> > 
> > There's two ways to handle this correctly, IMHO. One would be adding
> > SGRBG8_1X8 support to the mt9p031 driver. This is the correct way if the
> > device tree is configured for 8-bit parallel and there are only 8 data
> > lines connected between camera and SoC. As a quick hack, I think you
> > could just:
> > 
> >   sed "s/MEDIA_BUS_FMT_SGRBG12_1X12/MEDIA_BUS_FMT_SGRBG8_1X8/" -i drivers/media/i2c/mt9p031.c
> > 
> 
> 
> I tried that and it works well for my case. All pads in the pipeline
> now report their format as SGRBG8_1X8.
>
> However, now I am getting a Broken Pipe error from STREAMON when I try
> to run the pipeline:
> v4l2bufferpool gstv4l2bufferpool.c:677:gst_v4l2_buffer_pool_streamon:<v4l2src0:pool:src>
> [00m
> error with STREAMON 32 (Broken pipe)

What about format width and height, are they the same throughout the
pipeline? It didn't look that way in your last mail.

> 
> I also get a corresponding error on the dmesg:
> [ 1398.723524] ipu1_csi0: pipeline start failed with -32

That is the same issue. I assume link validation fails with -EPIPE
because frame dimensions differ between some connected source and sink
pad. If you have dynamic debug enabled, you can verify this by enabling
the relevant debug printks in drivers/media:

  echo "func __media_pipeline_start +p" > /sys/kernel/debug/dynamic_debug/control

This will show exactly which link causes the failure:

  imx-media capture-subsystem: link validation failed for 'ipu1_csi0_mux':2 -> 'ipu1_csi0':0, error -32
  ipu1_csi0: pipeline start failed with -32

Here width and height were different between connected source and sink
pad:

  media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
		[fmt:UYVY8_1X16/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]

  media-ctl --get-v4l2 "'ipu1_csi0':0"
		[fmt:UYVY8_1X16/1920x1080@1/60 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
		 crop.bounds:(0,0)/1920x1080
		 crop:(0,0)/1920x1080
		 compose.bounds:(0,0)/1920x1080
		 compose:(0,0)/1920x1080]

If you look at v4l2_subdev_link_validate_default in 
drivers/media/v4l2-core/v4l2-subdev.c, you can see that besides the
format code, it also checks width, height, and field setting.

regards
Philipp

Re: imx-media: MT9P031 Capture issues on IMX6

[Ibtsam Ul-Haq]

On Tue, Apr 17, 2018 at 10:34 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> Hi Ibtsam,
>
> On Tue, 2018-04-17 at 09:26 +0200, Ibtsam Ul-Haq wrote:
>> On Mon, Apr 16, 2018 at 11:30 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
>> > On Mon, 2018-04-16 at 09:54 +0200, Ibtsam Ul-Haq wrote:
>> > [...]
>> > > This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?
>> >
>> > Yes, see Documentation/media/uapi/v4l/pixfmt-srggb16.rst:
>> > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-srggb16.html
>> >
>> > > To be honest, I had not seen GR16 as FourCC before.
>> > > And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
>> > > do not know this FourCC:
>> > > v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
>> > > Unsupported fourcc 0x36315247 GR16
>> >
>> > The GStreamer V4L2 elements currently only support 8-bit per component
>> > Bayer formats.
>> >
>> > > Is there a way we can get by this?
>> >
>> > There's two ways to handle this correctly, IMHO. One would be adding
>> > SGRBG8_1X8 support to the mt9p031 driver. This is the correct way if the
>> > device tree is configured for 8-bit parallel and there are only 8 data
>> > lines connected between camera and SoC. As a quick hack, I think you
>> > could just:
>> >
>> >   sed "s/MEDIA_BUS_FMT_SGRBG12_1X12/MEDIA_BUS_FMT_SGRBG8_1X8/" -i drivers/media/i2c/mt9p031.c
>> >
>>
>>
>> I tried that and it works well for my case. All pads in the pipeline
>> now report their format as SGRBG8_1X8.
>>
>> However, now I am getting a Broken Pipe error from STREAMON when I try
>> to run the pipeline:
>> v4l2bufferpool gstv4l2bufferpool.c:677:gst_v4l2_buffer_pool_streamon:<v4l2src0:pool:src>
>> [00m
>> error with STREAMON 32 (Broken pipe)
>
> What about format width and height, are they the same throughout the
> pipeline? It didn't look that way in your last mail.
>

Indeed they were not the same. That was probably because the default
window width and height were 2592x1944.
So when I tried to set resolution to 640x480, it actually became
648x486, and the ipu1_csi0 expanded it to 656x486.

I have that changed now, so the pads now look to have the same width and height:

:~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
                [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb
                 crop:(16,54)/2560x1920]

:~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
                [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb
                 crop:(16,54)/2560x1920]

:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':1"
                [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb]

:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
                [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb]

:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
                [fmt:SGRBG8_1X8/640x480@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/640x480
                 compose.bounds:(0,0)/640x480
                 compose:(0,0)/640x480]

:~# media-ctl --get-v4l2 "'ipu1_csi0':2"
                [fmt:SGRBG8_1X8/640x480@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]

But now I am getting:
gst_v4l2_buffer_pool_poll (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
poll error 1: Invalid argument (22)

This is accompanied by dmesg errors:
[ 8056.756841] alloc_contig_range: [80400, 80496) PFNs busy
[ 8057.833501] ipu1_csi0: EOF timeout
[ 8058.953717] ipu1_csi0: wait last EOF timeout

It looks like there is some problem with memory allocation.


>>
>> I also get a corresponding error on the dmesg:
>> [ 1398.723524] ipu1_csi0: pipeline start failed with -32
>
> That is the same issue. I assume link validation fails with -EPIPE
> because frame dimensions differ between some connected source and sink
> pad. If you have dynamic debug enabled, you can verify this by enabling
> the relevant debug printks in drivers/media:
>
>   echo "func __media_pipeline_start +p" > /sys/kernel/debug/dynamic_debug/control
>
> This will show exactly which link causes the failure:
>
>   imx-media capture-subsystem: link validation failed for 'ipu1_csi0_mux':2 -> 'ipu1_csi0':0, error -32
>   ipu1_csi0: pipeline start failed with -32
>
> Here width and height were different between connected source and sink
> pad:
>
>   media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>                 [fmt:UYVY8_1X16/640x480 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
>
>   media-ctl --get-v4l2 "'ipu1_csi0':0"
>                 [fmt:UYVY8_1X16/1920x1080@1/60 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
>                  crop.bounds:(0,0)/1920x1080
>                  crop:(0,0)/1920x1080
>                  compose.bounds:(0,0)/1920x1080
>                  compose:(0,0)/1920x1080]
>
> If you look at v4l2_subdev_link_validate_default in
> drivers/media/v4l2-core/v4l2-subdev.c, you can see that besides the
> format code, it also checks width, height, and field setting.
>
> regards
> Philipp

Best regards,
Ibtsam Haq

Re: imx-media: MT9P031 Capture issues on IMX6

[Philipp Zabel]

On Tue, 2018-04-17 at 11:32 +0200, Ibtsam Ul-Haq wrote:
> On Tue, Apr 17, 2018 at 10:34 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> > Hi Ibtsam,
> > 
> > On Tue, 2018-04-17 at 09:26 +0200, Ibtsam Ul-Haq wrote:
> > > On Mon, Apr 16, 2018 at 11:30 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> > > > On Mon, 2018-04-16 at 09:54 +0200, Ibtsam Ul-Haq wrote:
> > > > [...]
> > > > > This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?
> > > > 
> > > > Yes, see Documentation/media/uapi/v4l/pixfmt-srggb16.rst:
> > > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-srggb16.html
> > > > 
> > > > > To be honest, I had not seen GR16 as FourCC before.
> > > > > And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
> > > > > do not know this FourCC:
> > > > > v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
> > > > > Unsupported fourcc 0x36315247 GR16
> > > > 
> > > > The GStreamer V4L2 elements currently only support 8-bit per component
> > > > Bayer formats.
> > > > 
> > > > > Is there a way we can get by this?
> > > > 
> > > > There's two ways to handle this correctly, IMHO. One would be adding
> > > > SGRBG8_1X8 support to the mt9p031 driver. This is the correct way if the
> > > > device tree is configured for 8-bit parallel and there are only 8 data
> > > > lines connected between camera and SoC. As a quick hack, I think you
> > > > could just:
> > > > 
> > > >   sed "s/MEDIA_BUS_FMT_SGRBG12_1X12/MEDIA_BUS_FMT_SGRBG8_1X8/" -i drivers/media/i2c/mt9p031.c
> > > > 
> > > 
> > > 
> > > I tried that and it works well for my case. All pads in the pipeline
> > > now report their format as SGRBG8_1X8.
> > > 
> > > However, now I am getting a Broken Pipe error from STREAMON when I try
> > > to run the pipeline:
> > > v4l2bufferpool gstv4l2bufferpool.c:677:gst_v4l2_buffer_pool_streamon:<v4l2src0:pool:src>
> > > [00m
> > > error with STREAMON 32 (Broken pipe)
> > 
> > What about format width and height, are they the same throughout the
> > pipeline? It didn't look that way in your last mail.
> > 
> 
> Indeed they were not the same. That was probably because the default
> window width and height were 2592x1944.
> So when I tried to set resolution to 640x480, it actually became
> 648x486, and the ipu1_csi0 expanded it to 656x486.

Yes, the CSI currently aligns width to a multiple of 16 pixels to
simplify handling DMA controller alignment restrictions.
This should probably be relaxed, especially for non-planar formats.

> I have that changed now, so the pads now look to have the same width and height:
> 
> :~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb
>                  crop:(16,54)/2560x1920]
> 
> :~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb
>                  crop:(16,54)/2560x1920]
> 
> :~# media-ctl --get-v4l2 "'ipu1_csi0_mux':1"
>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb]
> 
> :~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb]
> 
> :~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>                 [fmt:SGRBG8_1X8/640x480@1/30 field:none
> colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
>                  crop.bounds:(0,0)/640x480
>                  crop:(0,0)/640x480
>                  compose.bounds:(0,0)/640x480
>                  compose:(0,0)/640x480]
> 
> :~# media-ctl --get-v4l2 "'ipu1_csi0':2"
>                 [fmt:SGRBG8_1X8/640x480@1/30 field:none
> colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]
> 
> But now I am getting:
> gst_v4l2_buffer_pool_pthe handlingoll ():
> /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> poll error 1: Invalid argument (22)
> 
> This is accompanied by dmesg errors:
> [ 8056.756841] alloc_contig_range: [80400, 80496) PFNs busy

This is not a memory allocation problem. The above is a harmless info
message that points out a slight inefficiency in the CMA allocator.

> [ 8057.833501] ipu1_csi0: EOF timeout
> [ 8058.953717] ipu1_csi0: wait last EOF timeout

This is the problem. The driver believes all is configured correctly.
It starts the DMA transfer and waits for the End-Of-Frame interrupt.
The reason that doesn't happen could be the sensor failing to start
streaming, or the signal not arriving at the CSI properly. Can you
measure the pixel clock and vsync/hsync signals? Is pinctrl set up
correctly?

regards
Philipp

Re: imx-media: MT9P031 Capture issues on IMX6

[Ibtsam Ul-Haq]

On Tue, Apr 17, 2018 at 2:33 PM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
> On Tue, 2018-04-17 at 11:32 +0200, Ibtsam Ul-Haq wrote:
>> On Tue, Apr 17, 2018 at 10:34 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
>> > Hi Ibtsam,
>> >
>> > On Tue, 2018-04-17 at 09:26 +0200, Ibtsam Ul-Haq wrote:
>> > > On Mon, Apr 16, 2018 at 11:30 AM, Philipp Zabel <p.zabel@pengutronix.de> wrote:
>> > > > On Mon, 2018-04-16 at 09:54 +0200, Ibtsam Ul-Haq wrote:
>> > > > [...]
>> > > > > This indeed looks the case. But then, is 'GR16' the FourCC for 'SGRBG16'?
>> > > >
>> > > > Yes, see Documentation/media/uapi/v4l/pixfmt-srggb16.rst:
>> > > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-srggb16.html
>> > > >
>> > > > > To be honest, I had not seen GR16 as FourCC before.
>> > > > > And the Gstreamer debug logs (I used GST_DEBUG=5) also say that they
>> > > > > do not know this FourCC:
>> > > > > v4l2 gstv4l2object.c:1541:gst_v4l2_object_v4l2fourcc_to_bare_struct: [00m
>> > > > > Unsupported fourcc 0x36315247 GR16
>> > > >
>> > > > The GStreamer V4L2 elements currently only support 8-bit per component
>> > > > Bayer formats.
>> > > >
>> > > > > Is there a way we can get by this?
>> > > >
>> > > > There's two ways to handle this correctly, IMHO. One would be adding
>> > > > SGRBG8_1X8 support to the mt9p031 driver. This is the correct way if the
>> > > > device tree is configured for 8-bit parallel and there are only 8 data
>> > > > lines connected between camera and SoC. As a quick hack, I think you
>> > > > could just:
>> > > >
>> > > >   sed "s/MEDIA_BUS_FMT_SGRBG12_1X12/MEDIA_BUS_FMT_SGRBG8_1X8/" -i drivers/media/i2c/mt9p031.c
>> > > >
>> > >
>> > >
>> > > I tried that and it works well for my case. All pads in the pipeline
>> > > now report their format as SGRBG8_1X8.
>> > >
>> > > However, now I am getting a Broken Pipe error from STREAMON when I try
>> > > to run the pipeline:
>> > > v4l2bufferpool gstv4l2bufferpool.c:677:gst_v4l2_buffer_pool_streamon:<v4l2src0:pool:src>
>> > > [00m
>> > > error with STREAMON 32 (Broken pipe)
>> >
>> > What about format width and height, are they the same throughout the
>> > pipeline? It didn't look that way in your last mail.
>> >
>>
>> Indeed they were not the same. That was probably because the default
>> window width and height were 2592x1944.
>> So when I tried to set resolution to 640x480, it actually became
>> 648x486, and the ipu1_csi0 expanded it to 656x486.
>
> Yes, the CSI currently aligns width to a multiple of 16 pixels to
> simplify handling DMA controller alignment restrictions.
> This should probably be relaxed, especially for non-planar formats.
>
>> I have that changed now, so the pads now look to have the same width and height:
>>
>> :~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
>>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb
>>                  crop:(16,54)/2560x1920]
>>
>> :~# media-ctl --get-v4l2 "'mt9p031 0-0048':0"
>>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb
>>                  crop:(16,54)/2560x1920]
>>
>> :~# media-ctl --get-v4l2 "'ipu1_csi0_mux':1"
>>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb]
>>
>> :~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>>                 [fmt:SGRBG8_1X8/640x480 field:none colorspace:srgb]
>>
>> :~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>>                 [fmt:SGRBG8_1X8/640x480@1/30 field:none
>> colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
>>                  crop.bounds:(0,0)/640x480
>>                  crop:(0,0)/640x480
>>                  compose.bounds:(0,0)/640x480
>>                  compose:(0,0)/640x480]
>>
>> :~# media-ctl --get-v4l2 "'ipu1_csi0':2"
>>                 [fmt:SGRBG8_1X8/640x480@1/30 field:none
>> colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]
>>
>> But now I am getting:
>> gst_v4l2_buffer_pool_pthe handlingoll ():
>> /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
>> poll error 1: Invalid argument (22)
>>
>> This is accompanied by dmesg errors:
>> [ 8056.756841] alloc_contig_range: [80400, 80496) PFNs busy
>
> This is not a memory allocation problem. The above is a harmless info
> message that points out a slight inefficiency in the CMA allocator.
>
>> [ 8057.833501] ipu1_csi0: EOF timeout
>> [ 8058.953717] ipu1_csi0: wait last EOF timeout
>
> This is the problem. The driver believes all is configured correctly.
> It starts the DMA transfer and waits for the End-Of-Frame interrupt.
> The reason that doesn't happen could be the sensor failing to start
> streaming, or the signal not arriving at the CSI properly. Can you
> measure the pixel clock and vsync/hsync signals? Is pinctrl set up
> correctly?
>


I can see by using a logic analyzer that the PIXCLK does not look
nice. It looks similar to the issue mentioned here:
https://community.nxp.com/thread/454467

except that in my case it looks pulled up instead of down.
However I do not yet have a clue what causes this.
VSYNC and HSYNC waveforms look ok, until the whole capture is stopped
due to the error, after 14 frames.
The relevant pinctrl settings in the dts are:

    MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK    0x4001b0b0
    MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC        0x4001b0b0
    MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC        0x4001b0b0
    MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20            0x4000a0b0

And I have checked using "devregs" tool that these settings are indeed
applied to the relevant IOMUXC_SW_PAD_CTL and IOMUXC_SW_MUX_CTL
registers.


> regards
> Philipp

Best regards,
Ibtsam Haq

Re: imx-media: MT9P031 Capture issues on IMX6

[Fabio Estevam]

On Thu, Apr 19, 2018 at 1:55 PM, Ibtsam Ul-Haq
<ibtsam.haq.0x01@gmail.com> wrote:

> I can see by using a logic analyzer that the PIXCLK does not look
> nice. It looks similar to the issue mentioned here:
> https://community.nxp.com/thread/454467
>
> except that in my case it looks pulled up instead of down.
> However I do not yet have a clue what causes this.
> VSYNC and HSYNC waveforms look ok, until the whole capture is stopped
> due to the error, after 14 frames.
> The relevant pinctrl settings in the dts are:
>
>     MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK    0x4001b0b0
>     MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC        0x4001b0b0
>     MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC        0x4001b0b0

Not sure why you are setting the SION bit (bit 30) on the CSI pads.

Does it work better if you do not set it?

For your reference: this is what we do on imx6qdl-sabresd.dtsi:

MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0

Re: imx-media: MT9P031 Capture issues on IMX6

[Ibtsam Ul-Haq]

Hi Fabio,

On Thu, Apr 19, 2018 at 7:08 PM, Fabio Estevam <festevam@gmail.com> wrote:
> On Thu, Apr 19, 2018 at 1:55 PM, Ibtsam Ul-Haq
> <ibtsam.haq.0x01@gmail.com> wrote:
>
>> I can see by using a logic analyzer that the PIXCLK does not look
>> nice. It looks similar to the issue mentioned here:
>> https://community.nxp.com/thread/454467
>>
>> except that in my case it looks pulled up instead of down.
>> However I do not yet have a clue what causes this.
>> VSYNC and HSYNC waveforms look ok, until the whole capture is stopped
>> due to the error, after 14 frames.
>> The relevant pinctrl settings in the dts are:
>>
>>     MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK    0x4001b0b0
>>     MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC        0x4001b0b0
>>     MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC        0x4001b0b0
>
> Not sure why you are setting the SION bit (bit 30) on the CSI pads.
>
> Does it work better if you do not set it?
>


Thanks for your response.
The SION bit was this way in the dts provided by the board manufacturer.
But now I have tried it with the SION bit cleared, unfortunately it
didn't make any difference.
There is a new board revision coming soon with several layout changes.
So currently I am waiting, in the hope that we don't waste time
debugging it if it were a hardware problem.


> For your reference: this is what we do on imx6qdl-sabresd.dtsi:
>
> MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
> MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
> MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
> MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
> MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
> MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
> MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
> MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
> MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
> MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
> MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0

Best regards,
Ibtsam Haq