IMX CSI capture issues with tda1997x HDMI receiver

IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

Steve,

I'm testing IMX6 capture again with the tda1997x HDMI receiver found
on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6
CSI in a way where it can be configured for 8bit BT656 mode
(UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall
Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide
variety of resolutions, bus formats, interlaced modes, and
colorspaces.

I'm using the following for testing:
- Linux 4.20 +
  - media: imx-csi: Input connections to CSI should be optional
  - imx-media: Fixes for interlaced capture
  - media: imx: Stop stream before disabling IDMA channels
- v4l-utils 1.16.1
- gstreamer-1.14.1 (with the ability to test master as well)

I have a script that generates the media-ctl/v4l2-ctl commands based
on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI
decoder) and a 'mode' which I've defined as:
mode0: sensor -> mux -> csi -> /dev/videoN
mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a
mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN
mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN

The media-ctl topologies for each cpu/board combo are at
http://dev.gateworks.com/docs/linux/media/

I'm trying to test out simple v4l2-ctl based capture of a single frame
as well as capture and stream via both software JPEG encode and H264
hardware encode via CODA.

Please let me know of any changes that should be made to the commands
below even if only to purely help document things through clarity.

One of the issues I run into right away is image size: The imx.rst
docs state that the ic has a 'resize' limit of 1024x1024 but I think
I'm mislead by the word 'resize' and this also means you can't push
say 1080x720 through it (not resizing, just putting that on the src
pad) as it clips it to 1024x720. I believe some of Phillip's pending
patches may be aimed at rectifying this limitation?

Another issue I'm running into is colorspace conversion, specifically
colorimetry

Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0
MODE1:sensor->mux->csi->ic_prp->ic_prpenc
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
field:none colorspace:rec709
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]"
# capture device
media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
v4l2-ctl --device /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 1024/720
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 2048
        Size Image        : 1474560
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :
^^^ Note 1080x720 has been reduced to 1024x720 - ouch!
# capture 1 frame
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
^^^ works
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,format=UYVY ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
^^^ works (but of course squashed horizontally because of x=1024)
# stream H264/RTP/UDP (via coda)
# need to make sure we feed coda NV12/rec709
media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' #
fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709
ycbcr:601 quantization:lim-range
^^^ input colorspace is rec709 but colorimetry needs to be changed to 709
media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720
colorspace:rec709 ycbcr:709]'
^^^ no error but doing a get-v4l2 shows no change - is this all
because of trying to deal with 1080 instead of 1024?
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  v4l2h264enc output-io-mode=dmabuf-import ! \
  rtph264pay ! udpsink host=172.24.20.19 port=5001
v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
^^^ fails because of colorimetry not getting set

Let me remove the >1024 issue by using a 480p source...

Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0
MODE1:sensor->mux->csi->ic_prp->ic_prpenc
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
field:none colorspace:rec709
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/720x480]"
media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
  [fmt:AYUV8_1X32/720x480@1/25 field:none colorspace:rec709 xfer:709
ycbcr:601 quantization:lim-range]
# capture device
media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
v4l2-ctl --device /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 720/480
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1440
        Size Image        : 691200
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :
# capture 1 frame
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png
^^^ works
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,format=UYVY ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
# stream H264/RTP/UDP
media-ctl --get-v4l2 '"ipu1_ic_prpenc":0'
 [fmt:AYUV8_1X32/720x480@1/25 field:none colorspace:rec709 xfer:709
ycbcr:601 quantization:lim-range]
^^^ again, need to change colorimetry to 709
media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/720x480
colorspace:rec709 ycbcr:709]'
media-ctl --get-v4l2 '"ipu1_ic_prpenc":0'
 [fmt:AYUV8_1X32/720x480@1/25 field:none colorspace:rec709 xfer:709
ycbcr:601 quantization:lim-range]
^^^ still not changing
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  v4l2h264enc output-io-mode=dmabuf-import ! \
  rtph264pay ! udpsink host=172.24.20.19 port=5001
v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
^^^ fails because of colorimetry not getting set

Let's try an interlaced format...

Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0
MODE1:sensor->mux->csi->ic_prp->ic_prpenc
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
field:seq-tb colorspace:rec709
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
# capture device
media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
v4l2-ctl --device /dev/video1 --get-fmt-video
Format Video Capture:
        Width/Height      : 720/480
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1440
        Size Image        : 691200
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :
# capture 1 frame
v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
^^^ fails with ipu1_ic_prpvf: EOF timeout
^^^ this is almost identical to the adv7180 case which works... the
only exception I see is field:seq-tb vs field:alternate from the
source which should be fine (more specific than 'alternate')

Now lets go back to a 480p60 source but this time include the vdic
(which isn't necessary but should still work right?)

Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0
MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
field:none colorspace:rec709
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
# capture device
media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
v4l2-ctl --device /dev/video1 --get-fmt-video
Format Video Capture:
        Width/Height      : 720/480
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1440
        Size Image        : 691200
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :
# capture 1 frame
v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
ipu1_ic_prpvf: pipeline start failed with -32
^^^ maybe because we are feeding progressive to vdic? vdic isn't
needed here but it seems it should not fail to me

ok now I'm going to boot with a different device-tree that connects
the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be
1) better pixel bpp 2) colorimetry is rec709 so no conversion needed
for coda 3) ability to capture 1080p60 as I have confirmed tyring to
do so via bt656 exceeds the IMX pixel clock rate and causes CSI
corruption. Downsides to this mode is that I currently have a bug in
the tda1997x driver regarding this mode and interlaced capture.

Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0
MODE0:sensor->mux->csi
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480
field:none colorspace:rec709
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
                [fmt:UYVY8_1X16/720x480@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
# capture device
media-ctl -e 'ipu1_csi0 capture' # /dev/video4
v4l2-ctl --device /dev/video4 --get-fmt-video
Format Video Capture:
        Width/Height      : 720/480
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1440
        Size Image        : 691200
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: Rec. 709
        Quantization      : Limited Range
        Flags             : premultiplied-alpha
# capture 1 frame
v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1
convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video4 ! \
  video/x-raw,format=UYVY ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
# stream H264/RTP/UDP
media-ctl --get-v4l2 '"ipu1_csi0":0'
                [fmt:UYVY8_1X16/720x480@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
                 crop.bounds:(0,0)/720x480
                 crop:(0,0)/720x480
                 compose.bounds:(0,0)/720x480
                 compose:(0,0)/720x480]
^^^ no conv needed
media-ctl --get-v4l2 '"ipu1_csi0":0'
gst-launch-1.0 v4l2src device=/dev/video4 ! \
  v4l2h264enc output-io-mode=dmabuf-import ! \
  rtph264pay ! udpsink host=172.24.20.19 port=5001
v4l2src0: Internal data stream error.
^^^ fails... lets set the csi format again
media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/720x480
colorspace:rec709 ycbcr:709]'
media-ctl --get-v4l2 '"ipu1_csi0":0'
                [fmt:UYVY8_2X8/720x480@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
                 crop.bounds:(0,0)/720x480
                 crop:(0,0)/720x480
                 compose.bounds:(0,0)/720x480
                 compose:(0,0)/720x480]
^^^ wtf... the fcc is not showing UYVY8_2X8 which is just wrong and
even v4l2 frame capture will fail now

Any ideas on any of the above issues?

Best Regards,

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

Hi Tim, cc: Philipp,

On 1/23/19 3:21 PM, Tim Harvey wrote:
> Steve,
>
> I'm testing IMX6 capture again with the tda1997x HDMI receiver found
> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6
> CSI in a way where it can be configured for 8bit BT656 mode
> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall
> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide
> variety of resolutions, bus formats, interlaced modes, and
> colorspaces.
>
> I'm using the following for testing:
> - Linux 4.20 +
>    - media: imx-csi: Input connections to CSI should be optional
>    - imx-media: Fixes for interlaced capture
>    - media: imx: Stop stream before disabling IDMA channels
> - v4l-utils 1.16.1
> - gstreamer-1.14.1 (with the ability to test master as well)
>
> I have a script that generates the media-ctl/v4l2-ctl commands based
> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI
> decoder) and a 'mode' which I've defined as:
> mode0: sensor -> mux -> csi -> /dev/videoN
> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a
> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN
> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN
>
> The media-ctl topologies for each cpu/board combo are at
> http://dev.gateworks.com/docs/linux/media/
>
> I'm trying to test out simple v4l2-ctl based capture of a single frame
> as well as capture and stream via both software JPEG encode and H264
> hardware encode via CODA.
>
> Please let me know of any changes that should be made to the commands
> below even if only to purely help document things through clarity.
>
> One of the issues I run into right away is image size: The imx.rst
> docs state that the ic has a 'resize' limit of 1024x1024 but I think
> I'm mislead by the word 'resize' and this also means you can't push
> say 1080x720 through it (not resizing, just putting that on the src
> pad) as it clips it to 1024x720.

The IC is limited to a resized *output* frame of 1024x1024, no higher.

>   I believe some of Phillip's pending
> patches may be aimed at rectifying this limitation?

If you mean the mem2mem driver, yes. The mem2mem driver makes use the 
ipu-image-convert APIs which includes tiled resizing, so it will support 
greater-than 1024x1024 output frames. Latest status I've heard is that 
there is some misunderstanding of the behavior of rotation, that will 
require some rework of the mem2mem driver and ipu-image-convert (and 
sigh, the ic-prpencvf sub-device which also supports h/w rotation).

But you could actually grab the last posted version of mem2mem driver 
(v7). It works fine except for the rotation misunderstanding (but even 
that works but doesn't conform to v4l2 expected behavior). You will then 
have a gstreamer element v4l2convertN for >1024 resizing support.

Also recently merged from Philipp is the work to relax capture width 
alignment, by DMA'ing to padded frames. But that is not related to mem2mem.

>
> Another issue I'm running into is colorspace conversion, specifically
> colorimetry
>
> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0
> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
> field:none colorspace:rec709
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]"
> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]"
> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]"
> # capture device
> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
> v4l2-ctl --device /dev/video0 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 1024/720
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 2048
>          Size Image        : 1474560
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: ITU-R 601
>          Quantization      : Limited Range
>          Flags             :
> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch!

Yes as discussed above that is correct behavior, you'll need mem2mem 
device to support >1024 tiled scaling.

> # capture 1 frame
> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> ^^^ works
> # stream JPEG/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>    video/x-raw,format=UYVY ! \
>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> ^^^ works (but of course squashed horizontally because of x=1024)
> # stream H264/RTP/UDP (via coda)
> # need to make sure we feed coda NV12/rec709
> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' #
> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709
> ycbcr:601 quantization:lim-range
> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709
> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720
> colorspace:rec709 ycbcr:709]'
> ^^^ no error but doing a get-v4l2 shows no change - is this all
> because of trying to deal with 1080 instead of 1024?

No. The Image Converter can only Y`CbCr encode to ITU-R 601 
(V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr 
encoding when using an IC pipeline.

See imx_media_fill_default_mbus_fields().

> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>    v4l2h264enc output-io-mode=dmabuf-import ! \
>    rtph264pay ! udpsink host=172.24.20.19 port=5001
> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
> ^^^ fails because of colorimetry not getting set

Right, that is correct behavior also. See above.

Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace?

> <snip>
>
> Let's try an interlaced format...
>
> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0
> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
> field:seq-tb colorspace:rec709
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
> # capture device
> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
> v4l2-ctl --device /dev/video1 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 720/480
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1440
>          Size Image        : 691200
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: ITU-R 601
>          Quantization      : Limited Range
>          Flags             :
> # capture 1 frame
> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
> ^^^ fails with ipu1_ic_prpvf: EOF timeout
> ^^^ this is almost identical to the adv7180 case which works... the
> only exception I see is field:seq-tb vs field:alternate from the
> source which should be fine (more specific than 'alternate')

Hmm, can you send the complete pad formats from --get-v4l2. I don't see 
anything obviously wrong here.

> Now lets go back to a 480p60 source but this time include the vdic
> (which isn't necessary but should still work right?)

No. First, the CSI will only capture in bt.656 mode if it sees 
interlaced fields (bt.656 interlaced sync codes). Well, let me rephrase, 
the CSI does support progressive BT.656 but I have never tested that myself.

But more importantly, the VDIC must be given interlaced fields at its input.

>
> Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0
> MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
> field:none colorspace:rec709
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]"
> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
> # capture device
> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
> v4l2-ctl --device /dev/video1 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 720/480
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1440
>          Size Image        : 691200
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: ITU-R 601
>          Quantization      : Limited Range
>          Flags             :
> # capture 1 frame
> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
> ipu1_ic_prpvf: pipeline start failed with -32
> ^^^ maybe because we are feeding progressive to vdic? vdic isn't
> needed here but it seems it should not fail to me

this is a pipeline src->sink mismatch error, because the VDIC enforces 
fields at its input pad.

>
> ok now I'm going to boot with a different device-tree that connects
> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be
> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed
> for coda 3) ability to capture 1080p60 as I have confirmed tyring to
> do so via bt656 exceeds the IMX pixel clock rate and causes CSI
> corruption. Downsides to this mode is that I currently have a bug in
> the tda1997x driver regarding this mode and interlaced capture.
>
> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0
> MODE0:sensor->mux->csi
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480
> field:none colorspace:rec709
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]"
> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
> media-ctl --get-v4l2 '"ipu1_csi0":2'
>                  [fmt:UYVY8_1X16/720x480@1/30 field:none

This is passthrough, the CSI can only pass through input pixel data 
unmodified as "generic" data, when the input bus is 16-bit parallel, 
which it is in this case (UYVY8_1X16). So the pixel code at the CSI 
output pad is forced to same as input pad (UYVY8_1X16).

But aside from passthrough, that shouldn't be a problem...

> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> # capture device
> media-ctl -e 'ipu1_csi0 capture' # /dev/video4
> v4l2-ctl --device /dev/video4 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 720/480
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1440
>          Size Image        : 691200
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: Rec. 709
>          Quantization      : Limited Range
>          Flags             : premultiplied-alpha
> # capture 1 frame
> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1
> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png

So capture of 1 frame worked fine?

> # stream JPEG/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    video/x-raw,format=UYVY ! \
>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000

and this worked?

> # stream H264/RTP/UDP
> media-ctl --get-v4l2 '"ipu1_csi0":0'
>                  [fmt:UYVY8_1X16/720x480@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
>                   crop.bounds:(0,0)/720x480
>                   crop:(0,0)/720x480
>                   compose.bounds:(0,0)/720x480
>                   compose:(0,0)/720x480]
> ^^^ no conv needed
> media-ctl --get-v4l2 '"ipu1_csi0":0'

huh?

> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    v4l2h264enc output-io-mode=dmabuf-import ! \
>    rtph264pay ! udpsink host=172.24.20.19 port=5001
> v4l2src0: Internal data stream error.

Sorry, I don't have enough info to help here.


> ^^^ fails... lets set the csi format again
> media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/720x480
> colorspace:rec709 ycbcr:709]'
> media-ctl --get-v4l2 '"ipu1_csi0":0'
>                  [fmt:UYVY8_2X8/720x480@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
>                   crop.bounds:(0,0)/720x480
>                   crop:(0,0)/720x480
>                   compose.bounds:(0,0)/720x480
>                   compose:(0,0)/720x480]
> ^^^ wtf... the fcc is not showing UYVY8_2X8 which is just wrong and
> even v4l2 frame capture will fail now

The CSI does not support AYUV32 at its input pad, so it defaults to 
UYVY8_2X.

Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

Hi Tim,

On 1/25/19 3:57 PM, Steve Longerbeam wrote:
<snip>
>> Now lets go back to a 480p60 source but this time include the vdic
>> (which isn't necessary but should still work right?)
>
> No. First, the CSI will only capture in bt.656 mode if it sees 
> interlaced fields (bt.656 interlaced sync codes). Well, let me 
> rephrase, the CSI does support progressive BT.656 but I have never 
> tested that myself.

One more comment here. It would be great if you could test a progressive 
bt.656 sensor (example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 
IPU1_CSI0) since as I said I've never been able to test this myself.

Since it is progressive there's no need for the VDIC, so try these 
pipelines:

mode0: sensor -> mux -> csi -> /dev/videoN
mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN


Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Sun, Jan 27, 2019 at 2:36 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
> Hi Tim,
>
> On 1/25/19 3:57 PM, Steve Longerbeam wrote:
> <snip>
> >> Now lets go back to a 480p60 source but this time include the vdic
> >> (which isn't necessary but should still work right?)
> >
> > No. First, the CSI will only capture in bt.656 mode if it sees
> > interlaced fields (bt.656 interlaced sync codes). Well, let me
> > rephrase, the CSI does support progressive BT.656 but I have never
> > tested that myself.
>
> One more comment here. It would be great if you could test a progressive
> bt.656 sensor (example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656
> IPU1_CSI0) since as I said I've never been able to test this myself.
>
> Since it is progressive there's no need for the VDIC, so try these
> pipelines:
>
> mode0: sensor -> mux -> csi -> /dev/videoN
> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN
>

Steve,

These both work fine in all cases for 480p60Hz YUV via BT656.

When I use 480p60Hz 'RGB' via BT656 mode0 works fine as the CSI does
the colorspace conversion needed for coda but for mode1 I end up with
the itu601 colorimetery issue from before that I'm still trying to
find a solution for.

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

On 1/28/19 11:14 AM, Tim Harvey wrote:
> On Sun, Jan 27, 2019 at 2:36 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> Hi Tim,
>>
>> On 1/25/19 3:57 PM, Steve Longerbeam wrote:
>> <snip>
>>>> Now lets go back to a 480p60 source but this time include the vdic
>>>> (which isn't necessary but should still work right?)
>>> No. First, the CSI will only capture in bt.656 mode if it sees
>>> interlaced fields (bt.656 interlaced sync codes). Well, let me
>>> rephrase, the CSI does support progressive BT.656 but I have never
>>> tested that myself.
>> One more comment here. It would be great if you could test a progressive
>> bt.656 sensor (example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656
>> IPU1_CSI0) since as I said I've never been able to test this myself.
>>
>> Since it is progressive there's no need for the VDIC, so try these
>> pipelines:
>>
>> mode0: sensor -> mux -> csi -> /dev/videoN
>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN
>>
> Steve,
>
> These both work fine in all cases for 480p60Hz YUV via BT656.

Great! thanks for confirming.

Steve

>
> When I use 480p60Hz 'RGB' via BT656 mode0 works fine as the CSI does
> the colorspace conversion needed for coda but for mode1 I end up with
> the itu601 colorimetery issue from before that I'm still trying to
> find a solution for.
>
> Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
> Hi Tim, cc: Philipp,
>
> On 1/23/19 3:21 PM, Tim Harvey wrote:
> > Steve,
> >
> > I'm testing IMX6 capture again with the tda1997x HDMI receiver found
> > on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6
> > CSI in a way where it can be configured for 8bit BT656 mode
> > (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall
> > Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide
> > variety of resolutions, bus formats, interlaced modes, and
> > colorspaces.
> >
> > I'm using the following for testing:
> > - Linux 4.20 +
> >    - media: imx-csi: Input connections to CSI should be optional
> >    - imx-media: Fixes for interlaced capture
> >    - media: imx: Stop stream before disabling IDMA channels
> > - v4l-utils 1.16.1
> > - gstreamer-1.14.1 (with the ability to test master as well)
> >
> > I have a script that generates the media-ctl/v4l2-ctl commands based
> > on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI
> > decoder) and a 'mode' which I've defined as:
> > mode0: sensor -> mux -> csi -> /dev/videoN
> > mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a
> > mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN
> > mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN
> >
> > The media-ctl topologies for each cpu/board combo are at
> > http://dev.gateworks.com/docs/linux/media/
> >
> > I'm trying to test out simple v4l2-ctl based capture of a single frame
> > as well as capture and stream via both software JPEG encode and H264
> > hardware encode via CODA.
> >
> > Please let me know of any changes that should be made to the commands
> > below even if only to purely help document things through clarity.
> >
> > One of the issues I run into right away is image size: The imx.rst
> > docs state that the ic has a 'resize' limit of 1024x1024 but I think
> > I'm mislead by the word 'resize' and this also means you can't push
> > say 1080x720 through it (not resizing, just putting that on the src
> > pad) as it clips it to 1024x720.
>
> The IC is limited to a resized *output* frame of 1024x1024, no higher.
>

Ok so this means for anything >1024 I need to either use the csi to
downscale it by factors of 2 first, or use mem2mem solution which
means I can't use the VDIC (so no interlaced). I will try your patch
that adds registration of the mem2mem driver to the media-ctl api.

> >   I believe some of Phillip's pending
> > patches may be aimed at rectifying this limitation?
>
> If you mean the mem2mem driver, yes. The mem2mem driver makes use the
> ipu-image-convert APIs which includes tiled resizing, so it will support
> greater-than 1024x1024 output frames. Latest status I've heard is that
> there is some misunderstanding of the behavior of rotation, that will
> require some rework of the mem2mem driver and ipu-image-convert (and
> sigh, the ic-prpencvf sub-device which also supports h/w rotation).
>
> But you could actually grab the last posted version of mem2mem driver
> (v7). It works fine except for the rotation misunderstanding (but even
> that works but doesn't conform to v4l2 expected behavior). You will then
> have a gstreamer element v4l2convertN for >1024 resizing support.
>

Ok, so I've applied that to my kernel and now have a new /dev/video*
device for ipu_ic_pp. I'll start testing with it.

> Also recently merged from Philipp is the work to relax capture width
> alignment, by DMA'ing to padded frames. But that is not related to mem2mem.
>

Ok - I'll merge this one to so I'm testing with it.

> >
> > Another issue I'm running into is colorspace conversion, specifically
> > colorimetry
> >
> > Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0
> > MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
> > field:none colorspace:rec709
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]"
> > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]"
> > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]"
> > # capture device
> > media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
> > v4l2-ctl --device /dev/video0 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 1024/720
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 2048
> >          Size Image        : 1474560
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: ITU-R 601
> >          Quantization      : Limited Range
> >          Flags             :
> > ^^^ Note 1080x720 has been reduced to 1024x720 - ouch!
>
> Yes as discussed above that is correct behavior, you'll need mem2mem
> device to support >1024 tiled scaling.
>
> > # capture 1 frame
> > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> > ^^^ works
> > # stream JPEG/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video0 ! \
> >    video/x-raw,format=UYVY ! \
> >    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> > ^^^ works (but of course squashed horizontally because of x=1024)
> > # stream H264/RTP/UDP (via coda)
> > # need to make sure we feed coda NV12/rec709
> > media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' #
> > fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709
> > ycbcr:601 quantization:lim-range
> > ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709
> > media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720
> > colorspace:rec709 ycbcr:709]'
> > ^^^ no error but doing a get-v4l2 shows no change - is this all
> > because of trying to deal with 1080 instead of 1024?
>
> No. The Image Converter can only Y`CbCr encode to ITU-R 601
> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr
> encoding when using an IC pipeline.
>
> See imx_media_fill_default_mbus_fields().
>
> > gst-launch-1.0 v4l2src device=/dev/video0 ! \
> >    v4l2h264enc output-io-mode=dmabuf-import ! \
> >    rtph264pay ! udpsink host=172.24.20.19 port=5001
> > v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
> > ^^^ fails because of colorimetry not getting set
>
> Right, that is correct behavior also. See above.
>
> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace?
>

I will have to go back and look what the old downstream Freescale
kernel drivers did here... I could swear I could easily de-interlace
via VDIC and encode 1080i with that and gstreamer-imx.

> > <snip>
> >
> > Let's try an interlaced format...
> >
> > Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0
> > MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
> > field:seq-tb colorspace:rec709
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
> > media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
> > media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
> > media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
> > media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
> > media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
> > # capture device
> > media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
> > v4l2-ctl --device /dev/video1 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 720/480
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 1440
> >          Size Image        : 691200
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: ITU-R 601
> >          Quantization      : Limited Range
> >          Flags             :
> > # capture 1 frame
> > v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
> > ^^^ fails with ipu1_ic_prpvf: EOF timeout
> > ^^^ this is almost identical to the adv7180 case which works... the
> > only exception I see is field:seq-tb vs field:alternate from the
> > source which should be fine (more specific than 'alternate')
>
> Hmm, can you send the complete pad formats from --get-v4l2. I don't see
> anything obviously wrong here.
>

root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
                [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
                [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1"
                [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2"
                [fmt:AYUV8_1X32/720x480@1/60 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2"
                [fmt:AYUV8_1X32/720x480@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1"
                [fmt:AYUV8_1X32/720x480@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]

root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video
Format Video Capture:
        Width/Height      : 720/480
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1440
        Size Image        : 691200
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :

root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap
--stream-to=x.raw --stream-count=1
[  159.828654] ipu1_ic_prpvf: EOF timeout
VIDIOC_DQBUF: failed: Input/output error

> > Now lets go back to a 480p60 source but this time include the vdic
> > (which isn't necessary but should still work right?)
>
> No. First, the CSI will only capture in bt.656 mode if it sees
> interlaced fields (bt.656 interlaced sync codes). Well, let me rephrase,
> the CSI does support progressive BT.656 but I have never tested that myself.
>
> But more importantly, the VDIC must be given interlaced fields at its input.
>

ok, I understand now that VDIC can only accept interlaced fields, so
progressive should skip the VDIC.

> >
> > Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0
> > MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
> > field:none colorspace:rec709
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
> > media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
> > media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
> > media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
> > media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
> > media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
> > # capture device
> > media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
> > v4l2-ctl --device /dev/video1 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 720/480
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 1440
> >          Size Image        : 691200
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: ITU-R 601
> >          Quantization      : Limited Range
> >          Flags             :
> > # capture 1 frame
> > v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
> > ipu1_ic_prpvf: pipeline start failed with -32
> > ^^^ maybe because we are feeding progressive to vdic? vdic isn't
> > needed here but it seems it should not fail to me
>
> this is a pipeline src->sink mismatch error, because the VDIC enforces
> fields at its input pad.

right... as you said above vdic requires interlaced fields so
progressive must skip the vdic.

>
> >
> > ok now I'm going to boot with a different device-tree that connects
> > the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be
> > 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed
> > for coda 3) ability to capture 1080p60 as I have confirmed tyring to
> > do so via bt656 exceeds the IMX pixel clock rate and causes CSI
> > corruption. Downsides to this mode is that I currently have a bug in
> > the tda1997x driver regarding this mode and interlaced capture.
> >
> > Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0
> > MODE0:sensor->mux->csi
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480
> > field:none colorspace:rec709
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]"
> > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> >                  [fmt:UYVY8_1X16/720x480@1/30 field:none
>
> This is passthrough, the CSI can only pass through input pixel data
> unmodified as "generic" data, when the input bus is 16-bit parallel,
> which it is in this case (UYVY8_1X16). So the pixel code at the CSI
> output pad is forced to same as input pad (UYVY8_1X16).
>
> But aside from passthrough, that shouldn't be a problem...
>
> > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> > # capture device
> > media-ctl -e 'ipu1_csi0 capture' # /dev/video4
> > v4l2-ctl --device /dev/video4 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 720/480
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 1440
> >          Size Image        : 691200
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: Rec. 709
> >          Quantization      : Limited Range
> >          Flags             : premultiplied-alpha
> > # capture 1 frame
> > v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1
> > convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png
>
> So capture of 1 frame worked fine?

yes this worked fine

>
> > # stream JPEG/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >    video/x-raw,format=UYVY ! \
> >    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
>
> and this worked?

yes this worked fine

>
> > # stream H264/RTP/UDP
> > media-ctl --get-v4l2 '"ipu1_csi0":0'
> >                  [fmt:UYVY8_1X16/720x480@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
> >                   crop.bounds:(0,0)/720x480
> >                   crop:(0,0)/720x480
> >                   compose.bounds:(0,0)/720x480
> >                   compose:(0,0)/720x480]
> > ^^^ no conv needed
> > media-ctl --get-v4l2 '"ipu1_csi0":0'
>
> huh?

typo'd here and cut-and-pasted another occurrence of the media-ctl
--get-v4l2 command but my point here is that we have the correct
colorspace/colorimetry for coda so the H264 below should work fine

>
> > gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >    v4l2h264enc output-io-mode=dmabuf-import ! \
> >    rtph264pay ! udpsink host=172.24.20.19 port=5001
> > v4l2src0: Internal data stream error.
>
> Sorry, I don't have enough info to help here.
>

Here's the pad details:

root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
                [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
                [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2"
                [fmt:UYVY8_1X16/720x480@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]

root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video
Format Video Capture:
        Width/Height      : 720/480
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1440
        Size Image        : 691200
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: Rec. 709
        Quantization      : Limited Range
        Flags             :

So in this passthrough case v4l2-ctl stream capture works, gstreamer
capture with sw jpeg encode works, but gstreamer capture with h264
encode fails:

root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    v4l2h264enc output-io-mode=dmabuf-import ! \
>    rtph264pay ! udpsink host=172.24.20.19 port=5001
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop ():
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000416672
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

It would appear that this must have something to do with gstreamer.
The only difference here between this case and the adv7180 is that
field=none and colorimetry is itu601... I'm guessing the field=none is
causing an issue. I'll post a question to gstreamer-devel.

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

On 1/28/19 11:03 AM, Tim Harvey wrote:
> On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>> Hi Tim, cc: Philipp,
>>
>> On 1/23/19 3:21 PM, Tim Harvey wrote:
>>> Steve,
>>>
>>> I'm testing IMX6 capture again with the tda1997x HDMI receiver found
>>> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6
>>> CSI in a way where it can be configured for 8bit BT656 mode
>>> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall
>>> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide
>>> variety of resolutions, bus formats, interlaced modes, and
>>> colorspaces.
>>>
>>> I'm using the following for testing:
>>> - Linux 4.20 +
>>>     - media: imx-csi: Input connections to CSI should be optional
>>>     - imx-media: Fixes for interlaced capture
>>>     - media: imx: Stop stream before disabling IDMA channels
>>> - v4l-utils 1.16.1
>>> - gstreamer-1.14.1 (with the ability to test master as well)
>>>
>>> I have a script that generates the media-ctl/v4l2-ctl commands based
>>> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI
>>> decoder) and a 'mode' which I've defined as:
>>> mode0: sensor -> mux -> csi -> /dev/videoN
>>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a
>>> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN
>>> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN
>>>
>>> The media-ctl topologies for each cpu/board combo are at
>>> http://dev.gateworks.com/docs/linux/media/
>>>
>>> I'm trying to test out simple v4l2-ctl based capture of a single frame
>>> as well as capture and stream via both software JPEG encode and H264
>>> hardware encode via CODA.
>>>
>>> Please let me know of any changes that should be made to the commands
>>> below even if only to purely help document things through clarity.
>>>
>>> One of the issues I run into right away is image size: The imx.rst
>>> docs state that the ic has a 'resize' limit of 1024x1024 but I think
>>> I'm mislead by the word 'resize' and this also means you can't push
>>> say 1080x720 through it (not resizing, just putting that on the src
>>> pad) as it clips it to 1024x720.
>> The IC is limited to a resized *output* frame of 1024x1024, no higher.
>>
> Ok so this means for anything >1024 I need to either use the csi to
> downscale it by factors of 2 first, or use mem2mem solution which
> means I can't use the VDIC (so no interlaced).

Actually that isn't true. You can still use a VDIC pipeline for the 
1280x720 interlaced tda19971 source. It's just that you would need to 
downscale to <=1024 (either using the CSI /2, and/or using the IC prpvf 
scaler). Then use the mem2mem gstreamer element to upscale the prpvf 
capture output to something >1024.


>   I will try your patch
> that adds registration of the mem2mem driver to the media-ctl api.

Ok. That patch doesn't have anything functional to it, but it's nice to 
see the mem2mem device as part of the media graph.

>
>>>    I believe some of Phillip's pending
>>> patches may be aimed at rectifying this limitation?
>> If you mean the mem2mem driver, yes. The mem2mem driver makes use the
>> ipu-image-convert APIs which includes tiled resizing, so it will support
>> greater-than 1024x1024 output frames. Latest status I've heard is that
>> there is some misunderstanding of the behavior of rotation, that will
>> require some rework of the mem2mem driver and ipu-image-convert (and
>> sigh, the ic-prpencvf sub-device which also supports h/w rotation).
>>
>> But you could actually grab the last posted version of mem2mem driver
>> (v7). It works fine except for the rotation misunderstanding (but even
>> that works but doesn't conform to v4l2 expected behavior). You will then
>> have a gstreamer element v4l2convertN for >1024 resizing support.
>>
> Ok, so I've applied that to my kernel and now have a new /dev/video*
> device for ipu_ic_pp. I'll start testing with it.

Cool.

>
>> Also recently merged from Philipp is the work to relax capture width
>> alignment, by DMA'ing to padded frames. But that is not related to mem2mem.
>>
> Ok - I'll merge this one too so I'm testing with it.

Ok.

>
>>> Another issue I'm running into is colorspace conversion, specifically
>>> colorimetry
>>>
>>> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0
>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
>>> field:none colorspace:rec709
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]"
>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]"
>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]"
>>> # capture device
>>> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
>>> v4l2-ctl --device /dev/video0 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 1024/720
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 2048
>>>           Size Image        : 1474560
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: ITU-R 601
>>>           Quantization      : Limited Range
>>>           Flags             :
>>> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch!
>> Yes as discussed above that is correct behavior, you'll need mem2mem
>> device to support >1024 tiled scaling.
>>
>>> # capture 1 frame
>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
>>> ^^^ works
>>> # stream JPEG/RTP/UDP
>>> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>>>     video/x-raw,format=UYVY ! \
>>>     jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
>>> ^^^ works (but of course squashed horizontally because of x=1024)
>>> # stream H264/RTP/UDP (via coda)
>>> # need to make sure we feed coda NV12/rec709
>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' #
>>> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709
>>> ycbcr:601 quantization:lim-range
>>> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709
>>> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720
>>> colorspace:rec709 ycbcr:709]'
>>> ^^^ no error but doing a get-v4l2 shows no change - is this all
>>> because of trying to deal with 1080 instead of 1024?
>> No. The Image Converter can only Y`CbCr encode to ITU-R 601
>> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr
>> encoding when using an IC pipeline.
>>
>> See imx_media_fill_default_mbus_fields().
>>
>>> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>>>     v4l2h264enc output-io-mode=dmabuf-import ! \
>>>     rtph264pay ! udpsink host=172.24.20.19 port=5001
>>> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
>>> ^^^ fails because of colorimetry not getting set
>> Right, that is correct behavior also. See above.
>>
>> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace?
>>
> I will have to go back and look what the old downstream Freescale
> kernel drivers did here... I could swear I could easily de-interlace
> via VDIC and encode 1080i with that and gstreamer-imx.
>
>>> <snip>
>>>
>>> Let's try an interlaced format...
>>>
>>> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0
>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
>>> field:seq-tb colorspace:rec709
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
>>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
>>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
>>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
>>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
>>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
>>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
>>> # capture device
>>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
>>> v4l2-ctl --device /dev/video1 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 720/480
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 1440
>>>           Size Image        : 691200
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: ITU-R 601
>>>           Quantization      : Limited Range
>>>           Flags             :
>>> # capture 1 frame
>>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
>>> ^^^ fails with ipu1_ic_prpvf: EOF timeout
>>> ^^^ this is almost identical to the adv7180 case which works... the
>>> only exception I see is field:seq-tb vs field:alternate from the
>>> source which should be fine (more specific than 'alternate')
>> Hmm, can you send the complete pad formats from --get-v4l2. I don't see
>> anything obviously wrong here.
>>
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>                  [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>                  [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1"
>                  [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2"
>                  [fmt:AYUV8_1X32/720x480@1/60 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2"
>                  [fmt:AYUV8_1X32/720x480@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1"
>                  [fmt:AYUV8_1X32/720x480@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>
> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 720/480
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1440
>          Size Image        : 691200
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: ITU-R 601
>          Quantization      : Limited Range
>          Flags             :
>
> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap
> --stream-to=x.raw --stream-count=1
> [  159.828654] ipu1_ic_prpvf: EOF timeout
> VIDIOC_DQBUF: failed: Input/output error

Sorry everything looks fine, I don't know why that pipeline is failing 
to generate any data.

>
>>> Now lets go back to a 480p60 source but this time include the vdic
>>> (which isn't necessary but should still work right?)
>> No. First, the CSI will only capture in bt.656 mode if it sees
>> interlaced fields (bt.656 interlaced sync codes). Well, let me rephrase,
>> the CSI does support progressive BT.656 but I have never tested that myself.
>>
>> But more importantly, the VDIC must be given interlaced fields at its input.
>>
> ok, I understand now that VDIC can only accept interlaced fields, so
> progressive should skip the VDIC.
>
>>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0
>>> MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
>>> field:none colorspace:rec709
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
>>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
>>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
>>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]"
>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
>>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
>>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
>>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
>>> # capture device
>>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
>>> v4l2-ctl --device /dev/video1 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 720/480
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 1440
>>>           Size Image        : 691200
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: ITU-R 601
>>>           Quantization      : Limited Range
>>>           Flags             :
>>> # capture 1 frame
>>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
>>> ipu1_ic_prpvf: pipeline start failed with -32
>>> ^^^ maybe because we are feeding progressive to vdic? vdic isn't
>>> needed here but it seems it should not fail to me
>> this is a pipeline src->sink mismatch error, because the VDIC enforces
>> fields at its input pad.
> right... as you said above vdic requires interlaced fields so
> progressive must skip the vdic.
>
>>> ok now I'm going to boot with a different device-tree that connects
>>> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be
>>> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed
>>> for coda 3) ability to capture 1080p60 as I have confirmed tyring to
>>> do so via bt656 exceeds the IMX pixel clock rate and causes CSI
>>> corruption. Downsides to this mode is that I currently have a bug in
>>> the tda1997x driver regarding this mode and interlaced capture.
>>>
>>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0
>>> MODE0:sensor->mux->csi
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480
>>> field:none colorspace:rec709
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]"
>>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
>>> media-ctl --get-v4l2 '"ipu1_csi0":2'
>>>                   [fmt:UYVY8_1X16/720x480@1/30 field:none
>> This is passthrough, the CSI can only pass through input pixel data
>> unmodified as "generic" data, when the input bus is 16-bit parallel,
>> which it is in this case (UYVY8_1X16). So the pixel code at the CSI
>> output pad is forced to same as input pad (UYVY8_1X16).
>>
>> But aside from passthrough, that shouldn't be a problem...
>>
>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
>>> # capture device
>>> media-ctl -e 'ipu1_csi0 capture' # /dev/video4
>>> v4l2-ctl --device /dev/video4 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 720/480
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 1440
>>>           Size Image        : 691200
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: Rec. 709
>>>           Quantization      : Limited Range
>>>           Flags             : premultiplied-alpha
>>> # capture 1 frame
>>> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1
>>> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png
>> So capture of 1 frame worked fine?
> yes this worked fine
>
>>> # stream JPEG/RTP/UDP
>>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>>     video/x-raw,format=UYVY ! \
>>>     jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
>> and this worked?
> yes this worked fine
>
>>> # stream H264/RTP/UDP
>>> media-ctl --get-v4l2 '"ipu1_csi0":0'
>>>                   [fmt:UYVY8_1X16/720x480@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
>>>                    crop.bounds:(0,0)/720x480
>>>                    crop:(0,0)/720x480
>>>                    compose.bounds:(0,0)/720x480
>>>                    compose:(0,0)/720x480]
>>> ^^^ no conv needed
>>> media-ctl --get-v4l2 '"ipu1_csi0":0'
>> huh?
> typo'd here and cut-and-pasted another occurrence of the media-ctl
> --get-v4l2 command but my point here is that we have the correct
> colorspace/colorimetry for coda so the H264 below should work fine
>
>>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>>     v4l2h264enc output-io-mode=dmabuf-import ! \
>>>     rtph264pay ! udpsink host=172.24.20.19 port=5001
>>> v4l2src0: Internal data stream error.
>> Sorry, I don't have enough info to help here.
>>
> Here's the pad details:
>
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>                  [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>                  [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2"
>                  [fmt:UYVY8_1X16/720x480@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
>
> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 720/480
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1440
>          Size Image        : 691200
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: Rec. 709
>          Quantization      : Limited Range
>          Flags             :
>
> So in this passthrough case v4l2-ctl stream capture works, gstreamer
> capture with sw jpeg encode works, but gstreamer capture with h264
> encode fails:
>
> root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>     v4l2h264enc output-io-mode=dmabuf-import ! \
>>     rtph264pay ! udpsink host=172.24.20.19 port=5001
> Setting pipeline to PAUSED ...
> Pipeline is live and does not need PREROLL ...
> Setting pipeline to PLAYING ...
> ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> Internal data stream error.
> Additional debug info:
> gstbasesrc.c(3055): gst_base_src_loop ():
> /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> streaming stopped, reason not-negotiated (-4)
> Execution ended after 0:00:00.000416672
> Setting pipeline to PAUSED ...
> Setting pipeline to READY ...
> Setting pipeline to NULL ...
> Freeing pipeline ...
>
> It would appear that this must have something to do with gstreamer.
> The only difference here between this case and the adv7180 is that
> field=none and colorimetry is itu601... I'm guessing the field=none is
> causing an issue. I'll post a question to gstreamer-devel.

Ok. Given that v4l2-ctl stream capture works, gstreamer capture with sw 
jpeg encode works, but gstreamer capture with h264does not, this does 
sound like some issue with gstreamer.

Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Mon, Jan 28, 2019 at 3:04 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
>
>
> On 1/28/19 11:03 AM, Tim Harvey wrote:
> > On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
> >> Hi Tim, cc: Philipp,
> >>
> >> On 1/23/19 3:21 PM, Tim Harvey wrote:
> >>> Steve,
> >>>
> >>> I'm testing IMX6 capture again with the tda1997x HDMI receiver found
> >>> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6
> >>> CSI in a way where it can be configured for 8bit BT656 mode
> >>> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall
> >>> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide
> >>> variety of resolutions, bus formats, interlaced modes, and
> >>> colorspaces.
> >>>
> >>> I'm using the following for testing:
> >>> - Linux 4.20 +
> >>>     - media: imx-csi: Input connections to CSI should be optional
> >>>     - imx-media: Fixes for interlaced capture
> >>>     - media: imx: Stop stream before disabling IDMA channels
> >>> - v4l-utils 1.16.1
> >>> - gstreamer-1.14.1 (with the ability to test master as well)
> >>>
> >>> I have a script that generates the media-ctl/v4l2-ctl commands based
> >>> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI
> >>> decoder) and a 'mode' which I've defined as:
> >>> mode0: sensor -> mux -> csi -> /dev/videoN
> >>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a
> >>> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN
> >>> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN
> >>>
> >>> The media-ctl topologies for each cpu/board combo are at
> >>> http://dev.gateworks.com/docs/linux/media/
> >>>
> >>> I'm trying to test out simple v4l2-ctl based capture of a single frame
> >>> as well as capture and stream via both software JPEG encode and H264
> >>> hardware encode via CODA.
> >>>
> >>> Please let me know of any changes that should be made to the commands
> >>> below even if only to purely help document things through clarity.
> >>>
> >>> One of the issues I run into right away is image size: The imx.rst
> >>> docs state that the ic has a 'resize' limit of 1024x1024 but I think
> >>> I'm mislead by the word 'resize' and this also means you can't push
> >>> say 1080x720 through it (not resizing, just putting that on the src
> >>> pad) as it clips it to 1024x720.
> >> The IC is limited to a resized *output* frame of 1024x1024, no higher.
> >>
> > Ok so this means for anything >1024 I need to either use the csi to
> > downscale it by factors of 2 first, or use mem2mem solution which
> > means I can't use the VDIC (so no interlaced).
>
> Actually that isn't true. You can still use a VDIC pipeline for the
> 1280x720 interlaced tda19971 source. It's just that you would need to
> downscale to <=1024 (either using the CSI /2, and/or using the IC prpvf
> scaler). Then use the mem2mem gstreamer element to upscale the prpvf
> capture output to something >1024.
>

True, that would work and here are my mem2mem testing findings:

I've found that mem2mem from the CSI works well:
- Using the CSI to downscale 720p then going through mem2mem and
upscaling it (works perfectly!)
# 720p@60Hz YUV BT656
# query current timings and set to this
v4l2-ctl -d $(media-ctl -e "tda19971 2-0048") --set-dv-bt-timings=query
# get timings
media-ctl --get-v4l2 '"tda19971 2-0048":0' # [fmt:UYVY8_2X8/1280x720
field:none colorspace:srgb]
# reset media controller links
media-ctl --reset
# create links
media-ctl --link '"tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]'
media-ctl --link '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
media-ctl --link '"ipu1_csi0":2 -> "ipu1_csi0 capture":0[1]'
# set pad formats
media-ctl --set-v4l2 '"tda19971 2-0048":0[fmt:UYVY8_2X8/1280x720]'
media-ctl --set-v4l2 '"ipu1_csi0_mux":2[fmt:UYVY8_2X8/1280x720]'
media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/1280x720
colorspace:rec709 ycbcr:709]'
media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
media-ctl --get-v4l2 '"ipu1_csi0":2' # [fmt:AYUV8_1X32/640x360@1/30
field:none colorspace:rec709 xfer:709 ycbcr:709
quantization:lim-range]
# get video device
media-ctl -e 'ipu1_csi0 capture' # /dev/video4
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,format=UYVY ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
# ^^^^ works but video is 1/4scale 640x360
# get mem2mem device (here is perhaps a great reason to have the
mem2mem device in the media graph... so its easy to find the device
name)
grep ipu_ic_pp /sys/class/video4linux/*/name #
/sys/class/video4linux/video8/name:ipu_ic_pp mem2mem - so its
v4l2video8convert
# stream upscale via mem2mem then JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video4 ! \
  video/x-raw,format=UYVY ! \
  v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
# ^^^ works and video is back at 720p!
# stream H264/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video4 ! \
  v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
  v4l2h264enc output-io-mode=dmabuf-import ! \
  rtph264pay ! udpsink host=172.24.20.19 port=5001
# ^^^ works - 720p via coda

But when going through the IC we again run into the issue where the
output of the IC isn't a suitable colorspace:
# 720p@60Hz YUV BT656
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
field:none colorspace:srgb
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
# downscale because the IC can't do >1024
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' #
[fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb
ycbcr:601 quantization:lim-range]
# ^^^ note we are itu601 again b/c that's the only format the IC can ouput
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,format=UYVY ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
# ^^^ works... SW JPEG can handle itu601
# stream upscale via mem2mem then JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video4 ! \
  v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
  jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
ERROR: from element
/GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device
'/dev/video8' does not support 2:4:7:1 colorimetry
# ^^^ fails because mem2mem doesn't support itu601
# stream H264/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video4 ! \
  v4l2h264enc output-io-mode=dmabuf-import ! \
  rtph264pay ! udpsink host=172.24.20.19 port=5001
ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
Device '/dev/video9' does not support 2:4:7:1 colorimetry
# ^^^ coda has same issue... can't del with itu601

Am I perhaps missing a capsfilter to get the mem2mem driver to convert
the colorspace properly? If so, they the mem2mem driver could be used
to correct the colorspace to get IC output to coda working.

Also can we connect the mem2mem driver to the unused VDIC input in the
media controller so that we can use the VDIC to de-interlace content
captured from non IMX sources (ie PCI or USB capture devices)?

<snip>
> >
> >>> Another issue I'm running into is colorspace conversion, specifically
> >>> colorimetry
> >>>
> >>> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0
> >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> >>> # set sensor output pad to sensor source format
> >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> >>> # sensor format
> >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
> >>> field:none colorspace:rec709
> >>> # reset all links
> >>> media-ctl --reset
> >>> # setup links
> >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> >>> # configure pads
> >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
> >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
> >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]"
> >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]"
> >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]"
> >>> # capture device
> >>> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
> >>> v4l2-ctl --device /dev/video0 --get-fmt-video
> >>> Format Video Capture:
> >>>           Width/Height      : 1024/720
> >>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >>>           Field             : None
> >>>           Bytes per Line    : 2048
> >>>           Size Image        : 1474560
> >>>           Colorspace        : Rec. 709
> >>>           Transfer Function : Rec. 709
> >>>           YCbCr/HSV Encoding: ITU-R 601
> >>>           Quantization      : Limited Range
> >>>           Flags             :
> >>> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch!
> >> Yes as discussed above that is correct behavior, you'll need mem2mem
> >> device to support >1024 tiled scaling.
> >>
> >>> # capture 1 frame
> >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> >>> ^^^ works
> >>> # stream JPEG/RTP/UDP
> >>> gst-launch-1.0 v4l2src device=/dev/video0 ! \
> >>>     video/x-raw,format=UYVY ! \
> >>>     jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> >>> ^^^ works (but of course squashed horizontally because of x=1024)
> >>> # stream H264/RTP/UDP (via coda)
> >>> # need to make sure we feed coda NV12/rec709
> >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' #
> >>> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709
> >>> ycbcr:601 quantization:lim-range
> >>> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709
> >>> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720
> >>> colorspace:rec709 ycbcr:709]'
> >>> ^^^ no error but doing a get-v4l2 shows no change - is this all
> >>> because of trying to deal with 1080 instead of 1024?
> >> No. The Image Converter can only Y`CbCr encode to ITU-R 601
> >> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr
> >> encoding when using an IC pipeline.
> >>
> >> See imx_media_fill_default_mbus_fields().
> >>
> >>> gst-launch-1.0 v4l2src device=/dev/video0 ! \
> >>>     v4l2h264enc output-io-mode=dmabuf-import ! \
> >>>     rtph264pay ! udpsink host=172.24.20.19 port=5001
> >>> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
> >>> ^^^ fails because of colorimetry not getting set
> >> Right, that is correct behavior also. See above.
> >>
> >> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace?
> >>
> > I will have to go back and look what the old downstream Freescale
> > kernel drivers did here... I could swear I could easily de-interlace
> > via VDIC and encode 1080i with that and gstreamer-imx.

The old freescale kernel + gstreamer-imx does require using the IPU to
transform the colorspace before it goes to CODA but I'm still trying
to understand the gstreamer code. The pixel formats vs colorspace vs
colorimetry still confuse me.

So again, I'm struggling with how to get the output of the IC into coda.

> >
> >>> <snip>
> >>>
> >>> Let's try an interlaced format...
> >>>
> >>> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0
> >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> >>> # set sensor output pad to sensor source format
> >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> >>> # sensor format
> >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
> >>> field:seq-tb colorspace:rec709
> >>> # reset all links
> >>> media-ctl --reset
> >>> # setup links
> >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
> >>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
> >>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
> >>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
> >>> # configure pads
> >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
> >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
> >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
> >>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
> >>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
> >>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
> >>> # capture device
> >>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
> >>> v4l2-ctl --device /dev/video1 --get-fmt-video
> >>> Format Video Capture:
> >>>           Width/Height      : 720/480
> >>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >>>           Field             : None
> >>>           Bytes per Line    : 1440
> >>>           Size Image        : 691200
> >>>           Colorspace        : Rec. 709
> >>>           Transfer Function : Rec. 709
> >>>           YCbCr/HSV Encoding: ITU-R 601
> >>>           Quantization      : Limited Range
> >>>           Flags             :
> >>> # capture 1 frame
> >>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
> >>> ^^^ fails with ipu1_ic_prpvf: EOF timeout
> >>> ^^^ this is almost identical to the adv7180 case which works... the
> >>> only exception I see is field:seq-tb vs field:alternate from the
> >>> source which should be fine (more specific than 'alternate')
> >> Hmm, can you send the complete pad formats from --get-v4l2. I don't see
> >> anything obviously wrong here.
> >>
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
> >                  [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
> >                  [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1"
> >                  [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb
> > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2"
> >                  [fmt:AYUV8_1X32/720x480@1/60 field:none
> > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2"
> >                  [fmt:AYUV8_1X32/720x480@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1"
> >                  [fmt:AYUV8_1X32/720x480@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> >
> > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 720/480
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 1440
> >          Size Image        : 691200
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: ITU-R 601
> >          Quantization      : Limited Range
> >          Flags             :
> >
> > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap
> > --stream-to=x.raw --stream-count=1
> > [  159.828654] ipu1_ic_prpvf: EOF timeout
> > VIDIOC_DQBUF: failed: Input/output error
>
> Sorry everything looks fine, I don't know why that pipeline is failing
> to generate any data.
>

This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
still baffles me a bit but I've also found that any bt656 capture that
isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
the resolution checks in ipu_csi_init_interface() resulting in
'Unsupported interlaced video mode'. I'm not sure if
ipu_csi_set_bt_interlaced_codes() can be modified to support other
resolutions?

> >
<snip>
> >
> >>> ok now I'm going to boot with a different device-tree that connects
> >>> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be
> >>> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed
> >>> for coda 3) ability to capture 1080p60 as I have confirmed tyring to
> >>> do so via bt656 exceeds the IMX pixel clock rate and causes CSI
> >>> corruption. Downsides to this mode is that I currently have a bug in
> >>> the tda1997x driver regarding this mode and interlaced capture.
> >>>
> >>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0
> >>> MODE0:sensor->mux->csi
> >>> # set sensor output pad to sensor source format
> >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> >>> # sensor format
> >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480
> >>> field:none colorspace:rec709
> >>> # reset all links
> >>> media-ctl --reset
> >>> # setup links
> >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> >>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> >>> # configure pads
> >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]"
> >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]"
> >>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
> >>> media-ctl --get-v4l2 '"ipu1_csi0":2'
> >>>                   [fmt:UYVY8_1X16/720x480@1/30 field:none
> >> This is passthrough, the CSI can only pass through input pixel data
> >> unmodified as "generic" data, when the input bus is 16-bit parallel,
> >> which it is in this case (UYVY8_1X16). So the pixel code at the CSI
> >> output pad is forced to same as input pad (UYVY8_1X16).
> >>
> >> But aside from passthrough, that shouldn't be a problem...
> >>
> >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> >>> # capture device
> >>> media-ctl -e 'ipu1_csi0 capture' # /dev/video4
> >>> v4l2-ctl --device /dev/video4 --get-fmt-video
> >>> Format Video Capture:
> >>>           Width/Height      : 720/480
> >>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >>>           Field             : None
> >>>           Bytes per Line    : 1440
> >>>           Size Image        : 691200
> >>>           Colorspace        : Rec. 709
> >>>           Transfer Function : Rec. 709
> >>>           YCbCr/HSV Encoding: Rec. 709
> >>>           Quantization      : Limited Range
> >>>           Flags             : premultiplied-alpha
> >>> # capture 1 frame
> >>> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1
> >>> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png
> >> So capture of 1 frame worked fine?
> > yes this worked fine
> >
> >>> # stream JPEG/RTP/UDP
> >>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >>>     video/x-raw,format=UYVY ! \
> >>>     jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> >> and this worked?
> > yes this worked fine
> >
> >>> # stream H264/RTP/UDP
> >>> media-ctl --get-v4l2 '"ipu1_csi0":0'
> >>>                   [fmt:UYVY8_1X16/720x480@1/30 field:none
> >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
> >>>                    crop.bounds:(0,0)/720x480
> >>>                    crop:(0,0)/720x480
> >>>                    compose.bounds:(0,0)/720x480
> >>>                    compose:(0,0)/720x480]
> >>> ^^^ no conv needed
> >>> media-ctl --get-v4l2 '"ipu1_csi0":0'
> >> huh?
> > typo'd here and cut-and-pasted another occurrence of the media-ctl
> > --get-v4l2 command but my point here is that we have the correct
> > colorspace/colorimetry for coda so the H264 below should work fine
> >
> >>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >>>     v4l2h264enc output-io-mode=dmabuf-import ! \
> >>>     rtph264pay ! udpsink host=172.24.20.19 port=5001
> >>> v4l2src0: Internal data stream error.
> >> Sorry, I don't have enough info to help here.
> >>
> > Here's the pad details:
> >
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
> >                  [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
> >                  [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2"
> >                  [fmt:UYVY8_1X16/720x480@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> >
> > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 720/480
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 1440
> >          Size Image        : 691200
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: Rec. 709
> >          Quantization      : Limited Range
> >          Flags             :
> >
> > So in this passthrough case v4l2-ctl stream capture works, gstreamer
> > capture with sw jpeg encode works, but gstreamer capture with h264
> > encode fails:
> >
> > root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >>     v4l2h264enc output-io-mode=dmabuf-import ! \
> >>     rtph264pay ! udpsink host=172.24.20.19 port=5001
> > Setting pipeline to PAUSED ...
> > Pipeline is live and does not need PREROLL ...
> > Setting pipeline to PLAYING ...
> > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > Internal data stream error.
> > Additional debug info:
> > gstbasesrc.c(3055): gst_base_src_loop ():
> > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > streaming stopped, reason not-negotiated (-4)
> > Execution ended after 0:00:00.000416672
> > Setting pipeline to PAUSED ...
> > Setting pipeline to READY ...
> > Setting pipeline to NULL ...
> > Freeing pipeline ...
> >
> > It would appear that this must have something to do with gstreamer.
> > The only difference here between this case and the adv7180 is that
> > field=none and colorimetry is itu601... I'm guessing the field=none is
> > causing an issue. I'll post a question to gstreamer-devel.
>
> Ok. Given that v4l2-ctl stream capture works, gstreamer capture with sw
> jpeg encode works, but gstreamer capture with h264does not, this does
> sound like some issue with gstreamer.
>

While the 480p60Hz YUV via 16bit YUV sensor->mux->csi appears to work
from a V4L2 capture perspective I find that this input with the IC
causes 'pipeline start failed with -32' errors:

# imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
MODE1:sensor->mux->csi->ic_prp->ic_prpenc
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0'
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
# get framerate
v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
DV timings:
        Active width: 1280
        Active height: 720
        Total width: 1650
        Total height: 750
        Frame format: progressive
        Polarities: +vsync +hsync
        Pixelclock: 74250000 Hz (60.00 frames per second)
        Horizontal frontporch: 110
        Horizontal sync: 40
        Horizontal backporch: 220
        Vertical frontporch: 5
        Vertical sync: 5
        Vertical backporch: 20
        Standards: CTA-861
        CTA-861 VIC: 0
        Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC

# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
# capture device
media-ctl -e 'ipu1_ic_prpenc capture'
/dev/video0
v4l2-ctl --device /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 640/360
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1280
        Size Image        : 460800
        Colorspace        : Rec. 709
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :

# capture 1 frame
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
[  125.966980] ipu1_ic_prpenc: pipeline start failed with -32

Do you know what the failure is here?

And can you explain why I can't colorspace convert the following CSI
capture case?:

# imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0'
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
                [fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
^^^ still itu601

I have found in my testing when dealing with a BT656 RGB colorspace
input which I need to convert ot rec709 I have to set the 'input' pad
of the csi which doesn't make sense and if I do this with a 16-bit RGB
colorspace (above) it jacks up the fmt:
media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
                [fmt:UYVY8_2X8/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults
fmt to YUVY8_2X8

So there is something I'm still doing wrong to setup CSC.

Thanks,

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

On 1/30/19 5:18 PM, Tim Harvey wrote:
> On Mon, Jan 28, 2019 at 3:04 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>>
>>
>> On 1/28/19 11:03 AM, Tim Harvey wrote:
>>> On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>>>> Hi Tim, cc: Philipp,
>>>>
>>>> On 1/23/19 3:21 PM, Tim Harvey wrote:
>>>>> Steve,
>>>>>
>>>>> I'm testing IMX6 capture again with the tda1997x HDMI receiver found
>>>>> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6
>>>>> CSI in a way where it can be configured for 8bit BT656 mode
>>>>> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall
>>>>> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide
>>>>> variety of resolutions, bus formats, interlaced modes, and
>>>>> colorspaces.
>>>>>
>>>>> I'm using the following for testing:
>>>>> - Linux 4.20 +
>>>>>      - media: imx-csi: Input connections to CSI should be optional
>>>>>      - imx-media: Fixes for interlaced capture
>>>>>      - media: imx: Stop stream before disabling IDMA channels
>>>>> - v4l-utils 1.16.1
>>>>> - gstreamer-1.14.1 (with the ability to test master as well)
>>>>>
>>>>> I have a script that generates the media-ctl/v4l2-ctl commands based
>>>>> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI
>>>>> decoder) and a 'mode' which I've defined as:
>>>>> mode0: sensor -> mux -> csi -> /dev/videoN
>>>>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a
>>>>> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN
>>>>> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN
>>>>>
>>>>> The media-ctl topologies for each cpu/board combo are at
>>>>> http://dev.gateworks.com/docs/linux/media/
>>>>>
>>>>> I'm trying to test out simple v4l2-ctl based capture of a single frame
>>>>> as well as capture and stream via both software JPEG encode and H264
>>>>> hardware encode via CODA.
>>>>>
>>>>> Please let me know of any changes that should be made to the commands
>>>>> below even if only to purely help document things through clarity.
>>>>>
>>>>> One of the issues I run into right away is image size: The imx.rst
>>>>> docs state that the ic has a 'resize' limit of 1024x1024 but I think
>>>>> I'm mislead by the word 'resize' and this also means you can't push
>>>>> say 1080x720 through it (not resizing, just putting that on the src
>>>>> pad) as it clips it to 1024x720.
>>>> The IC is limited to a resized *output* frame of 1024x1024, no higher.
>>>>
>>> Ok so this means for anything >1024 I need to either use the csi to
>>> downscale it by factors of 2 first, or use mem2mem solution which
>>> means I can't use the VDIC (so no interlaced).
>> Actually that isn't true. You can still use a VDIC pipeline for the
>> 1280x720 interlaced tda19971 source. It's just that you would need to
>> downscale to <=1024 (either using the CSI /2, and/or using the IC prpvf
>> scaler). Then use the mem2mem gstreamer element to upscale the prpvf
>> capture output to something >1024.
>>
> True, that would work and here are my mem2mem testing findings:
>
> I've found that mem2mem from the CSI works well:
> - Using the CSI to downscale 720p then going through mem2mem and
> upscaling it (works perfectly!)

Good to hear!

> # 720p@60Hz YUV BT656
> # query current timings and set to this
> v4l2-ctl -d $(media-ctl -e "tda19971 2-0048") --set-dv-bt-timings=query
> # get timings
> media-ctl --get-v4l2 '"tda19971 2-0048":0' # [fmt:UYVY8_2X8/1280x720
> field:none colorspace:srgb]
> # reset media controller links
> media-ctl --reset
> # create links
> media-ctl --link '"tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]'
> media-ctl --link '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> media-ctl --link '"ipu1_csi0":2 -> "ipu1_csi0 capture":0[1]'
> # set pad formats
> media-ctl --set-v4l2 '"tda19971 2-0048":0[fmt:UYVY8_2X8/1280x720]'
> media-ctl --set-v4l2 '"ipu1_csi0_mux":2[fmt:UYVY8_2X8/1280x720]'
> media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/1280x720
> colorspace:rec709 ycbcr:709]'
> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale

It's not needed to do 1/2 scale in this use-case, since this pipeline is 
not sending to the IC.

But even for an IC pipeline, for this case (1280x720) you only need to 
downscale 1/2 in width, not in both width and height (the CSI can 
independently downscale in either dimension), E.g:

media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x720]' # 1/2 scale in width


> media-ctl --get-v4l2 '"ipu1_csi0":2' # [fmt:AYUV8_1X32/640x360@1/30
> field:none colorspace:rec709 xfer:709 ycbcr:709
> quantization:lim-range]
> # get video device
> media-ctl -e 'ipu1_csi0 capture' # /dev/video4
> # stream JPEG/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>    video/x-raw,format=UYVY ! \
>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> # ^^^^ works but video is 1/4scale 640x360
> # get mem2mem device (here is perhaps a great reason to have the
> mem2mem device in the media graph... so its easy to find the device
> name)
> grep ipu_ic_pp /sys/class/video4linux/*/name #
> /sys/class/video4linux/video8/name:ipu_ic_pp mem2mem - so its
> v4l2video8convert
> # stream upscale via mem2mem then JPEG/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    video/x-raw,format=UYVY ! \
>    v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> # ^^^ works and video is back at 720p!
> # stream H264/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
>    v4l2h264enc output-io-mode=dmabuf-import ! \
>    rtph264pay ! udpsink host=172.24.20.19 port=5001
> # ^^^ works - 720p via coda

Great.

>
> But when going through the IC we again run into the issue where the
> output of the IC isn't a suitable colorspace:
> # 720p@60Hz YUV BT656
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
> field:none colorspace:srgb
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> # downscale because the IC can't do >1024
> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' #
> [fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb
> ycbcr:601 quantization:lim-range]
> # ^^^ note we are itu601 again b/c that's the only format the IC can ouput
> # stream JPEG/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>    video/x-raw,format=UYVY ! \
>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> # ^^^ works... SW JPEG can handle itu601

Ok.

> # stream upscale via mem2mem then JPEG/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> ERROR: from element
> /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device
> '/dev/video8' does not support 2:4:7:1 colorimetry
> # ^^^ fails because mem2mem doesn't support itu601
> # stream H264/RTP/UDP
> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>    v4l2h264enc output-io-mode=dmabuf-import ! \
>    rtph264pay ! udpsink host=172.24.20.19 port=5001
> ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
> Device '/dev/video9' does not support 2:4:7:1 colorimetry
> # ^^^ coda has same issue... can't del with itu601

Well, just to see things working, try hacking 
imx_media_fill_default_mbus_fields() to set Rec. 709 encoding:

--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct 
v4l2_mbus_framefmt *tryfmt,
                 tryfmt->quantization = is_rgb ?
                         V4L2_QUANTIZATION_FULL_RANGE :
                         V4L2_QUANTIZATION_LIM_RANGE;
-               tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601;
+               tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709;
         }
  }
  EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields);


But of course that's not technically correct because the encoding in 
ipu-ic.c is BT.601.

The *real* way to fix this would be to allow programmable encodings in 
ipu-ic.c. But unfortunately the encodings are hardcoded (grep for 
ic_csc_rgb2ycbcr in ipu-ic.c).

The other option would be to change ic_csc_rgb2ycbcr to use the Rec. 709 
coefficients, and then the above patch is no longer a hack. The inverse 
encoding (ic_csc_ycbcr2rgb) would also need to be changed.


>
> Am I perhaps missing a capsfilter to get the mem2mem driver to convert
> the colorspace properly? If so, they the mem2mem driver could be used
> to correct the colorspace to get IC output to coda working.

Well, first I don't think the mem2mem driver is using the correct 
encoding. The mem2mem driver is making use of the IC encoding so it 
should be reporting and accepting only BT.601.

Philipp ^^


>
> Also can we connect the mem2mem driver to the unused VDIC input in the
> media controller so that we can use the VDIC to de-interlace content
> captured from non IMX sources (ie PCI or USB capture devices)?

Exactly! That's something I have been working on. But it's difficult to 
connect mem2mem to the unused VDIC IDMAC input pad because as of now the 
v4l2 mem2mem internal API's do not allow connecting to *existing* 
processing entities, and there are also issues with how sub-devices are 
to deal with mem2mem contexts.

I do have a WIP branch that creates a video output device that connects 
to the VDIC IDMAC input pad, which doesn't have the above issues. The 
only drawback with that is how gstreamer can make use of such an output 
device.


> <snip>
>>>>> Another issue I'm running into is colorspace conversion, specifically
>>>>> colorimetry
>>>>>
>>>>> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0
>>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>>>> # set sensor output pad to sensor source format
>>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>>>> # sensor format
>>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
>>>>> field:none colorspace:rec709
>>>>> # reset all links
>>>>> media-ctl --reset
>>>>> # setup links
>>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>>>> # configure pads
>>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
>>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
>>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]"
>>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]"
>>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]"
>>>>> # capture device
>>>>> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0
>>>>> v4l2-ctl --device /dev/video0 --get-fmt-video
>>>>> Format Video Capture:
>>>>>            Width/Height      : 1024/720
>>>>>            Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>>>            Field             : None
>>>>>            Bytes per Line    : 2048
>>>>>            Size Image        : 1474560
>>>>>            Colorspace        : Rec. 709
>>>>>            Transfer Function : Rec. 709
>>>>>            YCbCr/HSV Encoding: ITU-R 601
>>>>>            Quantization      : Limited Range
>>>>>            Flags             :
>>>>> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch!
>>>> Yes as discussed above that is correct behavior, you'll need mem2mem
>>>> device to support >1024 tiled scaling.
>>>>
>>>>> # capture 1 frame
>>>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
>>>>> ^^^ works
>>>>> # stream JPEG/RTP/UDP
>>>>> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>>>>>      video/x-raw,format=UYVY ! \
>>>>>      jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
>>>>> ^^^ works (but of course squashed horizontally because of x=1024)
>>>>> # stream H264/RTP/UDP (via coda)
>>>>> # need to make sure we feed coda NV12/rec709
>>>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' #
>>>>> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709
>>>>> ycbcr:601 quantization:lim-range
>>>>> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709
>>>>> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720
>>>>> colorspace:rec709 ycbcr:709]'
>>>>> ^^^ no error but doing a get-v4l2 shows no change - is this all
>>>>> because of trying to deal with 1080 instead of 1024?
>>>> No. The Image Converter can only Y`CbCr encode to ITU-R 601
>>>> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr
>>>> encoding when using an IC pipeline.
>>>>
>>>> See imx_media_fill_default_mbus_fields().
>>>>
>>>>> gst-launch-1.0 v4l2src device=/dev/video0 ! \
>>>>>      v4l2h264enc output-io-mode=dmabuf-import ! \
>>>>>      rtph264pay ! udpsink host=172.24.20.19 port=5001
>>>>> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry
>>>>> ^^^ fails because of colorimetry not getting set
>>>> Right, that is correct behavior also. See above.
>>>>
>>>> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace?
>>>>
>>> I will have to go back and look what the old downstream Freescale
>>> kernel drivers did here... I could swear I could easily de-interlace
>>> via VDIC and encode 1080i with that and gstreamer-imx.
> The old freescale kernel + gstreamer-imx does require using the IPU to
> transform the colorspace before it goes to CODA but I'm still trying
> to understand the gstreamer code. The pixel formats vs colorspace vs
> colorimetry still confuse me.
>
> So again, I'm struggling with how to get the output of the IC into coda.

See above, try just hacking with the above patch for now to get things 
working (albeit with incorrect Y'CbCr encoding).

>
>>>>> <snip>
>>>>>
>>>>> Let's try an interlaced format...
>>>>>
>>>>> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0
>>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>>>> # set sensor output pad to sensor source format
>>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>>>> # sensor format
>>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480
>>>>> field:seq-tb colorspace:rec709
>>>>> # reset all links
>>>>> media-ctl --reset
>>>>> # setup links
>>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]"
>>>>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]"
>>>>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]"
>>>>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]"
>>>>> # configure pads
>>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
>>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]"
>>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]"
>>>>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]"
>>>>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]"
>>>>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]"
>>>>> # capture device
>>>>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1
>>>>> v4l2-ctl --device /dev/video1 --get-fmt-video
>>>>> Format Video Capture:
>>>>>            Width/Height      : 720/480
>>>>>            Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>>>            Field             : None
>>>>>            Bytes per Line    : 1440
>>>>>            Size Image        : 691200
>>>>>            Colorspace        : Rec. 709
>>>>>            Transfer Function : Rec. 709
>>>>>            YCbCr/HSV Encoding: ITU-R 601
>>>>>            Quantization      : Limited Range
>>>>>            Flags             :
>>>>> # capture 1 frame
>>>>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1
>>>>> ^^^ fails with ipu1_ic_prpvf: EOF timeout
>>>>> ^^^ this is almost identical to the adv7180 case which works... the
>>>>> only exception I see is field:seq-tb vs field:alternate from the
>>>>> source which should be fine (more specific than 'alternate')
>>>> Hmm, can you send the complete pad formats from --get-v4l2. I don't see
>>>> anything obviously wrong here.
>>>>
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>>>                   [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>>>                   [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1"
>>>                   [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb
>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2"
>>>                   [fmt:AYUV8_1X32/720x480@1/60 field:none
>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2"
>>>                   [fmt:AYUV8_1X32/720x480@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1"
>>>                   [fmt:AYUV8_1X32/720x480@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>>
>>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 720/480
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 1440
>>>           Size Image        : 691200
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: ITU-R 601
>>>           Quantization      : Limited Range
>>>           Flags             :
>>>
>>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap
>>> --stream-to=x.raw --stream-count=1
>>> [  159.828654] ipu1_ic_prpvf: EOF timeout
>>> VIDIOC_DQBUF: failed: Input/output error
>> Sorry everything looks fine, I don't know why that pipeline is failing
>> to generate any data.
>>
> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
> still baffles me a bit but I've also found that any bt656 capture that
> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
> the resolution checks in ipu_csi_init_interface() resulting in
> 'Unsupported interlaced video mode'. I'm not sure if
> ipu_csi_set_bt_interlaced_codes() can be modified to support other
> resolutions?

Well, Bt.656 only defines standard definition NTSC and PAL.

> <snip>
>>>>> ok now I'm going to boot with a different device-tree that connects
>>>>> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be
>>>>> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed
>>>>> for coda 3) ability to capture 1080p60 as I have confirmed tyring to
>>>>> do so via bt656 exceeds the IMX pixel clock rate and causes CSI
>>>>> corruption. Downsides to this mode is that I currently have a bug in
>>>>> the tda1997x driver regarding this mode and interlaced capture.
>>>>>
>>>>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0
>>>>> MODE0:sensor->mux->csi
>>>>> # set sensor output pad to sensor source format
>>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>>>> # sensor format
>>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480
>>>>> field:none colorspace:rec709
>>>>> # reset all links
>>>>> media-ctl --reset
>>>>> # setup links
>>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>>>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
>>>>> # configure pads
>>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]"
>>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]"
>>>>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]"
>>>>> media-ctl --get-v4l2 '"ipu1_csi0":2'
>>>>>                    [fmt:UYVY8_1X16/720x480@1/30 field:none
>>>> This is passthrough, the CSI can only pass through input pixel data
>>>> unmodified as "generic" data, when the input bus is 16-bit parallel,
>>>> which it is in this case (UYVY8_1X16). So the pixel code at the CSI
>>>> output pad is forced to same as input pad (UYVY8_1X16).
>>>>
>>>> But aside from passthrough, that shouldn't be a problem...
>>>>
>>>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
>>>>> # capture device
>>>>> media-ctl -e 'ipu1_csi0 capture' # /dev/video4
>>>>> v4l2-ctl --device /dev/video4 --get-fmt-video
>>>>> Format Video Capture:
>>>>>            Width/Height      : 720/480
>>>>>            Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>>>            Field             : None
>>>>>            Bytes per Line    : 1440
>>>>>            Size Image        : 691200
>>>>>            Colorspace        : Rec. 709
>>>>>            Transfer Function : Rec. 709
>>>>>            YCbCr/HSV Encoding: Rec. 709
>>>>>            Quantization      : Limited Range
>>>>>            Flags             : premultiplied-alpha
>>>>> # capture 1 frame
>>>>> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1
>>>>> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png
>>>> So capture of 1 frame worked fine?
>>> yes this worked fine
>>>
>>>>> # stream JPEG/RTP/UDP
>>>>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>>>>      video/x-raw,format=UYVY ! \
>>>>>      jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
>>>> and this worked?
>>> yes this worked fine
>>>
>>>>> # stream H264/RTP/UDP
>>>>> media-ctl --get-v4l2 '"ipu1_csi0":0'
>>>>>                    [fmt:UYVY8_1X16/720x480@1/30 field:none
>>>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range
>>>>>                     crop.bounds:(0,0)/720x480
>>>>>                     crop:(0,0)/720x480
>>>>>                     compose.bounds:(0,0)/720x480
>>>>>                     compose:(0,0)/720x480]
>>>>> ^^^ no conv needed
>>>>> media-ctl --get-v4l2 '"ipu1_csi0":0'
>>>> huh?
>>> typo'd here and cut-and-pasted another occurrence of the media-ctl
>>> --get-v4l2 command but my point here is that we have the correct
>>> colorspace/colorimetry for coda so the H264 below should work fine
>>>
>>>>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>>>>      v4l2h264enc output-io-mode=dmabuf-import ! \
>>>>>      rtph264pay ! udpsink host=172.24.20.19 port=5001
>>>>> v4l2src0: Internal data stream error.
>>>> Sorry, I don't have enough info to help here.
>>>>
>>> Here's the pad details:
>>>
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>>>                   [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>>>                   [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709]
>>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2"
>>>                   [fmt:UYVY8_1X16/720x480@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
>>>
>>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 720/480
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 1440
>>>           Size Image        : 691200
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: Rec. 709
>>>           Quantization      : Limited Range
>>>           Flags             :
>>>
>>> So in this passthrough case v4l2-ctl stream capture works, gstreamer
>>> capture with sw jpeg encode works, but gstreamer capture with h264
>>> encode fails:
>>>
>>> root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>>>      v4l2h264enc output-io-mode=dmabuf-import ! \
>>>>      rtph264pay ! udpsink host=172.24.20.19 port=5001
>>> Setting pipeline to PAUSED ...
>>> Pipeline is live and does not need PREROLL ...
>>> Setting pipeline to PLAYING ...
>>> ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
>>> Internal data stream error.
>>> Additional debug info:
>>> gstbasesrc.c(3055): gst_base_src_loop ():
>>> /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
>>> streaming stopped, reason not-negotiated (-4)
>>> Execution ended after 0:00:00.000416672
>>> Setting pipeline to PAUSED ...
>>> Setting pipeline to READY ...
>>> Setting pipeline to NULL ...
>>> Freeing pipeline ...
>>>
>>> It would appear that this must have something to do with gstreamer.
>>> The only difference here between this case and the adv7180 is that
>>> field=none and colorimetry is itu601... I'm guessing the field=none is
>>> causing an issue. I'll post a question to gstreamer-devel.
>> Ok. Given that v4l2-ctl stream capture works, gstreamer capture with sw
>> jpeg encode works, but gstreamer capture with h264does not, this does
>> sound like some issue with gstreamer.
>>
> While the 480p60Hz YUV via 16bit YUV sensor->mux->csi appears to work
> from a V4L2 capture perspective I find that this input with the IC
> causes 'pipeline start failed with -32' errors:
>
> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> # get framerate
> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
> DV timings:
>          Active width: 1280
>          Active height: 720
>          Total width: 1650
>          Total height: 750
>          Frame format: progressive
>          Polarities: +vsync +hsync
>          Pixelclock: 74250000 Hz (60.00 frames per second)
>          Horizontal frontporch: 110
>          Horizontal sync: 40
>          Horizontal backporch: 220
>          Vertical frontporch: 5
>          Vertical sync: 5
>          Vertical backporch: 20
>          Standards: CTA-861
>          CTA-861 VIC: 0
>          Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
>
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
> # capture device
> media-ctl -e 'ipu1_ic_prpenc capture'
> /dev/video0
> v4l2-ctl --device /dev/video0 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 640/360
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1280
>          Size Image        : 460800
>          Colorspace        : Rec. 709
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: ITU-R 601
>          Quantization      : Limited Range
>          Flags             :
>
> # capture 1 frame
> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
>
> Do you know what the failure is here?

You are /2 downscaling in the CSI, but did not set the compose window at 
the input pad, e.g:

media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale



>
> And can you explain why I can't colorspace convert the following CSI
> capture case?:
>
> # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> media-ctl --get-v4l2 '"ipu1_csi0":2'
>                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> ^^^ still itu601
>
> I have found in my testing when dealing with a BT656 RGB colorspace
> input which I need to convert ot rec709 I have to set the 'input' pad
> of the csi which doesn't make sense and if I do this with a 16-bit RGB
> colorspace (above) it jacks up the fmt:
> media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> media-ctl --get-v4l2 '"ipu1_csi0":2'
>                  [fmt:UYVY8_2X8/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults
> fmt to YUVY8_2X8
>
> So there is something I'm still doing wrong to setup CSC.

Sorry I guess I don't understand your question here. Correct the CSI 
does not accept AYUV32 at its input pad.

Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

Hi Tim, cc: Philipp,

On 2/2/19 11:10 AM, Steve Longerbeam wrote:
>
> <snip>
> On 1/30/19 5:18 PM, Tim Harvey wrote:
>
>> # stream upscale via mem2mem then JPEG/RTP/UDP
>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>    v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
>>    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
>> ERROR: from element
>> /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device
>> '/dev/video8' does not support 2:4:7:1 colorimetry
>> # ^^^ fails because mem2mem doesn't support itu601
>> # stream H264/RTP/UDP
>> gst-launch-1.0 v4l2src device=/dev/video4 ! \
>>    v4l2h264enc output-io-mode=dmabuf-import ! \
>>    rtph264pay ! udpsink host=172.24.20.19 port=5001
>> ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
>> Device '/dev/video9' does not support 2:4:7:1 colorimetry
>> # ^^^ coda has same issue... can't del with itu601
>
> Well, just to see things working, try hacking 
> imx_media_fill_default_mbus_fields() to set Rec. 709 encoding:
>
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct 
> v4l2_mbus_framefmt *tryfmt,
>                 tryfmt->quantization = is_rgb ?
>                         V4L2_QUANTIZATION_FULL_RANGE :
>                         V4L2_QUANTIZATION_LIM_RANGE;
> -               tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601;
> +               tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709;
>         }
>  }
>  EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields);
>
>
> But of course that's not technically correct because the encoding in 
> ipu-ic.c is BT.601.
>
> The *real* way to fix this would be to allow programmable encodings in 
> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for 
> ic_csc_rgb2ycbcr in ipu-ic.c).

I went ahead and implemented this!

The IC will now support both BT.601 and BT.709 encoding. I will post a 
patchset, please test!


>>
>> Am I perhaps missing a capsfilter to get the mem2mem driver to convert
>> the colorspace properly? If so, they the mem2mem driver could be used
>> to correct the colorspace to get IC output to coda working.
>
> Well, first I don't think the mem2mem driver is using the correct 
> encoding. The mem2mem driver is making use of the IC encoding so it 
> should be reporting and accepting only BT.601.

The mem2mem driver can now require BT.709 and/or BT.601 encoding with 
the above patchset.

Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
>
<snip>
>
> >
> > But when going through the IC we again run into the issue where the
> > output of the IC isn't a suitable colorspace:
> > # 720p@60Hz YUV BT656
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
> > field:none colorspace:srgb
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> > # downscale because the IC can't do >1024
> > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' #
> > [fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb
> > ycbcr:601 quantization:lim-range]
> > # ^^^ note we are itu601 again b/c that's the only format the IC can ouput
> > # stream JPEG/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video0 ! \
> >    video/x-raw,format=UYVY ! \
> >    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> > # ^^^ works... SW JPEG can handle itu601
>
> Ok.
>
> > # stream upscale via mem2mem then JPEG/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >    v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
> >    jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> > ERROR: from element
> > /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device
> > '/dev/video8' does not support 2:4:7:1 colorimetry
> > # ^^^ fails because mem2mem doesn't support itu601
> > # stream H264/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video4 ! \
> >    v4l2h264enc output-io-mode=dmabuf-import ! \
> >    rtph264pay ! udpsink host=172.24.20.19 port=5001
> > ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
> > Device '/dev/video9' does not support 2:4:7:1 colorimetry
> > # ^^^ coda has same issue... can't del with itu601
>
> Well, just to see things working, try hacking
> imx_media_fill_default_mbus_fields() to set Rec. 709 encoding:
>
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct
> v4l2_mbus_framefmt *tryfmt,
>                  tryfmt->quantization = is_rgb ?
>                          V4L2_QUANTIZATION_FULL_RANGE :
>                          V4L2_QUANTIZATION_LIM_RANGE;
> -               tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601;
> +               tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709;
>          }
>   }
>   EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields);
>
>
> But of course that's not technically correct because the encoding in
> ipu-ic.c is BT.601.
>
> The *real* way to fix this would be to allow programmable encodings in
> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
> ic_csc_rgb2ycbcr in ipu-ic.c).
>

Ok, I saw that you went ahead and worked on this (thanks!) and that
you have a bt.709.v2 branch... is that one ready for testing?

> The other option would be to change ic_csc_rgb2ycbcr to use the Rec. 709
> coefficients, and then the above patch is no longer a hack. The inverse
> encoding (ic_csc_ycbcr2rgb) would also need to be changed.
>
>
> >
> > Am I perhaps missing a capsfilter to get the mem2mem driver to convert
> > the colorspace properly? If so, they the mem2mem driver could be used
> > to correct the colorspace to get IC output to coda working.
>
> Well, first I don't think the mem2mem driver is using the correct
> encoding. The mem2mem driver is making use of the IC encoding so it
> should be reporting and accepting only BT.601.
>
> Philipp ^^
>
>
> >
> > Also can we connect the mem2mem driver to the unused VDIC input in the
> > media controller so that we can use the VDIC to de-interlace content
> > captured from non IMX sources (ie PCI or USB capture devices)?
>
> Exactly! That's something I have been working on. But it's difficult to
> connect mem2mem to the unused VDIC IDMAC input pad because as of now the
> v4l2 mem2mem internal API's do not allow connecting to *existing*
> processing entities, and there are also issues with how sub-devices are
> to deal with mem2mem contexts.
>
> I do have a WIP branch that creates a video output device that connects
> to the VDIC IDMAC input pad, which doesn't have the above issues. The
> only drawback with that is how gstreamer can make use of such an output
> device.
>

ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch?

I also noticed you have a add-fim-to-prpencvf branch... are you
working on adding FIM to ipu_ic_prp/enc still? That would be nice to
have to deal with sync loss/regain in analog decoders going through
VDIC de-interlacing.

>
<snip>
> >>
> > This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
> > still baffles me a bit but I've also found that any bt656 capture that
> > isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
> > the resolution checks in ipu_csi_init_interface() resulting in
> > 'Unsupported interlaced video mode'. I'm not sure if
> > ipu_csi_set_bt_interlaced_codes() can be modified to support other
> > resolutions?
>
> Well, Bt.656 only defines standard definition NTSC and PAL.
>

That is true. Do you know of any other sensors that use higher
resolutions with BT656 SAV/EAV encoding? The BT656 mode does work well
for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6
pixel clock and can't be used).

I could dig into the bt656 spec to try and understand the various
codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I
suppose but I'm still not sure I want to push up a device-tree config
that describes the tda1997x to CSI connection as 8bit BT656 as I would
prefer to describe it as 16bit YUV instead (as I may be close to
getting that format working well if we can get the IC able to output
rec709).

<snip>

> >
> > # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
> > MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> >                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> > # get framerate
> > v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
> > DV timings:
> >          Active width: 1280
> >          Active height: 720
> >          Total width: 1650
> >          Total height: 750
> >          Frame format: progressive
> >          Polarities: +vsync +hsync
> >          Pixelclock: 74250000 Hz (60.00 frames per second)
> >          Horizontal frontporch: 110
> >          Horizontal sync: 40
> >          Horizontal backporch: 220
> >          Vertical frontporch: 5
> >          Vertical sync: 5
> >          Vertical backporch: 20
> >          Standards: CTA-861
> >          CTA-861 VIC: 0
> >          Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
> >
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
> > # capture device
> > media-ctl -e 'ipu1_ic_prpenc capture'
> > /dev/video0
> > v4l2-ctl --device /dev/video0 --get-fmt-video
> > Format Video Capture:
> >          Width/Height      : 640/360
> >          Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >          Field             : None
> >          Bytes per Line    : 1280
> >          Size Image        : 460800
> >          Colorspace        : Rec. 709
> >          Transfer Function : Rec. 709
> >          YCbCr/HSV Encoding: ITU-R 601
> >          Quantization      : Limited Range
> >          Flags             :
> >
> > # capture 1 frame
> > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> > [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
> >
> > Do you know what the failure is here?
>
> You are /2 downscaling in the CSI, but did not set the compose window at
> the input pad, e.g:
>
> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
>

I don't think this is the issue. Note that I 'was' able to get the
same pipeline with the div-by-2 downscale without compose working with
720p bt656 YUV.

Here is the 720p 16bit YUV bus again with the compose:
# 720p60Hz YUV 16bit YUV bus
# imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
MODE1:sensor->mux->csi->ic_prp->ic_prpenc
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
BT timings set
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0'
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
media-ctl --get-v4l2 "'ipu1_csi0':0"
                [fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/1280x720
                 crop:(0,0)/1280x720
                 compose.bounds:(0,0)/1280x720
                 compose:(0,0)/640x360]
media-ctl --get-v4l2 "'ipu1_csi0':1"
                [fmt:UYVY8_1X16/640x360@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
media-ctl --get-v4l2 "'ipu1_ic_prp':1"
                [fmt:AYUV8_1X32/640x480@1/30 field:none
colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
                [fmt:AYUV8_1X32/640x360@1001/30000 field:none
colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
media-ctl -e 'ipu1_ic_prpenc capture'
/dev/video0
v4l2-ctl --device /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 640/360
        Pixel Format      : 'UYVY' (UYVY 4:2:2)
        Field             : None
        Bytes per Line    : 1280
        Size Image        : 460800
        Colorspace        : SMPTE 170M
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Limited Range
        Flags             :
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
[ 1279.017148] ipu1_ic_prpenc: pipeline start failed with -32

>
>
> >
> > And can you explain why I can't colorspace convert the following CSI
> > capture case?:
> >
> > # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> >                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> >                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> > ^^^ still itu601
> >
> > I have found in my testing when dealing with a BT656 RGB colorspace
> > input which I need to convert ot rec709 I have to set the 'input' pad
> > of the csi which doesn't make sense and if I do this with a 16-bit RGB
> > colorspace (above) it jacks up the fmt:
> > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> >                  [fmt:UYVY8_2X8/1280x720@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> > ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults
> > fmt to YUVY8_2X8
> >
> > So there is something I'm still doing wrong to setup CSC.
>
> Sorry I guess I don't understand your question here. Correct the CSI
> does not accept AYUV32 at its input pad.
>

What I was asking here is what is the correct way to configure for
ITU601 to REC709 CSC at the CSI?  I would expect to set the REC709
format on the output pad of the CSI but it seems to require me to set
it on the input pad.

Consider:
# imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0'
                 [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
                [fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
^^^ ITU601, but I want REC709
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
               [fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
^^^ No invalid argument but its still ITU601 at CSI output... but what
I've found is if I set it at the CSI input the ycbcr changes but the
fmt gets jacked up (as expected)
media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
                [fmt:UYVY8_2X8/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI
doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8. However, again
I expect to have set the CSC on the output pad. If I do this on a
BT656 sensor bus it works because UYVY8_2X8 is valid.

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

On 2/5/19 11:16 AM, Tim Harvey wrote:
> On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com>  wrote:
>
> <snip>
>> The *real* way to fix this would be to allow programmable encodings in
>> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
>> ic_csc_rgb2ycbcr in ipu-ic.c).
>>
> Ok, I saw that you went ahead and worked on this (thanks!) and that
> you have a bt.709.v2 branch... is that one ready for testing?

Yes. I tried to test it too, but there is some regression in captured 
images, it could be something in ov5640.c (I'm testing with the SabreSD 
and the OV5640), or something else recently added. The regression looks 
like a stride problem, I'm hoping it wasn't the recently introduced 
compose window changes to relax width alignment. Have you tested with 
those commits and a prpenc/vf pipeline?

>> <snip>
>>> Also can we connect the mem2mem driver to the unused VDIC input in the
>>> media controller so that we can use the VDIC to de-interlace content
>>> captured from non IMX sources (ie PCI or USB capture devices)?
>> Exactly! That's something I have been working on. But it's difficult to
>> connect mem2mem to the unused VDIC IDMAC input pad because as of now the
>> v4l2 mem2mem internal API's do not allow connecting to *existing*
>> processing entities, and there are also issues with how sub-devices are
>> to deal with mem2mem contexts.
>>
>> I do have a WIP branch that creates a video output device that connects
>> to the VDIC IDMAC input pad, which doesn't have the above issues. The
>> only drawback with that is how gstreamer can make use of such an output
>> device.
>>
> ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch?

The output-vdic branch. It's almost ready to go, there is only some 
strange issue with low and medium motion-compensation modes (captured 
images show "snow").


> I also noticed you have a add-fim-to-prpencvf branch... are you
> working on adding FIM to ipu_ic_prp/enc still? That would be nice to
> have to deal with sync loss/regain in analog decoders going through
> VDIC de-interlacing.

I've been trying to get this working, I ran into some locking issues 
when enabling lock debug options. Still trying to find a solution.

> <snip>
>>> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
>>> still baffles me a bit but I've also found that any bt656 capture that
>>> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
>>> the resolution checks in ipu_csi_init_interface() resulting in
>>> 'Unsupported interlaced video mode'. I'm not sure if
>>> ipu_csi_set_bt_interlaced_codes() can be modified to support other
>>> resolutions?
>> Well, Bt.656 only defines standard definition NTSC and PAL.
>>
> That is true. Do you know of any other sensors that use higher
> resolutions with BT656 SAV/EAV encoding?

Nope all I have for testing is the NTSC/PAL ADV7180 on the SabreAuto.

>   The BT656 mode does work well
> for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6
> pixel clock and can't be used).
>
> I could dig into the bt656 spec to try and understand the various
> codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I
> suppose but I'm still not sure I want to push up a device-tree config
> that describes the tda1997x to CSI connection as 8bit BT656 as I would
> prefer to describe it as 16bit YUV instead (as I may be close to
> getting that format working well if we can get the IC able to output
> rec709).
>
> <snip>
>
>>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>>> # get framerate
>>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
>>> DV timings:
>>>           Active width: 1280
>>>           Active height: 720
>>>           Total width: 1650
>>>           Total height: 750
>>>           Frame format: progressive
>>>           Polarities: +vsync +hsync
>>>           Pixelclock: 74250000 Hz (60.00 frames per second)
>>>           Horizontal frontporch: 110
>>>           Horizontal sync: 40
>>>           Horizontal backporch: 220
>>>           Vertical frontporch: 5
>>>           Vertical sync: 5
>>>           Vertical backporch: 20
>>>           Standards: CTA-861
>>>           CTA-861 VIC: 0
>>>           Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
>>>
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
>>> # capture device
>>> media-ctl -e 'ipu1_ic_prpenc capture'
>>> /dev/video0
>>> v4l2-ctl --device /dev/video0 --get-fmt-video
>>> Format Video Capture:
>>>           Width/Height      : 640/360
>>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>           Field             : None
>>>           Bytes per Line    : 1280
>>>           Size Image        : 460800
>>>           Colorspace        : Rec. 709
>>>           Transfer Function : Rec. 709
>>>           YCbCr/HSV Encoding: ITU-R 601
>>>           Quantization      : Limited Range
>>>           Flags             :
>>>
>>> # capture 1 frame
>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
>>> [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
>>>
>>> Do you know what the failure is here?
>> You are /2 downscaling in the CSI, but did not set the compose window at
>> the input pad, e.g:
>>
>> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
>>
> I don't think this is the issue. Note that I 'was' able to get the
> same pipeline with the div-by-2 downscale without compose working with
> 720p bt656 YUV.
>
> Here is the 720p 16bit YUV bus again with the compose:
> # 720p60Hz YUV 16bit YUV bus
> # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> BT timings set
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> media-ctl --get-v4l2 "'ipu1_csi0':0"
>                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>                   crop.bounds:(0,0)/1280x720
>                   crop:(0,0)/1280x720
>                   compose.bounds:(0,0)/1280x720
>                   compose:(0,0)/640x360]
> media-ctl --get-v4l2 "'ipu1_csi0':1"
>                  [fmt:UYVY8_1X16/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>                  [fmt:AYUV8_1X32/640x480@1/30 field:none

Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting 
ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to 
default formats.

> colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
> media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
>                  [fmt:AYUV8_1X32/640x360@1001/30000 field:none
> colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
> media-ctl -e 'ipu1_ic_prpenc capture'
> /dev/video0
> v4l2-ctl --device /dev/video0 --get-fmt-video
> Format Video Capture:
>          Width/Height      : 640/360
>          Pixel Format      : 'UYVY' (UYVY 4:2:2)
>          Field             : None
>          Bytes per Line    : 1280
>          Size Image        : 460800
>          Colorspace        : SMPTE 170M
>          Transfer Function : Rec. 709
>          YCbCr/HSV Encoding: ITU-R 601
>          Quantization      : Limited Range
>          Flags             :
> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> [ 1279.017148] ipu1_ic_prpenc: pipeline start failed with -32
>
>>> And can you explain why I can't colorspace convert the following CSI
>>> capture case?:
>>>
>>> # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
>>> media-ctl --get-v4l2 '"ipu1_csi0":2'
>>>                   [fmt:UYVY8_1X16/1280x720@1/30 field:none
>>> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
>>> ^^^ still itu601
>>>
>>> I have found in my testing when dealing with a BT656 RGB colorspace
>>> input which I need to convert ot rec709 I have to set the 'input' pad
>>> of the csi which doesn't make sense and if I do this with a 16-bit RGB
>>> colorspace (above) it jacks up the fmt:
>>> media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
>>> media-ctl --get-v4l2 '"ipu1_csi0":2'
>>>                   [fmt:UYVY8_2X8/1280x720@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
>>> ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults
>>> fmt to YUVY8_2X8
>>>
>>> So there is something I'm still doing wrong to setup CSC.
>> Sorry I guess I don't understand your question here. Correct the CSI
>> does not accept AYUV32 at its input pad.
>>
> What I was asking here is what is the correct way to configure for
> ITU601 to REC709 CSC at the CSI?  I would expect to set the REC709
> format on the output pad of the CSI but it seems to require me to set
> it on the input pad.

Right, the driver only allows setting colorimetry at the sink pads, and 
propagates them to the source pads. So you'll need to set colorimetry 
params at the CSI sink pad.


> Consider:
> # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi
> # set sensor output pad to sensor source format
> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> # sensor format
> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
> # reset all links
> media-ctl --reset
> # setup links
> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> # configure pads
> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]"
> media-ctl --get-v4l2 '"ipu1_csi0":2'
>                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> ^^^ ITU601, but I want REC709
> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> media-ctl --get-v4l2 '"ipu1_csi0":2'
>                 [fmt:UYVY8_1X16/1280x720@1/30 field:none
> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> ^^^ No invalid argument but its still ITU601 at CSI output... but what
> I've found is if I set it at the CSI input the ycbcr changes but the
> fmt gets jacked up (as expected)
> media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
>                  [fmt:UYVY8_2X8/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI
> doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8.

Why doesn't the CSI accept UYVY8_1X16 at its sink pad?

This doesn't work?:

media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 colorspace:rec709 ycbcr:709]"


Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
>
>
> On 2/5/19 11:16 AM, Tim Harvey wrote:
> > On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com>  wrote:
> >
> > <snip>
> >> The *real* way to fix this would be to allow programmable encodings in
> >> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
> >> ic_csc_rgb2ycbcr in ipu-ic.c).
> >>
> > Ok, I saw that you went ahead and worked on this (thanks!) and that
> > you have a bt.709.v2 branch... is that one ready for testing?
>
> Yes. I tried to test it too, but there is some regression in captured
> images, it could be something in ov5640.c (I'm testing with the SabreSD
> and the OV5640), or something else recently added. The regression looks
> like a stride problem, I'm hoping it wasn't the recently introduced
> compose window changes to relax width alignment. Have you tested with
> those commits and a prpenc/vf pipeline?

I have been testing v4.20 with:
media: imx: lift CSI and PRP ENC/VF width alignment restriction
media: imx: set compose rectangle to mbus format
media: imx: add capture compose rectangle
imx: imx-media: register mem2mem device with media controller
media: imx: add mem2mem device
media: imx-csi: Skip first few frames from a BT.656 source
media: imx.rst: Update doc to reflect fixes to interlaced capture
media: imx: Allow interweave with top/bottom lines swapped
media: imx-csi: Move crop/compose reset after filling default mbus fields
media: imx: vdic: rely on VDIC for correct field order
media: imx-csi: Allow skipping odd chroma rows for YVU420
media: imx: interweave and odd-chroma-row skip are incompatible
media: imx-csi: Double crop height for alternate fields at sink
media: imx: Fix field negotiation
gpu: ipu-v3: Add planar support to interlaced scan
gpu: ipu-csi: Swap fields according to input/output field types
media: videodev2.h: Add more field helper macros
media: imx: prpencvf: Stop upstream before disabling IDMA channel
media: imx: csi: Stop upstream before disabling IDMA channel
media: imx: csi: Disable CSI immediately after last EOF
media: imx-csi: Input connections to CSI should be optional

What kind of artifacts are you seeing?

I noticed your 'media: imx: Add support for BT.709 encoding' series
don't apply cleanly to 4.20 that I'm working with so perhaps its
something else. I'll bump up to linux-media, apply your bt.709.v2 and
see what I get. The BT.709 support is critical for me otherwise I
can't use coda with any pipelines that go through the IC with the
tda1997x HDMI decoder.

>
> >> <snip>
> >>> Also can we connect the mem2mem driver to the unused VDIC input in the
> >>> media controller so that we can use the VDIC to de-interlace content
> >>> captured from non IMX sources (ie PCI or USB capture devices)?
> >> Exactly! That's something I have been working on. But it's difficult to
> >> connect mem2mem to the unused VDIC IDMAC input pad because as of now the
> >> v4l2 mem2mem internal API's do not allow connecting to *existing*
> >> processing entities, and there are also issues with how sub-devices are
> >> to deal with mem2mem contexts.
> >>
> >> I do have a WIP branch that creates a video output device that connects
> >> to the VDIC IDMAC input pad, which doesn't have the above issues. The
> >> only drawback with that is how gstreamer can make use of such an output
> >> device.
> >>
> > ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch?
>
> The output-vdic branch. It's almost ready to go, there is only some
> strange issue with low and medium motion-compensation modes (captured
> images show "snow").
>
>
> > I also noticed you have a add-fim-to-prpencvf branch... are you
> > working on adding FIM to ipu_ic_prp/enc still? That would be nice to
> > have to deal with sync loss/regain in analog decoders going through
> > VDIC de-interlacing.
>
> I've been trying to get this working, I ran into some locking issues
> when enabling lock debug options. Still trying to find a solution.
>
> > <snip>
> >>> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
> >>> still baffles me a bit but I've also found that any bt656 capture that
> >>> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
> >>> the resolution checks in ipu_csi_init_interface() resulting in
> >>> 'Unsupported interlaced video mode'. I'm not sure if
> >>> ipu_csi_set_bt_interlaced_codes() can be modified to support other
> >>> resolutions?
> >> Well, Bt.656 only defines standard definition NTSC and PAL.
> >>
> > That is true. Do you know of any other sensors that use higher
> > resolutions with BT656 SAV/EAV encoding?
>
> Nope all I have for testing is the NTSC/PAL ADV7180 on the SabreAuto.
>
> >   The BT656 mode does work well
> > for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6
> > pixel clock and can't be used).
> >
> > I could dig into the bt656 spec to try and understand the various
> > codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I
> > suppose but I'm still not sure I want to push up a device-tree config
> > that describes the tda1997x to CSI connection as 8bit BT656 as I would
> > prefer to describe it as 16bit YUV instead (as I may be close to
> > getting that format working well if we can get the IC able to output
> > rec709).
> >
> > <snip>
> >
> >>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
> >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> >>> # set sensor output pad to sensor source format
> >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> >>> # sensor format
> >>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
> >>>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> >>> # get framerate
> >>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
> >>> DV timings:
> >>>           Active width: 1280
> >>>           Active height: 720
> >>>           Total width: 1650
> >>>           Total height: 750
> >>>           Frame format: progressive
> >>>           Polarities: +vsync +hsync
> >>>           Pixelclock: 74250000 Hz (60.00 frames per second)
> >>>           Horizontal frontporch: 110
> >>>           Horizontal sync: 40
> >>>           Horizontal backporch: 220
> >>>           Vertical frontporch: 5
> >>>           Vertical sync: 5
> >>>           Vertical backporch: 20
> >>>           Standards: CTA-861
> >>>           CTA-861 VIC: 0
> >>>           Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
> >>>
> >>> # reset all links
> >>> media-ctl --reset
> >>> # setup links
> >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> >>> # configure pads
> >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
> >>> # capture device
> >>> media-ctl -e 'ipu1_ic_prpenc capture'
> >>> /dev/video0
> >>> v4l2-ctl --device /dev/video0 --get-fmt-video
> >>> Format Video Capture:
> >>>           Width/Height      : 640/360
> >>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
> >>>           Field             : None
> >>>           Bytes per Line    : 1280
> >>>           Size Image        : 460800
> >>>           Colorspace        : Rec. 709
> >>>           Transfer Function : Rec. 709
> >>>           YCbCr/HSV Encoding: ITU-R 601
> >>>           Quantization      : Limited Range
> >>>           Flags             :
> >>>
> >>> # capture 1 frame
> >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> >>> [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
> >>>
> >>> Do you know what the failure is here?
> >> You are /2 downscaling in the CSI, but did not set the compose window at
> >> the input pad, e.g:
> >>
> >> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
> >>
> > I don't think this is the issue. Note that I 'was' able to get the
> > same pipeline with the div-by-2 downscale without compose working with
> > 720p bt656 YUV.
> >
> > Here is the 720p 16bit YUV bus again with the compose:
> > # 720p60Hz YUV 16bit YUV bus
> > # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
> > MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > BT timings set
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> >                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
> > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > media-ctl --get-v4l2 "'ipu1_csi0':0"
> >                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
> >                   crop.bounds:(0,0)/1280x720
> >                   crop:(0,0)/1280x720
> >                   compose.bounds:(0,0)/1280x720
> >                   compose:(0,0)/640x360]
> > media-ctl --get-v4l2 "'ipu1_csi0':1"
> >                  [fmt:UYVY8_1X16/640x360@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> > media-ctl --get-v4l2 "'ipu1_ic_prp':1"
> >                  [fmt:AYUV8_1X32/640x480@1/30 field:none
>
> Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting
> ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to
> default formats.

but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]

here's a sequence of set/get:
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0
[fmt:UYVY8_1X16/1280x720 field:none]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2
[fmt:UYVY8_1X16/1280x720 field:none]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
                [fmt:UYVY8_2X8/640x360@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/640x360
                 crop:(0,0)/640x360
                 compose.bounds:(0,0)/640x360
                 compose:(0,0)/640x360]
^^^ the compose setting defaults so I really don't think the next cmd
does anything
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0
[compose:(0,0)/640x360]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
                [fmt:UYVY8_2X8/640x360@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/640x360
                 crop:(0,0)/640x360
                 compose.bounds:(0,0)/640x360
                 compose:(0,0)/640x360]
^^^ see, same as above (but I like adding the compose to pipeline
configuration instructions for documentation purposes; makes it
obvious they can select the window)
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
                [fmt:AYUV8_1X32/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
^^^ but yes, this one isn't getting set properly to 640x360... and it
looks like its at the input:
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
                [fmt:AYUV8_1X32/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]

my understanding is if I set it on the output pad it will propagate
back to the input pad. I'm also wondering why I don't get an error if
it doesn't change.

If I setup the input pad manually and continue on:
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
                [fmt:AYUV8_1X32/640x360@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
                [fmt:AYUV8_1X32/640x360@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1
[fmt:AYUV32/640x360]"
root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
                [fmt:AYUV8_1X32/640x360@1001/30000 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap
--stream-to=x.raw --stream-count=1
[ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32
^^^ still fails

If I can get pipelines that go through the IC working with UYVY8_1X16
then I will use that for the bus format and ditch the UYVY8_2X16 bt656
which has the limitations of not supporting 1080p60 (not to mention
the fact that anything other than SD resolutions are out of the bt656
as discussed), interlaced formats.

>
<snip>
>
> >>
> > What I was asking here is what is the correct way to configure for
> > ITU601 to REC709 CSC at the CSI?  I would expect to set the REC709
> > format on the output pad of the CSI but it seems to require me to set
> > it on the input pad.
>
> Right, the driver only allows setting colorimetry at the sink pads, and
> propagates them to the source pads. So you'll need to set colorimetry
> params at the CSI sink pad.

oh gosh... I think I had that backwards!

>
> > Consider:
> > # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> >                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> >                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> > ^^^ ITU601, but I want REC709
> > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> >                 [fmt:UYVY8_1X16/1280x720@1/30 field:none

So the above was wrong as I was trying to set it on the CSI 'output'
pad not its 'input'. Is there no error reporting all the way back to
let the user know this was an invalid choice?

> > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> > ^^^ No invalid argument but its still ITU601 at CSI output... but what
> > I've found is if I set it at the CSI input the ycbcr changes but the
> > fmt gets jacked up (as expected)
> > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> >                  [fmt:UYVY8_2X8/1280x720@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> > ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI
> > doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8.
>
> Why doesn't the CSI accept UYVY8_1X16 at its sink pad?
>
> This doesn't work?:
>
> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 colorspace:rec709 ycbcr:709]"
>

that works!

I was trying to set AYUV32 at its input pad which it doesn't support.

Again, do you know why errors don't propagate back to media-ctl? Most
of my difficulties have been not realizing what was valid and
understanding that I have to read back the format after I set it to
determine failures.

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Wed, Feb 6, 2019 at 8:31 AM Tim Harvey <tharvey@gateworks.com> wrote:
>
> On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
> >
> >
> >
> > On 2/5/19 11:16 AM, Tim Harvey wrote:
> > > On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com>  wrote:
> > >
> > > <snip>
> > >> The *real* way to fix this would be to allow programmable encodings in
> > >> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
> > >> ic_csc_rgb2ycbcr in ipu-ic.c).
> > >>
> > > Ok, I saw that you went ahead and worked on this (thanks!) and that
> > > you have a bt.709.v2 branch... is that one ready for testing?
> >
> > Yes. I tried to test it too, but there is some regression in captured
> > images, it could be something in ov5640.c (I'm testing with the SabreSD
> > and the OV5640), or something else recently added. The regression looks
> > like a stride problem, I'm hoping it wasn't the recently introduced
> > compose window changes to relax width alignment. Have you tested with
> > those commits and a prpenc/vf pipeline?
>
> I have been testing v4.20 with:
> media: imx: lift CSI and PRP ENC/VF width alignment restriction
> media: imx: set compose rectangle to mbus format
> media: imx: add capture compose rectangle
> imx: imx-media: register mem2mem device with media controller
> media: imx: add mem2mem device
> media: imx-csi: Skip first few frames from a BT.656 source
> media: imx.rst: Update doc to reflect fixes to interlaced capture
> media: imx: Allow interweave with top/bottom lines swapped
> media: imx-csi: Move crop/compose reset after filling default mbus fields
> media: imx: vdic: rely on VDIC for correct field order
> media: imx-csi: Allow skipping odd chroma rows for YVU420
> media: imx: interweave and odd-chroma-row skip are incompatible
> media: imx-csi: Double crop height for alternate fields at sink
> media: imx: Fix field negotiation
> gpu: ipu-v3: Add planar support to interlaced scan
> gpu: ipu-csi: Swap fields according to input/output field types
> media: videodev2.h: Add more field helper macros
> media: imx: prpencvf: Stop upstream before disabling IDMA channel
> media: imx: csi: Stop upstream before disabling IDMA channel
> media: imx: csi: Disable CSI immediately after last EOF
> media: imx-csi: Input connections to CSI should be optional
>
> What kind of artifacts are you seeing?
>
> I noticed your 'media: imx: Add support for BT.709 encoding' series
> don't apply cleanly to 4.20 that I'm working with so perhaps its
> something else. I'll bump up to linux-media, apply your bt.709.v2 and
> see what I get. The BT.709 support is critical for me otherwise I
> can't use coda with any pipelines that go through the IC with the
> tda1997x HDMI decoder.
>

Steve,

I'm testing your bt.709.v2 branch and it works great! I can now
request ycbcr=709 at the CSI input pad and rec709 propagates all the
way through the rest of the pipeline and works with coda. I don't see
any artifacts so I'm still curious what your seeing?

I did notice it adds a regression in that RGB->RGB ends up erroring
out with 'Unsupported YCbCr encoding' in ipu_ic_task_init. I'll
respond to your patch with details and a fix.

> >
<snip>
> > >
> > >>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
> > >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > >>> # set sensor output pad to sensor source format
> > >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > >>> # sensor format
> > >>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
> > >>>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> > >>> # get framerate
> > >>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
> > >>> DV timings:
> > >>>           Active width: 1280
> > >>>           Active height: 720
> > >>>           Total width: 1650
> > >>>           Total height: 750
> > >>>           Frame format: progressive
> > >>>           Polarities: +vsync +hsync
> > >>>           Pixelclock: 74250000 Hz (60.00 frames per second)
> > >>>           Horizontal frontporch: 110
> > >>>           Horizontal sync: 40
> > >>>           Horizontal backporch: 220
> > >>>           Vertical frontporch: 5
> > >>>           Vertical sync: 5
> > >>>           Vertical backporch: 20
> > >>>           Standards: CTA-861
> > >>>           CTA-861 VIC: 0
> > >>>           Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
> > >>>
> > >>> # reset all links
> > >>> media-ctl --reset
> > >>> # setup links
> > >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > >>> # configure pads
> > >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> > >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
> > >>> # capture device
> > >>> media-ctl -e 'ipu1_ic_prpenc capture'
> > >>> /dev/video0
> > >>> v4l2-ctl --device /dev/video0 --get-fmt-video
> > >>> Format Video Capture:
> > >>>           Width/Height      : 640/360
> > >>>           Pixel Format      : 'UYVY' (UYVY 4:2:2)
> > >>>           Field             : None
> > >>>           Bytes per Line    : 1280
> > >>>           Size Image        : 460800
> > >>>           Colorspace        : Rec. 709
> > >>>           Transfer Function : Rec. 709
> > >>>           YCbCr/HSV Encoding: ITU-R 601
> > >>>           Quantization      : Limited Range
> > >>>           Flags             :
> > >>>
> > >>> # capture 1 frame
> > >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> > >>> [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
> > >>>
> > >>> Do you know what the failure is here?
> > >> You are /2 downscaling in the CSI, but did not set the compose window at
> > >> the input pad, e.g:
> > >>
> > >> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
> > >>
> > > I don't think this is the issue. Note that I 'was' able to get the
> > > same pipeline with the div-by-2 downscale without compose working with
> > > 720p bt656 YUV.
> > >
> > > Here is the 720p 16bit YUV bus again with the compose:
> > > # 720p60Hz YUV 16bit YUV bus
> > > # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
> > > MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > > BT timings set
> > > # sensor format
> > > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> > >                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> > > # reset all links
> > > media-ctl --reset
> > > # setup links
> > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > > # configure pads
> > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > > media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > > media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
> > > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > > media-ctl --get-v4l2 "'ipu1_csi0':0"
> > >                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
> > >                   crop.bounds:(0,0)/1280x720
> > >                   crop:(0,0)/1280x720
> > >                   compose.bounds:(0,0)/1280x720
> > >                   compose:(0,0)/640x360]
> > > media-ctl --get-v4l2 "'ipu1_csi0':1"
> > >                  [fmt:UYVY8_1X16/640x360@1/30 field:none
> > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> > > media-ctl --get-v4l2 "'ipu1_ic_prp':1"
> > >                  [fmt:AYUV8_1X32/640x480@1/30 field:none
> >
> > Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting
> > ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to
> > default formats.
>
> but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]
>
> here's a sequence of set/get:
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0
> [fmt:UYVY8_1X16/1280x720 field:none]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>                 [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2
> [fmt:UYVY8_1X16/1280x720 field:none]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>                 [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>                 [fmt:UYVY8_2X8/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>                  crop.bounds:(0,0)/640x360
>                  crop:(0,0)/640x360
>                  compose.bounds:(0,0)/640x360
>                  compose:(0,0)/640x360]
> ^^^ the compose setting defaults so I really don't think the next cmd
> does anything
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0
> [compose:(0,0)/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>                 [fmt:UYVY8_2X8/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>                  crop.bounds:(0,0)/640x360
>                  crop:(0,0)/640x360
>                  compose.bounds:(0,0)/640x360
>                  compose:(0,0)/640x360]
> ^^^ see, same as above (but I like adding the compose to pipeline
> configuration instructions for documentation purposes; makes it
> obvious they can select the window)
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>                 [fmt:AYUV8_1X32/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> ^^^ but yes, this one isn't getting set properly to 640x360... and it
> looks like its at the input:
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
>                 [fmt:AYUV8_1X32/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>
> my understanding is if I set it on the output pad it will propagate
> back to the input pad. I'm also wondering why I don't get an error if
> it doesn't change.
>
> If I setup the input pad manually and continue on:
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
>                 [fmt:AYUV8_1X32/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>                 [fmt:AYUV8_1X32/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1
> [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
>                 [fmt:AYUV8_1X32/640x360@1001/30000 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap
> --stream-to=x.raw --stream-count=1
> [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32
> ^^^ still fails
>

Ok there is definitely something wrong when using the IC with
UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me
like the ipu1_ic_prp isn't negotiating its format properly. You can't
re-create this because you don't have any UYVY8_1X16 (passthrough)
sensors right?

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

On 2/7/19 5:15 PM, Tim Harvey wrote:
> On Wed, Feb 6, 2019 at 8:31 AM Tim Harvey <tharvey@gateworks.com> wrote:
>> On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>>>
>>>
>>> On 2/5/19 11:16 AM, Tim Harvey wrote:
>>>> On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com>  wrote:
>>>>
>>>> <snip>
>>>>> The *real* way to fix this would be to allow programmable encodings in
>>>>> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
>>>>> ic_csc_rgb2ycbcr in ipu-ic.c).
>>>>>
>>>> Ok, I saw that you went ahead and worked on this (thanks!) and that
>>>> you have a bt.709.v2 branch... is that one ready for testing?
>>> Yes. I tried to test it too, but there is some regression in captured
>>> images, it could be something in ov5640.c (I'm testing with the SabreSD
>>> and the OV5640), or something else recently added. The regression looks
>>> like a stride problem, I'm hoping it wasn't the recently introduced
>>> compose window changes to relax width alignment. Have you tested with
>>> those commits and a prpenc/vf pipeline?
>> I have been testing v4.20 with:
>> media: imx: lift CSI and PRP ENC/VF width alignment restriction
>> media: imx: set compose rectangle to mbus format
>> media: imx: add capture compose rectangle
>> imx: imx-media: register mem2mem device with media controller
>> media: imx: add mem2mem device
>> media: imx-csi: Skip first few frames from a BT.656 source
>> media: imx.rst: Update doc to reflect fixes to interlaced capture
>> media: imx: Allow interweave with top/bottom lines swapped
>> media: imx-csi: Move crop/compose reset after filling default mbus fields
>> media: imx: vdic: rely on VDIC for correct field order
>> media: imx-csi: Allow skipping odd chroma rows for YVU420
>> media: imx: interweave and odd-chroma-row skip are incompatible
>> media: imx-csi: Double crop height for alternate fields at sink
>> media: imx: Fix field negotiation
>> gpu: ipu-v3: Add planar support to interlaced scan
>> gpu: ipu-csi: Swap fields according to input/output field types
>> media: videodev2.h: Add more field helper macros
>> media: imx: prpencvf: Stop upstream before disabling IDMA channel
>> media: imx: csi: Stop upstream before disabling IDMA channel
>> media: imx: csi: Disable CSI immediately after last EOF
>> media: imx-csi: Input connections to CSI should be optional
>>
>> What kind of artifacts are you seeing?
>>
>> I noticed your 'media: imx: Add support for BT.709 encoding' series
>> don't apply cleanly to 4.20 that I'm working with so perhaps its
>> something else. I'll bump up to linux-media, apply your bt.709.v2 and
>> see what I get. The BT.709 support is critical for me otherwise I
>> can't use coda with any pipelines that go through the IC with the
>> tda1997x HDMI decoder.
>>
> Steve,
>
> I'm testing your bt.709.v2 branch and it works great! I can now
> request ycbcr=709 at the CSI input pad and rec709 propagates all the
> way through the rest of the pipeline and works with coda. I don't see
> any artifacts so I'm still curious what your seeing?

Our emails crossed, see my last reply. Great to hear rec709 is working 
for you.

>
> I did notice it adds a regression in that RGB->RGB ends up erroring
> out with 'Unsupported YCbCr encoding' in ipu_ic_task_init. I'll
> respond to your patch with details and a fix.

Ok, I take a look before submitting v2.

>
> <snip>
>>>>>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
>>>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>>>>> # set sensor output pad to sensor source format
>>>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>>>>> # sensor format
>>>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>>>>                    [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>>>>>> # get framerate
>>>>>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
>>>>>> DV timings:
>>>>>>            Active width: 1280
>>>>>>            Active height: 720
>>>>>>            Total width: 1650
>>>>>>            Total height: 750
>>>>>>            Frame format: progressive
>>>>>>            Polarities: +vsync +hsync
>>>>>>            Pixelclock: 74250000 Hz (60.00 frames per second)
>>>>>>            Horizontal frontporch: 110
>>>>>>            Horizontal sync: 40
>>>>>>            Horizontal backporch: 220
>>>>>>            Vertical frontporch: 5
>>>>>>            Vertical sync: 5
>>>>>>            Vertical backporch: 20
>>>>>>            Standards: CTA-861
>>>>>>            CTA-861 VIC: 0
>>>>>>            Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
>>>>>>
>>>>>> # reset all links
>>>>>> media-ctl --reset
>>>>>> # setup links
>>>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>>>>> # configure pads
>>>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
>>>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
>>>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
>>>>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
>>>>>> # capture device
>>>>>> media-ctl -e 'ipu1_ic_prpenc capture'
>>>>>> /dev/video0
>>>>>> v4l2-ctl --device /dev/video0 --get-fmt-video
>>>>>> Format Video Capture:
>>>>>>            Width/Height      : 640/360
>>>>>>            Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>>>>            Field             : None
>>>>>>            Bytes per Line    : 1280
>>>>>>            Size Image        : 460800
>>>>>>            Colorspace        : Rec. 709
>>>>>>            Transfer Function : Rec. 709
>>>>>>            YCbCr/HSV Encoding: ITU-R 601
>>>>>>            Quantization      : Limited Range
>>>>>>            Flags             :
>>>>>>
>>>>>> # capture 1 frame
>>>>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
>>>>>> [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
>>>>>>
>>>>>> Do you know what the failure is here?
>>>>> You are /2 downscaling in the CSI, but did not set the compose window at
>>>>> the input pad, e.g:
>>>>>
>>>>> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
>>>>>
>>>> I don't think this is the issue. Note that I 'was' able to get the
>>>> same pipeline with the div-by-2 downscale without compose working with
>>>> 720p bt656 YUV.
>>>>
>>>> Here is the 720p 16bit YUV bus again with the compose:
>>>> # 720p60Hz YUV 16bit YUV bus
>>>> # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>>> BT timings set
>>>> # sensor format
>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>>>> # reset all links
>>>> media-ctl --reset
>>>> # setup links
>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>>> # configure pads
>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>> media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
>>>> media-ctl --get-v4l2 "'ipu1_csi0':0"
>>>>                   [fmt:UYVY8_1X16/1280x720@1/30 field:none
>>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>>>>                    crop.bounds:(0,0)/1280x720
>>>>                    crop:(0,0)/1280x720
>>>>                    compose.bounds:(0,0)/1280x720
>>>>                    compose:(0,0)/640x360]
>>>> media-ctl --get-v4l2 "'ipu1_csi0':1"
>>>>                   [fmt:UYVY8_1X16/640x360@1/30 field:none
>>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>>> media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>>>>                   [fmt:AYUV8_1X32/640x480@1/30 field:none
>>> Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting
>>> ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to
>>> default formats.
>> but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]
>>
>> here's a sequence of set/get:
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0
>> [fmt:UYVY8_1X16/1280x720 field:none]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2
>> [fmt:UYVY8_1X16/1280x720 field:none]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>>                  [fmt:UYVY8_2X8/640x360@1/30 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>>                   crop.bounds:(0,0)/640x360
>>                   crop:(0,0)/640x360
>>                   compose.bounds:(0,0)/640x360
>>                   compose:(0,0)/640x360]
>> ^^^ the compose setting defaults so I really don't think the next cmd
>> does anything
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0
>> [compose:(0,0)/640x360]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>>                  [fmt:UYVY8_2X8/640x360@1/30 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>>                   crop.bounds:(0,0)/640x360
>>                   crop:(0,0)/640x360
>>                   compose.bounds:(0,0)/640x360
>>                   compose:(0,0)/640x360]
>> ^^^ see, same as above (but I like adding the compose to pipeline
>> configuration instructions for documentation purposes; makes it
>> obvious they can select the window)
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>>                  [fmt:AYUV8_1X32/1280x720@1/30 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>> ^^^ but yes, this one isn't getting set properly to 640x360... and it
>> looks like its at the input:
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
>>                  [fmt:AYUV8_1X32/1280x720@1/30 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>
>> my understanding is if I set it on the output pad it will propagate
>> back to the input pad. I'm also wondering why I don't get an error if
>> it doesn't change.
>>
>> If I setup the input pad manually and continue on:
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
>>                  [fmt:AYUV8_1X32/640x360@1/30 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>>                  [fmt:AYUV8_1X32/640x360@1/30 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1
>> [fmt:AYUV32/640x360]"
>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
>>                  [fmt:AYUV8_1X32/640x360@1001/30000 field:none
>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap
>> --stream-to=x.raw --stream-count=1
>> [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32
>> ^^^ still fails
>>
> Ok there is definitely something wrong when using the IC with
> UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me
> like the ipu1_ic_prp isn't negotiating its format properly. You can't
> re-create this because you don't have any UYVY8_1X16 (passthrough)
> sensors right?

Sorry, maybe I didn't mention this, but passthrough cannot go though the 
IPU, you can only send passthrough pixels out the CSI directly to 
/dev/videoN interface (the ipu_csi:2 pad).

Steve

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Thu, Feb 7, 2019 at 5:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
>
<snip>
> >>
> > Ok there is definitely something wrong when using the IC with
> > UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me
> > like the ipu1_ic_prp isn't negotiating its format properly. You can't
> > re-create this because you don't have any UYVY8_1X16 (passthrough)
> > sensors right?
>
> Sorry, maybe I didn't mention this, but passthrough cannot go though the
> IPU, you can only send passthrough pixels out the CSI directly to
> /dev/videoN interface (the ipu_csi:2 pad).
>

crud... this has been my issue all along with that set of UYVY8_1X16
pipelines then. So this means the mem2mem driver also won't be able to
handle 16bit pixel formats as well. So while I can downscale this by
multiples of 2 (independent width/height), CSC convert it from
srgb/bt.601 to yuv/bt.709, and even pixel reorder it within YUV  via
the ipu_csi entitty I'll never be able to deinterlace, scale with
flexibility, flip/rotate or do a RGB->YUV CSC on it as those are all
features of the IC path.

I'm still struggling with what mbus format to configure the
sensor<->csi interconnect in the device-tree. I was leaning towards
16bit but now that I realize that can't be used with the IPU I'm
thinking the bt656 is more flexible (accept it has the limitation of
not being able to do 1080p60 due to pixel clock and also can't handle
interlaced due bt656 codes). If I end up wanting to switch between
tda1997x sensor bus formats I'm not even sure the best way to deal
with that (two dts I suppose and allowing user to select which one
they use).

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

On 2/8/19 1:23 PM, Tim Harvey wrote:
> On Thu, Feb 7, 2019 at 5:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>>
> <snip>
>>> Ok there is definitely something wrong when using the IC with
>>> UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me
>>> like the ipu1_ic_prp isn't negotiating its format properly. You can't
>>> re-create this because you don't have any UYVY8_1X16 (passthrough)
>>> sensors right?
>> Sorry, maybe I didn't mention this, but passthrough cannot go though the
>> IPU, you can only send passthrough pixels out the CSI directly to
>> /dev/videoN interface (the ipu_csi:2 pad).
>>
> crud... this has been my issue all along with that set of UYVY8_1X16
> pipelines then. So this means the mem2mem driver also won't be able to
> handle 16bit pixel formats as well.

Ugh, let me rephrase. UYVY8_1X16 incoming on a parallel bus (not MIPI 
CSI-2) to the CSI cannot be sent through the IPU, those pixels must be 
sent directly to the ipu_csi:2 pad to /dev/videoN. At least according 
the the imx6 register manual.

But that's not a limitation of the mem2mem driver because the pixels are 
coming from a memory buffer and not a 16-bi parallel bus via the CSI 
(and in any case mem2mem does not deal in media bus codes, it speaks 
V4L2_PIX_FMT_* which contains no bus info). So yes, you can receive 
UYVY8_1X16 on the CSI parallel bus, routed to ipu_csi:2 pad to 
/dev/videoN, and then pipe that to mem2mem v4l2convertN element in a 
gstreamer pipeline.



>   So while I can downscale this by
> multiples of 2 (independent width/height), CSC convert it from
> srgb/bt.601 to yuv/bt.709, and even pixel reorder it within YUV  via
> the ipu_csi entitty I'll never be able to deinterlace,

that's true, currently you can't use the VDIC to do motion compensated 
de-interlacing, but you can still make use of the IDMAC interweave 
feature to deinterlace without motion compensation.

>   scale with
> flexibility, flip/rotate or do a RGB->YUV CSC on it as those are all
> features of the IC path.

mem2mem v4l2convertN element will do those for you (flexible up/down 
scaling, flip/rotate, and CSC).

Steve



>
> I'm still struggling with what mbus format to configure the
> sensor<->csi interconnect in the device-tree. I was leaning towards
> 16bit but now that I realize that can't be used with the IPU I'm
> thinking the bt656 is more flexible (accept it has the limitation of
> not being able to do 1080p60 due to pixel clock and also can't handle
> interlaced due bt656 codes). If I end up wanting to switch between
> tda1997x sensor bus formats I'm not even sure the best way to deal
> with that (two dts I suppose and allowing user to select which one
> they use).
>
> Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Tim Harvey]

On Fri, Feb 8, 2019 at 3:22 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>
>
>
> On 2/8/19 1:23 PM, Tim Harvey wrote:
> > On Thu, Feb 7, 2019 at 5:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
> >>
> > <snip>
> >>> Ok there is definitely something wrong when using the IC with
> >>> UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me
> >>> like the ipu1_ic_prp isn't negotiating its format properly. You can't
> >>> re-create this because you don't have any UYVY8_1X16 (passthrough)
> >>> sensors right?
> >> Sorry, maybe I didn't mention this, but passthrough cannot go though the
> >> IPU, you can only send passthrough pixels out the CSI directly to
> >> /dev/videoN interface (the ipu_csi:2 pad).
> >>
> > crud... this has been my issue all along with that set of UYVY8_1X16
> > pipelines then. So this means the mem2mem driver also won't be able to
> > handle 16bit pixel formats as well.
>
> Ugh, let me rephrase. UYVY8_1X16 incoming on a parallel bus (not MIPI
> CSI-2) to the CSI cannot be sent through the IPU, those pixels must be
> sent directly to the ipu_csi:2 pad to /dev/videoN. At least according
> the the imx6 register manual.
>
> But that's not a limitation of the mem2mem driver because the pixels are
> coming from a memory buffer and not a 16-bi parallel bus via the CSI
> (and in any case mem2mem does not deal in media bus codes, it speaks
> V4L2_PIX_FMT_* which contains no bus info). So yes, you can receive
> UYVY8_1X16 on the CSI parallel bus, routed to ipu_csi:2 pad to
> /dev/videoN, and then pipe that to mem2mem v4l2convertN element in a
> gstreamer pipeline.
>
>
>
> >   So while I can downscale this by
> > multiples of 2 (independent width/height), CSC convert it from
> > srgb/bt.601 to yuv/bt.709, and even pixel reorder it within YUV  via
> > the ipu_csi entitty I'll never be able to deinterlace,
>
> that's true, currently you can't use the VDIC to do motion compensated
> de-interlacing, but you can still make use of the IDMAC interweave
> feature to deinterlace without motion compensation.
>
> >   scale with
> > flexibility, flip/rotate or do a RGB->YUV CSC on it as those are all
> > features of the IC path.
>
> mem2mem v4l2convertN element will do those for you (flexible up/down
> scaling, flip/rotate, and CSC).
>

ah... ok right using mem2mem is precisely that 'mem to mem' and not
direct CSI to IPU.

Ok so i experimented with some pipelines and perhaps am setting them
up wrong. I'll respond to the imx-media mem2mem patch with my
observations and expiriments.

Thanks,

Tim

Re: IMX CSI capture issues with tda1997x HDMI receiver

[Steve Longerbeam]

Hi Tim,

On 2/6/19 8:31 AM, Tim Harvey wrote:
> On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote:
>>
>>
>> On 2/5/19 11:16 AM, Tim Harvey wrote:
>>> On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com>  wrote:
>>>
>>> <snip>
>>>> The *real* way to fix this would be to allow programmable encodings in
>>>> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
>>>> ic_csc_rgb2ycbcr in ipu-ic.c).
>>>>
>>> Ok, I saw that you went ahead and worked on this (thanks!) and that
>>> you have a bt.709.v2 branch... is that one ready for testing?
>> Yes. I tried to test it too, but there is some regression in captured
>> images, it could be something in ov5640.c (I'm testing with the SabreSD
>> and the OV5640), or something else recently added. The regression looks
>> like a stride problem, I'm hoping it wasn't the recently introduced
>> compose window changes to relax width alignment. Have you tested with
>> those commits and a prpenc/vf pipeline?
> I have been testing v4.20 with:
> media: imx: lift CSI and PRP ENC/VF width alignment restriction
> media: imx: set compose rectangle to mbus format
> media: imx: add capture compose rectangle
> imx: imx-media: register mem2mem device with media controller
> media: imx: add mem2mem device
> media: imx-csi: Skip first few frames from a BT.656 source
> media: imx.rst: Update doc to reflect fixes to interlaced capture
> media: imx: Allow interweave with top/bottom lines swapped
> media: imx-csi: Move crop/compose reset after filling default mbus fields
> media: imx: vdic: rely on VDIC for correct field order
> media: imx-csi: Allow skipping odd chroma rows for YVU420
> media: imx: interweave and odd-chroma-row skip are incompatible
> media: imx-csi: Double crop height for alternate fields at sink
> media: imx: Fix field negotiation
> gpu: ipu-v3: Add planar support to interlaced scan
> gpu: ipu-csi: Swap fields according to input/output field types
> media: videodev2.h: Add more field helper macros
> media: imx: prpencvf: Stop upstream before disabling IDMA channel
> media: imx: csi: Stop upstream before disabling IDMA channel
> media: imx: csi: Disable CSI immediately after last EOF
> media: imx-csi: Input connections to CSI should be optional
>
> What kind of artifacts are you seeing?

It was the wrong-stride kind of artifact ("striations" ?? :)

As it turns out the regression was indeed related to the capture compose 
window changes, I submitted a patch that fixes it ("media: imx: Set 
capture compose rectangle in capture_device_set_format").

>
> I noticed your 'media: imx: Add support for BT.709 encoding' series
> don't apply cleanly to 4.20 that I'm working with so perhaps its
> something else. I'll bump up to linux-media, apply your bt.709.v2 and
> see what I get. The BT.709 support is critical for me otherwise I
> can't use coda with any pipelines that go through the IC with the
> tda1997x HDMI decoder.

With the regression fix above, I was able to verify the BT.709 encode 
and inverse encode matrices produce good image colors, with the 
bt.709.v2 branch. So I will submit this today.

>>>> <snip>
>>>>> Also can we connect the mem2mem driver to the unused VDIC input in the
>>>>> media controller so that we can use the VDIC to de-interlace content
>>>>> captured from non IMX sources (ie PCI or USB capture devices)?
>>>> Exactly! That's something I have been working on. But it's difficult to
>>>> connect mem2mem to the unused VDIC IDMAC input pad because as of now the
>>>> v4l2 mem2mem internal API's do not allow connecting to *existing*
>>>> processing entities, and there are also issues with how sub-devices are
>>>> to deal with mem2mem contexts.
>>>>
>>>> I do have a WIP branch that creates a video output device that connects
>>>> to the VDIC IDMAC input pad, which doesn't have the above issues. The
>>>> only drawback with that is how gstreamer can make use of such an output
>>>> device.
>>>>
>>> ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch?
>> The output-vdic branch. It's almost ready to go, there is only some
>> strange issue with low and medium motion-compensation modes (captured
>> images show "snow").
>>
>>
>>> I also noticed you have a add-fim-to-prpencvf branch... are you
>>> working on adding FIM to ipu_ic_prp/enc still? That would be nice to
>>> have to deal with sync loss/regain in analog decoders going through
>>> VDIC de-interlacing.
>> I've been trying to get this working, I ran into some locking issues
>> when enabling lock debug options. Still trying to find a solution.
>>
>>> <snip>
>>>>> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
>>>>> still baffles me a bit but I've also found that any bt656 capture that
>>>>> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
>>>>> the resolution checks in ipu_csi_init_interface() resulting in
>>>>> 'Unsupported interlaced video mode'. I'm not sure if
>>>>> ipu_csi_set_bt_interlaced_codes() can be modified to support other
>>>>> resolutions?
>>>> Well, Bt.656 only defines standard definition NTSC and PAL.
>>>>
>>> That is true. Do you know of any other sensors that use higher
>>> resolutions with BT656 SAV/EAV encoding?
>> Nope all I have for testing is the NTSC/PAL ADV7180 on the SabreAuto.
>>
>>>    The BT656 mode does work well
>>> for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6
>>> pixel clock and can't be used).
>>>
>>> I could dig into the bt656 spec to try and understand the various
>>> codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I
>>> suppose but I'm still not sure I want to push up a device-tree config
>>> that describes the tda1997x to CSI connection as 8bit BT656 as I would
>>> prefer to describe it as 16bit YUV instead (as I may be close to
>>> getting that format working well if we can get the IC able to output
>>> rec709).
>>>
>>> <snip>
>>>
>>>>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
>>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>>>> # set sensor output pad to sensor source format
>>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>>>> # sensor format
>>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>>>                    [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>>>>> # get framerate
>>>>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
>>>>> DV timings:
>>>>>            Active width: 1280
>>>>>            Active height: 720
>>>>>            Total width: 1650
>>>>>            Total height: 750
>>>>>            Frame format: progressive
>>>>>            Polarities: +vsync +hsync
>>>>>            Pixelclock: 74250000 Hz (60.00 frames per second)
>>>>>            Horizontal frontporch: 110
>>>>>            Horizontal sync: 40
>>>>>            Horizontal backporch: 220
>>>>>            Vertical frontporch: 5
>>>>>            Vertical sync: 5
>>>>>            Vertical backporch: 20
>>>>>            Standards: CTA-861
>>>>>            CTA-861 VIC: 0
>>>>>            Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
>>>>>
>>>>> # reset all links
>>>>> media-ctl --reset
>>>>> # setup links
>>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>>>> # configure pads
>>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
>>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
>>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
>>>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
>>>>> # capture device
>>>>> media-ctl -e 'ipu1_ic_prpenc capture'
>>>>> /dev/video0
>>>>> v4l2-ctl --device /dev/video0 --get-fmt-video
>>>>> Format Video Capture:
>>>>>            Width/Height      : 640/360
>>>>>            Pixel Format      : 'UYVY' (UYVY 4:2:2)
>>>>>            Field             : None
>>>>>            Bytes per Line    : 1280
>>>>>            Size Image        : 460800
>>>>>            Colorspace        : Rec. 709
>>>>>            Transfer Function : Rec. 709
>>>>>            YCbCr/HSV Encoding: ITU-R 601
>>>>>            Quantization      : Limited Range
>>>>>            Flags             :
>>>>>
>>>>> # capture 1 frame
>>>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
>>>>> [  125.966980] ipu1_ic_prpenc: pipeline start failed with -32
>>>>>
>>>>> Do you know what the failure is here?
>>>> You are /2 downscaling in the CSI, but did not set the compose window at
>>>> the input pad, e.g:
>>>>
>>>> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
>>>>
>>> I don't think this is the issue. Note that I 'was' able to get the
>>> same pipeline with the div-by-2 downscale without compose working with
>>> 720p bt656 YUV.
>>>
>>> Here is the 720p 16bit YUV bus again with the compose:
>>> # 720p60Hz YUV 16bit YUV bus
>>> # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> BT timings set
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>                   [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
>>> media-ctl --get-v4l2 "'ipu1_csi0':0"
>>>                   [fmt:UYVY8_1X16/1280x720@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>>>                    crop.bounds:(0,0)/1280x720
>>>                    crop:(0,0)/1280x720
>>>                    compose.bounds:(0,0)/1280x720
>>>                    compose:(0,0)/640x360]
>>> media-ctl --get-v4l2 "'ipu1_csi0':1"
>>>                   [fmt:UYVY8_1X16/640x360@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
>>> media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>>>                   [fmt:AYUV8_1X32/640x480@1/30 field:none
>> Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting
>> ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to
>> default formats.
> but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]
>
> here's a sequence of set/get:
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0
> [fmt:UYVY8_1X16/1280x720 field:none]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0"
>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2
> [fmt:UYVY8_1X16/1280x720 field:none]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2"
>                  [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>                  [fmt:UYVY8_2X8/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>                   crop.bounds:(0,0)/640x360
>                   crop:(0,0)/640x360
>                   compose.bounds:(0,0)/640x360
>                   compose:(0,0)/640x360]
> ^^^ the compose setting defaults so I really don't think the next cmd
> does anything
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0
> [compose:(0,0)/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0"
>                  [fmt:UYVY8_2X8/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
>                   crop.bounds:(0,0)/640x360
>                   crop:(0,0)/640x360
>                   compose.bounds:(0,0)/640x360
>                   compose:(0,0)/640x360]
> ^^^ see, same as above (but I like adding the compose to pipeline
> configuration instructions for documentation purposes; makes it
> obvious they can select the window)
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>                  [fmt:AYUV8_1X32/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> ^^^ but yes, this one isn't getting set properly to 640x360... and it
> looks like its at the input:
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
>                  [fmt:AYUV8_1X32/1280x720@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]

No no -- the ipu_ic_prp entity cannot scale, the ipu_ic_prpenc and 
ipu_ic_prpvf scale. Think of ipu_ic_prp as a mux, taking input from 
either CSI0, CSI1, or the VDIC. It can then route that data to PRPENC, 
PRPVF, or both simultaneously.


>
> my understanding is if I set it on the output pad it will propagate
> back to the input pad. I'm also wondering why I don't get an error if
> it doesn't change.

Nope, the other way around. The subdev drivers propagate formats from 
their sink pads to their source pads. And you can rely on media-ctl to 
propagate formats from a subdev's source pad to the next linked subdev's 
sink pad.

In other words, media-ctl does inter-entity format propagation 
(source->sink), and the kernel media driver does intra-entity 
propagation (sink->source).

>
> If I setup the input pad manually and continue on:
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0"
>                  [fmt:AYUV8_1X32/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1"
>                  [fmt:AYUV8_1X32/640x360@1/30 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1
> [fmt:AYUV32/640x360]"
> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
>                  [fmt:AYUV8_1X32/640x360@1001/30000 field:none
> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap
> --stream-to=x.raw --stream-count=1
> [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32
> ^^^ still fails

Again ipu1_ic_prp can't scale. You need to set 640x360 at the 
ipu1_ic_prpenc:1 pad.

>
> If I can get pipelines that go through the IC working with UYVY8_1X16
> then I will use that for the bus format and ditch the UYVY8_2X16 bt656
> which has the limitations of not supporting 1080p60 (not to mention
> the fact that anything other than SD resolutions are out of the bt656
> as discussed), interlaced formats.
>
> <snip>
>>> What I was asking here is what is the correct way to configure for
>>> ITU601 to REC709 CSC at the CSI?  I would expect to set the REC709
>>> format on the output pad of the CSI but it seems to require me to set
>>> it on the input pad.
>> Right, the driver only allows setting colorimetry at the sink pads, and
>> propagates them to the source pads. So you'll need to set colorimetry
>> params at the CSI sink pad.
> oh gosh... I think I had that backwards!

Yep :)

I admit and agree a lot of this is pretty cryptic :-/

But it does actually make sense that imx-media subdevs don't allow 
different colorspace parameters between their sink and source pads, 
because they _cannot_ modify colorspace. There's no support for say, 
transcoding the YCbCr encoding from BT.601 at a sink to BT.709 at the 
source, or quantization, or any of the other colorimetry parameters.

So it is correct behavior to accept userspace-requested colorspace 
params at the sink pad, and force/propagate the colospace params from 
sink pad to source pad.

So what this all means is that, the easiest way to set colorspace all 
throughout the pipeline, is to set it at the external sensor source pad 
(the tda19971 in your case). Media-ctl will then take care to propagate 
that colorimetry to the sink pad of the next linked subdevice, and 
imx-media will take care to propagate colorimetry inside each subdevice 
from sink to source.

>
>>> Consider:
>>> # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi
>>> # set sensor output pad to sensor source format
>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
>>> # sensor format
>>> media-ctl --get-v4l2 '"tda19971 2-0048":0'
>>>                    [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
>>> # reset all links
>>> media-ctl --reset
>>> # setup links
>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
>>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
>>> # configure pads
>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
>>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]"
>>> media-ctl --get-v4l2 '"ipu1_csi0":2'
>>>                   [fmt:UYVY8_1X16/1280x720@1/30 field:none
>>> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
>>> ^^^ ITU601, but I want REC709
>>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
>>> media-ctl --get-v4l2 '"ipu1_csi0":2'
>>>                  [fmt:UYVY8_1X16/1280x720@1/30 field:none
> So the above was wrong as I was trying to set it on the CSI 'output'
> pad not its 'input'. Is there no error reporting all the way back to
> let the user know this was an invalid choice?

Nope :-/

>
>>> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
>>> ^^^ No invalid argument but its still ITU601 at CSI output... but what
>>> I've found is if I set it at the CSI input the ycbcr changes but the
>>> fmt gets jacked up (as expected)
>>> media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
>>>                   [fmt:UYVY8_2X8/1280x720@1/30 field:none
>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
>>> ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI
>>> doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8.
>> Why doesn't the CSI accept UYVY8_1X16 at its sink pad?
>>
>> This doesn't work?:
>>
>> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 colorspace:rec709 ycbcr:709]"
>>
> that works!
>
> I was trying to set AYUV32 at its input pad which it doesn't support.

Right.

Keep in mind also, that the i.MX IPU internally handles only ARGB32 or 
AYUV32 on an internal 32-bit bus. So the IPU internal pads (ipu_csi:1, 
ipu_csi:2, and all pads of ipu_vdic, ipu_prp, ipu_prpenc, and ipu_prpvf) 
only accept media bus codes ARGB8888_1X32 or AYUV8_1X32.

But ipu_csi:0 is a pad external to the IPU, so it accepts other media 
bus codes like UYVY8_1X16.

For example, say you want to setup a pipeline that converts pixels from 
a sensor that sends RGB565_2X8_LE, to YUV space. So you would need 
ipu_prpenc or ipu_prpvf to carry out RGB->YUV CSC. At the ipu_prpencvf:1 
source pad, you must set the mbus code to AYUV8_1X32. The IDMAC can then 
carry out pixel reordering _within YUV space_, to produce many of the 
other YUV pixel formats like YUYV, YV12, 422P, etc. The IDMAC format is 
set at the capture interface with 'v4l2-ctl -dN 
--set-fmt-video=pixelformat=..."

Here is such an example on the SabreSD with the OV5640, which is 
transmitting RGB565_2X8_LE and BT.709 YCbCr encoded, and ipu1_ic_prpenc 
color converts that to AYUV8_1X32, and finally at the capture interface 
/dev/video0, the format is set to YV12 (in fact, this is the pipeline I 
used to test the new BT.709 encoding matrix):

'ov5640 1-003c':0
         [fmt:RGB565_2X8_LE/1024x768@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'imx6-mipi-csi2':0
         [fmt:RGB565_2X8_LE/1024x768 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'imx6-mipi-csi2':2
         [fmt:RGB565_2X8_LE/1024x768 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'ipu1_csi1':0
         [fmt:RGB565_2X8_LE/1024x768@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range
          crop.bounds:(0,0)/1024x768
          crop:(0,0)/1024x768
          compose.bounds:(0,0)/1024x768
          compose:(0,0)/1024x768]

'ipu1_csi1':1
         [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'ipu1_ic_prp':0
         [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'ipu1_ic_prp':1
         [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'ipu1_ic_prpenc':0
         [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

'ipu1_ic_prpenc':1
         [fmt:AYUV8_1X32/800x600@1/30 field:none colorspace:srgb 
xfer:srgb ycbcr:709 quantization:full-range]

/dev/video0:0
Format Video Capture:
     Width/Height      : 800/600
     Pixel Format      : 'YV12'
     Field             : None
     Bytes per Line    : 800
     Size Image        : 720000
     Colorspace        : sRGB
     Transfer Function : sRGB
     YCbCr/HSV Encoding: Rec. 709
     Quantization      : Full Range
     Flags             : 30ad6792



>
> Again, do you know why errors don't propagate back to media-ctl? Most
> of my difficulties have been not realizing what was valid and
> understanding that I have to read back the format after I set it to
> determine failures.

Yeah, like most media drivers, imx-media will force parameters to 
something supported when userland asks for something unsupported, 
without letting the user know that happened. I agree it can be difficult.

Steve