* IMX CSI capture issues with tda1997x HDMI receiver @ 2019-01-23 23:21 Tim Harvey 2019-01-25 23:57 ` Steve Longerbeam 0 siblings, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-01-23 23:21 UTC (permalink / raw) To: linux-media, Steve Longerbeam, Philipp Zabel Steve, I'm testing IMX6 capture again with the tda1997x HDMI receiver found on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6 CSI in a way where it can be configured for 8bit BT656 mode (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide variety of resolutions, bus formats, interlaced modes, and colorspaces. I'm using the following for testing: - Linux 4.20 + - media: imx-csi: Input connections to CSI should be optional - imx-media: Fixes for interlaced capture - media: imx: Stop stream before disabling IDMA channels - v4l-utils 1.16.1 - gstreamer-1.14.1 (with the ability to test master as well) I have a script that generates the media-ctl/v4l2-ctl commands based on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI decoder) and a 'mode' which I've defined as: mode0: sensor -> mux -> csi -> /dev/videoN mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN The media-ctl topologies for each cpu/board combo are at http://dev.gateworks.com/docs/linux/media/ I'm trying to test out simple v4l2-ctl based capture of a single frame as well as capture and stream via both software JPEG encode and H264 hardware encode via CODA. Please let me know of any changes that should be made to the commands below even if only to purely help document things through clarity. One of the issues I run into right away is image size: The imx.rst docs state that the ic has a 'resize' limit of 1024x1024 but I think I'm mislead by the word 'resize' and this also means you can't push say 1080x720 through it (not resizing, just putting that on the src pad) as it clips it to 1024x720. I believe some of Phillip's pending patches may be aimed at rectifying this limitation? Another issue I'm running into is colorspace conversion, specifically colorimetry Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0 MODE1:sensor->mux->csi->ic_prp->ic_prpenc # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 field:none colorspace:rec709 # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]" media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]" # capture device media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 v4l2-ctl --device /dev/video0 --get-fmt-video Format Video Capture: Width/Height : 1024/720 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 2048 Size Image : 1474560 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : ^^^ Note 1080x720 has been reduced to 1024x720 - ouch! # capture 1 frame v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 ^^^ works # stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video0 ! \ video/x-raw,format=UYVY ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 ^^^ works (but of course squashed horizontally because of x=1024) # stream H264/RTP/UDP (via coda) # need to make sure we feed coda NV12/rec709 media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' # fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709 media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]' ^^^ no error but doing a get-v4l2 shows no change - is this all because of trying to deal with 1080 instead of 1024? gst-launch-1.0 v4l2src device=/dev/video0 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=172.24.20.19 port=5001 v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry ^^^ fails because of colorimetry not getting set Let me remove the >1024 issue by using a 480p source... Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0 MODE1:sensor->mux->csi->ic_prp->ic_prpenc # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 field:none colorspace:rec709 # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]" media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/720x480]" media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' [fmt:AYUV8_1X32/720x480@1/25 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] # capture device media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 v4l2-ctl --device /dev/video0 --get-fmt-video Format Video Capture: Width/Height : 720/480 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1440 Size Image : 691200 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : # capture 1 frame v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png ^^^ works # stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video0 ! \ video/x-raw,format=UYVY ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 # stream H264/RTP/UDP media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' [fmt:AYUV8_1X32/720x480@1/25 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] ^^^ again, need to change colorimetry to 709 media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/720x480 colorspace:rec709 ycbcr:709]' media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' [fmt:AYUV8_1X32/720x480@1/25 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] ^^^ still not changing gst-launch-1.0 v4l2src device=/dev/video0 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=172.24.20.19 port=5001 v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry ^^^ fails because of colorimetry not getting set Let's try an interlaced format... Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0 MODE1:sensor->mux->csi->ic_prp->ic_prpenc # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709 # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]" media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" # capture device media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 v4l2-ctl --device /dev/video1 --get-fmt-video Format Video Capture: Width/Height : 720/480 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1440 Size Image : 691200 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : # capture 1 frame v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 ^^^ fails with ipu1_ic_prpvf: EOF timeout ^^^ this is almost identical to the adv7180 case which works... the only exception I see is field:seq-tb vs field:alternate from the source which should be fine (more specific than 'alternate') Now lets go back to a 480p60 source but this time include the vdic (which isn't necessary but should still work right?) Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0 MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 field:none colorspace:rec709 # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]" media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" # capture device media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 v4l2-ctl --device /dev/video1 --get-fmt-video Format Video Capture: Width/Height : 720/480 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1440 Size Image : 691200 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : # capture 1 frame v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 ipu1_ic_prpvf: pipeline start failed with -32 ^^^ maybe because we are feeding progressive to vdic? vdic isn't needed here but it seems it should not fail to me ok now I'm going to boot with a different device-tree that connects the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed for coda 3) ability to capture 1080p60 as I have confirmed tyring to do so via bt656 exceeds the IMX pixel clock rate and causes CSI corruption. Downsides to this mode is that I currently have a bug in the tda1997x driver regarding this mode and interlaced capture. Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0 MODE0:sensor->mux->csi # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480 field:none colorspace:rec709 # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" media-ctl --get-v4l2 '"ipu1_csi0":2' [fmt:UYVY8_1X16/720x480@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] # capture device media-ctl -e 'ipu1_csi0 capture' # /dev/video4 v4l2-ctl --device /dev/video4 --get-fmt-video Format Video Capture: Width/Height : 720/480 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1440 Size Image : 691200 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: Rec. 709 Quantization : Limited Range Flags : premultiplied-alpha # capture 1 frame v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1 convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png # stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video4 ! \ video/x-raw,format=UYVY ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 # stream H264/RTP/UDP media-ctl --get-v4l2 '"ipu1_csi0":0' [fmt:UYVY8_1X16/720x480@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range crop.bounds:(0,0)/720x480 crop:(0,0)/720x480 compose.bounds:(0,0)/720x480 compose:(0,0)/720x480] ^^^ no conv needed media-ctl --get-v4l2 '"ipu1_csi0":0' gst-launch-1.0 v4l2src device=/dev/video4 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=172.24.20.19 port=5001 v4l2src0: Internal data stream error. ^^^ fails... lets set the csi format again media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/720x480 colorspace:rec709 ycbcr:709]' media-ctl --get-v4l2 '"ipu1_csi0":0' [fmt:UYVY8_2X8/720x480@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range crop.bounds:(0,0)/720x480 crop:(0,0)/720x480 compose.bounds:(0,0)/720x480 compose:(0,0)/720x480] ^^^ wtf... the fcc is not showing UYVY8_2X8 which is just wrong and even v4l2 frame capture will fail now Any ideas on any of the above issues? Best Regards, Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-23 23:21 IMX CSI capture issues with tda1997x HDMI receiver Tim Harvey @ 2019-01-25 23:57 ` Steve Longerbeam 2019-01-27 22:36 ` Steve Longerbeam 2019-01-28 19:03 ` Tim Harvey 0 siblings, 2 replies; 19+ messages in thread From: Steve Longerbeam @ 2019-01-25 23:57 UTC (permalink / raw) To: Tim Harvey, Philipp Zabel; +Cc: linux-media Hi Tim, cc: Philipp, On 1/23/19 3:21 PM, Tim Harvey wrote: > Steve, > > I'm testing IMX6 capture again with the tda1997x HDMI receiver found > on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6 > CSI in a way where it can be configured for 8bit BT656 mode > (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall > Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide > variety of resolutions, bus formats, interlaced modes, and > colorspaces. > > I'm using the following for testing: > - Linux 4.20 + > - media: imx-csi: Input connections to CSI should be optional > - imx-media: Fixes for interlaced capture > - media: imx: Stop stream before disabling IDMA channels > - v4l-utils 1.16.1 > - gstreamer-1.14.1 (with the ability to test master as well) > > I have a script that generates the media-ctl/v4l2-ctl commands based > on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI > decoder) and a 'mode' which I've defined as: > mode0: sensor -> mux -> csi -> /dev/videoN > mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a > mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN > mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN > > The media-ctl topologies for each cpu/board combo are at > http://dev.gateworks.com/docs/linux/media/ > > I'm trying to test out simple v4l2-ctl based capture of a single frame > as well as capture and stream via both software JPEG encode and H264 > hardware encode via CODA. > > Please let me know of any changes that should be made to the commands > below even if only to purely help document things through clarity. > > One of the issues I run into right away is image size: The imx.rst > docs state that the ic has a 'resize' limit of 1024x1024 but I think > I'm mislead by the word 'resize' and this also means you can't push > say 1080x720 through it (not resizing, just putting that on the src > pad) as it clips it to 1024x720. The IC is limited to a resized *output* frame of 1024x1024, no higher. > I believe some of Phillip's pending > patches may be aimed at rectifying this limitation? If you mean the mem2mem driver, yes. The mem2mem driver makes use the ipu-image-convert APIs which includes tiled resizing, so it will support greater-than 1024x1024 output frames. Latest status I've heard is that there is some misunderstanding of the behavior of rotation, that will require some rework of the mem2mem driver and ipu-image-convert (and sigh, the ic-prpencvf sub-device which also supports h/w rotation). But you could actually grab the last posted version of mem2mem driver (v7). It works fine except for the rotation misunderstanding (but even that works but doesn't conform to v4l2 expected behavior). You will then have a gstreamer element v4l2convertN for >1024 resizing support. Also recently merged from Philipp is the work to relax capture width alignment, by DMA'ing to padded frames. But that is not related to mem2mem. > > Another issue I'm running into is colorspace conversion, specifically > colorimetry > > Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0 > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 > field:none colorspace:rec709 > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]" > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]" > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]" > # capture device > media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 > v4l2-ctl --device /dev/video0 --get-fmt-video > Format Video Capture: > Width/Height : 1024/720 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 2048 > Size Image : 1474560 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: ITU-R 601 > Quantization : Limited Range > Flags : > ^^^ Note 1080x720 has been reduced to 1024x720 - ouch! Yes as discussed above that is correct behavior, you'll need mem2mem device to support >1024 tiled scaling. > # capture 1 frame > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > ^^^ works > # stream JPEG/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > video/x-raw,format=UYVY ! \ > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > ^^^ works (but of course squashed horizontally because of x=1024) > # stream H264/RTP/UDP (via coda) > # need to make sure we feed coda NV12/rec709 > media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' # > fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709 > ycbcr:601 quantization:lim-range > ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709 > media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720 > colorspace:rec709 ycbcr:709]' > ^^^ no error but doing a get-v4l2 shows no change - is this all > because of trying to deal with 1080 instead of 1024? No. The Image Converter can only Y`CbCr encode to ITU-R 601 (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr encoding when using an IC pipeline. See imx_media_fill_default_mbus_fields(). > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > v4l2h264enc output-io-mode=dmabuf-import ! \ > rtph264pay ! udpsink host=172.24.20.19 port=5001 > v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry > ^^^ fails because of colorimetry not getting set Right, that is correct behavior also. See above. Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace? > <snip> > > Let's try an interlaced format... > > Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0 > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 > field:seq-tb colorspace:rec709 > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" > media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" > media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" > media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]" > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" > media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" > media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" > media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" > # capture device > media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 > v4l2-ctl --device /dev/video1 --get-fmt-video > Format Video Capture: > Width/Height : 720/480 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1440 > Size Image : 691200 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: ITU-R 601 > Quantization : Limited Range > Flags : > # capture 1 frame > v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 > ^^^ fails with ipu1_ic_prpvf: EOF timeout > ^^^ this is almost identical to the adv7180 case which works... the > only exception I see is field:seq-tb vs field:alternate from the > source which should be fine (more specific than 'alternate') Hmm, can you send the complete pad formats from --get-v4l2. I don't see anything obviously wrong here. > Now lets go back to a 480p60 source but this time include the vdic > (which isn't necessary but should still work right?) No. First, the CSI will only capture in bt.656 mode if it sees interlaced fields (bt.656 interlaced sync codes). Well, let me rephrase, the CSI does support progressive BT.656 but I have never tested that myself. But more importantly, the VDIC must be given interlaced fields at its input. > > Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0 > MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 > field:none colorspace:rec709 > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" > media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" > media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" > media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]" > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" > media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" > media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" > media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" > # capture device > media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 > v4l2-ctl --device /dev/video1 --get-fmt-video > Format Video Capture: > Width/Height : 720/480 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1440 > Size Image : 691200 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: ITU-R 601 > Quantization : Limited Range > Flags : > # capture 1 frame > v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 > ipu1_ic_prpvf: pipeline start failed with -32 > ^^^ maybe because we are feeding progressive to vdic? vdic isn't > needed here but it seems it should not fail to me this is a pipeline src->sink mismatch error, because the VDIC enforces fields at its input pad. > > ok now I'm going to boot with a different device-tree that connects > the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be > 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed > for coda 3) ability to capture 1080p60 as I have confirmed tyring to > do so via bt656 exceeds the IMX pixel clock rate and causes CSI > corruption. Downsides to this mode is that I currently have a bug in > the tda1997x driver regarding this mode and interlaced capture. > > Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0 > MODE0:sensor->mux->csi > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480 > field:none colorspace:rec709 > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]" > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" > media-ctl --get-v4l2 '"ipu1_csi0":2' > [fmt:UYVY8_1X16/720x480@1/30 field:none This is passthrough, the CSI can only pass through input pixel data unmodified as "generic" data, when the input bus is 16-bit parallel, which it is in this case (UYVY8_1X16). So the pixel code at the CSI output pad is forced to same as input pad (UYVY8_1X16). But aside from passthrough, that shouldn't be a problem... > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > # capture device > media-ctl -e 'ipu1_csi0 capture' # /dev/video4 > v4l2-ctl --device /dev/video4 --get-fmt-video > Format Video Capture: > Width/Height : 720/480 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1440 > Size Image : 691200 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: Rec. 709 > Quantization : Limited Range > Flags : premultiplied-alpha > # capture 1 frame > v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1 > convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png So capture of 1 frame worked fine? > # stream JPEG/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > video/x-raw,format=UYVY ! \ > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 and this worked? > # stream H264/RTP/UDP > media-ctl --get-v4l2 '"ipu1_csi0":0' > [fmt:UYVY8_1X16/720x480@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range > crop.bounds:(0,0)/720x480 > crop:(0,0)/720x480 > compose.bounds:(0,0)/720x480 > compose:(0,0)/720x480] > ^^^ no conv needed > media-ctl --get-v4l2 '"ipu1_csi0":0' huh? > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > v4l2h264enc output-io-mode=dmabuf-import ! \ > rtph264pay ! udpsink host=172.24.20.19 port=5001 > v4l2src0: Internal data stream error. Sorry, I don't have enough info to help here. > ^^^ fails... lets set the csi format again > media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/720x480 > colorspace:rec709 ycbcr:709]' > media-ctl --get-v4l2 '"ipu1_csi0":0' > [fmt:UYVY8_2X8/720x480@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range > crop.bounds:(0,0)/720x480 > crop:(0,0)/720x480 > compose.bounds:(0,0)/720x480 > compose:(0,0)/720x480] > ^^^ wtf... the fcc is not showing UYVY8_2X8 which is just wrong and > even v4l2 frame capture will fail now The CSI does not support AYUV32 at its input pad, so it defaults to UYVY8_2X. Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-25 23:57 ` Steve Longerbeam @ 2019-01-27 22:36 ` Steve Longerbeam 2019-01-28 19:14 ` Tim Harvey 2019-01-28 19:03 ` Tim Harvey 1 sibling, 1 reply; 19+ messages in thread From: Steve Longerbeam @ 2019-01-27 22:36 UTC (permalink / raw) To: Tim Harvey, Philipp Zabel; +Cc: linux-media Hi Tim, On 1/25/19 3:57 PM, Steve Longerbeam wrote: <snip> >> Now lets go back to a 480p60 source but this time include the vdic >> (which isn't necessary but should still work right?) > > No. First, the CSI will only capture in bt.656 mode if it sees > interlaced fields (bt.656 interlaced sync codes). Well, let me > rephrase, the CSI does support progressive BT.656 but I have never > tested that myself. One more comment here. It would be great if you could test a progressive bt.656 sensor (example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0) since as I said I've never been able to test this myself. Since it is progressive there's no need for the VDIC, so try these pipelines: mode0: sensor -> mux -> csi -> /dev/videoN mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-27 22:36 ` Steve Longerbeam @ 2019-01-28 19:14 ` Tim Harvey 2019-01-28 23:05 ` Steve Longerbeam 0 siblings, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-01-28 19:14 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Sun, Jan 27, 2019 at 2:36 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > Hi Tim, > > On 1/25/19 3:57 PM, Steve Longerbeam wrote: > <snip> > >> Now lets go back to a 480p60 source but this time include the vdic > >> (which isn't necessary but should still work right?) > > > > No. First, the CSI will only capture in bt.656 mode if it sees > > interlaced fields (bt.656 interlaced sync codes). Well, let me > > rephrase, the CSI does support progressive BT.656 but I have never > > tested that myself. > > One more comment here. It would be great if you could test a progressive > bt.656 sensor (example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 > IPU1_CSI0) since as I said I've never been able to test this myself. > > Since it is progressive there's no need for the VDIC, so try these > pipelines: > > mode0: sensor -> mux -> csi -> /dev/videoN > mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN > Steve, These both work fine in all cases for 480p60Hz YUV via BT656. When I use 480p60Hz 'RGB' via BT656 mode0 works fine as the CSI does the colorspace conversion needed for coda but for mode1 I end up with the itu601 colorimetery issue from before that I'm still trying to find a solution for. Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-28 19:14 ` Tim Harvey @ 2019-01-28 23:05 ` Steve Longerbeam 0 siblings, 0 replies; 19+ messages in thread From: Steve Longerbeam @ 2019-01-28 23:05 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media On 1/28/19 11:14 AM, Tim Harvey wrote: > On Sun, Jan 27, 2019 at 2:36 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >> Hi Tim, >> >> On 1/25/19 3:57 PM, Steve Longerbeam wrote: >> <snip> >>>> Now lets go back to a 480p60 source but this time include the vdic >>>> (which isn't necessary but should still work right?) >>> No. First, the CSI will only capture in bt.656 mode if it sees >>> interlaced fields (bt.656 interlaced sync codes). Well, let me >>> rephrase, the CSI does support progressive BT.656 but I have never >>> tested that myself. >> One more comment here. It would be great if you could test a progressive >> bt.656 sensor (example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 >> IPU1_CSI0) since as I said I've never been able to test this myself. >> >> Since it is progressive there's no need for the VDIC, so try these >> pipelines: >> >> mode0: sensor -> mux -> csi -> /dev/videoN >> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN >> > Steve, > > These both work fine in all cases for 480p60Hz YUV via BT656. Great! thanks for confirming. Steve > > When I use 480p60Hz 'RGB' via BT656 mode0 works fine as the CSI does > the colorspace conversion needed for coda but for mode1 I end up with > the itu601 colorimetery issue from before that I'm still trying to > find a solution for. > > Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-25 23:57 ` Steve Longerbeam 2019-01-27 22:36 ` Steve Longerbeam @ 2019-01-28 19:03 ` Tim Harvey 2019-01-28 23:04 ` Steve Longerbeam 1 sibling, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-01-28 19:03 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > Hi Tim, cc: Philipp, > > On 1/23/19 3:21 PM, Tim Harvey wrote: > > Steve, > > > > I'm testing IMX6 capture again with the tda1997x HDMI receiver found > > on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6 > > CSI in a way where it can be configured for 8bit BT656 mode > > (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall > > Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide > > variety of resolutions, bus formats, interlaced modes, and > > colorspaces. > > > > I'm using the following for testing: > > - Linux 4.20 + > > - media: imx-csi: Input connections to CSI should be optional > > - imx-media: Fixes for interlaced capture > > - media: imx: Stop stream before disabling IDMA channels > > - v4l-utils 1.16.1 > > - gstreamer-1.14.1 (with the ability to test master as well) > > > > I have a script that generates the media-ctl/v4l2-ctl commands based > > on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI > > decoder) and a 'mode' which I've defined as: > > mode0: sensor -> mux -> csi -> /dev/videoN > > mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a > > mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN > > mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN > > > > The media-ctl topologies for each cpu/board combo are at > > http://dev.gateworks.com/docs/linux/media/ > > > > I'm trying to test out simple v4l2-ctl based capture of a single frame > > as well as capture and stream via both software JPEG encode and H264 > > hardware encode via CODA. > > > > Please let me know of any changes that should be made to the commands > > below even if only to purely help document things through clarity. > > > > One of the issues I run into right away is image size: The imx.rst > > docs state that the ic has a 'resize' limit of 1024x1024 but I think > > I'm mislead by the word 'resize' and this also means you can't push > > say 1080x720 through it (not resizing, just putting that on the src > > pad) as it clips it to 1024x720. > > The IC is limited to a resized *output* frame of 1024x1024, no higher. > Ok so this means for anything >1024 I need to either use the csi to downscale it by factors of 2 first, or use mem2mem solution which means I can't use the VDIC (so no interlaced). I will try your patch that adds registration of the mem2mem driver to the media-ctl api. > > I believe some of Phillip's pending > > patches may be aimed at rectifying this limitation? > > If you mean the mem2mem driver, yes. The mem2mem driver makes use the > ipu-image-convert APIs which includes tiled resizing, so it will support > greater-than 1024x1024 output frames. Latest status I've heard is that > there is some misunderstanding of the behavior of rotation, that will > require some rework of the mem2mem driver and ipu-image-convert (and > sigh, the ic-prpencvf sub-device which also supports h/w rotation). > > But you could actually grab the last posted version of mem2mem driver > (v7). It works fine except for the rotation misunderstanding (but even > that works but doesn't conform to v4l2 expected behavior). You will then > have a gstreamer element v4l2convertN for >1024 resizing support. > Ok, so I've applied that to my kernel and now have a new /dev/video* device for ipu_ic_pp. I'll start testing with it. > Also recently merged from Philipp is the work to relax capture width > alignment, by DMA'ing to padded frames. But that is not related to mem2mem. > Ok - I'll merge this one to so I'm testing with it. > > > > Another issue I'm running into is colorspace conversion, specifically > > colorimetry > > > > Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0 > > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 > > field:none colorspace:rec709 > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]" > > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]" > > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]" > > # capture device > > media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 > > v4l2-ctl --device /dev/video0 --get-fmt-video > > Format Video Capture: > > Width/Height : 1024/720 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 2048 > > Size Image : 1474560 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: ITU-R 601 > > Quantization : Limited Range > > Flags : > > ^^^ Note 1080x720 has been reduced to 1024x720 - ouch! > > Yes as discussed above that is correct behavior, you'll need mem2mem > device to support >1024 tiled scaling. > > > # capture 1 frame > > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > > ^^^ works > > # stream JPEG/RTP/UDP > > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > > video/x-raw,format=UYVY ! \ > > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > > ^^^ works (but of course squashed horizontally because of x=1024) > > # stream H264/RTP/UDP (via coda) > > # need to make sure we feed coda NV12/rec709 > > media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' # > > fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709 > > ycbcr:601 quantization:lim-range > > ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709 > > media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720 > > colorspace:rec709 ycbcr:709]' > > ^^^ no error but doing a get-v4l2 shows no change - is this all > > because of trying to deal with 1080 instead of 1024? > > No. The Image Converter can only Y`CbCr encode to ITU-R 601 > (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr > encoding when using an IC pipeline. > > See imx_media_fill_default_mbus_fields(). > > > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > > v4l2h264enc output-io-mode=dmabuf-import ! \ > > rtph264pay ! udpsink host=172.24.20.19 port=5001 > > v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry > > ^^^ fails because of colorimetry not getting set > > Right, that is correct behavior also. See above. > > Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace? > I will have to go back and look what the old downstream Freescale kernel drivers did here... I could swear I could easily de-interlace via VDIC and encode 1080i with that and gstreamer-imx. > > <snip> > > > > Let's try an interlaced format... > > > > Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0 > > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 > > field:seq-tb colorspace:rec709 > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" > > media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" > > media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" > > media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]" > > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" > > media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" > > media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" > > media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" > > # capture device > > media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 > > v4l2-ctl --device /dev/video1 --get-fmt-video > > Format Video Capture: > > Width/Height : 720/480 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 1440 > > Size Image : 691200 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: ITU-R 601 > > Quantization : Limited Range > > Flags : > > # capture 1 frame > > v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 > > ^^^ fails with ipu1_ic_prpvf: EOF timeout > > ^^^ this is almost identical to the adv7180 case which works... the > > only exception I see is field:seq-tb vs field:alternate from the > > source which should be fine (more specific than 'alternate') > > Hmm, can you send the complete pad formats from --get-v4l2. I don't see > anything obviously wrong here. > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1" [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2" [fmt:AYUV8_1X32/720x480@1/60 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2" [fmt:AYUV8_1X32/720x480@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1" [fmt:AYUV8_1X32/720x480@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video Format Video Capture: Width/Height : 720/480 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1440 Size Image : 691200 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 [ 159.828654] ipu1_ic_prpvf: EOF timeout VIDIOC_DQBUF: failed: Input/output error > > Now lets go back to a 480p60 source but this time include the vdic > > (which isn't necessary but should still work right?) > > No. First, the CSI will only capture in bt.656 mode if it sees > interlaced fields (bt.656 interlaced sync codes). Well, let me rephrase, > the CSI does support progressive BT.656 but I have never tested that myself. > > But more importantly, the VDIC must be given interlaced fields at its input. > ok, I understand now that VDIC can only accept interlaced fields, so progressive should skip the VDIC. > > > > Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0 > > MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 > > field:none colorspace:rec709 > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" > > media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" > > media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" > > media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]" > > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" > > media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" > > media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" > > media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" > > # capture device > > media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 > > v4l2-ctl --device /dev/video1 --get-fmt-video > > Format Video Capture: > > Width/Height : 720/480 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 1440 > > Size Image : 691200 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: ITU-R 601 > > Quantization : Limited Range > > Flags : > > # capture 1 frame > > v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 > > ipu1_ic_prpvf: pipeline start failed with -32 > > ^^^ maybe because we are feeding progressive to vdic? vdic isn't > > needed here but it seems it should not fail to me > > this is a pipeline src->sink mismatch error, because the VDIC enforces > fields at its input pad. right... as you said above vdic requires interlaced fields so progressive must skip the vdic. > > > > > ok now I'm going to boot with a different device-tree that connects > > the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be > > 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed > > for coda 3) ability to capture 1080p60 as I have confirmed tyring to > > do so via bt656 exceeds the IMX pixel clock rate and causes CSI > > corruption. Downsides to this mode is that I currently have a bug in > > the tda1997x driver regarding this mode and interlaced capture. > > > > Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0 > > MODE0:sensor->mux->csi > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480 > > field:none colorspace:rec709 > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]" > > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" > > media-ctl --get-v4l2 '"ipu1_csi0":2' > > [fmt:UYVY8_1X16/720x480@1/30 field:none > > This is passthrough, the CSI can only pass through input pixel data > unmodified as "generic" data, when the input bus is 16-bit parallel, > which it is in this case (UYVY8_1X16). So the pixel code at the CSI > output pad is forced to same as input pad (UYVY8_1X16). > > But aside from passthrough, that shouldn't be a problem... > > > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > > # capture device > > media-ctl -e 'ipu1_csi0 capture' # /dev/video4 > > v4l2-ctl --device /dev/video4 --get-fmt-video > > Format Video Capture: > > Width/Height : 720/480 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 1440 > > Size Image : 691200 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: Rec. 709 > > Quantization : Limited Range > > Flags : premultiplied-alpha > > # capture 1 frame > > v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1 > > convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png > > So capture of 1 frame worked fine? yes this worked fine > > > # stream JPEG/RTP/UDP > > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > > video/x-raw,format=UYVY ! \ > > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > > and this worked? yes this worked fine > > > # stream H264/RTP/UDP > > media-ctl --get-v4l2 '"ipu1_csi0":0' > > [fmt:UYVY8_1X16/720x480@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range > > crop.bounds:(0,0)/720x480 > > crop:(0,0)/720x480 > > compose.bounds:(0,0)/720x480 > > compose:(0,0)/720x480] > > ^^^ no conv needed > > media-ctl --get-v4l2 '"ipu1_csi0":0' > > huh? typo'd here and cut-and-pasted another occurrence of the media-ctl --get-v4l2 command but my point here is that we have the correct colorspace/colorimetry for coda so the H264 below should work fine > > > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > > v4l2h264enc output-io-mode=dmabuf-import ! \ > > rtph264pay ! udpsink host=172.24.20.19 port=5001 > > v4l2src0: Internal data stream error. > > Sorry, I don't have enough info to help here. > Here's the pad details: root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2" [fmt:UYVY8_1X16/720x480@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video Format Video Capture: Width/Height : 720/480 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1440 Size Image : 691200 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: Rec. 709 Quantization : Limited Range Flags : So in this passthrough case v4l2-ctl stream capture works, gstreamer capture with sw jpeg encode works, but gstreamer capture with h264 encode fails: root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \ > v4l2h264enc output-io-mode=dmabuf-import ! \ > rtph264pay ! udpsink host=172.24.20.19 port=5001 Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error. Additional debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: streaming stopped, reason not-negotiated (-4) Execution ended after 0:00:00.000416672 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... It would appear that this must have something to do with gstreamer. The only difference here between this case and the adv7180 is that field=none and colorimetry is itu601... I'm guessing the field=none is causing an issue. I'll post a question to gstreamer-devel. Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-28 19:03 ` Tim Harvey @ 2019-01-28 23:04 ` Steve Longerbeam 2019-01-31 1:18 ` Tim Harvey 0 siblings, 1 reply; 19+ messages in thread From: Steve Longerbeam @ 2019-01-28 23:04 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media On 1/28/19 11:03 AM, Tim Harvey wrote: > On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >> Hi Tim, cc: Philipp, >> >> On 1/23/19 3:21 PM, Tim Harvey wrote: >>> Steve, >>> >>> I'm testing IMX6 capture again with the tda1997x HDMI receiver found >>> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6 >>> CSI in a way where it can be configured for 8bit BT656 mode >>> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall >>> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide >>> variety of resolutions, bus formats, interlaced modes, and >>> colorspaces. >>> >>> I'm using the following for testing: >>> - Linux 4.20 + >>> - media: imx-csi: Input connections to CSI should be optional >>> - imx-media: Fixes for interlaced capture >>> - media: imx: Stop stream before disabling IDMA channels >>> - v4l-utils 1.16.1 >>> - gstreamer-1.14.1 (with the ability to test master as well) >>> >>> I have a script that generates the media-ctl/v4l2-ctl commands based >>> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI >>> decoder) and a 'mode' which I've defined as: >>> mode0: sensor -> mux -> csi -> /dev/videoN >>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a >>> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN >>> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN >>> >>> The media-ctl topologies for each cpu/board combo are at >>> http://dev.gateworks.com/docs/linux/media/ >>> >>> I'm trying to test out simple v4l2-ctl based capture of a single frame >>> as well as capture and stream via both software JPEG encode and H264 >>> hardware encode via CODA. >>> >>> Please let me know of any changes that should be made to the commands >>> below even if only to purely help document things through clarity. >>> >>> One of the issues I run into right away is image size: The imx.rst >>> docs state that the ic has a 'resize' limit of 1024x1024 but I think >>> I'm mislead by the word 'resize' and this also means you can't push >>> say 1080x720 through it (not resizing, just putting that on the src >>> pad) as it clips it to 1024x720. >> The IC is limited to a resized *output* frame of 1024x1024, no higher. >> > Ok so this means for anything >1024 I need to either use the csi to > downscale it by factors of 2 first, or use mem2mem solution which > means I can't use the VDIC (so no interlaced). Actually that isn't true. You can still use a VDIC pipeline for the 1280x720 interlaced tda19971 source. It's just that you would need to downscale to <=1024 (either using the CSI /2, and/or using the IC prpvf scaler). Then use the mem2mem gstreamer element to upscale the prpvf capture output to something >1024. > I will try your patch > that adds registration of the mem2mem driver to the media-ctl api. Ok. That patch doesn't have anything functional to it, but it's nice to see the mem2mem device as part of the media graph. > >>> I believe some of Phillip's pending >>> patches may be aimed at rectifying this limitation? >> If you mean the mem2mem driver, yes. The mem2mem driver makes use the >> ipu-image-convert APIs which includes tiled resizing, so it will support >> greater-than 1024x1024 output frames. Latest status I've heard is that >> there is some misunderstanding of the behavior of rotation, that will >> require some rework of the mem2mem driver and ipu-image-convert (and >> sigh, the ic-prpencvf sub-device which also supports h/w rotation). >> >> But you could actually grab the last posted version of mem2mem driver >> (v7). It works fine except for the rotation misunderstanding (but even >> that works but doesn't conform to v4l2 expected behavior). You will then >> have a gstreamer element v4l2convertN for >1024 resizing support. >> > Ok, so I've applied that to my kernel and now have a new /dev/video* > device for ipu_ic_pp. I'll start testing with it. Cool. > >> Also recently merged from Philipp is the work to relax capture width >> alignment, by DMA'ing to padded frames. But that is not related to mem2mem. >> > Ok - I'll merge this one too so I'm testing with it. Ok. > >>> Another issue I'm running into is colorspace conversion, specifically >>> colorimetry >>> >>> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0 >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 >>> field:none colorspace:rec709 >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]" >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]" >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]" >>> # capture device >>> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 >>> v4l2-ctl --device /dev/video0 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 1024/720 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 2048 >>> Size Image : 1474560 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: ITU-R 601 >>> Quantization : Limited Range >>> Flags : >>> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch! >> Yes as discussed above that is correct behavior, you'll need mem2mem >> device to support >1024 tiled scaling. >> >>> # capture 1 frame >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 >>> ^^^ works >>> # stream JPEG/RTP/UDP >>> gst-launch-1.0 v4l2src device=/dev/video0 ! \ >>> video/x-raw,format=UYVY ! \ >>> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 >>> ^^^ works (but of course squashed horizontally because of x=1024) >>> # stream H264/RTP/UDP (via coda) >>> # need to make sure we feed coda NV12/rec709 >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' # >>> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709 >>> ycbcr:601 quantization:lim-range >>> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709 >>> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720 >>> colorspace:rec709 ycbcr:709]' >>> ^^^ no error but doing a get-v4l2 shows no change - is this all >>> because of trying to deal with 1080 instead of 1024? >> No. The Image Converter can only Y`CbCr encode to ITU-R 601 >> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr >> encoding when using an IC pipeline. >> >> See imx_media_fill_default_mbus_fields(). >> >>> gst-launch-1.0 v4l2src device=/dev/video0 ! \ >>> v4l2h264enc output-io-mode=dmabuf-import ! \ >>> rtph264pay ! udpsink host=172.24.20.19 port=5001 >>> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry >>> ^^^ fails because of colorimetry not getting set >> Right, that is correct behavior also. See above. >> >> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace? >> > I will have to go back and look what the old downstream Freescale > kernel drivers did here... I could swear I could easily de-interlace > via VDIC and encode 1080i with that and gstreamer-imx. > >>> <snip> >>> >>> Let's try an interlaced format... >>> >>> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0 >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 >>> field:seq-tb colorspace:rec709 >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" >>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" >>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" >>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]" >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" >>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" >>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" >>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" >>> # capture device >>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 >>> v4l2-ctl --device /dev/video1 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 720/480 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 1440 >>> Size Image : 691200 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: ITU-R 601 >>> Quantization : Limited Range >>> Flags : >>> # capture 1 frame >>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 >>> ^^^ fails with ipu1_ic_prpvf: EOF timeout >>> ^^^ this is almost identical to the adv7180 case which works... the >>> only exception I see is field:seq-tb vs field:alternate from the >>> source which should be fine (more specific than 'alternate') >> Hmm, can you send the complete pad formats from --get-v4l2. I don't see >> anything obviously wrong here. >> > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" > [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" > [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1" > [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2" > [fmt:AYUV8_1X32/720x480@1/60 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2" > [fmt:AYUV8_1X32/720x480@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1" > [fmt:AYUV8_1X32/720x480@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video > Format Video Capture: > Width/Height : 720/480 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1440 > Size Image : 691200 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: ITU-R 601 > Quantization : Limited Range > Flags : > > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap > --stream-to=x.raw --stream-count=1 > [ 159.828654] ipu1_ic_prpvf: EOF timeout > VIDIOC_DQBUF: failed: Input/output error Sorry everything looks fine, I don't know why that pipeline is failing to generate any data. > >>> Now lets go back to a 480p60 source but this time include the vdic >>> (which isn't necessary but should still work right?) >> No. First, the CSI will only capture in bt.656 mode if it sees >> interlaced fields (bt.656 interlaced sync codes). Well, let me rephrase, >> the CSI does support progressive BT.656 but I have never tested that myself. >> >> But more importantly, the VDIC must be given interlaced fields at its input. >> > ok, I understand now that VDIC can only accept interlaced fields, so > progressive should skip the VDIC. > >>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via BT656 IPU1_CSI0 >>> MODE1:sensor->mux->csi->vdic->ic_prp->ic_prpvf >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 >>> field:none colorspace:rec709 >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" >>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" >>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" >>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:none]" >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" >>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" >>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" >>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" >>> # capture device >>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 >>> v4l2-ctl --device /dev/video1 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 720/480 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 1440 >>> Size Image : 691200 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: ITU-R 601 >>> Quantization : Limited Range >>> Flags : >>> # capture 1 frame >>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 >>> ipu1_ic_prpvf: pipeline start failed with -32 >>> ^^^ maybe because we are feeding progressive to vdic? vdic isn't >>> needed here but it seems it should not fail to me >> this is a pipeline src->sink mismatch error, because the VDIC enforces >> fields at its input pad. > right... as you said above vdic requires interlaced fields so > progressive must skip the vdic. > >>> ok now I'm going to boot with a different device-tree that connects >>> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be >>> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed >>> for coda 3) ability to capture 1080p60 as I have confirmed tyring to >>> do so via bt656 exceeds the IMX pixel clock rate and causes CSI >>> corruption. Downsides to this mode is that I currently have a bug in >>> the tda1997x driver regarding this mode and interlaced capture. >>> >>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0 >>> MODE0:sensor->mux->csi >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480 >>> field:none colorspace:rec709 >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]" >>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" >>> media-ctl --get-v4l2 '"ipu1_csi0":2' >>> [fmt:UYVY8_1X16/720x480@1/30 field:none >> This is passthrough, the CSI can only pass through input pixel data >> unmodified as "generic" data, when the input bus is 16-bit parallel, >> which it is in this case (UYVY8_1X16). So the pixel code at the CSI >> output pad is forced to same as input pad (UYVY8_1X16). >> >> But aside from passthrough, that shouldn't be a problem... >> >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] >>> # capture device >>> media-ctl -e 'ipu1_csi0 capture' # /dev/video4 >>> v4l2-ctl --device /dev/video4 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 720/480 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 1440 >>> Size Image : 691200 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: Rec. 709 >>> Quantization : Limited Range >>> Flags : premultiplied-alpha >>> # capture 1 frame >>> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1 >>> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png >> So capture of 1 frame worked fine? > yes this worked fine > >>> # stream JPEG/RTP/UDP >>> gst-launch-1.0 v4l2src device=/dev/video4 ! \ >>> video/x-raw,format=UYVY ! \ >>> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 >> and this worked? > yes this worked fine > >>> # stream H264/RTP/UDP >>> media-ctl --get-v4l2 '"ipu1_csi0":0' >>> [fmt:UYVY8_1X16/720x480@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range >>> crop.bounds:(0,0)/720x480 >>> crop:(0,0)/720x480 >>> compose.bounds:(0,0)/720x480 >>> compose:(0,0)/720x480] >>> ^^^ no conv needed >>> media-ctl --get-v4l2 '"ipu1_csi0":0' >> huh? > typo'd here and cut-and-pasted another occurrence of the media-ctl > --get-v4l2 command but my point here is that we have the correct > colorspace/colorimetry for coda so the H264 below should work fine > >>> gst-launch-1.0 v4l2src device=/dev/video4 ! \ >>> v4l2h264enc output-io-mode=dmabuf-import ! \ >>> rtph264pay ! udpsink host=172.24.20.19 port=5001 >>> v4l2src0: Internal data stream error. >> Sorry, I don't have enough info to help here. >> > Here's the pad details: > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" > [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" > [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2" > [fmt:UYVY8_1X16/720x480@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video > Format Video Capture: > Width/Height : 720/480 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1440 > Size Image : 691200 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: Rec. 709 > Quantization : Limited Range > Flags : > > So in this passthrough case v4l2-ctl stream capture works, gstreamer > capture with sw jpeg encode works, but gstreamer capture with h264 > encode fails: > > root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \ >> v4l2h264enc output-io-mode=dmabuf-import ! \ >> rtph264pay ! udpsink host=172.24.20.19 port=5001 > Setting pipeline to PAUSED ... > Pipeline is live and does not need PREROLL ... > Setting pipeline to PLAYING ... > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: > Internal data stream error. > Additional debug info: > gstbasesrc.c(3055): gst_base_src_loop (): > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: > streaming stopped, reason not-negotiated (-4) > Execution ended after 0:00:00.000416672 > Setting pipeline to PAUSED ... > Setting pipeline to READY ... > Setting pipeline to NULL ... > Freeing pipeline ... > > It would appear that this must have something to do with gstreamer. > The only difference here between this case and the adv7180 is that > field=none and colorimetry is itu601... I'm guessing the field=none is > causing an issue. I'll post a question to gstreamer-devel. Ok. Given that v4l2-ctl stream capture works, gstreamer capture with sw jpeg encode works, but gstreamer capture with h264does not, this does sound like some issue with gstreamer. Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-28 23:04 ` Steve Longerbeam @ 2019-01-31 1:18 ` Tim Harvey 2019-02-02 19:10 ` Steve Longerbeam 0 siblings, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-01-31 1:18 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Mon, Jan 28, 2019 at 3:04 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > > > On 1/28/19 11:03 AM, Tim Harvey wrote: > > On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > >> Hi Tim, cc: Philipp, > >> > >> On 1/23/19 3:21 PM, Tim Harvey wrote: > >>> Steve, > >>> > >>> I'm testing IMX6 capture again with the tda1997x HDMI receiver found > >>> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6 > >>> CSI in a way where it can be configured for 8bit BT656 mode > >>> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall > >>> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide > >>> variety of resolutions, bus formats, interlaced modes, and > >>> colorspaces. > >>> > >>> I'm using the following for testing: > >>> - Linux 4.20 + > >>> - media: imx-csi: Input connections to CSI should be optional > >>> - imx-media: Fixes for interlaced capture > >>> - media: imx: Stop stream before disabling IDMA channels > >>> - v4l-utils 1.16.1 > >>> - gstreamer-1.14.1 (with the ability to test master as well) > >>> > >>> I have a script that generates the media-ctl/v4l2-ctl commands based > >>> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI > >>> decoder) and a 'mode' which I've defined as: > >>> mode0: sensor -> mux -> csi -> /dev/videoN > >>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a > >>> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN > >>> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN > >>> > >>> The media-ctl topologies for each cpu/board combo are at > >>> http://dev.gateworks.com/docs/linux/media/ > >>> > >>> I'm trying to test out simple v4l2-ctl based capture of a single frame > >>> as well as capture and stream via both software JPEG encode and H264 > >>> hardware encode via CODA. > >>> > >>> Please let me know of any changes that should be made to the commands > >>> below even if only to purely help document things through clarity. > >>> > >>> One of the issues I run into right away is image size: The imx.rst > >>> docs state that the ic has a 'resize' limit of 1024x1024 but I think > >>> I'm mislead by the word 'resize' and this also means you can't push > >>> say 1080x720 through it (not resizing, just putting that on the src > >>> pad) as it clips it to 1024x720. > >> The IC is limited to a resized *output* frame of 1024x1024, no higher. > >> > > Ok so this means for anything >1024 I need to either use the csi to > > downscale it by factors of 2 first, or use mem2mem solution which > > means I can't use the VDIC (so no interlaced). > > Actually that isn't true. You can still use a VDIC pipeline for the > 1280x720 interlaced tda19971 source. It's just that you would need to > downscale to <=1024 (either using the CSI /2, and/or using the IC prpvf > scaler). Then use the mem2mem gstreamer element to upscale the prpvf > capture output to something >1024. > True, that would work and here are my mem2mem testing findings: I've found that mem2mem from the CSI works well: - Using the CSI to downscale 720p then going through mem2mem and upscaling it (works perfectly!) # 720p@60Hz YUV BT656 # query current timings and set to this v4l2-ctl -d $(media-ctl -e "tda19971 2-0048") --set-dv-bt-timings=query # get timings media-ctl --get-v4l2 '"tda19971 2-0048":0' # [fmt:UYVY8_2X8/1280x720 field:none colorspace:srgb] # reset media controller links media-ctl --reset # create links media-ctl --link '"tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]' media-ctl --link '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' media-ctl --link '"ipu1_csi0":2 -> "ipu1_csi0 capture":0[1]' # set pad formats media-ctl --set-v4l2 '"tda19971 2-0048":0[fmt:UYVY8_2X8/1280x720]' media-ctl --set-v4l2 '"ipu1_csi0_mux":2[fmt:UYVY8_2X8/1280x720]' media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]' media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale media-ctl --get-v4l2 '"ipu1_csi0":2' # [fmt:AYUV8_1X32/640x360@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] # get video device media-ctl -e 'ipu1_csi0 capture' # /dev/video4 # stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video0 ! \ video/x-raw,format=UYVY ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 # ^^^^ works but video is 1/4scale 640x360 # get mem2mem device (here is perhaps a great reason to have the mem2mem device in the media graph... so its easy to find the device name) grep ipu_ic_pp /sys/class/video4linux/*/name # /sys/class/video4linux/video8/name:ipu_ic_pp mem2mem - so its v4l2video8convert # stream upscale via mem2mem then JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video4 ! \ video/x-raw,format=UYVY ! \ v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 # ^^^ works and video is back at 720p! # stream H264/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video4 ! \ v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=172.24.20.19 port=5001 # ^^^ works - 720p via coda But when going through the IC we again run into the issue where the output of the IC isn't a suitable colorspace: # 720p@60Hz YUV BT656 # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 field:none colorspace:srgb # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" # downscale because the IC can't do >1024 media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' # [fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] # ^^^ note we are itu601 again b/c that's the only format the IC can ouput # stream JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video0 ! \ video/x-raw,format=UYVY ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 # ^^^ works... SW JPEG can handle itu601 # stream upscale via mem2mem then JPEG/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video4 ! \ v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 ERROR: from element /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device '/dev/video8' does not support 2:4:7:1 colorimetry # ^^^ fails because mem2mem doesn't support itu601 # stream H264/RTP/UDP gst-launch-1.0 v4l2src device=/dev/video4 ! \ v4l2h264enc output-io-mode=dmabuf-import ! \ rtph264pay ! udpsink host=172.24.20.19 port=5001 ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: Device '/dev/video9' does not support 2:4:7:1 colorimetry # ^^^ coda has same issue... can't del with itu601 Am I perhaps missing a capsfilter to get the mem2mem driver to convert the colorspace properly? If so, they the mem2mem driver could be used to correct the colorspace to get IC output to coda working. Also can we connect the mem2mem driver to the unused VDIC input in the media controller so that we can use the VDIC to de-interlace content captured from non IMX sources (ie PCI or USB capture devices)? <snip> > > > >>> Another issue I'm running into is colorspace conversion, specifically > >>> colorimetry > >>> > >>> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0 > >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc > >>> # set sensor output pad to sensor source format > >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > >>> # sensor format > >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 > >>> field:none colorspace:rec709 > >>> # reset all links > >>> media-ctl --reset > >>> # setup links > >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > >>> # configure pads > >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" > >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" > >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]" > >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]" > >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]" > >>> # capture device > >>> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 > >>> v4l2-ctl --device /dev/video0 --get-fmt-video > >>> Format Video Capture: > >>> Width/Height : 1024/720 > >>> Pixel Format : 'UYVY' (UYVY 4:2:2) > >>> Field : None > >>> Bytes per Line : 2048 > >>> Size Image : 1474560 > >>> Colorspace : Rec. 709 > >>> Transfer Function : Rec. 709 > >>> YCbCr/HSV Encoding: ITU-R 601 > >>> Quantization : Limited Range > >>> Flags : > >>> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch! > >> Yes as discussed above that is correct behavior, you'll need mem2mem > >> device to support >1024 tiled scaling. > >> > >>> # capture 1 frame > >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > >>> ^^^ works > >>> # stream JPEG/RTP/UDP > >>> gst-launch-1.0 v4l2src device=/dev/video0 ! \ > >>> video/x-raw,format=UYVY ! \ > >>> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > >>> ^^^ works (but of course squashed horizontally because of x=1024) > >>> # stream H264/RTP/UDP (via coda) > >>> # need to make sure we feed coda NV12/rec709 > >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' # > >>> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709 > >>> ycbcr:601 quantization:lim-range > >>> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709 > >>> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720 > >>> colorspace:rec709 ycbcr:709]' > >>> ^^^ no error but doing a get-v4l2 shows no change - is this all > >>> because of trying to deal with 1080 instead of 1024? > >> No. The Image Converter can only Y`CbCr encode to ITU-R 601 > >> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr > >> encoding when using an IC pipeline. > >> > >> See imx_media_fill_default_mbus_fields(). > >> > >>> gst-launch-1.0 v4l2src device=/dev/video0 ! \ > >>> v4l2h264enc output-io-mode=dmabuf-import ! \ > >>> rtph264pay ! udpsink host=172.24.20.19 port=5001 > >>> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry > >>> ^^^ fails because of colorimetry not getting set > >> Right, that is correct behavior also. See above. > >> > >> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace? > >> > > I will have to go back and look what the old downstream Freescale > > kernel drivers did here... I could swear I could easily de-interlace > > via VDIC and encode 1080i with that and gstreamer-imx. The old freescale kernel + gstreamer-imx does require using the IPU to transform the colorspace before it goes to CODA but I'm still trying to understand the gstreamer code. The pixel formats vs colorspace vs colorimetry still confuse me. So again, I'm struggling with how to get the output of the IC into coda. > > > >>> <snip> > >>> > >>> Let's try an interlaced format... > >>> > >>> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0 > >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc > >>> # set sensor output pad to sensor source format > >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > >>> # sensor format > >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 > >>> field:seq-tb colorspace:rec709 > >>> # reset all links > >>> media-ctl --reset > >>> # setup links > >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" > >>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" > >>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" > >>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" > >>> # configure pads > >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]" > >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]" > >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" > >>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" > >>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" > >>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" > >>> # capture device > >>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 > >>> v4l2-ctl --device /dev/video1 --get-fmt-video > >>> Format Video Capture: > >>> Width/Height : 720/480 > >>> Pixel Format : 'UYVY' (UYVY 4:2:2) > >>> Field : None > >>> Bytes per Line : 1440 > >>> Size Image : 691200 > >>> Colorspace : Rec. 709 > >>> Transfer Function : Rec. 709 > >>> YCbCr/HSV Encoding: ITU-R 601 > >>> Quantization : Limited Range > >>> Flags : > >>> # capture 1 frame > >>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 > >>> ^^^ fails with ipu1_ic_prpvf: EOF timeout > >>> ^^^ this is almost identical to the adv7180 case which works... the > >>> only exception I see is field:seq-tb vs field:alternate from the > >>> source which should be fine (more specific than 'alternate') > >> Hmm, can you send the complete pad formats from --get-v4l2. I don't see > >> anything obviously wrong here. > >> > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" > > [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" > > [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1" > > [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2" > > [fmt:AYUV8_1X32/720x480@1/60 field:none > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2" > > [fmt:AYUV8_1X32/720x480@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1" > > [fmt:AYUV8_1X32/720x480@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > > > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video > > Format Video Capture: > > Width/Height : 720/480 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 1440 > > Size Image : 691200 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: ITU-R 601 > > Quantization : Limited Range > > Flags : > > > > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap > > --stream-to=x.raw --stream-count=1 > > [ 159.828654] ipu1_ic_prpvf: EOF timeout > > VIDIOC_DQBUF: failed: Input/output error > > Sorry everything looks fine, I don't know why that pipeline is failing > to generate any data. > This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc) still baffles me a bit but I've also found that any bt656 capture that isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of the resolution checks in ipu_csi_init_interface() resulting in 'Unsupported interlaced video mode'. I'm not sure if ipu_csi_set_bt_interlaced_codes() can be modified to support other resolutions? > > <snip> > > > >>> ok now I'm going to boot with a different device-tree that connects > >>> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be > >>> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed > >>> for coda 3) ability to capture 1080p60 as I have confirmed tyring to > >>> do so via bt656 exceeds the IMX pixel clock rate and causes CSI > >>> corruption. Downsides to this mode is that I currently have a bug in > >>> the tda1997x driver regarding this mode and interlaced capture. > >>> > >>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0 > >>> MODE0:sensor->mux->csi > >>> # set sensor output pad to sensor source format > >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > >>> # sensor format > >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480 > >>> field:none colorspace:rec709 > >>> # reset all links > >>> media-ctl --reset > >>> # setup links > >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > >>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > >>> # configure pads > >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]" > >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]" > >>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" > >>> media-ctl --get-v4l2 '"ipu1_csi0":2' > >>> [fmt:UYVY8_1X16/720x480@1/30 field:none > >> This is passthrough, the CSI can only pass through input pixel data > >> unmodified as "generic" data, when the input bus is 16-bit parallel, > >> which it is in this case (UYVY8_1X16). So the pixel code at the CSI > >> output pad is forced to same as input pad (UYVY8_1X16). > >> > >> But aside from passthrough, that shouldn't be a problem... > >> > >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > >>> # capture device > >>> media-ctl -e 'ipu1_csi0 capture' # /dev/video4 > >>> v4l2-ctl --device /dev/video4 --get-fmt-video > >>> Format Video Capture: > >>> Width/Height : 720/480 > >>> Pixel Format : 'UYVY' (UYVY 4:2:2) > >>> Field : None > >>> Bytes per Line : 1440 > >>> Size Image : 691200 > >>> Colorspace : Rec. 709 > >>> Transfer Function : Rec. 709 > >>> YCbCr/HSV Encoding: Rec. 709 > >>> Quantization : Limited Range > >>> Flags : premultiplied-alpha > >>> # capture 1 frame > >>> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1 > >>> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png > >> So capture of 1 frame worked fine? > > yes this worked fine > > > >>> # stream JPEG/RTP/UDP > >>> gst-launch-1.0 v4l2src device=/dev/video4 ! \ > >>> video/x-raw,format=UYVY ! \ > >>> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > >> and this worked? > > yes this worked fine > > > >>> # stream H264/RTP/UDP > >>> media-ctl --get-v4l2 '"ipu1_csi0":0' > >>> [fmt:UYVY8_1X16/720x480@1/30 field:none > >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range > >>> crop.bounds:(0,0)/720x480 > >>> crop:(0,0)/720x480 > >>> compose.bounds:(0,0)/720x480 > >>> compose:(0,0)/720x480] > >>> ^^^ no conv needed > >>> media-ctl --get-v4l2 '"ipu1_csi0":0' > >> huh? > > typo'd here and cut-and-pasted another occurrence of the media-ctl > > --get-v4l2 command but my point here is that we have the correct > > colorspace/colorimetry for coda so the H264 below should work fine > > > >>> gst-launch-1.0 v4l2src device=/dev/video4 ! \ > >>> v4l2h264enc output-io-mode=dmabuf-import ! \ > >>> rtph264pay ! udpsink host=172.24.20.19 port=5001 > >>> v4l2src0: Internal data stream error. > >> Sorry, I don't have enough info to help here. > >> > > Here's the pad details: > > > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" > > [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" > > [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] > > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2" > > [fmt:UYVY8_1X16/720x480@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > > > > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video > > Format Video Capture: > > Width/Height : 720/480 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 1440 > > Size Image : 691200 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: Rec. 709 > > Quantization : Limited Range > > Flags : > > > > So in this passthrough case v4l2-ctl stream capture works, gstreamer > > capture with sw jpeg encode works, but gstreamer capture with h264 > > encode fails: > > > > root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \ > >> v4l2h264enc output-io-mode=dmabuf-import ! \ > >> rtph264pay ! udpsink host=172.24.20.19 port=5001 > > Setting pipeline to PAUSED ... > > Pipeline is live and does not need PREROLL ... > > Setting pipeline to PLAYING ... > > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: > > Internal data stream error. > > Additional debug info: > > gstbasesrc.c(3055): gst_base_src_loop (): > > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: > > streaming stopped, reason not-negotiated (-4) > > Execution ended after 0:00:00.000416672 > > Setting pipeline to PAUSED ... > > Setting pipeline to READY ... > > Setting pipeline to NULL ... > > Freeing pipeline ... > > > > It would appear that this must have something to do with gstreamer. > > The only difference here between this case and the adv7180 is that > > field=none and colorimetry is itu601... I'm guessing the field=none is > > causing an issue. I'll post a question to gstreamer-devel. > > Ok. Given that v4l2-ctl stream capture works, gstreamer capture with sw > jpeg encode works, but gstreamer capture with h264does not, this does > sound like some issue with gstreamer. > While the 480p60Hz YUV via 16bit YUV sensor->mux->csi appears to work from a V4L2 capture perspective I find that this input with the IC causes 'pipeline start failed with -32' errors: # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 MODE1:sensor->mux->csi->ic_prp->ic_prpenc # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] # get framerate v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings DV timings: Active width: 1280 Active height: 720 Total width: 1650 Total height: 750 Frame format: progressive Polarities: +vsync +hsync Pixelclock: 74250000 Hz (60.00 frames per second) Horizontal frontporch: 110 Horizontal sync: 40 Horizontal backporch: 220 Vertical frontporch: 5 Vertical sync: 5 Vertical backporch: 20 Standards: CTA-861 CTA-861 VIC: 0 Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' # capture device media-ctl -e 'ipu1_ic_prpenc capture' /dev/video0 v4l2-ctl --device /dev/video0 --get-fmt-video Format Video Capture: Width/Height : 640/360 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1280 Size Image : 460800 Colorspace : Rec. 709 Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : # capture 1 frame v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 Do you know what the failure is here? And can you explain why I can't colorspace convert the following CSI capture case?: # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" media-ctl --get-v4l2 '"ipu1_csi0":2' [fmt:UYVY8_1X16/1280x720@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] ^^^ still itu601 I have found in my testing when dealing with a BT656 RGB colorspace input which I need to convert ot rec709 I have to set the 'input' pad of the csi which doesn't make sense and if I do this with a 16-bit RGB colorspace (above) it jacks up the fmt: media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" media-ctl --get-v4l2 '"ipu1_csi0":2' [fmt:UYVY8_2X8/1280x720@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults fmt to YUVY8_2X8 So there is something I'm still doing wrong to setup CSC. Thanks, Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-01-31 1:18 ` Tim Harvey @ 2019-02-02 19:10 ` Steve Longerbeam 2019-02-03 19:37 ` Steve Longerbeam 2019-02-05 19:16 ` Tim Harvey 0 siblings, 2 replies; 19+ messages in thread From: Steve Longerbeam @ 2019-02-02 19:10 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media On 1/30/19 5:18 PM, Tim Harvey wrote: > On Mon, Jan 28, 2019 at 3:04 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >> >> >> On 1/28/19 11:03 AM, Tim Harvey wrote: >>> On Fri, Jan 25, 2019 at 3:57 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >>>> Hi Tim, cc: Philipp, >>>> >>>> On 1/23/19 3:21 PM, Tim Harvey wrote: >>>>> Steve, >>>>> >>>>> I'm testing IMX6 capture again with the tda1997x HDMI receiver found >>>>> on Gateworks GW54xx and GW551x boards. This is hooked up to the IMX6 >>>>> CSI in a way where it can be configured for 8bit BT656 mode >>>>> (UYVY8_2X8) or 16bit YUV mode (UYVY8_1X16). Also I have a Marshall >>>>> Electronics V-SG4K-HDI HDMI signal generator here so I can test a wide >>>>> variety of resolutions, bus formats, interlaced modes, and >>>>> colorspaces. >>>>> >>>>> I'm using the following for testing: >>>>> - Linux 4.20 + >>>>> - media: imx-csi: Input connections to CSI should be optional >>>>> - imx-media: Fixes for interlaced capture >>>>> - media: imx: Stop stream before disabling IDMA channels >>>>> - v4l-utils 1.16.1 >>>>> - gstreamer-1.14.1 (with the ability to test master as well) >>>>> >>>>> I have a script that generates the media-ctl/v4l2-ctl commands based >>>>> on providing a sensor ('adv7180' analog decoder or 'tda1997x' HDMI >>>>> decoder) and a 'mode' which I've defined as: >>>>> mode0: sensor -> mux -> csi -> /dev/videoN >>>>> mode1: sensor -> mux -> csi -> ic_prp -> ic_prpenc -> /dev/videoN (provides a >>>>> mode2: sensor -> mux -> csi -> ic_prp -> ic_prpvf -> /dev/videoN >>>>> mode3: sensor -> mux -> csi -> vdic -> ic_prp -> ic_prpvf -> /dev/videoN >>>>> >>>>> The media-ctl topologies for each cpu/board combo are at >>>>> http://dev.gateworks.com/docs/linux/media/ >>>>> >>>>> I'm trying to test out simple v4l2-ctl based capture of a single frame >>>>> as well as capture and stream via both software JPEG encode and H264 >>>>> hardware encode via CODA. >>>>> >>>>> Please let me know of any changes that should be made to the commands >>>>> below even if only to purely help document things through clarity. >>>>> >>>>> One of the issues I run into right away is image size: The imx.rst >>>>> docs state that the ic has a 'resize' limit of 1024x1024 but I think >>>>> I'm mislead by the word 'resize' and this also means you can't push >>>>> say 1080x720 through it (not resizing, just putting that on the src >>>>> pad) as it clips it to 1024x720. >>>> The IC is limited to a resized *output* frame of 1024x1024, no higher. >>>> >>> Ok so this means for anything >1024 I need to either use the csi to >>> downscale it by factors of 2 first, or use mem2mem solution which >>> means I can't use the VDIC (so no interlaced). >> Actually that isn't true. You can still use a VDIC pipeline for the >> 1280x720 interlaced tda19971 source. It's just that you would need to >> downscale to <=1024 (either using the CSI /2, and/or using the IC prpvf >> scaler). Then use the mem2mem gstreamer element to upscale the prpvf >> capture output to something >1024. >> > True, that would work and here are my mem2mem testing findings: > > I've found that mem2mem from the CSI works well: > - Using the CSI to downscale 720p then going through mem2mem and > upscaling it (works perfectly!) Good to hear! > # 720p@60Hz YUV BT656 > # query current timings and set to this > v4l2-ctl -d $(media-ctl -e "tda19971 2-0048") --set-dv-bt-timings=query > # get timings > media-ctl --get-v4l2 '"tda19971 2-0048":0' # [fmt:UYVY8_2X8/1280x720 > field:none colorspace:srgb] > # reset media controller links > media-ctl --reset > # create links > media-ctl --link '"tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]' > media-ctl --link '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' > media-ctl --link '"ipu1_csi0":2 -> "ipu1_csi0 capture":0[1]' > # set pad formats > media-ctl --set-v4l2 '"tda19971 2-0048":0[fmt:UYVY8_2X8/1280x720]' > media-ctl --set-v4l2 '"ipu1_csi0_mux":2[fmt:UYVY8_2X8/1280x720]' > media-ctl --set-v4l2 '"ipu1_csi0":0[fmt:AYUV32/1280x720 > colorspace:rec709 ycbcr:709]' > media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale It's not needed to do 1/2 scale in this use-case, since this pipeline is not sending to the IC. But even for an IC pipeline, for this case (1280x720) you only need to downscale 1/2 in width, not in both width and height (the CSI can independently downscale in either dimension), E.g: media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x720]' # 1/2 scale in width > media-ctl --get-v4l2 '"ipu1_csi0":2' # [fmt:AYUV8_1X32/640x360@1/30 > field:none colorspace:rec709 xfer:709 ycbcr:709 > quantization:lim-range] > # get video device > media-ctl -e 'ipu1_csi0 capture' # /dev/video4 > # stream JPEG/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > video/x-raw,format=UYVY ! \ > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > # ^^^^ works but video is 1/4scale 640x360 > # get mem2mem device (here is perhaps a great reason to have the > mem2mem device in the media graph... so its easy to find the device > name) > grep ipu_ic_pp /sys/class/video4linux/*/name # > /sys/class/video4linux/video8/name:ipu_ic_pp mem2mem - so its > v4l2video8convert > # stream upscale via mem2mem then JPEG/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > video/x-raw,format=UYVY ! \ > v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > # ^^^ works and video is back at 720p! > # stream H264/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ > v4l2h264enc output-io-mode=dmabuf-import ! \ > rtph264pay ! udpsink host=172.24.20.19 port=5001 > # ^^^ works - 720p via coda Great. > > But when going through the IC we again run into the issue where the > output of the IC isn't a suitable colorspace: > # 720p@60Hz YUV BT656 > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 > field:none colorspace:srgb > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" > # downscale because the IC can't do >1024 > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' # > [fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb > ycbcr:601 quantization:lim-range] > # ^^^ note we are itu601 again b/c that's the only format the IC can ouput > # stream JPEG/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > video/x-raw,format=UYVY ! \ > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > # ^^^ works... SW JPEG can handle itu601 Ok. > # stream upscale via mem2mem then JPEG/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > ERROR: from element > /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device > '/dev/video8' does not support 2:4:7:1 colorimetry > # ^^^ fails because mem2mem doesn't support itu601 > # stream H264/RTP/UDP > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > v4l2h264enc output-io-mode=dmabuf-import ! \ > rtph264pay ! udpsink host=172.24.20.19 port=5001 > ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: > Device '/dev/video9' does not support 2:4:7:1 colorimetry > # ^^^ coda has same issue... can't del with itu601 Well, just to see things working, try hacking imx_media_fill_default_mbus_fields() to set Rec. 709 encoding: --- a/drivers/staging/media/imx/imx-media-utils.c +++ b/drivers/staging/media/imx/imx-media-utils.c @@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct v4l2_mbus_framefmt *tryfmt, tryfmt->quantization = is_rgb ? V4L2_QUANTIZATION_FULL_RANGE : V4L2_QUANTIZATION_LIM_RANGE; - tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601; + tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709; } } EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields); But of course that's not technically correct because the encoding in ipu-ic.c is BT.601. The *real* way to fix this would be to allow programmable encodings in ipu-ic.c. But unfortunately the encodings are hardcoded (grep for ic_csc_rgb2ycbcr in ipu-ic.c). The other option would be to change ic_csc_rgb2ycbcr to use the Rec. 709 coefficients, and then the above patch is no longer a hack. The inverse encoding (ic_csc_ycbcr2rgb) would also need to be changed. > > Am I perhaps missing a capsfilter to get the mem2mem driver to convert > the colorspace properly? If so, they the mem2mem driver could be used > to correct the colorspace to get IC output to coda working. Well, first I don't think the mem2mem driver is using the correct encoding. The mem2mem driver is making use of the IC encoding so it should be reporting and accepting only BT.601. Philipp ^^ > > Also can we connect the mem2mem driver to the unused VDIC input in the > media controller so that we can use the VDIC to de-interlace content > captured from non IMX sources (ie PCI or USB capture devices)? Exactly! That's something I have been working on. But it's difficult to connect mem2mem to the unused VDIC IDMAC input pad because as of now the v4l2 mem2mem internal API's do not allow connecting to *existing* processing entities, and there are also issues with how sub-devices are to deal with mem2mem contexts. I do have a WIP branch that creates a video output device that connects to the VDIC IDMAC input pad, which doesn't have the above issues. The only drawback with that is how gstreamer can make use of such an output device. > <snip> >>>>> Another issue I'm running into is colorspace conversion, specifically >>>>> colorimetry >>>>> >>>>> Example: imx6q-gw54xx tda19971 720p60Hz YUV via BT656 IPU1_CSI0 >>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>>>> # set sensor output pad to sensor source format >>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>>>> # sensor format >>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 >>>>> field:none colorspace:rec709 >>>>> # reset all links >>>>> media-ctl --reset >>>>> # setup links >>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>>>> # configure pads >>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" >>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" >>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/1280x720]" >>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/1280x720]" >>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/1280x720]" >>>>> # capture device >>>>> media-ctl -e 'ipu1_ic_prpenc capture' # /dev/video0 >>>>> v4l2-ctl --device /dev/video0 --get-fmt-video >>>>> Format Video Capture: >>>>> Width/Height : 1024/720 >>>>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>>>> Field : None >>>>> Bytes per Line : 2048 >>>>> Size Image : 1474560 >>>>> Colorspace : Rec. 709 >>>>> Transfer Function : Rec. 709 >>>>> YCbCr/HSV Encoding: ITU-R 601 >>>>> Quantization : Limited Range >>>>> Flags : >>>>> ^^^ Note 1080x720 has been reduced to 1024x720 - ouch! >>>> Yes as discussed above that is correct behavior, you'll need mem2mem >>>> device to support >1024 tiled scaling. >>>> >>>>> # capture 1 frame >>>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 >>>>> ^^^ works >>>>> # stream JPEG/RTP/UDP >>>>> gst-launch-1.0 v4l2src device=/dev/video0 ! \ >>>>> video/x-raw,format=UYVY ! \ >>>>> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 >>>>> ^^^ works (but of course squashed horizontally because of x=1024) >>>>> # stream H264/RTP/UDP (via coda) >>>>> # need to make sure we feed coda NV12/rec709 >>>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":0' # >>>>> fmt:AYUV8_1X32/1280x720@1/25 field:none colorspace:rec709 xfer:709 >>>>> ycbcr:601 quantization:lim-range >>>>> ^^^ input colorspace is rec709 but colorimetry needs to be changed to 709 >>>>> media-ctl --set-v4l2 '"ipu1_ic_prpenc":0[fmt:AYUV32/1280x720 >>>>> colorspace:rec709 ycbcr:709]' >>>>> ^^^ no error but doing a get-v4l2 shows no change - is this all >>>>> because of trying to deal with 1080 instead of 1024? >>>> No. The Image Converter can only Y`CbCr encode to ITU-R 601 >>>> (V4L2_YCBCR_ENC_601). So the driver won't accept Rec. 709 Y`CbCr >>>> encoding when using an IC pipeline. >>>> >>>> See imx_media_fill_default_mbus_fields(). >>>> >>>>> gst-launch-1.0 v4l2src device=/dev/video0 ! \ >>>>> v4l2h264enc output-io-mode=dmabuf-import ! \ >>>>> rtph264pay ! udpsink host=172.24.20.19 port=5001 >>>>> v4l2src0: Device '/dev/video0' does not support 2:0:0:0 colorimetry >>>>> ^^^ fails because of colorimetry not getting set >>>> Right, that is correct behavior also. See above. >>>> >>>> Philipp, can the coda driver accept V4L2_YCBCR_ENC_601 colorspace? >>>> >>> I will have to go back and look what the old downstream Freescale >>> kernel drivers did here... I could swear I could easily de-interlace >>> via VDIC and encode 1080i with that and gstreamer-imx. > The old freescale kernel + gstreamer-imx does require using the IPU to > transform the colorspace before it goes to CODA but I'm still trying > to understand the gstreamer code. The pixel formats vs colorspace vs > colorimetry still confuse me. > > So again, I'm struggling with how to get the output of the IC into coda. See above, try just hacking with the above patch for now to get things working (albeit with incorrect Y'CbCr encoding). > >>>>> <snip> >>>>> >>>>> Let's try an interlaced format... >>>>> >>>>> Example: imx6q-gw54xx tda19971 480i60Hz YUV via BT656 IPU1_CSI0 >>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>>>> # set sensor output pad to sensor source format >>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>>>> # sensor format >>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/720x480 >>>>> field:seq-tb colorspace:rec709 >>>>> # reset all links >>>>> media-ctl --reset >>>>> # setup links >>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" >>>>> media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" >>>>> media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" >>>>> media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" >>>>> # configure pads >>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/720x480 field:seq-tb]" >>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/720x480 field:seq-tb]" >>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x480]" >>>>> media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x480]" >>>>> media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x480]" >>>>> media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x480]" >>>>> # capture device >>>>> media-ctl -e 'ipu1_ic_prpvf capture' # /dev/video1 >>>>> v4l2-ctl --device /dev/video1 --get-fmt-video >>>>> Format Video Capture: >>>>> Width/Height : 720/480 >>>>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>>>> Field : None >>>>> Bytes per Line : 1440 >>>>> Size Image : 691200 >>>>> Colorspace : Rec. 709 >>>>> Transfer Function : Rec. 709 >>>>> YCbCr/HSV Encoding: ITU-R 601 >>>>> Quantization : Limited Range >>>>> Flags : >>>>> # capture 1 frame >>>>> v4l2-ctl --device /dev/video1 --stream-mmap --stream-to=x.raw --stream-count=1 >>>>> ^^^ fails with ipu1_ic_prpvf: EOF timeout >>>>> ^^^ this is almost identical to the adv7180 case which works... the >>>>> only exception I see is field:seq-tb vs field:alternate from the >>>>> source which should be fine (more specific than 'alternate') >>>> Hmm, can you send the complete pad formats from --get-v4l2. I don't see >>>> anything obviously wrong here. >>>> >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" >>> [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" >>> [fmt:UYVY8_2X8/720x480 field:seq-tb colorspace:rec709] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':1" >>> [fmt:AYUV8_1X32/720x480@1/30 field:seq-tb >>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_vdic':2" >>> [fmt:AYUV8_1X32/720x480@1/60 field:none >>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':2" >>> [fmt:AYUV8_1X32/720x480@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpvf':1" >>> [fmt:AYUV8_1X32/720x480@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >>> >>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 720/480 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 1440 >>> Size Image : 691200 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: ITU-R 601 >>> Quantization : Limited Range >>> Flags : >>> >>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video1 --stream-mmap >>> --stream-to=x.raw --stream-count=1 >>> [ 159.828654] ipu1_ic_prpvf: EOF timeout >>> VIDIOC_DQBUF: failed: Input/output error >> Sorry everything looks fine, I don't know why that pipeline is failing >> to generate any data. >> > This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc) > still baffles me a bit but I've also found that any bt656 capture that > isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of > the resolution checks in ipu_csi_init_interface() resulting in > 'Unsupported interlaced video mode'. I'm not sure if > ipu_csi_set_bt_interlaced_codes() can be modified to support other > resolutions? Well, Bt.656 only defines standard definition NTSC and PAL. > <snip> >>>>> ok now I'm going to boot with a different device-tree that connects >>>>> the tda1997x to IMX6 via 16bit YUV. Advantages of this mode should be >>>>> 1) better pixel bpp 2) colorimetry is rec709 so no conversion needed >>>>> for coda 3) ability to capture 1080p60 as I have confirmed tyring to >>>>> do so via bt656 exceeds the IMX pixel clock rate and causes CSI >>>>> corruption. Downsides to this mode is that I currently have a bug in >>>>> the tda1997x driver regarding this mode and interlaced capture. >>>>> >>>>> Example: imx6q-gw54xx tda19971 480p60Hz YUV via 16bit YUV IPU1_CSI0 >>>>> MODE0:sensor->mux->csi >>>>> # set sensor output pad to sensor source format >>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>>>> # sensor format >>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_1X16/720x480 >>>>> field:none colorspace:rec709 >>>>> # reset all links >>>>> media-ctl --reset >>>>> # setup links >>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>>>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" >>>>> # configure pads >>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/720x480 field:none]" >>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/720x480 field:none]" >>>>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" >>>>> media-ctl --get-v4l2 '"ipu1_csi0":2' >>>>> [fmt:UYVY8_1X16/720x480@1/30 field:none >>>> This is passthrough, the CSI can only pass through input pixel data >>>> unmodified as "generic" data, when the input bus is 16-bit parallel, >>>> which it is in this case (UYVY8_1X16). So the pixel code at the CSI >>>> output pad is forced to same as input pad (UYVY8_1X16). >>>> >>>> But aside from passthrough, that shouldn't be a problem... >>>> >>>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] >>>>> # capture device >>>>> media-ctl -e 'ipu1_csi0 capture' # /dev/video4 >>>>> v4l2-ctl --device /dev/video4 --get-fmt-video >>>>> Format Video Capture: >>>>> Width/Height : 720/480 >>>>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>>>> Field : None >>>>> Bytes per Line : 1440 >>>>> Size Image : 691200 >>>>> Colorspace : Rec. 709 >>>>> Transfer Function : Rec. 709 >>>>> YCbCr/HSV Encoding: Rec. 709 >>>>> Quantization : Limited Range >>>>> Flags : premultiplied-alpha >>>>> # capture 1 frame >>>>> v4l2-ctl --device /dev/video4 --stream-mmap --stream-to=x.raw --stream-count=1 >>>>> convert -size 720x480 -depth 16 uyvy:x.raw /var/www/html/files/frame.png >>>> So capture of 1 frame worked fine? >>> yes this worked fine >>> >>>>> # stream JPEG/RTP/UDP >>>>> gst-launch-1.0 v4l2src device=/dev/video4 ! \ >>>>> video/x-raw,format=UYVY ! \ >>>>> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 >>>> and this worked? >>> yes this worked fine >>> >>>>> # stream H264/RTP/UDP >>>>> media-ctl --get-v4l2 '"ipu1_csi0":0' >>>>> [fmt:UYVY8_1X16/720x480@1/30 field:none >>>>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range >>>>> crop.bounds:(0,0)/720x480 >>>>> crop:(0,0)/720x480 >>>>> compose.bounds:(0,0)/720x480 >>>>> compose:(0,0)/720x480] >>>>> ^^^ no conv needed >>>>> media-ctl --get-v4l2 '"ipu1_csi0":0' >>>> huh? >>> typo'd here and cut-and-pasted another occurrence of the media-ctl >>> --get-v4l2 command but my point here is that we have the correct >>> colorspace/colorimetry for coda so the H264 below should work fine >>> >>>>> gst-launch-1.0 v4l2src device=/dev/video4 ! \ >>>>> v4l2h264enc output-io-mode=dmabuf-import ! \ >>>>> rtph264pay ! udpsink host=172.24.20.19 port=5001 >>>>> v4l2src0: Internal data stream error. >>>> Sorry, I don't have enough info to help here. >>>> >>> Here's the pad details: >>> >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" >>> [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" >>> [fmt:UYVY8_1X16/720x480 field:none colorspace:rec709] >>> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':2" >>> [fmt:UYVY8_1X16/720x480@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] >>> >>> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video4 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 720/480 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 1440 >>> Size Image : 691200 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: Rec. 709 >>> Quantization : Limited Range >>> Flags : >>> >>> So in this passthrough case v4l2-ctl stream capture works, gstreamer >>> capture with sw jpeg encode works, but gstreamer capture with h264 >>> encode fails: >>> >>> root@imx6q-gw5404:~# gst-launch-1.0 v4l2src device=/dev/video4 ! \ >>>> v4l2h264enc output-io-mode=dmabuf-import ! \ >>>> rtph264pay ! udpsink host=172.24.20.19 port=5001 >>> Setting pipeline to PAUSED ... >>> Pipeline is live and does not need PREROLL ... >>> Setting pipeline to PLAYING ... >>> ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: >>> Internal data stream error. >>> Additional debug info: >>> gstbasesrc.c(3055): gst_base_src_loop (): >>> /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: >>> streaming stopped, reason not-negotiated (-4) >>> Execution ended after 0:00:00.000416672 >>> Setting pipeline to PAUSED ... >>> Setting pipeline to READY ... >>> Setting pipeline to NULL ... >>> Freeing pipeline ... >>> >>> It would appear that this must have something to do with gstreamer. >>> The only difference here between this case and the adv7180 is that >>> field=none and colorimetry is itu601... I'm guessing the field=none is >>> causing an issue. I'll post a question to gstreamer-devel. >> Ok. Given that v4l2-ctl stream capture works, gstreamer capture with sw >> jpeg encode works, but gstreamer capture with h264does not, this does >> sound like some issue with gstreamer. >> > While the 480p60Hz YUV via 16bit YUV sensor->mux->csi appears to work > from a V4L2 capture perspective I find that this input with the IC > causes 'pipeline start failed with -32' errors: > > # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > # get framerate > v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings > DV timings: > Active width: 1280 > Active height: 720 > Total width: 1650 > Total height: 750 > Frame format: progressive > Polarities: +vsync +hsync > Pixelclock: 74250000 Hz (60.00 frames per second) > Horizontal frontporch: 110 > Horizontal sync: 40 > Horizontal backporch: 220 > Vertical frontporch: 5 > Vertical sync: 5 > Vertical backporch: 20 > Standards: CTA-861 > CTA-861 VIC: 0 > Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC > > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' > # capture device > media-ctl -e 'ipu1_ic_prpenc capture' > /dev/video0 > v4l2-ctl --device /dev/video0 --get-fmt-video > Format Video Capture: > Width/Height : 640/360 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1280 > Size Image : 460800 > Colorspace : Rec. 709 > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: ITU-R 601 > Quantization : Limited Range > Flags : > > # capture 1 frame > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 > > Do you know what the failure is here? You are /2 downscaling in the CSI, but did not set the compose window at the input pad, e.g: media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale > > And can you explain why I can't colorspace convert the following CSI > capture case?: > > # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' > [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > media-ctl --get-v4l2 '"ipu1_csi0":2' > [fmt:UYVY8_1X16/1280x720@1/30 field:none > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] > ^^^ still itu601 > > I have found in my testing when dealing with a BT656 RGB colorspace > input which I need to convert ot rec709 I have to set the 'input' pad > of the csi which doesn't make sense and if I do this with a 16-bit RGB > colorspace (above) it jacks up the fmt: > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > media-ctl --get-v4l2 '"ipu1_csi0":2' > [fmt:UYVY8_2X8/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults > fmt to YUVY8_2X8 > > So there is something I'm still doing wrong to setup CSC. Sorry I guess I don't understand your question here. Correct the CSI does not accept AYUV32 at its input pad. Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-02 19:10 ` Steve Longerbeam @ 2019-02-03 19:37 ` Steve Longerbeam 2019-02-05 19:16 ` Tim Harvey 1 sibling, 0 replies; 19+ messages in thread From: Steve Longerbeam @ 2019-02-03 19:37 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media Hi Tim, cc: Philipp, On 2/2/19 11:10 AM, Steve Longerbeam wrote: > > <snip> > On 1/30/19 5:18 PM, Tim Harvey wrote: > >> # stream upscale via mem2mem then JPEG/RTP/UDP >> gst-launch-1.0 v4l2src device=/dev/video4 ! \ >> v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ >> jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 >> ERROR: from element >> /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device >> '/dev/video8' does not support 2:4:7:1 colorimetry >> # ^^^ fails because mem2mem doesn't support itu601 >> # stream H264/RTP/UDP >> gst-launch-1.0 v4l2src device=/dev/video4 ! \ >> v4l2h264enc output-io-mode=dmabuf-import ! \ >> rtph264pay ! udpsink host=172.24.20.19 port=5001 >> ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: >> Device '/dev/video9' does not support 2:4:7:1 colorimetry >> # ^^^ coda has same issue... can't del with itu601 > > Well, just to see things working, try hacking > imx_media_fill_default_mbus_fields() to set Rec. 709 encoding: > > --- a/drivers/staging/media/imx/imx-media-utils.c > +++ b/drivers/staging/media/imx/imx-media-utils.c > @@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct > v4l2_mbus_framefmt *tryfmt, > tryfmt->quantization = is_rgb ? > V4L2_QUANTIZATION_FULL_RANGE : > V4L2_QUANTIZATION_LIM_RANGE; > - tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601; > + tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709; > } > } > EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields); > > > But of course that's not technically correct because the encoding in > ipu-ic.c is BT.601. > > The *real* way to fix this would be to allow programmable encodings in > ipu-ic.c. But unfortunately the encodings are hardcoded (grep for > ic_csc_rgb2ycbcr in ipu-ic.c). I went ahead and implemented this! The IC will now support both BT.601 and BT.709 encoding. I will post a patchset, please test! >> >> Am I perhaps missing a capsfilter to get the mem2mem driver to convert >> the colorspace properly? If so, they the mem2mem driver could be used >> to correct the colorspace to get IC output to coda working. > > Well, first I don't think the mem2mem driver is using the correct > encoding. The mem2mem driver is making use of the IC encoding so it > should be reporting and accepting only BT.601. The mem2mem driver can now require BT.709 and/or BT.601 encoding with the above patchset. Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-02 19:10 ` Steve Longerbeam 2019-02-03 19:37 ` Steve Longerbeam @ 2019-02-05 19:16 ` Tim Harvey 2019-02-05 23:52 ` Steve Longerbeam 1 sibling, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-02-05 19:16 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > <snip> > > > > > But when going through the IC we again run into the issue where the > > output of the IC isn't a suitable colorspace: > > # 720p@60Hz YUV BT656 > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720 > > field:none colorspace:srgb > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" > > # downscale because the IC can't do >1024 > > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' # > > [fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb > > ycbcr:601 quantization:lim-range] > > # ^^^ note we are itu601 again b/c that's the only format the IC can ouput > > # stream JPEG/RTP/UDP > > gst-launch-1.0 v4l2src device=/dev/video0 ! \ > > video/x-raw,format=UYVY ! \ > > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > > # ^^^ works... SW JPEG can handle itu601 > > Ok. > > > # stream upscale via mem2mem then JPEG/RTP/UDP > > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > > v4l2video8convert ! video/x-raw,width=1280,height=720 ! \ > > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000 > > ERROR: from element > > /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device > > '/dev/video8' does not support 2:4:7:1 colorimetry > > # ^^^ fails because mem2mem doesn't support itu601 > > # stream H264/RTP/UDP > > gst-launch-1.0 v4l2src device=/dev/video4 ! \ > > v4l2h264enc output-io-mode=dmabuf-import ! \ > > rtph264pay ! udpsink host=172.24.20.19 port=5001 > > ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: > > Device '/dev/video9' does not support 2:4:7:1 colorimetry > > # ^^^ coda has same issue... can't del with itu601 > > Well, just to see things working, try hacking > imx_media_fill_default_mbus_fields() to set Rec. 709 encoding: > > --- a/drivers/staging/media/imx/imx-media-utils.c > +++ b/drivers/staging/media/imx/imx-media-utils.c > @@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct > v4l2_mbus_framefmt *tryfmt, > tryfmt->quantization = is_rgb ? > V4L2_QUANTIZATION_FULL_RANGE : > V4L2_QUANTIZATION_LIM_RANGE; > - tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601; > + tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709; > } > } > EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields); > > > But of course that's not technically correct because the encoding in > ipu-ic.c is BT.601. > > The *real* way to fix this would be to allow programmable encodings in > ipu-ic.c. But unfortunately the encodings are hardcoded (grep for > ic_csc_rgb2ycbcr in ipu-ic.c). > Ok, I saw that you went ahead and worked on this (thanks!) and that you have a bt.709.v2 branch... is that one ready for testing? > The other option would be to change ic_csc_rgb2ycbcr to use the Rec. 709 > coefficients, and then the above patch is no longer a hack. The inverse > encoding (ic_csc_ycbcr2rgb) would also need to be changed. > > > > > > Am I perhaps missing a capsfilter to get the mem2mem driver to convert > > the colorspace properly? If so, they the mem2mem driver could be used > > to correct the colorspace to get IC output to coda working. > > Well, first I don't think the mem2mem driver is using the correct > encoding. The mem2mem driver is making use of the IC encoding so it > should be reporting and accepting only BT.601. > > Philipp ^^ > > > > > > Also can we connect the mem2mem driver to the unused VDIC input in the > > media controller so that we can use the VDIC to de-interlace content > > captured from non IMX sources (ie PCI or USB capture devices)? > > Exactly! That's something I have been working on. But it's difficult to > connect mem2mem to the unused VDIC IDMAC input pad because as of now the > v4l2 mem2mem internal API's do not allow connecting to *existing* > processing entities, and there are also issues with how sub-devices are > to deal with mem2mem contexts. > > I do have a WIP branch that creates a video output device that connects > to the VDIC IDMAC input pad, which doesn't have the above issues. The > only drawback with that is how gstreamer can make use of such an output > device. > ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch? I also noticed you have a add-fim-to-prpencvf branch... are you working on adding FIM to ipu_ic_prp/enc still? That would be nice to have to deal with sync loss/regain in analog decoders going through VDIC de-interlacing. > <snip> > >> > > This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc) > > still baffles me a bit but I've also found that any bt656 capture that > > isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of > > the resolution checks in ipu_csi_init_interface() resulting in > > 'Unsupported interlaced video mode'. I'm not sure if > > ipu_csi_set_bt_interlaced_codes() can be modified to support other > > resolutions? > > Well, Bt.656 only defines standard definition NTSC and PAL. > That is true. Do you know of any other sensors that use higher resolutions with BT656 SAV/EAV encoding? The BT656 mode does work well for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6 pixel clock and can't be used). I could dig into the bt656 spec to try and understand the various codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I suppose but I'm still not sure I want to push up a device-tree config that describes the tda1997x to CSI connection as 8bit BT656 as I would prefer to describe it as 16bit YUV instead (as I may be close to getting that format working well if we can get the IC able to output rec709). <snip> > > > > # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 > > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' > > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > > # get framerate > > v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings > > DV timings: > > Active width: 1280 > > Active height: 720 > > Total width: 1650 > > Total height: 750 > > Frame format: progressive > > Polarities: +vsync +hsync > > Pixelclock: 74250000 Hz (60.00 frames per second) > > Horizontal frontporch: 110 > > Horizontal sync: 40 > > Horizontal backporch: 220 > > Vertical frontporch: 5 > > Vertical sync: 5 > > Vertical backporch: 20 > > Standards: CTA-861 > > CTA-861 VIC: 0 > > Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC > > > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" > > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' > > # capture device > > media-ctl -e 'ipu1_ic_prpenc capture' > > /dev/video0 > > v4l2-ctl --device /dev/video0 --get-fmt-video > > Format Video Capture: > > Width/Height : 640/360 > > Pixel Format : 'UYVY' (UYVY 4:2:2) > > Field : None > > Bytes per Line : 1280 > > Size Image : 460800 > > Colorspace : Rec. 709 > > Transfer Function : Rec. 709 > > YCbCr/HSV Encoding: ITU-R 601 > > Quantization : Limited Range > > Flags : > > > > # capture 1 frame > > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > > [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 > > > > Do you know what the failure is here? > > You are /2 downscaling in the CSI, but did not set the compose window at > the input pad, e.g: > > media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale > I don't think this is the issue. Note that I 'was' able to get the same pipeline with the div-by-2 downscale without compose working with 720p bt656 YUV. Here is the 720p 16bit YUV bus again with the compose: # 720p60Hz YUV 16bit YUV bus # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0 MODE1:sensor->mux->csi->ic_prp->ic_prpenc v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query BT timings set # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]" media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" media-ctl --get-v4l2 "'ipu1_csi0':0" [fmt:UYVY8_1X16/1280x720@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range crop.bounds:(0,0)/1280x720 crop:(0,0)/1280x720 compose.bounds:(0,0)/1280x720 compose:(0,0)/640x360] media-ctl --get-v4l2 "'ipu1_csi0':1" [fmt:UYVY8_1X16/640x360@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] media-ctl --get-v4l2 "'ipu1_ic_prp':1" [fmt:AYUV8_1X32/640x480@1/30 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] media-ctl --get-v4l2 "'ipu1_ic_prpenc':1" [fmt:AYUV8_1X32/640x360@1001/30000 field:none colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] media-ctl -e 'ipu1_ic_prpenc capture' /dev/video0 v4l2-ctl --device /dev/video0 --get-fmt-video Format Video Capture: Width/Height : 640/360 Pixel Format : 'UYVY' (UYVY 4:2:2) Field : None Bytes per Line : 1280 Size Image : 460800 Colorspace : SMPTE 170M Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Limited Range Flags : v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 [ 1279.017148] ipu1_ic_prpenc: pipeline start failed with -32 > > > > > > And can you explain why I can't colorspace convert the following CSI > > capture case?: > > > > # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' > > [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > > media-ctl --get-v4l2 '"ipu1_csi0":2' > > [fmt:UYVY8_1X16/1280x720@1/30 field:none > > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] > > ^^^ still itu601 > > > > I have found in my testing when dealing with a BT656 RGB colorspace > > input which I need to convert ot rec709 I have to set the 'input' pad > > of the csi which doesn't make sense and if I do this with a 16-bit RGB > > colorspace (above) it jacks up the fmt: > > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > > media-ctl --get-v4l2 '"ipu1_csi0":2' > > [fmt:UYVY8_2X8/1280x720@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > > ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults > > fmt to YUVY8_2X8 > > > > So there is something I'm still doing wrong to setup CSC. > > Sorry I guess I don't understand your question here. Correct the CSI > does not accept AYUV32 at its input pad. > What I was asking here is what is the correct way to configure for ITU601 to REC709 CSC at the CSI? I would expect to set the REC709 format on the output pad of the CSI but it seems to require me to set it on the input pad. Consider: # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi # set sensor output pad to sensor source format v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query # sensor format media-ctl --get-v4l2 '"tda19971 2-0048":0' [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] # reset all links media-ctl --reset # setup links media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" # configure pads media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]" media-ctl --get-v4l2 '"ipu1_csi0":2' [fmt:UYVY8_1X16/1280x720@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] ^^^ ITU601, but I want REC709 media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" media-ctl --get-v4l2 '"ipu1_csi0":2' [fmt:UYVY8_1X16/1280x720@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] ^^^ No invalid argument but its still ITU601 at CSI output... but what I've found is if I set it at the CSI input the ycbcr changes but the fmt gets jacked up (as expected) media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" [fmt:UYVY8_2X8/1280x720@1/30 field:none colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8. However, again I expect to have set the CSC on the output pad. If I do this on a BT656 sensor bus it works because UYVY8_2X8 is valid. Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-05 19:16 ` Tim Harvey @ 2019-02-05 23:52 ` Steve Longerbeam 2019-02-06 16:31 ` Tim Harvey 0 siblings, 1 reply; 19+ messages in thread From: Steve Longerbeam @ 2019-02-05 23:52 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media On 2/5/19 11:16 AM, Tim Harvey wrote: > On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com> wrote: > > <snip> >> The *real* way to fix this would be to allow programmable encodings in >> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for >> ic_csc_rgb2ycbcr in ipu-ic.c). >> > Ok, I saw that you went ahead and worked on this (thanks!) and that > you have a bt.709.v2 branch... is that one ready for testing? Yes. I tried to test it too, but there is some regression in captured images, it could be something in ov5640.c (I'm testing with the SabreSD and the OV5640), or something else recently added. The regression looks like a stride problem, I'm hoping it wasn't the recently introduced compose window changes to relax width alignment. Have you tested with those commits and a prpenc/vf pipeline? >> <snip> >>> Also can we connect the mem2mem driver to the unused VDIC input in the >>> media controller so that we can use the VDIC to de-interlace content >>> captured from non IMX sources (ie PCI or USB capture devices)? >> Exactly! That's something I have been working on. But it's difficult to >> connect mem2mem to the unused VDIC IDMAC input pad because as of now the >> v4l2 mem2mem internal API's do not allow connecting to *existing* >> processing entities, and there are also issues with how sub-devices are >> to deal with mem2mem contexts. >> >> I do have a WIP branch that creates a video output device that connects >> to the VDIC IDMAC input pad, which doesn't have the above issues. The >> only drawback with that is how gstreamer can make use of such an output >> device. >> > ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch? The output-vdic branch. It's almost ready to go, there is only some strange issue with low and medium motion-compensation modes (captured images show "snow"). > I also noticed you have a add-fim-to-prpencvf branch... are you > working on adding FIM to ipu_ic_prp/enc still? That would be nice to > have to deal with sync loss/regain in analog decoders going through > VDIC de-interlacing. I've been trying to get this working, I ran into some locking issues when enabling lock debug options. Still trying to find a solution. > <snip> >>> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc) >>> still baffles me a bit but I've also found that any bt656 capture that >>> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of >>> the resolution checks in ipu_csi_init_interface() resulting in >>> 'Unsupported interlaced video mode'. I'm not sure if >>> ipu_csi_set_bt_interlaced_codes() can be modified to support other >>> resolutions? >> Well, Bt.656 only defines standard definition NTSC and PAL. >> > That is true. Do you know of any other sensors that use higher > resolutions with BT656 SAV/EAV encoding? Nope all I have for testing is the NTSC/PAL ADV7180 on the SabreAuto. > The BT656 mode does work well > for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6 > pixel clock and can't be used). > > I could dig into the bt656 spec to try and understand the various > codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I > suppose but I'm still not sure I want to push up a device-tree config > that describes the tda1997x to CSI connection as 8bit BT656 as I would > prefer to describe it as 16bit YUV instead (as I may be close to > getting that format working well if we can get the IC able to output > rec709). > > <snip> > >>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >>> # get framerate >>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings >>> DV timings: >>> Active width: 1280 >>> Active height: 720 >>> Total width: 1650 >>> Total height: 750 >>> Frame format: progressive >>> Polarities: +vsync +hsync >>> Pixelclock: 74250000 Hz (60.00 frames per second) >>> Horizontal frontporch: 110 >>> Horizontal sync: 40 >>> Horizontal backporch: 220 >>> Vertical frontporch: 5 >>> Vertical sync: 5 >>> Vertical backporch: 20 >>> Standards: CTA-861 >>> CTA-861 VIC: 0 >>> Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC >>> >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' >>> # capture device >>> media-ctl -e 'ipu1_ic_prpenc capture' >>> /dev/video0 >>> v4l2-ctl --device /dev/video0 --get-fmt-video >>> Format Video Capture: >>> Width/Height : 640/360 >>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>> Field : None >>> Bytes per Line : 1280 >>> Size Image : 460800 >>> Colorspace : Rec. 709 >>> Transfer Function : Rec. 709 >>> YCbCr/HSV Encoding: ITU-R 601 >>> Quantization : Limited Range >>> Flags : >>> >>> # capture 1 frame >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 >>> [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 >>> >>> Do you know what the failure is here? >> You are /2 downscaling in the CSI, but did not set the compose window at >> the input pad, e.g: >> >> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale >> > I don't think this is the issue. Note that I 'was' able to get the > same pipeline with the div-by-2 downscale without compose working with > 720p bt656 YUV. > > Here is the 720p 16bit YUV bus again with the compose: > # 720p60Hz YUV 16bit YUV bus > # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0 > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > BT timings set > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]" > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > media-ctl --get-v4l2 "'ipu1_csi0':0" > [fmt:UYVY8_1X16/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > crop.bounds:(0,0)/1280x720 > crop:(0,0)/1280x720 > compose.bounds:(0,0)/1280x720 > compose:(0,0)/640x360] > media-ctl --get-v4l2 "'ipu1_csi0':1" > [fmt:UYVY8_1X16/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > media-ctl --get-v4l2 "'ipu1_ic_prp':1" > [fmt:AYUV8_1X32/640x480@1/30 field:none Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to default formats. > colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] > media-ctl --get-v4l2 "'ipu1_ic_prpenc':1" > [fmt:AYUV8_1X32/640x360@1001/30000 field:none > colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range] > media-ctl -e 'ipu1_ic_prpenc capture' > /dev/video0 > v4l2-ctl --device /dev/video0 --get-fmt-video > Format Video Capture: > Width/Height : 640/360 > Pixel Format : 'UYVY' (UYVY 4:2:2) > Field : None > Bytes per Line : 1280 > Size Image : 460800 > Colorspace : SMPTE 170M > Transfer Function : Rec. 709 > YCbCr/HSV Encoding: ITU-R 601 > Quantization : Limited Range > Flags : > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > [ 1279.017148] ipu1_ic_prpenc: pipeline start failed with -32 > >>> And can you explain why I can't colorspace convert the following CSI >>> capture case?: >>> >>> # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" >>> media-ctl --get-v4l2 '"ipu1_csi0":2' >>> [fmt:UYVY8_1X16/1280x720@1/30 field:none >>> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] >>> ^^^ still itu601 >>> >>> I have found in my testing when dealing with a BT656 RGB colorspace >>> input which I need to convert ot rec709 I have to set the 'input' pad >>> of the csi which doesn't make sense and if I do this with a 16-bit RGB >>> colorspace (above) it jacks up the fmt: >>> media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" >>> media-ctl --get-v4l2 '"ipu1_csi0":2' >>> [fmt:UYVY8_2X8/1280x720@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] >>> ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults >>> fmt to YUVY8_2X8 >>> >>> So there is something I'm still doing wrong to setup CSC. >> Sorry I guess I don't understand your question here. Correct the CSI >> does not accept AYUV32 at its input pad. >> > What I was asking here is what is the correct way to configure for > ITU601 to REC709 CSC at the CSI? I would expect to set the REC709 > format on the output pad of the CSI but it seems to require me to set > it on the input pad. Right, the driver only allows setting colorimetry at the sink pads, and propagates them to the source pads. So you'll need to set colorimetry params at the CSI sink pad. > Consider: > # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi > # set sensor output pad to sensor source format > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > # sensor format > media-ctl --get-v4l2 '"tda19971 2-0048":0' > [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] > # reset all links > media-ctl --reset > # setup links > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > # configure pads > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]" > media-ctl --get-v4l2 '"ipu1_csi0":2' > [fmt:UYVY8_1X16/1280x720@1/30 field:none > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] > ^^^ ITU601, but I want REC709 > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > media-ctl --get-v4l2 '"ipu1_csi0":2' > [fmt:UYVY8_1X16/1280x720@1/30 field:none > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] > ^^^ No invalid argument but its still ITU601 at CSI output... but what > I've found is if I set it at the CSI input the ycbcr changes but the > fmt gets jacked up (as expected) > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > [fmt:UYVY8_2X8/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI > doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8. Why doesn't the CSI accept UYVY8_1X16 at its sink pad? This doesn't work?: media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 colorspace:rec709 ycbcr:709]" Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-05 23:52 ` Steve Longerbeam @ 2019-02-06 16:31 ` Tim Harvey 2019-02-08 1:15 ` Tim Harvey 2019-02-08 1:48 ` Steve Longerbeam 0 siblings, 2 replies; 19+ messages in thread From: Tim Harvey @ 2019-02-06 16:31 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > > > On 2/5/19 11:16 AM, Tim Harvey wrote: > > On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com> wrote: > > > > <snip> > >> The *real* way to fix this would be to allow programmable encodings in > >> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for > >> ic_csc_rgb2ycbcr in ipu-ic.c). > >> > > Ok, I saw that you went ahead and worked on this (thanks!) and that > > you have a bt.709.v2 branch... is that one ready for testing? > > Yes. I tried to test it too, but there is some regression in captured > images, it could be something in ov5640.c (I'm testing with the SabreSD > and the OV5640), or something else recently added. The regression looks > like a stride problem, I'm hoping it wasn't the recently introduced > compose window changes to relax width alignment. Have you tested with > those commits and a prpenc/vf pipeline? I have been testing v4.20 with: media: imx: lift CSI and PRP ENC/VF width alignment restriction media: imx: set compose rectangle to mbus format media: imx: add capture compose rectangle imx: imx-media: register mem2mem device with media controller media: imx: add mem2mem device media: imx-csi: Skip first few frames from a BT.656 source media: imx.rst: Update doc to reflect fixes to interlaced capture media: imx: Allow interweave with top/bottom lines swapped media: imx-csi: Move crop/compose reset after filling default mbus fields media: imx: vdic: rely on VDIC for correct field order media: imx-csi: Allow skipping odd chroma rows for YVU420 media: imx: interweave and odd-chroma-row skip are incompatible media: imx-csi: Double crop height for alternate fields at sink media: imx: Fix field negotiation gpu: ipu-v3: Add planar support to interlaced scan gpu: ipu-csi: Swap fields according to input/output field types media: videodev2.h: Add more field helper macros media: imx: prpencvf: Stop upstream before disabling IDMA channel media: imx: csi: Stop upstream before disabling IDMA channel media: imx: csi: Disable CSI immediately after last EOF media: imx-csi: Input connections to CSI should be optional What kind of artifacts are you seeing? I noticed your 'media: imx: Add support for BT.709 encoding' series don't apply cleanly to 4.20 that I'm working with so perhaps its something else. I'll bump up to linux-media, apply your bt.709.v2 and see what I get. The BT.709 support is critical for me otherwise I can't use coda with any pipelines that go through the IC with the tda1997x HDMI decoder. > > >> <snip> > >>> Also can we connect the mem2mem driver to the unused VDIC input in the > >>> media controller so that we can use the VDIC to de-interlace content > >>> captured from non IMX sources (ie PCI or USB capture devices)? > >> Exactly! That's something I have been working on. But it's difficult to > >> connect mem2mem to the unused VDIC IDMAC input pad because as of now the > >> v4l2 mem2mem internal API's do not allow connecting to *existing* > >> processing entities, and there are also issues with how sub-devices are > >> to deal with mem2mem contexts. > >> > >> I do have a WIP branch that creates a video output device that connects > >> to the VDIC IDMAC input pad, which doesn't have the above issues. The > >> only drawback with that is how gstreamer can make use of such an output > >> device. > >> > > ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch? > > The output-vdic branch. It's almost ready to go, there is only some > strange issue with low and medium motion-compensation modes (captured > images show "snow"). > > > > I also noticed you have a add-fim-to-prpencvf branch... are you > > working on adding FIM to ipu_ic_prp/enc still? That would be nice to > > have to deal with sync loss/regain in analog decoders going through > > VDIC de-interlacing. > > I've been trying to get this working, I ran into some locking issues > when enabling lock debug options. Still trying to find a solution. > > > <snip> > >>> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc) > >>> still baffles me a bit but I've also found that any bt656 capture that > >>> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of > >>> the resolution checks in ipu_csi_init_interface() resulting in > >>> 'Unsupported interlaced video mode'. I'm not sure if > >>> ipu_csi_set_bt_interlaced_codes() can be modified to support other > >>> resolutions? > >> Well, Bt.656 only defines standard definition NTSC and PAL. > >> > > That is true. Do you know of any other sensors that use higher > > resolutions with BT656 SAV/EAV encoding? > > Nope all I have for testing is the NTSC/PAL ADV7180 on the SabreAuto. > > > The BT656 mode does work well > > for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6 > > pixel clock and can't be used). > > > > I could dig into the bt656 spec to try and understand the various > > codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I > > suppose but I'm still not sure I want to push up a device-tree config > > that describes the tda1997x to CSI connection as 8bit BT656 as I would > > prefer to describe it as 16bit YUV instead (as I may be close to > > getting that format working well if we can get the IC able to output > > rec709). > > > > <snip> > > > >>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 > >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc > >>> # set sensor output pad to sensor source format > >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > >>> # sensor format > >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' > >>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > >>> # get framerate > >>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings > >>> DV timings: > >>> Active width: 1280 > >>> Active height: 720 > >>> Total width: 1650 > >>> Total height: 750 > >>> Frame format: progressive > >>> Polarities: +vsync +hsync > >>> Pixelclock: 74250000 Hz (60.00 frames per second) > >>> Horizontal frontporch: 110 > >>> Horizontal sync: 40 > >>> Horizontal backporch: 220 > >>> Vertical frontporch: 5 > >>> Vertical sync: 5 > >>> Vertical backporch: 20 > >>> Standards: CTA-861 > >>> CTA-861 VIC: 0 > >>> Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC > >>> > >>> # reset all links > >>> media-ctl --reset > >>> # setup links > >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > >>> # configure pads > >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" > >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' > >>> # capture device > >>> media-ctl -e 'ipu1_ic_prpenc capture' > >>> /dev/video0 > >>> v4l2-ctl --device /dev/video0 --get-fmt-video > >>> Format Video Capture: > >>> Width/Height : 640/360 > >>> Pixel Format : 'UYVY' (UYVY 4:2:2) > >>> Field : None > >>> Bytes per Line : 1280 > >>> Size Image : 460800 > >>> Colorspace : Rec. 709 > >>> Transfer Function : Rec. 709 > >>> YCbCr/HSV Encoding: ITU-R 601 > >>> Quantization : Limited Range > >>> Flags : > >>> > >>> # capture 1 frame > >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > >>> [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 > >>> > >>> Do you know what the failure is here? > >> You are /2 downscaling in the CSI, but did not set the compose window at > >> the input pad, e.g: > >> > >> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale > >> > > I don't think this is the issue. Note that I 'was' able to get the > > same pipeline with the div-by-2 downscale without compose working with > > 720p bt656 YUV. > > > > Here is the 720p 16bit YUV bus again with the compose: > > # 720p60Hz YUV 16bit YUV bus > > # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0 > > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > BT timings set > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' > > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]" > > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > > media-ctl --get-v4l2 "'ipu1_csi0':0" > > [fmt:UYVY8_1X16/1280x720@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > > crop.bounds:(0,0)/1280x720 > > crop:(0,0)/1280x720 > > compose.bounds:(0,0)/1280x720 > > compose:(0,0)/640x360] > > media-ctl --get-v4l2 "'ipu1_csi0':1" > > [fmt:UYVY8_1X16/640x360@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > media-ctl --get-v4l2 "'ipu1_ic_prp':1" > > [fmt:AYUV8_1X32/640x480@1/30 field:none > > Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting > ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to > default formats. but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ] here's a sequence of set/get: root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" [fmt:UYVY8_2X8/640x360@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range crop.bounds:(0,0)/640x360 crop:(0,0)/640x360 compose.bounds:(0,0)/640x360 compose:(0,0)/640x360] ^^^ the compose setting defaults so I really don't think the next cmd does anything root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [compose:(0,0)/640x360]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" [fmt:UYVY8_2X8/640x360@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range crop.bounds:(0,0)/640x360 crop:(0,0)/640x360 compose.bounds:(0,0)/640x360 compose:(0,0)/640x360] ^^^ see, same as above (but I like adding the compose to pipeline configuration instructions for documentation purposes; makes it obvious they can select the window) root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" [fmt:AYUV8_1X32/1280x720@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] ^^^ but yes, this one isn't getting set properly to 640x360... and it looks like its at the input: root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" [fmt:AYUV8_1X32/1280x720@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] my understanding is if I set it on the output pad it will propagate back to the input pad. I'm also wondering why I don't get an error if it doesn't change. If I setup the input pad manually and continue on: root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" [fmt:AYUV8_1X32/640x360@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" [fmt:AYUV8_1X32/640x360@1/30 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360]" root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1" [fmt:AYUV8_1X32/640x360@1001/30000 field:none colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32 ^^^ still fails If I can get pipelines that go through the IC working with UYVY8_1X16 then I will use that for the bus format and ditch the UYVY8_2X16 bt656 which has the limitations of not supporting 1080p60 (not to mention the fact that anything other than SD resolutions are out of the bt656 as discussed), interlaced formats. > <snip> > > >> > > What I was asking here is what is the correct way to configure for > > ITU601 to REC709 CSC at the CSI? I would expect to set the REC709 > > format on the output pad of the CSI but it seems to require me to set > > it on the input pad. > > Right, the driver only allows setting colorimetry at the sink pads, and > propagates them to the source pads. So you'll need to set colorimetry > params at the CSI sink pad. oh gosh... I think I had that backwards! > > > Consider: > > # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi > > # set sensor output pad to sensor source format > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > # sensor format > > media-ctl --get-v4l2 '"tda19971 2-0048":0' > > [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] > > # reset all links > > media-ctl --reset > > # setup links > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" > > # configure pads > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]" > > media-ctl --get-v4l2 '"ipu1_csi0":2' > > [fmt:UYVY8_1X16/1280x720@1/30 field:none > > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] > > ^^^ ITU601, but I want REC709 > > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > > media-ctl --get-v4l2 '"ipu1_csi0":2' > > [fmt:UYVY8_1X16/1280x720@1/30 field:none So the above was wrong as I was trying to set it on the CSI 'output' pad not its 'input'. Is there no error reporting all the way back to let the user know this was an invalid choice? > > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] > > ^^^ No invalid argument but its still ITU601 at CSI output... but what > > I've found is if I set it at the CSI input the ycbcr changes but the > > fmt gets jacked up (as expected) > > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" > > [fmt:UYVY8_2X8/1280x720@1/30 field:none > > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] > > ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI > > doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8. > > Why doesn't the CSI accept UYVY8_1X16 at its sink pad? > > This doesn't work?: > > media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 colorspace:rec709 ycbcr:709]" > that works! I was trying to set AYUV32 at its input pad which it doesn't support. Again, do you know why errors don't propagate back to media-ctl? Most of my difficulties have been not realizing what was valid and understanding that I have to read back the format after I set it to determine failures. Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-06 16:31 ` Tim Harvey @ 2019-02-08 1:15 ` Tim Harvey 2019-02-08 1:53 ` Steve Longerbeam 2019-02-08 1:48 ` Steve Longerbeam 1 sibling, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-02-08 1:15 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Wed, Feb 6, 2019 at 8:31 AM Tim Harvey <tharvey@gateworks.com> wrote: > > On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > > > > > > > On 2/5/19 11:16 AM, Tim Harvey wrote: > > > On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com> wrote: > > > > > > <snip> > > >> The *real* way to fix this would be to allow programmable encodings in > > >> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for > > >> ic_csc_rgb2ycbcr in ipu-ic.c). > > >> > > > Ok, I saw that you went ahead and worked on this (thanks!) and that > > > you have a bt.709.v2 branch... is that one ready for testing? > > > > Yes. I tried to test it too, but there is some regression in captured > > images, it could be something in ov5640.c (I'm testing with the SabreSD > > and the OV5640), or something else recently added. The regression looks > > like a stride problem, I'm hoping it wasn't the recently introduced > > compose window changes to relax width alignment. Have you tested with > > those commits and a prpenc/vf pipeline? > > I have been testing v4.20 with: > media: imx: lift CSI and PRP ENC/VF width alignment restriction > media: imx: set compose rectangle to mbus format > media: imx: add capture compose rectangle > imx: imx-media: register mem2mem device with media controller > media: imx: add mem2mem device > media: imx-csi: Skip first few frames from a BT.656 source > media: imx.rst: Update doc to reflect fixes to interlaced capture > media: imx: Allow interweave with top/bottom lines swapped > media: imx-csi: Move crop/compose reset after filling default mbus fields > media: imx: vdic: rely on VDIC for correct field order > media: imx-csi: Allow skipping odd chroma rows for YVU420 > media: imx: interweave and odd-chroma-row skip are incompatible > media: imx-csi: Double crop height for alternate fields at sink > media: imx: Fix field negotiation > gpu: ipu-v3: Add planar support to interlaced scan > gpu: ipu-csi: Swap fields according to input/output field types > media: videodev2.h: Add more field helper macros > media: imx: prpencvf: Stop upstream before disabling IDMA channel > media: imx: csi: Stop upstream before disabling IDMA channel > media: imx: csi: Disable CSI immediately after last EOF > media: imx-csi: Input connections to CSI should be optional > > What kind of artifacts are you seeing? > > I noticed your 'media: imx: Add support for BT.709 encoding' series > don't apply cleanly to 4.20 that I'm working with so perhaps its > something else. I'll bump up to linux-media, apply your bt.709.v2 and > see what I get. The BT.709 support is critical for me otherwise I > can't use coda with any pipelines that go through the IC with the > tda1997x HDMI decoder. > Steve, I'm testing your bt.709.v2 branch and it works great! I can now request ycbcr=709 at the CSI input pad and rec709 propagates all the way through the rest of the pipeline and works with coda. I don't see any artifacts so I'm still curious what your seeing? I did notice it adds a regression in that RGB->RGB ends up erroring out with 'Unsupported YCbCr encoding' in ipu_ic_task_init. I'll respond to your patch with details and a fix. > > <snip> > > > > > >>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 > > >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc > > >>> # set sensor output pad to sensor source format > > >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > >>> # sensor format > > >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' > > >>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > > >>> # get framerate > > >>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings > > >>> DV timings: > > >>> Active width: 1280 > > >>> Active height: 720 > > >>> Total width: 1650 > > >>> Total height: 750 > > >>> Frame format: progressive > > >>> Polarities: +vsync +hsync > > >>> Pixelclock: 74250000 Hz (60.00 frames per second) > > >>> Horizontal frontporch: 110 > > >>> Horizontal sync: 40 > > >>> Horizontal backporch: 220 > > >>> Vertical frontporch: 5 > > >>> Vertical sync: 5 > > >>> Vertical backporch: 20 > > >>> Standards: CTA-861 > > >>> CTA-861 VIC: 0 > > >>> Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC > > >>> > > >>> # reset all links > > >>> media-ctl --reset > > >>> # setup links > > >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > > >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > > >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > > >>> # configure pads > > >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > > >>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" > > >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > > >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > > >>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' > > >>> # capture device > > >>> media-ctl -e 'ipu1_ic_prpenc capture' > > >>> /dev/video0 > > >>> v4l2-ctl --device /dev/video0 --get-fmt-video > > >>> Format Video Capture: > > >>> Width/Height : 640/360 > > >>> Pixel Format : 'UYVY' (UYVY 4:2:2) > > >>> Field : None > > >>> Bytes per Line : 1280 > > >>> Size Image : 460800 > > >>> Colorspace : Rec. 709 > > >>> Transfer Function : Rec. 709 > > >>> YCbCr/HSV Encoding: ITU-R 601 > > >>> Quantization : Limited Range > > >>> Flags : > > >>> > > >>> # capture 1 frame > > >>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 > > >>> [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 > > >>> > > >>> Do you know what the failure is here? > > >> You are /2 downscaling in the CSI, but did not set the compose window at > > >> the input pad, e.g: > > >> > > >> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale > > >> > > > I don't think this is the issue. Note that I 'was' able to get the > > > same pipeline with the div-by-2 downscale without compose working with > > > 720p bt656 YUV. > > > > > > Here is the 720p 16bit YUV bus again with the compose: > > > # 720p60Hz YUV 16bit YUV bus > > > # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0 > > > MODE1:sensor->mux->csi->ic_prp->ic_prpenc > > > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query > > > BT timings set > > > # sensor format > > > media-ctl --get-v4l2 '"tda19971 2-0048":0' > > > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > > > # reset all links > > > media-ctl --reset > > > # setup links > > > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" > > > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" > > > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" > > > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" > > > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" > > > # configure pads > > > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" > > > media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]" > > > media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]" > > > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" > > > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" > > > media-ctl --get-v4l2 "'ipu1_csi0':0" > > > [fmt:UYVY8_1X16/1280x720@1/30 field:none > > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > > > crop.bounds:(0,0)/1280x720 > > > crop:(0,0)/1280x720 > > > compose.bounds:(0,0)/1280x720 > > > compose:(0,0)/640x360] > > > media-ctl --get-v4l2 "'ipu1_csi0':1" > > > [fmt:UYVY8_1X16/640x360@1/30 field:none > > > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > > media-ctl --get-v4l2 "'ipu1_ic_prp':1" > > > [fmt:AYUV8_1X32/640x480@1/30 field:none > > > > Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting > > ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to > > default formats. > > but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ] > > here's a sequence of set/get: > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0 > [fmt:UYVY8_1X16/1280x720 field:none]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2 > [fmt:UYVY8_1X16/1280x720 field:none]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" > [fmt:UYVY8_2X8/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > crop.bounds:(0,0)/640x360 > crop:(0,0)/640x360 > compose.bounds:(0,0)/640x360 > compose:(0,0)/640x360] > ^^^ the compose setting defaults so I really don't think the next cmd > does anything > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 > [compose:(0,0)/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" > [fmt:UYVY8_2X8/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > crop.bounds:(0,0)/640x360 > crop:(0,0)/640x360 > compose.bounds:(0,0)/640x360 > compose:(0,0)/640x360] > ^^^ see, same as above (but I like adding the compose to pipeline > configuration instructions for documentation purposes; makes it > obvious they can select the window) > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" > [fmt:AYUV8_1X32/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > ^^^ but yes, this one isn't getting set properly to 640x360... and it > looks like its at the input: > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" > [fmt:AYUV8_1X32/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > > my understanding is if I set it on the output pad it will propagate > back to the input pad. I'm also wondering why I don't get an error if > it doesn't change. > > If I setup the input pad manually and continue on: > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" > [fmt:AYUV8_1X32/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" > [fmt:AYUV8_1X32/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1 > [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1" > [fmt:AYUV8_1X32/640x360@1001/30000 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap > --stream-to=x.raw --stream-count=1 > [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32 > ^^^ still fails > Ok there is definitely something wrong when using the IC with UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me like the ipu1_ic_prp isn't negotiating its format properly. You can't re-create this because you don't have any UYVY8_1X16 (passthrough) sensors right? Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-08 1:15 ` Tim Harvey @ 2019-02-08 1:53 ` Steve Longerbeam 2019-02-08 21:23 ` Tim Harvey 0 siblings, 1 reply; 19+ messages in thread From: Steve Longerbeam @ 2019-02-08 1:53 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media On 2/7/19 5:15 PM, Tim Harvey wrote: > On Wed, Feb 6, 2019 at 8:31 AM Tim Harvey <tharvey@gateworks.com> wrote: >> On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >>> >>> >>> On 2/5/19 11:16 AM, Tim Harvey wrote: >>>> On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com> wrote: >>>> >>>> <snip> >>>>> The *real* way to fix this would be to allow programmable encodings in >>>>> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for >>>>> ic_csc_rgb2ycbcr in ipu-ic.c). >>>>> >>>> Ok, I saw that you went ahead and worked on this (thanks!) and that >>>> you have a bt.709.v2 branch... is that one ready for testing? >>> Yes. I tried to test it too, but there is some regression in captured >>> images, it could be something in ov5640.c (I'm testing with the SabreSD >>> and the OV5640), or something else recently added. The regression looks >>> like a stride problem, I'm hoping it wasn't the recently introduced >>> compose window changes to relax width alignment. Have you tested with >>> those commits and a prpenc/vf pipeline? >> I have been testing v4.20 with: >> media: imx: lift CSI and PRP ENC/VF width alignment restriction >> media: imx: set compose rectangle to mbus format >> media: imx: add capture compose rectangle >> imx: imx-media: register mem2mem device with media controller >> media: imx: add mem2mem device >> media: imx-csi: Skip first few frames from a BT.656 source >> media: imx.rst: Update doc to reflect fixes to interlaced capture >> media: imx: Allow interweave with top/bottom lines swapped >> media: imx-csi: Move crop/compose reset after filling default mbus fields >> media: imx: vdic: rely on VDIC for correct field order >> media: imx-csi: Allow skipping odd chroma rows for YVU420 >> media: imx: interweave and odd-chroma-row skip are incompatible >> media: imx-csi: Double crop height for alternate fields at sink >> media: imx: Fix field negotiation >> gpu: ipu-v3: Add planar support to interlaced scan >> gpu: ipu-csi: Swap fields according to input/output field types >> media: videodev2.h: Add more field helper macros >> media: imx: prpencvf: Stop upstream before disabling IDMA channel >> media: imx: csi: Stop upstream before disabling IDMA channel >> media: imx: csi: Disable CSI immediately after last EOF >> media: imx-csi: Input connections to CSI should be optional >> >> What kind of artifacts are you seeing? >> >> I noticed your 'media: imx: Add support for BT.709 encoding' series >> don't apply cleanly to 4.20 that I'm working with so perhaps its >> something else. I'll bump up to linux-media, apply your bt.709.v2 and >> see what I get. The BT.709 support is critical for me otherwise I >> can't use coda with any pipelines that go through the IC with the >> tda1997x HDMI decoder. >> > Steve, > > I'm testing your bt.709.v2 branch and it works great! I can now > request ycbcr=709 at the CSI input pad and rec709 propagates all the > way through the rest of the pipeline and works with coda. I don't see > any artifacts so I'm still curious what your seeing? Our emails crossed, see my last reply. Great to hear rec709 is working for you. > > I did notice it adds a regression in that RGB->RGB ends up erroring > out with 'Unsupported YCbCr encoding' in ipu_ic_task_init. I'll > respond to your patch with details and a fix. Ok, I take a look before submitting v2. > > <snip> >>>>>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 >>>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>>>>> # set sensor output pad to sensor source format >>>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>>>>> # sensor format >>>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>>>>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >>>>>> # get framerate >>>>>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings >>>>>> DV timings: >>>>>> Active width: 1280 >>>>>> Active height: 720 >>>>>> Total width: 1650 >>>>>> Total height: 750 >>>>>> Frame format: progressive >>>>>> Polarities: +vsync +hsync >>>>>> Pixelclock: 74250000 Hz (60.00 frames per second) >>>>>> Horizontal frontporch: 110 >>>>>> Horizontal sync: 40 >>>>>> Horizontal backporch: 220 >>>>>> Vertical frontporch: 5 >>>>>> Vertical sync: 5 >>>>>> Vertical backporch: 20 >>>>>> Standards: CTA-861 >>>>>> CTA-861 VIC: 0 >>>>>> Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC >>>>>> >>>>>> # reset all links >>>>>> media-ctl --reset >>>>>> # setup links >>>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>>>>> # configure pads >>>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" >>>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" >>>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" >>>>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' >>>>>> # capture device >>>>>> media-ctl -e 'ipu1_ic_prpenc capture' >>>>>> /dev/video0 >>>>>> v4l2-ctl --device /dev/video0 --get-fmt-video >>>>>> Format Video Capture: >>>>>> Width/Height : 640/360 >>>>>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>>>>> Field : None >>>>>> Bytes per Line : 1280 >>>>>> Size Image : 460800 >>>>>> Colorspace : Rec. 709 >>>>>> Transfer Function : Rec. 709 >>>>>> YCbCr/HSV Encoding: ITU-R 601 >>>>>> Quantization : Limited Range >>>>>> Flags : >>>>>> >>>>>> # capture 1 frame >>>>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 >>>>>> [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 >>>>>> >>>>>> Do you know what the failure is here? >>>>> You are /2 downscaling in the CSI, but did not set the compose window at >>>>> the input pad, e.g: >>>>> >>>>> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale >>>>> >>>> I don't think this is the issue. Note that I 'was' able to get the >>>> same pipeline with the div-by-2 downscale without compose working with >>>> 720p bt656 YUV. >>>> >>>> Here is the 720p 16bit YUV bus again with the compose: >>>> # 720p60Hz YUV 16bit YUV bus >>>> # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0 >>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>>> BT timings set >>>> # sensor format >>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >>>> # reset all links >>>> media-ctl --reset >>>> # setup links >>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>>> # configure pads >>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>>> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>>> media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]" >>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" >>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" >>>> media-ctl --get-v4l2 "'ipu1_csi0':0" >>>> [fmt:UYVY8_1X16/1280x720@1/30 field:none >>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range >>>> crop.bounds:(0,0)/1280x720 >>>> crop:(0,0)/1280x720 >>>> compose.bounds:(0,0)/1280x720 >>>> compose:(0,0)/640x360] >>>> media-ctl --get-v4l2 "'ipu1_csi0':1" >>>> [fmt:UYVY8_1X16/640x360@1/30 field:none >>>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >>>> media-ctl --get-v4l2 "'ipu1_ic_prp':1" >>>> [fmt:AYUV8_1X32/640x480@1/30 field:none >>> Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting >>> ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to >>> default formats. >> but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ] >> >> here's a sequence of set/get: >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0 >> [fmt:UYVY8_1X16/1280x720 field:none]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" >> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2 >> [fmt:UYVY8_1X16/1280x720 field:none]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" >> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" >> [fmt:UYVY8_2X8/640x360@1/30 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range >> crop.bounds:(0,0)/640x360 >> crop:(0,0)/640x360 >> compose.bounds:(0,0)/640x360 >> compose:(0,0)/640x360] >> ^^^ the compose setting defaults so I really don't think the next cmd >> does anything >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 >> [compose:(0,0)/640x360]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" >> [fmt:UYVY8_2X8/640x360@1/30 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range >> crop.bounds:(0,0)/640x360 >> crop:(0,0)/640x360 >> compose.bounds:(0,0)/640x360 >> compose:(0,0)/640x360] >> ^^^ see, same as above (but I like adding the compose to pipeline >> configuration instructions for documentation purposes; makes it >> obvious they can select the window) >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" >> [fmt:AYUV8_1X32/1280x720@1/30 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >> ^^^ but yes, this one isn't getting set properly to 640x360... and it >> looks like its at the input: >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" >> [fmt:AYUV8_1X32/1280x720@1/30 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >> >> my understanding is if I set it on the output pad it will propagate >> back to the input pad. I'm also wondering why I don't get an error if >> it doesn't change. >> >> If I setup the input pad manually and continue on: >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" >> [fmt:AYUV8_1X32/640x360@1/30 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" >> [fmt:AYUV8_1X32/640x360@1/30 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >> root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1 >> [fmt:AYUV32/640x360]" >> root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1" >> [fmt:AYUV8_1X32/640x360@1001/30000 field:none >> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >> root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap >> --stream-to=x.raw --stream-count=1 >> [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32 >> ^^^ still fails >> > Ok there is definitely something wrong when using the IC with > UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me > like the ipu1_ic_prp isn't negotiating its format properly. You can't > re-create this because you don't have any UYVY8_1X16 (passthrough) > sensors right? Sorry, maybe I didn't mention this, but passthrough cannot go though the IPU, you can only send passthrough pixels out the CSI directly to /dev/videoN interface (the ipu_csi:2 pad). Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-08 1:53 ` Steve Longerbeam @ 2019-02-08 21:23 ` Tim Harvey 2019-02-08 23:22 ` Steve Longerbeam 0 siblings, 1 reply; 19+ messages in thread From: Tim Harvey @ 2019-02-08 21:23 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Thu, Feb 7, 2019 at 5:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > <snip> > >> > > Ok there is definitely something wrong when using the IC with > > UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me > > like the ipu1_ic_prp isn't negotiating its format properly. You can't > > re-create this because you don't have any UYVY8_1X16 (passthrough) > > sensors right? > > Sorry, maybe I didn't mention this, but passthrough cannot go though the > IPU, you can only send passthrough pixels out the CSI directly to > /dev/videoN interface (the ipu_csi:2 pad). > crud... this has been my issue all along with that set of UYVY8_1X16 pipelines then. So this means the mem2mem driver also won't be able to handle 16bit pixel formats as well. So while I can downscale this by multiples of 2 (independent width/height), CSC convert it from srgb/bt.601 to yuv/bt.709, and even pixel reorder it within YUV via the ipu_csi entitty I'll never be able to deinterlace, scale with flexibility, flip/rotate or do a RGB->YUV CSC on it as those are all features of the IC path. I'm still struggling with what mbus format to configure the sensor<->csi interconnect in the device-tree. I was leaning towards 16bit but now that I realize that can't be used with the IPU I'm thinking the bt656 is more flexible (accept it has the limitation of not being able to do 1080p60 due to pixel clock and also can't handle interlaced due bt656 codes). If I end up wanting to switch between tda1997x sensor bus formats I'm not even sure the best way to deal with that (two dts I suppose and allowing user to select which one they use). Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-08 21:23 ` Tim Harvey @ 2019-02-08 23:22 ` Steve Longerbeam 2019-02-11 17:13 ` Tim Harvey 0 siblings, 1 reply; 19+ messages in thread From: Steve Longerbeam @ 2019-02-08 23:22 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media On 2/8/19 1:23 PM, Tim Harvey wrote: > On Thu, Feb 7, 2019 at 5:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >> > <snip> >>> Ok there is definitely something wrong when using the IC with >>> UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me >>> like the ipu1_ic_prp isn't negotiating its format properly. You can't >>> re-create this because you don't have any UYVY8_1X16 (passthrough) >>> sensors right? >> Sorry, maybe I didn't mention this, but passthrough cannot go though the >> IPU, you can only send passthrough pixels out the CSI directly to >> /dev/videoN interface (the ipu_csi:2 pad). >> > crud... this has been my issue all along with that set of UYVY8_1X16 > pipelines then. So this means the mem2mem driver also won't be able to > handle 16bit pixel formats as well. Ugh, let me rephrase. UYVY8_1X16 incoming on a parallel bus (not MIPI CSI-2) to the CSI cannot be sent through the IPU, those pixels must be sent directly to the ipu_csi:2 pad to /dev/videoN. At least according the the imx6 register manual. But that's not a limitation of the mem2mem driver because the pixels are coming from a memory buffer and not a 16-bi parallel bus via the CSI (and in any case mem2mem does not deal in media bus codes, it speaks V4L2_PIX_FMT_* which contains no bus info). So yes, you can receive UYVY8_1X16 on the CSI parallel bus, routed to ipu_csi:2 pad to /dev/videoN, and then pipe that to mem2mem v4l2convertN element in a gstreamer pipeline. > So while I can downscale this by > multiples of 2 (independent width/height), CSC convert it from > srgb/bt.601 to yuv/bt.709, and even pixel reorder it within YUV via > the ipu_csi entitty I'll never be able to deinterlace, that's true, currently you can't use the VDIC to do motion compensated de-interlacing, but you can still make use of the IDMAC interweave feature to deinterlace without motion compensation. > scale with > flexibility, flip/rotate or do a RGB->YUV CSC on it as those are all > features of the IC path. mem2mem v4l2convertN element will do those for you (flexible up/down scaling, flip/rotate, and CSC). Steve > > I'm still struggling with what mbus format to configure the > sensor<->csi interconnect in the device-tree. I was leaning towards > 16bit but now that I realize that can't be used with the IPU I'm > thinking the bt656 is more flexible (accept it has the limitation of > not being able to do 1080p60 due to pixel clock and also can't handle > interlaced due bt656 codes). If I end up wanting to switch between > tda1997x sensor bus formats I'm not even sure the best way to deal > with that (two dts I suppose and allowing user to select which one > they use). > > Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-08 23:22 ` Steve Longerbeam @ 2019-02-11 17:13 ` Tim Harvey 0 siblings, 0 replies; 19+ messages in thread From: Tim Harvey @ 2019-02-11 17:13 UTC (permalink / raw) To: Steve Longerbeam; +Cc: Philipp Zabel, linux-media On Fri, Feb 8, 2019 at 3:22 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > > > > On 2/8/19 1:23 PM, Tim Harvey wrote: > > On Thu, Feb 7, 2019 at 5:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: > >> > > <snip> > >>> Ok there is definitely something wrong when using the IC with > >>> UYVY8_1X16 (passthrough) which works with UYVY8_2X8. It looks to me > >>> like the ipu1_ic_prp isn't negotiating its format properly. You can't > >>> re-create this because you don't have any UYVY8_1X16 (passthrough) > >>> sensors right? > >> Sorry, maybe I didn't mention this, but passthrough cannot go though the > >> IPU, you can only send passthrough pixels out the CSI directly to > >> /dev/videoN interface (the ipu_csi:2 pad). > >> > > crud... this has been my issue all along with that set of UYVY8_1X16 > > pipelines then. So this means the mem2mem driver also won't be able to > > handle 16bit pixel formats as well. > > Ugh, let me rephrase. UYVY8_1X16 incoming on a parallel bus (not MIPI > CSI-2) to the CSI cannot be sent through the IPU, those pixels must be > sent directly to the ipu_csi:2 pad to /dev/videoN. At least according > the the imx6 register manual. > > But that's not a limitation of the mem2mem driver because the pixels are > coming from a memory buffer and not a 16-bi parallel bus via the CSI > (and in any case mem2mem does not deal in media bus codes, it speaks > V4L2_PIX_FMT_* which contains no bus info). So yes, you can receive > UYVY8_1X16 on the CSI parallel bus, routed to ipu_csi:2 pad to > /dev/videoN, and then pipe that to mem2mem v4l2convertN element in a > gstreamer pipeline. > > > > > So while I can downscale this by > > multiples of 2 (independent width/height), CSC convert it from > > srgb/bt.601 to yuv/bt.709, and even pixel reorder it within YUV via > > the ipu_csi entitty I'll never be able to deinterlace, > > that's true, currently you can't use the VDIC to do motion compensated > de-interlacing, but you can still make use of the IDMAC interweave > feature to deinterlace without motion compensation. > > > scale with > > flexibility, flip/rotate or do a RGB->YUV CSC on it as those are all > > features of the IC path. > > mem2mem v4l2convertN element will do those for you (flexible up/down > scaling, flip/rotate, and CSC). > ah... ok right using mem2mem is precisely that 'mem to mem' and not direct CSI to IPU. Ok so i experimented with some pipelines and perhaps am setting them up wrong. I'll respond to the imx-media mem2mem patch with my observations and expiriments. Thanks, Tim ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: IMX CSI capture issues with tda1997x HDMI receiver 2019-02-06 16:31 ` Tim Harvey 2019-02-08 1:15 ` Tim Harvey @ 2019-02-08 1:48 ` Steve Longerbeam 1 sibling, 0 replies; 19+ messages in thread From: Steve Longerbeam @ 2019-02-08 1:48 UTC (permalink / raw) To: Tim Harvey; +Cc: Philipp Zabel, linux-media Hi Tim, On 2/6/19 8:31 AM, Tim Harvey wrote: > On Tue, Feb 5, 2019 at 3:54 PM Steve Longerbeam <slongerbeam@gmail.com> wrote: >> >> >> On 2/5/19 11:16 AM, Tim Harvey wrote: >>> On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam<slongerbeam@gmail.com> wrote: >>> >>> <snip> >>>> The *real* way to fix this would be to allow programmable encodings in >>>> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for >>>> ic_csc_rgb2ycbcr in ipu-ic.c). >>>> >>> Ok, I saw that you went ahead and worked on this (thanks!) and that >>> you have a bt.709.v2 branch... is that one ready for testing? >> Yes. I tried to test it too, but there is some regression in captured >> images, it could be something in ov5640.c (I'm testing with the SabreSD >> and the OV5640), or something else recently added. The regression looks >> like a stride problem, I'm hoping it wasn't the recently introduced >> compose window changes to relax width alignment. Have you tested with >> those commits and a prpenc/vf pipeline? > I have been testing v4.20 with: > media: imx: lift CSI and PRP ENC/VF width alignment restriction > media: imx: set compose rectangle to mbus format > media: imx: add capture compose rectangle > imx: imx-media: register mem2mem device with media controller > media: imx: add mem2mem device > media: imx-csi: Skip first few frames from a BT.656 source > media: imx.rst: Update doc to reflect fixes to interlaced capture > media: imx: Allow interweave with top/bottom lines swapped > media: imx-csi: Move crop/compose reset after filling default mbus fields > media: imx: vdic: rely on VDIC for correct field order > media: imx-csi: Allow skipping odd chroma rows for YVU420 > media: imx: interweave and odd-chroma-row skip are incompatible > media: imx-csi: Double crop height for alternate fields at sink > media: imx: Fix field negotiation > gpu: ipu-v3: Add planar support to interlaced scan > gpu: ipu-csi: Swap fields according to input/output field types > media: videodev2.h: Add more field helper macros > media: imx: prpencvf: Stop upstream before disabling IDMA channel > media: imx: csi: Stop upstream before disabling IDMA channel > media: imx: csi: Disable CSI immediately after last EOF > media: imx-csi: Input connections to CSI should be optional > > What kind of artifacts are you seeing? It was the wrong-stride kind of artifact ("striations" ?? :) As it turns out the regression was indeed related to the capture compose window changes, I submitted a patch that fixes it ("media: imx: Set capture compose rectangle in capture_device_set_format"). > > I noticed your 'media: imx: Add support for BT.709 encoding' series > don't apply cleanly to 4.20 that I'm working with so perhaps its > something else. I'll bump up to linux-media, apply your bt.709.v2 and > see what I get. The BT.709 support is critical for me otherwise I > can't use coda with any pipelines that go through the IC with the > tda1997x HDMI decoder. With the regression fix above, I was able to verify the BT.709 encode and inverse encode matrices produce good image colors, with the bt.709.v2 branch. So I will submit this today. >>>> <snip> >>>>> Also can we connect the mem2mem driver to the unused VDIC input in the >>>>> media controller so that we can use the VDIC to de-interlace content >>>>> captured from non IMX sources (ie PCI or USB capture devices)? >>>> Exactly! That's something I have been working on. But it's difficult to >>>> connect mem2mem to the unused VDIC IDMAC input pad because as of now the >>>> v4l2 mem2mem internal API's do not allow connecting to *existing* >>>> processing entities, and there are also issues with how sub-devices are >>>> to deal with mem2mem contexts. >>>> >>>> I do have a WIP branch that creates a video output device that connects >>>> to the VDIC IDMAC input pad, which doesn't have the above issues. The >>>> only drawback with that is how gstreamer can make use of such an output >>>> device. >>>> >>> ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch? >> The output-vdic branch. It's almost ready to go, there is only some >> strange issue with low and medium motion-compensation modes (captured >> images show "snow"). >> >> >>> I also noticed you have a add-fim-to-prpencvf branch... are you >>> working on adding FIM to ipu_ic_prp/enc still? That would be nice to >>> have to deal with sync loss/regain in analog decoders going through >>> VDIC de-interlacing. >> I've been trying to get this working, I ran into some locking issues >> when enabling lock debug options. Still trying to find a solution. >> >>> <snip> >>>>> This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc) >>>>> still baffles me a bit but I've also found that any bt656 capture that >>>>> isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of >>>>> the resolution checks in ipu_csi_init_interface() resulting in >>>>> 'Unsupported interlaced video mode'. I'm not sure if >>>>> ipu_csi_set_bt_interlaced_codes() can be modified to support other >>>>> resolutions? >>>> Well, Bt.656 only defines standard definition NTSC and PAL. >>>> >>> That is true. Do you know of any other sensors that use higher >>> resolutions with BT656 SAV/EAV encoding? >> Nope all I have for testing is the NTSC/PAL ADV7180 on the SabreAuto. >> >>> The BT656 mode does work well >>> for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6 >>> pixel clock and can't be used). >>> >>> I could dig into the bt656 spec to try and understand the various >>> codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I >>> suppose but I'm still not sure I want to push up a device-tree config >>> that describes the tda1997x to CSI connection as 8bit BT656 as I would >>> prefer to describe it as 16bit YUV instead (as I may be close to >>> getting that format working well if we can get the IC able to output >>> rec709). >>> >>> <snip> >>> >>>>> # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0 >>>>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>>>> # set sensor output pad to sensor source format >>>>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>>>> # sensor format >>>>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>>>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >>>>> # get framerate >>>>> v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings >>>>> DV timings: >>>>> Active width: 1280 >>>>> Active height: 720 >>>>> Total width: 1650 >>>>> Total height: 750 >>>>> Frame format: progressive >>>>> Polarities: +vsync +hsync >>>>> Pixelclock: 74250000 Hz (60.00 frames per second) >>>>> Horizontal frontporch: 110 >>>>> Horizontal sync: 40 >>>>> Horizontal backporch: 220 >>>>> Vertical frontporch: 5 >>>>> Vertical sync: 5 >>>>> Vertical backporch: 20 >>>>> Standards: CTA-861 >>>>> CTA-861 VIC: 0 >>>>> Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC >>>>> >>>>> # reset all links >>>>> media-ctl --reset >>>>> # setup links >>>>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>>>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>>>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>>>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>>>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>>>> # configure pads >>>>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>>>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>>>> media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]" >>>>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" >>>>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" >>>>> media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' >>>>> # capture device >>>>> media-ctl -e 'ipu1_ic_prpenc capture' >>>>> /dev/video0 >>>>> v4l2-ctl --device /dev/video0 --get-fmt-video >>>>> Format Video Capture: >>>>> Width/Height : 640/360 >>>>> Pixel Format : 'UYVY' (UYVY 4:2:2) >>>>> Field : None >>>>> Bytes per Line : 1280 >>>>> Size Image : 460800 >>>>> Colorspace : Rec. 709 >>>>> Transfer Function : Rec. 709 >>>>> YCbCr/HSV Encoding: ITU-R 601 >>>>> Quantization : Limited Range >>>>> Flags : >>>>> >>>>> # capture 1 frame >>>>> v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1 >>>>> [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32 >>>>> >>>>> Do you know what the failure is here? >>>> You are /2 downscaling in the CSI, but did not set the compose window at >>>> the input pad, e.g: >>>> >>>> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale >>>> >>> I don't think this is the issue. Note that I 'was' able to get the >>> same pipeline with the div-by-2 downscale without compose working with >>> 720p bt656 YUV. >>> >>> Here is the 720p 16bit YUV bus again with the compose: >>> # 720p60Hz YUV 16bit YUV bus >>> # imx6q-gw54xx tda19971 2-0048 IPU1_CSI0 >>> MODE1:sensor->mux->csi->ic_prp->ic_prpenc >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> BT timings set >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" >>> media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" >>> media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]" >>> media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]" >>> media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]" >>> media-ctl --get-v4l2 "'ipu1_csi0':0" >>> [fmt:UYVY8_1X16/1280x720@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range >>> crop.bounds:(0,0)/1280x720 >>> crop:(0,0)/1280x720 >>> compose.bounds:(0,0)/1280x720 >>> compose:(0,0)/640x360] >>> media-ctl --get-v4l2 "'ipu1_csi0':1" >>> [fmt:UYVY8_1X16/640x360@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] >>> media-ctl --get-v4l2 "'ipu1_ic_prp':1" >>> [fmt:AYUV8_1X32/640x480@1/30 field:none >> Ah, ipu1_ic_prp:1 pad is 640x480, it looks like you missed setting >> ipu1_csi0:1 pad format, so that ipu1_ic_prp pads are still set to >> default formats. > but I do set it above: media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ] > > here's a sequence of set/get: > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'tda19971 2-0048':0 > [fmt:UYVY8_1X16/1280x720 field:none]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'tda19971 2-0048':0" > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0_mux':2 > [fmt:UYVY8_1X16/1280x720 field:none]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0_mux':2" > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709] > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" > [fmt:UYVY8_2X8/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > crop.bounds:(0,0)/640x360 > crop:(0,0)/640x360 > compose.bounds:(0,0)/640x360 > compose:(0,0)/640x360] > ^^^ the compose setting defaults so I really don't think the next cmd > does anything > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_csi0':0 > [compose:(0,0)/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_csi0':0" > [fmt:UYVY8_2X8/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range > crop.bounds:(0,0)/640x360 > crop:(0,0)/640x360 > compose.bounds:(0,0)/640x360 > compose:(0,0)/640x360] > ^^^ see, same as above (but I like adding the compose to pipeline > configuration instructions for documentation purposes; makes it > obvious they can select the window) > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':1 [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" > [fmt:AYUV8_1X32/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > ^^^ but yes, this one isn't getting set properly to 640x360... and it > looks like its at the input: > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" > [fmt:AYUV8_1X32/1280x720@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] No no -- the ipu_ic_prp entity cannot scale, the ipu_ic_prpenc and ipu_ic_prpvf scale. Think of ipu_ic_prp as a mux, taking input from either CSI0, CSI1, or the VDIC. It can then route that data to PRPENC, PRPVF, or both simultaneously. > > my understanding is if I set it on the output pad it will propagate > back to the input pad. I'm also wondering why I don't get an error if > it doesn't change. Nope, the other way around. The subdev drivers propagate formats from their sink pads to their source pads. And you can rely on media-ctl to propagate formats from a subdev's source pad to the next linked subdev's sink pad. In other words, media-ctl does inter-entity format propagation (source->sink), and the kernel media driver does intra-entity propagation (sink->source). > > If I setup the input pad manually and continue on: > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prp':0 [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':0" > [fmt:AYUV8_1X32/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prp':1" > [fmt:AYUV8_1X32/640x360@1/30 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# media-ctl --set-v4l2 "'ipu1_ic_prpenc':1 > [fmt:AYUV32/640x360]" > root@imx6q-gw5404:~# media-ctl --get-v4l2 "'ipu1_ic_prpenc':1" > [fmt:AYUV8_1X32/640x360@1001/30000 field:none > colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range] > root@imx6q-gw5404:~# v4l2-ctl --device /dev/video0 --stream-mmap > --stream-to=x.raw --stream-count=1 > [ 4695.356802] ipu1_ic_prpenc: pipeline start failed with -32 > ^^^ still fails Again ipu1_ic_prp can't scale. You need to set 640x360 at the ipu1_ic_prpenc:1 pad. > > If I can get pipelines that go through the IC working with UYVY8_1X16 > then I will use that for the bus format and ditch the UYVY8_2X16 bt656 > which has the limitations of not supporting 1080p60 (not to mention > the fact that anything other than SD resolutions are out of the bt656 > as discussed), interlaced formats. > > <snip> >>> What I was asking here is what is the correct way to configure for >>> ITU601 to REC709 CSC at the CSI? I would expect to set the REC709 >>> format on the output pad of the CSI but it seems to require me to set >>> it on the input pad. >> Right, the driver only allows setting colorimetry at the sink pads, and >> propagates them to the source pads. So you'll need to set colorimetry >> params at the CSI sink pad. > oh gosh... I think I had that backwards! Yep :) I admit and agree a lot of this is pretty cryptic :-/ But it does actually make sense that imx-media subdevs don't allow different colorspace parameters between their sink and source pads, because they _cannot_ modify colorspace. There's no support for say, transcoding the YCbCr encoding from BT.601 at a sink to BT.709 at the source, or quantization, or any of the other colorimetry parameters. So it is correct behavior to accept userspace-requested colorspace params at the sink pad, and force/propagate the colospace params from sink pad to source pad. So what this all means is that, the easiest way to set colorspace all throughout the pipeline, is to set it at the external sensor source pad (the tda19971 in your case). Media-ctl will then take care to propagate that colorimetry to the sink pad of the next linked subdevice, and imx-media will take care to propagate colorimetry inside each subdevice from sink to source. > >>> Consider: >>> # imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi >>> # set sensor output pad to sensor source format >>> v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query >>> # sensor format >>> media-ctl --get-v4l2 '"tda19971 2-0048":0' >>> [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb] >>> # reset all links >>> media-ctl --reset >>> # setup links >>> media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]" >>> media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" >>> media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" >>> # configure pads >>> media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]" >>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]" >>> media-ctl --get-v4l2 '"ipu1_csi0":2' >>> [fmt:UYVY8_1X16/1280x720@1/30 field:none >>> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] >>> ^^^ ITU601, but I want REC709 >>> media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" >>> media-ctl --get-v4l2 '"ipu1_csi0":2' >>> [fmt:UYVY8_1X16/1280x720@1/30 field:none > So the above was wrong as I was trying to set it on the CSI 'output' > pad not its 'input'. Is there no error reporting all the way back to > let the user know this was an invalid choice? Nope :-/ > >>> colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range] >>> ^^^ No invalid argument but its still ITU601 at CSI output... but what >>> I've found is if I set it at the CSI input the ycbcr changes but the >>> fmt gets jacked up (as expected) >>> media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]" >>> [fmt:UYVY8_2X8/1280x720@1/30 field:none >>> colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range] >>> ^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI >>> doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8. >> Why doesn't the CSI accept UYVY8_1X16 at its sink pad? >> >> This doesn't work?: >> >> media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 colorspace:rec709 ycbcr:709]" >> > that works! > > I was trying to set AYUV32 at its input pad which it doesn't support. Right. Keep in mind also, that the i.MX IPU internally handles only ARGB32 or AYUV32 on an internal 32-bit bus. So the IPU internal pads (ipu_csi:1, ipu_csi:2, and all pads of ipu_vdic, ipu_prp, ipu_prpenc, and ipu_prpvf) only accept media bus codes ARGB8888_1X32 or AYUV8_1X32. But ipu_csi:0 is a pad external to the IPU, so it accepts other media bus codes like UYVY8_1X16. For example, say you want to setup a pipeline that converts pixels from a sensor that sends RGB565_2X8_LE, to YUV space. So you would need ipu_prpenc or ipu_prpvf to carry out RGB->YUV CSC. At the ipu_prpencvf:1 source pad, you must set the mbus code to AYUV8_1X32. The IDMAC can then carry out pixel reordering _within YUV space_, to produce many of the other YUV pixel formats like YUYV, YV12, 422P, etc. The IDMAC format is set at the capture interface with 'v4l2-ctl -dN --set-fmt-video=pixelformat=..." Here is such an example on the SabreSD with the OV5640, which is transmitting RGB565_2X8_LE and BT.709 YCbCr encoded, and ipu1_ic_prpenc color converts that to AYUV8_1X32, and finally at the capture interface /dev/video0, the format is set to YV12 (in fact, this is the pipeline I used to test the new BT.709 encoding matrix): 'ov5640 1-003c':0 [fmt:RGB565_2X8_LE/1024x768@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'imx6-mipi-csi2':0 [fmt:RGB565_2X8_LE/1024x768 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'imx6-mipi-csi2':2 [fmt:RGB565_2X8_LE/1024x768 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'ipu1_csi1':0 [fmt:RGB565_2X8_LE/1024x768@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range crop.bounds:(0,0)/1024x768 crop:(0,0)/1024x768 compose.bounds:(0,0)/1024x768 compose:(0,0)/1024x768] 'ipu1_csi1':1 [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'ipu1_ic_prp':0 [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'ipu1_ic_prp':1 [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'ipu1_ic_prpenc':0 [fmt:ARGB8888_1X32/1024x768@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] 'ipu1_ic_prpenc':1 [fmt:AYUV8_1X32/800x600@1/30 field:none colorspace:srgb xfer:srgb ycbcr:709 quantization:full-range] /dev/video0:0 Format Video Capture: Width/Height : 800/600 Pixel Format : 'YV12' Field : None Bytes per Line : 800 Size Image : 720000 Colorspace : sRGB Transfer Function : sRGB YCbCr/HSV Encoding: Rec. 709 Quantization : Full Range Flags : 30ad6792 > > Again, do you know why errors don't propagate back to media-ctl? Most > of my difficulties have been not realizing what was valid and > understanding that I have to read back the format after I set it to > determine failures. Yeah, like most media drivers, imx-media will force parameters to something supported when userland asks for something unsupported, without letting the user know that happened. I agree it can be difficult. Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2019-02-11 17:13 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-01-23 23:21 IMX CSI capture issues with tda1997x HDMI receiver Tim Harvey 2019-01-25 23:57 ` Steve Longerbeam 2019-01-27 22:36 ` Steve Longerbeam 2019-01-28 19:14 ` Tim Harvey 2019-01-28 23:05 ` Steve Longerbeam 2019-01-28 19:03 ` Tim Harvey 2019-01-28 23:04 ` Steve Longerbeam 2019-01-31 1:18 ` Tim Harvey 2019-02-02 19:10 ` Steve Longerbeam 2019-02-03 19:37 ` Steve Longerbeam 2019-02-05 19:16 ` Tim Harvey 2019-02-05 23:52 ` Steve Longerbeam 2019-02-06 16:31 ` Tim Harvey 2019-02-08 1:15 ` Tim Harvey 2019-02-08 1:53 ` Steve Longerbeam 2019-02-08 21:23 ` Tim Harvey 2019-02-08 23:22 ` Steve Longerbeam 2019-02-11 17:13 ` Tim Harvey 2019-02-08 1:48 ` Steve Longerbeam
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).