Coda: imx53 plays video with incorrect colors

Coda: imx53 plays video with incorrect colors

[Fabio Estevam]

Hi,

I am testing video playback on an imx53-qsb running 5.10.6, but I am
not getting the correct colors in the TVE output. This is the result:

https://www.dropbox.com/s/a4ifivpoi663dkd/mx53.mp4?dl=0

The original video is this one:
http://cdn.clipcanvas.com/sample/clipcanvas_14348_offline.mp4

I resized the TVE output to 1024x576 so that scaling is not needed for now.

The Gstreamer pipeline I am using is:

# gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux !
h264parse ! v4l2h264dec ! kmssink

Gstreamer version is 1.18.2.

Any ideas on how to get the video playback with the correct colors?

Thanks,

Fabio Estevam

Re: Coda: imx53 plays video with incorrect colors

[Fabio Estevam]

On Thu, Jan 14, 2021 at 9:20 AM Fabio Estevam <festevam@gmail.com> wrote:
>
> Hi,
>
> I am testing video playback on an imx53-qsb running 5.10.6, but I am
> not getting the correct colors in the TVE output. This is the result:
>
> https://www.dropbox.com/s/a4ifivpoi663dkd/mx53.mp4?dl=0
>
> The original video is this one:
> http://cdn.clipcanvas.com/sample/clipcanvas_14348_offline.mp4
>
> I resized the TVE output to 1024x576 so that scaling is not needed for now.
>
> The Gstreamer pipeline I am using is:
>
> # gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux !
> h264parse ! v4l2h264dec ! kmssink
>
> Gstreamer version is 1.18.2.
>
> Any ideas on how to get the video playback with the correct colors?

I forgot to mention that this incorrect color behavior is seen with
other video files too.

Actually, I was not able to see video playback with correct colors
using mainline kernel on i.MX53 yet.

Thanks

Re: Coda: imx53 plays video with incorrect colors

[Nicolas Dufresne]

Le jeudi 14 janvier 2021 à 12:36 -0300, Fabio Estevam a écrit :
> On Thu, Jan 14, 2021 at 9:20 AM Fabio Estevam <festevam@gmail.com> wrote:
> > 
> > Hi,
> > 
> > I am testing video playback on an imx53-qsb running 5.10.6, but I am
> > not getting the correct colors in the TVE output. This is the result:
> > 
> > https://www.dropbox.com/s/a4ifivpoi663dkd/mx53.mp4?dl=0
> > 
> > The original video is this one:
> > http://cdn.clipcanvas.com/sample/clipcanvas_14348_offline.mp4
> > 
> > I resized the TVE output to 1024x576 so that scaling is not needed for now.
> > 
> > The Gstreamer pipeline I am using is:
> > 
> > # gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux !
> > h264parse ! v4l2h264dec ! kmssink
> > 
> > Gstreamer version is 1.18.2.
> > 
> > Any ideas on how to get the video playback with the correct colors?
> 
> I forgot to mention that this incorrect color behavior is seen with
> other video files too.
> 
> Actually, I was not able to see video playback with correct colors
> using mainline kernel on i.MX53 yet.

Perhaps you would explain in detail what isn't color correct ? To debug this,
you probably want to inspect the caps and the colorimetry negotiated between
each element (use -v in gst-launch-1.0). It's quite possible that the decoder is
ignoring upstream colors and get badly hinted by the driver, or that kmssink is
pnot passing colorimetry to the driver.

> 
> Thanks

Re: Coda: imx53 plays video with incorrect colors

[Fabio Estevam]

Hi Nicolas,

On Sat, Jan 16, 2021 at 11:46 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:

> Perhaps you would explain in detail what isn't color correct ? To debug this,

Here is the movie showing the incorrect colors:

https://www.dropbox.com/s/a4ifivpoi663dkd/mx53.mp4?dl=0

The original video is this one:
http://cdn.clipcanvas.com/sample/clipcanvas_14348_offline.mp4

Please note that the color mismatch is not related to this specific
video sample.
It happens with all videos I have tried.

> you probably want to inspect the caps and the colorimetry negotiated between
> each element (use -v in gst-launch-1.0). It's quite possible that the decoder is
> ignoring upstream colors and get badly hinted by the driver, or that kmssink is
> pnot passing colorimetry to the driver.

Here is the -v output:

# gst-launch-1.0 -v filesrc location=/media/clip.mp4 ! qtdemux !
h264parse ! v4l2h264dec ! kmssink
Setting pipeline to PAUSED ...
[   15.113196] msm msm: [drm:adreno_request_fw] loaded
qcom/yamato_pm4.fw from new location
[   15.124377] msm msm: [drm:adreno_request_fw] loaded
qcom/yamato_pfp.fw from new location
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 1024
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 576
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps =
video/x-h264, stream-format=(string)avc, alignment=(string)au,
level=(string)3.1, profile=(string)main,
codec_data=(buffer)014d401fffe10
023674d401f967200800936028100000e100002bf203460016e40016e45ef7c1e1108a24001000468de3c80,
width=(int)1024, height=(int)576, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps =
video/x-h264, stream-format=(string)byte-stream, alignment=(string)au,
level=(string)3.1, profile=(string)main, width=(int)1024, height=(
int)576, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)25/1,
chroma-format=(string)4:2:0, bit-depth-luma=(uint)8,
bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:sink: caps =
video/x-h264, stream-format=(string)byte-stream, alignment=(string)au,
level=(string)3.1, profile=(string)main, width=(int)1024,
height=(int)576, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction)25/1, chroma-format=(string)4:2:0,
bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:src: caps =
video/x-raw, format=(string)I420, width=(int)1024, height=(int)576,
interlace-mode=(string)progressive, multiview-mode=(string)mono,
multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono,
pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jp
eg, colorimetry=(string)bt601, framerate=(fraction)25/1
/GstPipeline:pipeline0/GstKMSSink:kmssink0.GstPad:sink: caps =
video/x-raw, format=(string)I420, width=(int)1024, height=(int)576,
interlace-mode=(string)progressive, multiview-mode=(string)mono,
multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono,
pixel-aspect-ratio=(fraction)1/1,
chroma-site=(string)jpeg,colorimetry=(string)bt601,
framerate=(fraction)25/1
Pipeline is PREROLLED ...0 %)
Setting pipeline to PLAYING ...
New clock: GstSystemClock

Any adjustment I need to do in the Gstreamer pipeline to negotiate the
correct colors to the display?

Thanks

Re: Coda: imx53 plays video with incorrect colors

[Fabio Estevam]

Hi Philipp,

On Sun, Jan 17, 2021 at 10:18 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:

> At first sight there is nothing wrong from GStreamer happening. It picks i420, pass it to KMS and it comes out wrong, first suspect would be the display driver. Understand that yuv formats are often unused and untested as most display server / compositer uses rgb type of formats converting with GPU.
>

On i.MX53 I see this 'wrong color' behavior when playing videos into
TVE as well as parallel display.

Are you able to playback video successfully on i.MX53? If so, could
you please share your Gstreamer pipeline?

Thanks

Re: Coda: imx53 plays video with incorrect colors

[Philipp Zabel]

Hi Fabio,

On Mon, 2021-01-18 at 07:57 -0300, Fabio Estevam wrote:
> Hi Philipp,
> 
> On Sun, Jan 17, 2021 at 10:18 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> 
> > At first sight there is nothing wrong from GStreamer happening. It
> > picks i420, pass it to KMS and it comes out wrong, first suspect
> > would be the display driver. Understand that yuv formats are often
> > unused and untested as most display server / compositer uses rgb
> > type of formats converting with GPU.

This is both a bug and a missing feature in the imx-drm driver. The
driver doesn't support color space conversion on the VGA output and
doesn't report that correctly.

> On i.MX53 I see this 'wrong color' behavior when playing videos into
> TVE as well as parallel display.
>
> Are you able to playback video successfully on i.MX53? If so, could
> you please share your Gstreamer pipeline?

I can reproduce this with

  gst-launch-1.0 videotestsrc ! video/x-raw,format=I420 ! kmssink

and the same for NV12 or YUY2. We are missing color space conversion on
the VGA output.

The IPU only supports color space conversion on one output via the DP.
This path is hard-coded to DI0 in the driver. The direct DC path to DI1,
which TVE/VGA is connected to, does not support CSC.

The driver could be modified to switch the DP->DI0/DC->DI1 mapping
around to DP->DI1/DC->DI0 when required. As a simple test, you can
switch statically with:

----------8<----------
diff --git a/drivers/gpu/ipu-v3/ipu-common.c b/drivers/gpu/ipu-v3/ipu-common.c
index d166ee262ce4..f7c86ef0bed7 100644
--- a/drivers/gpu/ipu-v3/ipu-common.c
+++ b/drivers/gpu/ipu-v3/ipu-common.c
@@ -1118,18 +1118,18 @@ static struct ipu_platform_reg client_reg[] = {
 	}, {
 		.pdata = {
 			.di = 0,
-			.dc = 5,
-			.dp = IPU_DP_FLOW_SYNC_BG,
-			.dma[0] = IPUV3_CHANNEL_MEM_BG_SYNC,
+			.dc = 1,
+			.dp = -EINVAL,
+			.dma[0] = IPUV3_CHANNEL_MEM_DC_SYNC,
 			.dma[1] = IPUV3_CHANNEL_MEM_FG_SYNC,
 		},
 		.name = "imx-ipuv3-crtc",
 	}, {
 		.pdata = {
 			.di = 1,
-			.dc = 1,
-			.dp = -EINVAL,
-			.dma[0] = IPUV3_CHANNEL_MEM_DC_SYNC,
+			.dc = 5,
+			.dp = IPU_DP_FLOW_SYNC_BG,
+			.dma[0] = IPUV3_CHANNEL_MEM_BG_SYNC,
 			.dma[1] = -EINVAL,
 		},
 		.name = "imx-ipuv3-crtc",
diff --git a/drivers/gpu/ipu-v3/ipu-dc.c b/drivers/gpu/ipu-v3/ipu-dc.c
index 34b4075a6a8e..0a7f48e4aa37 100644
--- a/drivers/gpu/ipu-v3/ipu-dc.c
+++ b/drivers/gpu/ipu-v3/ipu-dc.c
@@ -364,9 +364,9 @@ int ipu_dc_init(struct ipu_soc *ipu, struct device *dev,
 
 	writel(DC_WR_CH_CONF_WORD_SIZE_24 | DC_WR_CH_CONF_DISP_ID_PARALLEL(1) |
 			DC_WR_CH_CONF_PROG_DI_ID,
-			priv->channels[1].base + DC_WR_CH_CONF);
-	writel(DC_WR_CH_CONF_WORD_SIZE_24 | DC_WR_CH_CONF_DISP_ID_PARALLEL(0),
 			priv->channels[5].base + DC_WR_CH_CONF);
+	writel(DC_WR_CH_CONF_WORD_SIZE_24 | DC_WR_CH_CONF_DISP_ID_PARALLEL(0),
+			priv->channels[1].base + DC_WR_CH_CONF);
 
 	writel(DC_GEN_SYNC_1_6_SYNC | DC_GEN_SYNC_PRIORITY_1,
 		priv->dc_reg + DC_GEN);
---------->8----------

That should enable CSC (and overlay plane) on DI1, and lose it on DI0.

Or, as a workaround, add a v4l2convert element and use the IC to convert
to BGRx between decoder and kmssink.

regards
Philipp

Re: Coda: imx53 plays video with incorrect colors

[Fabio Estevam]

Hi Philipp,

Thanks for your reply.

On Mon, Jan 18, 2021 at 9:40 AM Philipp Zabel <p.zabel@pengutronix.de> wrote:

> The driver could be modified to switch the DP->DI0/DC->DI1 mapping
> around to DP->DI1/DC->DI0 when required. As a simple test, you can
> switch statically with:

It does change the colors but still does not play the video with the
correct colors. Looks like it plays in black-and-white now.

> Or, as a workaround, add a v4l2convert element and use the IC to convert
> to BGRx between decoder and kmssink.

Yes, I have tried to do this, but it says that v4l2convert does not
support bt601 colorimetry, and then a segfault occurs:

# gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux !
h264parse ! v4l2h264dec ! v4l2convert ! video/x-raw,format=BGRx !
kmssink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/v4l2convert:v4l2convert0:
Device '/dev/video4' does not support bt601 colorimetry
Additional debug info:
../sys/v4l2/gstv4l2object.c(4032): gst_v4l2_object_set_format_full ():
/GstPipeline:pipeline0/v4l2convert:v4l2convert0:
Device wants 2:4:5:4 colorimetry
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Caught SIGSEGV
exec gdb failed: No such file or directory
Spinning.  Please run 'gdb gst-launch-1.0 217' to continue debugging,
Ctrl-C to quit, or Ctrl-\ to dump core.

Is the Gstreamer pipeline above correct?

Thanks

Re: Coda: imx53 plays video with incorrect colors

[Philipp Zabel]

Hi Fabio,

On Mon, 2021-01-18 at 10:28 -0300, Fabio Estevam wrote:
> Hi Philipp,
> 
> Thanks for your reply.
> 
> On Mon, Jan 18, 2021 at 9:40 AM Philipp Zabel <p.zabel@pengutronix.de> wrote:
> 
> > The driver could be modified to switch the DP->DI0/DC->DI1 mapping
> > around to DP->DI1/DC->DI0 when required. As a simple test, you can
> > switch statically with:
> 
> It does change the colors but still does not play the video with the
> correct colors. Looks like it plays in black-and-white now.

Please try forcing decoder output to NV12 instead of I420.

> > Or, as a workaround, add a v4l2convert element and use the IC to convert
> > to BGRx between decoder and kmssink.
> 
> Yes, I have tried to do this, but it says that v4l2convert does not
> support bt601 colorimetry, and then a segfault occurs:
> 
> # gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux !
> h264parse ! v4l2h264dec ! v4l2convert ! video/x-raw,format=BGRx !
> kmssink
> Setting pipeline to PAUSED ...
> Pipeline is PREROLLING ...
> ERROR: from element /GstPipeline:pipeline0/v4l2convert:v4l2convert0:
> Device '/dev/video4' does not support bt601 colorimetry
> Additional debug info:
> ../sys/v4l2/gstv4l2object.c(4032): gst_v4l2_object_set_format_full ():
> /GstPipeline:pipeline0/v4l2convert:v4l2convert0:
> Device wants 2:4:5:4 colorimetry
> ERROR: pipeline doesn't want to preroll.
> Setting pipeline to NULL ...
> Caught SIGSEGV
> exec gdb failed: No such file or directory
> Spinning.  Please run 'gdb gst-launch-1.0 217' to continue debugging,
> Ctrl-C to quit, or Ctrl-\ to dump core.
> 
> Is the Gstreamer pipeline above correct?

Yes. Please try if the following patch makes it work:

----------8<----------
From c45afcaf6fbef56a86dce19200c06df78718db60 Mon Sep 17 00:00:00 2001
From: Philipp Zabel <p.zabel@pengutronix.de>
Date: Mon, 18 Jan 2021 15:54:43 +0100
Subject: [PATCH] v4l2object: handle GST_VIDEO_TRANSFER_BT601

V4L2 makes no difference between the BT.601 and BT.709 transfer
functions [1], but GStreamer does since 1.18 [2].

Adapt gst_v4l2_object_get_colorspace() and
gst_v4l2_object_set_format_full().

[1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#colorspace-smpte-170m-v4l2-colorspace-smpte170m
[2] https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/merge_requests/724
---
 sys/v4l2/gstv4l2object.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sys/v4l2/gstv4l2object.c b/sys/v4l2/gstv4l2object.c
index ea4363e17303..b13216d75836 100644
--- a/sys/v4l2/gstv4l2object.c
+++ b/sys/v4l2/gstv4l2object.c
@@ -2334,7 +2334,7 @@ gst_v4l2_object_get_colorspace (struct v4l2_format *fmt,
     case V4L2_COLORSPACE_SMPTE170M:
       cinfo->range = GST_VIDEO_COLOR_RANGE_16_235;
       cinfo->matrix = GST_VIDEO_COLOR_MATRIX_BT601;
-      cinfo->transfer = GST_VIDEO_TRANSFER_BT709;
+      cinfo->transfer = GST_VIDEO_TRANSFER_BT601;
       cinfo->primaries = GST_VIDEO_COLOR_PRIMARIES_SMPTE170M;
       break;
     case V4L2_COLORSPACE_REC709:
@@ -2463,6 +2463,8 @@ gst_v4l2_object_get_colorspace (struct v4l2_format *fmt,
     case V4L2_XFER_FUNC_709:
       if (colorspace == V4L2_COLORSPACE_BT2020 && fmt->fmt.pix.height >= 2160)
         cinfo->transfer = GST_VIDEO_TRANSFER_BT2020_12;
+      else if (colorspace == V4L2_COLORSPACE_SMPTE170M)
+        cinfo->transfer = GST_VIDEO_TRANSFER_BT601;
       else
         cinfo->transfer = GST_VIDEO_TRANSFER_BT709;
       break;
@@ -3855,6 +3857,7 @@ gst_v4l2_object_set_format_full (GstV4l2Object * v4l2object, GstCaps * caps,
     case GST_VIDEO_TRANSFER_GAMMA10:
       transfer = V4L2_XFER_FUNC_NONE;
       break;
+    case GST_VIDEO_TRANSFER_BT601:
     case GST_VIDEO_TRANSFER_BT2020_12:
     case GST_VIDEO_TRANSFER_BT709:
       transfer = V4L2_XFER_FUNC_709;
-- 
2.20.1
---------->8----------

This may not be the correct solution. GStreamer could keep choosing
BT709 as told by V4L2, and use the new
gst_video_color_transfer_is_equivalent() function to test for
equivalence instead.

regards
Philipp

Re: Coda: imx53 plays video with incorrect colors

[Fabio Estevam]

Hi Philipp,

On Mon, Jan 18, 2021 at 12:29 PM Philipp Zabel <p.zabel@pengutronix.de> wrote:

> Please try forcing decoder output to NV12 instead of I420.

Your suggestion works, thank you. After applying your IPU patch and
using this Gstreamer pipeline:

gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux ! h264parse
! v4l2h264dec ! video/x-raw,format=NV12  !  kmssink

The video is played with correct colors.

Can we make the NV12 format be used automatically so that a simple
'gst-play-1.0 /media/clip.mp4' works?

> Yes. Please try if the following patch makes it work:

With this Gstreamer patch, I don't see the segfault anymore, but it
plays in black-and-white.

Thanks

Re: Coda: imx53 plays video with incorrect colors

[Philipp Zabel]

Hi Fabio,

On Mon, 2021-01-18 at 15:36 -0300, Fabio Estevam wrote:
> Hi Philipp,
> 
> On Mon, Jan 18, 2021 at 12:29 PM Philipp Zabel <p.zabel@pengutronix.de> wrote:
> 
> > Please try forcing decoder output to NV12 instead of I420.
> 
> Your suggestion works, thank you. After applying your IPU patch and
> using this Gstreamer pipeline:
> 
> gst-launch-1.0 filesrc location=/media/clip.mp4 ! qtdemux ! h264parse
> ! v4l2h264dec ! video/x-raw,format=NV12  !  kmssink
> 
> The video is played with correct colors.

Ok, thank you. It looks to me like the I420 issue is either a bug in
coda-dev, or an interaction issue between GStreamer and coda-dev, where
coda-dev doesn't let GStreamer change the format at a point where it
expects to be able to. The b/w image (with slight discolorations in a
regular pattern) could very well be actual NV12 interpreted as I420 by
GStreamer (and thus kmssink). I'll have closer look.

> Can we make the NV12 format be used automatically so that a simple
> 'gst-play-1.0 /media/clip.mp4' works?

That would be a matter of increasing the rank of the NV12 format
in gst_v4l2_object_format_get_rank() above that of I420 and YV12.

> > Yes. Please try if the following patch makes it work:
> 
> With this Gstreamer patch, I don't see the segfault anymore, but it
> plays in black-and-white.

Thanks for testing. I've opened

https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/856

to track this.

regards
Philipp