tc358743 - first rx'd frame is incomplete

tc358743 - first rx'd frame is incomplete

[Jakub Vaněk]

Hello,

I think I might have found a bug in the tc358743 driver.
The issue is that only the bottom part of the first captured
frame is received by userspace V4L2 applications. All frames
following the first one are received without this issue.

I'm observing this on a Raspberry Pi 4 w/ their 5.4.83 kernel.
To communicate with the chip, I'm using the unicam+tc358743
kernel drivers. The chip itself is on the Auvidea B101 board.

I can reproduce the problem by running the following commands
with the HDMI-CSI bridge running (1024x768@60Hz input):

# sync timings between HDMI and CSI
$ v4l2-ctl --device /dev/video0 \
           --set-dv-bt-timings query
# capture one raw frame without skipping the first one:
$ v4l2-ctl --device /dev/video0 \
           --stream-mmap=1 \
           --stream-to=frame.raw \
           --stream-count=1

Then, to convert from raw pixels to PNG, I would run one of these
ImageMagick commands (depending on the format - size of YUV frame
would be 2*w*h bytes, size of RGB 3*w*h bytes):

# from yuv to png (set WxH to HDMI input resolution):
$ convert -size WxH -depth 8 -colorspace YUV \
          -sampling-factor 4:2:2 yuv:frame.raw frame.png

# from rgb to png (set WxH to HDMI input resolution):
$ convert -size WxH -depth 8 -colorspace RGB rgb:frame.raw \
          -separate +channel -swap 0,2 -combine -gamma 2.2 frame.png

The result is that the remaining part of the frame is shifted upwards
and the bottom of the picture is green (YUV) or black (RGB).

In this case, I can work the issue around with --stream-skip=1.
However, it gets a bit more complicated when GStreamer is involved.

I have already created a ticket for the Raspberry kernel here:
https://github.com/raspberrypi/linux/issues/4058,
but given that tc358743 is a mainline driver, it would be better
to solve this issue in mainline first.

Can anyone reproduce this on other kernels or boards?
There is some chance that this problem is Raspberry-specific.

If it turns out that the issue happens on multiple boards,
a fix was suggested by Dave Stevenson - implementing
v4l2_subdev_sensor_ops's g_skip_frames() in the tc358743 driver.
This would signal to CSI2 RX drivers that the first frame is invalid
and needs to be skipped.

With best regards

Jakub

Re: tc358743 - first rx'd frame is incomplete

[Mats Randgaard (matrandg)]

Hi Jakub,
Great that you have found and documented this issue! The product this driver originally was written for skips 20 frames when it starts streaming, so this first half frame is completely hidden. My colleague said he doesn’t know why the number is so high, but remember there were issues when the video stream was started. This might be related to other parts of the video pipeline.
 
Regards,
Mats Randgaard
 
----------------------------------------------------------------

From: Jakub Vaněk <linuxtardis@gmail.com>
Date: Thursday, 21 January 2021 at 12:58
To: linux-media@vger.kernel.org <linux-media@vger.kernel.org>
Cc: Mats Randgaard (matrandg) <matrandg@cisco.com>, Dave Stevenson <dave.stevenson@raspberrypi.com>
Subject: tc358743 - first rx'd frame is incomplete
Hello,

I think I might have found a bug in the tc358743 driver.
The issue is that only the bottom part of the first captured
frame is received by userspace V4L2 applications. All frames
following the first one are received without this issue.

I'm observing this on a Raspberry Pi 4 w/ their 5.4.83 kernel.
To communicate with the chip, I'm using the unicam+tc358743
kernel drivers. The chip itself is on the Auvidea B101 board.

I can reproduce the problem by running the following commands
with the HDMI-CSI bridge running (1024x768@60Hz input):

# sync timings between HDMI and CSI
$ v4l2-ctl --device /dev/video0 \
           --set-dv-bt-timings query
# capture one raw frame without skipping the first one:
$ v4l2-ctl --device /dev/video0 \
           --stream-mmap=1 \
           --stream-to=frame.raw \
           --stream-count=1

Then, to convert from raw pixels to PNG, I would run one of these
ImageMagick commands (depending on the format - size of YUV frame
would be 2*w*h bytes, size of RGB 3*w*h bytes):

# from yuv to png (set WxH to HDMI input resolution):
$ convert -size WxH -depth 8 -colorspace YUV \
          -sampling-factor 4:2:2 yuv:frame.raw frame.png

# from rgb to png (set WxH to HDMI input resolution):
$ convert -size WxH -depth 8 -colorspace RGB rgb:frame.raw \
          -separate +channel -swap 0,2 -combine -gamma 2.2 frame.png

The result is that the remaining part of the frame is shifted upwards
and the bottom of the picture is green (YUV) or black (RGB).

In this case, I can work the issue around with --stream-skip=1.
However, it gets a bit more complicated when GStreamer is involved.

I have already created a ticket for the Raspberry kernel here:
https://github.com/raspberrypi/linux/issues/4058,
but given that tc358743 is a mainline driver, it would be better
to solve this issue in mainline first.

Can anyone reproduce this on other kernels or boards?
There is some chance that this problem is Raspberry-specific.

If it turns out that the issue happens on multiple boards,
a fix was suggested by Dave Stevenson - implementing
v4l2_subdev_sensor_ops's g_skip_frames() in the tc358743 driver.
This would signal to CSI2 RX drivers that the first frame is invalid
and needs to be skipped.

With best regards

Jakub