Video4Linux: Call an I2C subdev function to start a stream after a CSI2 driver has been started

Video4Linux: Call an I2C subdev function to start a stream after a CSI2 driver has been started

[Terry Barnaby]

Hi,

We are working on a system that uses a NXP IMX8MP SOC with a TP2855 
analogue video front end chip that can capture 1920x1080p25 or PAL 
720x576i25 analogue video streams which is generally working using 
gstreamer as the higher level software driving this.

We have an intermittent video stream start up problem which we believe 
is down to:

1. The TP2855 is initialised first and its subdev *_s_stream() call is 
called to start the video input.

2. The NXP CSI2 video input hardware is then started: 
imx8-mipi-csi2-sam.c: mipi_csis_s_stream().

3. The TP2855 enables the CSI2 clock after its *_s_stream() call, 
probably in hardware after its PLL's have locked, by sending a CSI2 
start sequence on the CSI2 clock pair. This clock is then a continuous 
clock ie. it does not go into low power mode during horizontal/vertical 
blanking.

4. The NXP CSI2 video input hardware, based on some Samsung IP, does not 
see the CSI2 clock unless it sees the CSI2 start sequence and depending 
on timings it may not see this at video pipeline startup and we get not 
video stream from the CSI2 hardware.

I was hoping the subdev *_s_stream() call would be after all of the 
hardware's pipeline was setup so I could instigate a CSI2 clock restart 
in the TP2855 driver, but unfortunately this is called before the CSI2 
hardware is setup.

I can add a one shot timer to do this in the TP2855 subdev *_s_stream() 
call, but obviously this is not ideal. Is there anyway to get a subdev 
function called in the video4linux API automatically once all of the 
video streams hardware is setup ?

Any ideas/recommendations ?

Terry

Re: Video4Linux: Call an I2C subdev function to start a stream after a CSI2 driver has been started

[Kieran Bingham]

Hi Terry,

Quoting Terry Barnaby (2022-05-10 14:35:38)
> Hi,
> 
> We are working on a system that uses a NXP IMX8MP SOC with a TP2855 
> analogue video front end chip that can capture 1920x1080p25 or PAL 
> 720x576i25 analogue video streams which is generally working using 
> gstreamer as the higher level software driving this.
> 
> We have an intermittent video stream start up problem which we believe 
> is down to:
> 
> 1. The TP2855 is initialised first and its subdev *_s_stream() call is 
> called to start the video input.
> 
> 2. The NXP CSI2 video input hardware is then started: 
> imx8-mipi-csi2-sam.c: mipi_csis_s_stream().

I don't think I can see this file in the latest linux sources. Are you
working on an out of tree BSP driver supported by a vendor?

If so - you might be able to contact them directly for support, but if
we don't have the source code it can be hard to support your issue.

--
Regards

Kieran


> 
> 3. The TP2855 enables the CSI2 clock after its *_s_stream() call, 
> probably in hardware after its PLL's have locked, by sending a CSI2 
> start sequence on the CSI2 clock pair. This clock is then a continuous 
> clock ie. it does not go into low power mode during horizontal/vertical 
> blanking.
> 
> 4. The NXP CSI2 video input hardware, based on some Samsung IP, does not 
> see the CSI2 clock unless it sees the CSI2 start sequence and depending 
> on timings it may not see this at video pipeline startup and we get not 
> video stream from the CSI2 hardware.
> 
> I was hoping the subdev *_s_stream() call would be after all of the 
> hardware's pipeline was setup so I could instigate a CSI2 clock restart 
> in the TP2855 driver, but unfortunately this is called before the CSI2 
> hardware is setup.
> 
> I can add a one shot timer to do this in the TP2855 subdev *_s_stream() 
> call, but obviously this is not ideal. Is there anyway to get a subdev 
> function called in the video4linux API automatically once all of the 
> video streams hardware is setup ?
> 
> Any ideas/recommendations ?
> 
> Terry
> 
>

Re: Video4Linux: Call an I2C subdev function to start a stream after a CSI2 driver has been started

[Terry Barnaby]

Hi Kiera,

Thanks for you response. Yes its from the NXP Yocto distribution with 
kernel at: git://source.codeaurora.org/external/imx/linux-imx.git

But I wasn't after specific information/help with this platform, I was 
really trying to find out if there is a mechanism within the 
mainline/standard Video4Linux kernel API's such that a video front end 
subdev can get some function called once the video pipeline has been 
setup ie. All hardware modules configured and ready to run a video stream.

For example there is the s_stream() function pointer in 
v4l2_subdev_video_ops, but that function is called before the later (in 
pipeline) CSI2 module is started and I need a function to be called 
after all of the pipelines hardware has been initialised to get around 
this hardware feature/issue.

Terry
On 10/05/2022 16:33, Kieran Bingham wrote:
> Hi Terry,
>
> Quoting Terry Barnaby (2022-05-10 14:35:38)
>> Hi,
>>
>> We are working on a system that uses a NXP IMX8MP SOC with a TP2855
>> analogue video front end chip that can capture 1920x1080p25 or PAL
>> 720x576i25 analogue video streams which is generally working using
>> gstreamer as the higher level software driving this.
>>
>> We have an intermittent video stream start up problem which we believe
>> is down to:
>>
>> 1. The TP2855 is initialised first and its subdev *_s_stream() call is
>> called to start the video input.
>>
>> 2. The NXP CSI2 video input hardware is then started:
>> imx8-mipi-csi2-sam.c: mipi_csis_s_stream().
> I don't think I can see this file in the latest linux sources. Are you
> working on an out of tree BSP driver supported by a vendor?
>
> If so - you might be able to contact them directly for support, but if
> we don't have the source code it can be hard to support your issue.
>
> --
> Regards
>
> Kieran
>
>
>> 3. The TP2855 enables the CSI2 clock after its *_s_stream() call,
>> probably in hardware after its PLL's have locked, by sending a CSI2
>> start sequence on the CSI2 clock pair. This clock is then a continuous
>> clock ie. it does not go into low power mode during horizontal/vertical
>> blanking.
>>
>> 4. The NXP CSI2 video input hardware, based on some Samsung IP, does not
>> see the CSI2 clock unless it sees the CSI2 start sequence and depending
>> on timings it may not see this at video pipeline startup and we get not
>> video stream from the CSI2 hardware.
>>
>> I was hoping the subdev *_s_stream() call would be after all of the
>> hardware's pipeline was setup so I could instigate a CSI2 clock restart
>> in the TP2855 driver, but unfortunately this is called before the CSI2
>> hardware is setup.
>>
>> I can add a one shot timer to do this in the TP2855 subdev *_s_stream()
>> call, but obviously this is not ideal. Is there anyway to get a subdev
>> function called in the video4linux API automatically once all of the
>> video streams hardware is setup ?
>>
>> Any ideas/recommendations ?
>>
>> Terry
>>
>>

Re: Video4Linux: Call an I2C subdev function to start a stream after a CSI2 driver has been started

[Terry Barnaby]

On 10/05/2022 16:33, Kieran Bingham wrote:
> Hi Terry,
>
> Quoting Terry Barnaby (2022-05-10 14:35:38)
>> Hi,
>>
>> We are working on a system that uses a NXP IMX8MP SOC with a TP2855
>> analogue video front end chip that can capture 1920x1080p25 or PAL
>> 720x576i25 analogue video streams which is generally working using
>> gstreamer as the higher level software driving this.
>>
>> We have an intermittent video stream start up problem which we believe
>> is down to:
>>
>> 1. The TP2855 is initialised first and its subdev *_s_stream() call is
>> called to start the video input.
>>
>> 2. The NXP CSI2 video input hardware is then started:
>> imx8-mipi-csi2-sam.c: mipi_csis_s_stream().
> I don't think I can see this file in the latest linux sources. Are you
> working on an out of tree BSP driver supported by a vendor?
>
> If so - you might be able to contact them directly for support, but if
> we don't have the source code it can be hard to support your issue.
>
> --
> Regards
>
> Kieran

Sorry, top posted in previous email (all different groups methods). I 
have repeated my response here.

Hi Kiera,

Thanks for you response. Yes its from the NXP Yocto distribution with 
kernel at: git://source.codeaurora.org/external/imx/linux-imx.git

But I wasn't after specific information/help with this platform, I was 
really trying to find out if there is a mechanism within the 
mainline/standard Video4Linux kernel API's such that a video front end 
subdev can get some function called once the video pipeline has been 
setup ie. All hardware modules configured and ready to run a video stream.

For example there is the s_stream() function pointer in 
v4l2_subdev_video_ops, but that function is called before the later (in 
pipeline) CSI2 module is started and I need a function to be called 
after all of the pipelines hardware has been initialised to get around 
this hardware feature/issue.

Terry

>
>
>> 3. The TP2855 enables the CSI2 clock after its *_s_stream() call,
>> probably in hardware after its PLL's have locked, by sending a CSI2
>> start sequence on the CSI2 clock pair. This clock is then a continuous
>> clock ie. it does not go into low power mode during horizontal/vertical
>> blanking.
>>
>> 4. The NXP CSI2 video input hardware, based on some Samsung IP, does not
>> see the CSI2 clock unless it sees the CSI2 start sequence and depending
>> on timings it may not see this at video pipeline startup and we get not
>> video stream from the CSI2 hardware.
>>
>> I was hoping the subdev *_s_stream() call would be after all of the
>> hardware's pipeline was setup so I could instigate a CSI2 clock restart
>> in the TP2855 driver, but unfortunately this is called before the CSI2
>> hardware is setup.
>>
>> I can add a one shot timer to do this in the TP2855 subdev *_s_stream()
>> call, but obviously this is not ideal. Is there anyway to get a subdev
>> function called in the video4linux API automatically once all of the
>> video streams hardware is setup ?
>>
>> Any ideas/recommendations ?
>>
>> Terry
>>
>>

Re: Video4Linux: Call an I2C subdev function to start a stream after a CSI2 driver has been started

[Kieran Bingham]

Hi Terry,

Quoting Terry Barnaby (2022-05-10 17:01:28)
> On 10/05/2022 16:33, Kieran Bingham wrote:
> > Hi Terry,
> >
> > Quoting Terry Barnaby (2022-05-10 14:35:38)
> >> Hi,
> >>
> >> We are working on a system that uses a NXP IMX8MP SOC with a TP2855
> >> analogue video front end chip that can capture 1920x1080p25 or PAL
> >> 720x576i25 analogue video streams which is generally working using
> >> gstreamer as the higher level software driving this.
> >>
> >> We have an intermittent video stream start up problem which we believe
> >> is down to:
> >>
> >> 1. The TP2855 is initialised first and its subdev *_s_stream() call is
> >> called to start the video input.
> >>
> >> 2. The NXP CSI2 video input hardware is then started:
> >> imx8-mipi-csi2-sam.c: mipi_csis_s_stream().
> > I don't think I can see this file in the latest linux sources. Are you
> > working on an out of tree BSP driver supported by a vendor?
> >
> > If so - you might be able to contact them directly for support, but if
> > we don't have the source code it can be hard to support your issue.
> >
> > --
> > Regards
> >
> > Kieran
> 
> Sorry, top posted in previous email (all different groups methods). I 
> have repeated my response here.
> 
> Hi Kiera,
> 
> Thanks for you response. Yes its from the NXP Yocto distribution with 
> kernel at: git://source.codeaurora.org/external/imx/linux-imx.git
> 
> But I wasn't after specific information/help with this platform, I was 
> really trying to find out if there is a mechanism within the 
> mainline/standard Video4Linux kernel API's such that a video front end 
> subdev can get some function called once the video pipeline has been 
> setup ie. All hardware modules configured and ready to run a video stream.
> 
> For example there is the s_stream() function pointer in 
> v4l2_subdev_video_ops, but that function is called before the later (in 
> pipeline) CSI2 module is started and I need a function to be called 
> after all of the pipelines hardware has been initialised to get around 
> this hardware feature/issue.

There's quite a distinction between 'ready to run a stream' ... and
'running a stream', and your description below references actions that
occur in out of tree drivers, so it's not possible to get a clear
picture here of what is really happening without extending support to an
out of tree vendor driver.

The v4l2_async_notifier framework allows hooks that know when the
pipeline is 'ready to run a stream' if I understand your statements
correctly, but the mention of s_stream() makes me wonder if you're
trying to do something /after/ you have started the stream.

--
Kieran



> 
> Terry
> 
> >
> >
> >> 3. The TP2855 enables the CSI2 clock after its *_s_stream() call,
> >> probably in hardware after its PLL's have locked, by sending a CSI2
> >> start sequence on the CSI2 clock pair. This clock is then a continuous
> >> clock ie. it does not go into low power mode during horizontal/vertical
> >> blanking.
> >>
> >> 4. The NXP CSI2 video input hardware, based on some Samsung IP, does not
> >> see the CSI2 clock unless it sees the CSI2 start sequence and depending
> >> on timings it may not see this at video pipeline startup and we get not
> >> video stream from the CSI2 hardware.
> >>
> >> I was hoping the subdev *_s_stream() call would be after all of the
> >> hardware's pipeline was setup so I could instigate a CSI2 clock restart
> >> in the TP2855 driver, but unfortunately this is called before the CSI2
> >> hardware is setup.
> >>
> >> I can add a one shot timer to do this in the TP2855 subdev *_s_stream()
> >> call, but obviously this is not ideal. Is there anyway to get a subdev
> >> function called in the video4linux API automatically once all of the
> >> video streams hardware is setup ?
> >>
> >> Any ideas/recommendations ?
> >>
> >> Terry
> >>
> >>
>