Looking for device driver advice

Looking for device driver advice

[Patrick Doyle]

I am looking for advice regarding the construction of a device driver
for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
interface on a Soc (a Microchip/Atmel SAMAD2x device.)

The Sony imager is controlled and configured via I2C, as is the
Toshiba converter.  I could write a single driver that configures both
devices and treats them as a single device that just happens to use 2
i2c addresses.  I could use the i2c_new_dummy() API to construct the
device abstraction for the second physical device at probe time for
the first physical device.

Or I could do something smarter (or at least different), specifying
the two devices independently via my device tree file, perhaps linking
them together via "port" nodes.  Currently, I use the "port" node
concept to link an i2c imager to the Image System Controller (isc)
node in the SAMA5 device.  Perhaps that generalizes to a chain of
nodes linked together... I don't know.

I'm also not sure how these two devices might play into V4L2's
"subdev" concept.  Are they separate, independent sub devices of the
ISC, or are they a single sub device.

Any thoughts, intuition, pointers to existing code that addresses
questions such as these, would be welcome.

Thanks.

--wpd

Re: Looking for device driver advice

[Hans Verkuil]

Hi Patrick,

On 04/10/2017 10:13 PM, Patrick Doyle wrote:
> I am looking for advice regarding the construction of a device driver
> for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
> MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
> interface on a Soc (a Microchip/Atmel SAMAD2x device.)
> 
> The Sony imager is controlled and configured via I2C, as is the
> Toshiba converter.  I could write a single driver that configures both
> devices and treats them as a single device that just happens to use 2
> i2c addresses.  I could use the i2c_new_dummy() API to construct the
> device abstraction for the second physical device at probe time for
> the first physical device.
> 
> Or I could do something smarter (or at least different), specifying
> the two devices independently via my device tree file, perhaps linking
> them together via "port" nodes.  Currently, I use the "port" node
> concept to link an i2c imager to the Image System Controller (isc)
> node in the SAMA5 device.  Perhaps that generalizes to a chain of
> nodes linked together... I don't know.

That would be the right solution. Unfortunately the atmel-isc.c driver
(at least the version in the mainline kernel) only supports a single
subdev device. At least, as far as I can see.

What you have is a video pipeline of 2 subdevs and the atmel-isc as DMA
engine:

imx241 -> tc358748 -> atmel-isc

connected in the device tree by ports.

Looking at the code I think both subdev drivers would be loaded, but
the atmel-isc driver would only call ops from the tc358748.

The v4l2_subdev_call functions in atmel-isc should most likely be replaced
by v4l2_device_call_all().

But I don't have the tc358748 datasheet with the register information, so
I am not sure if this is sufficient.

> I'm also not sure how these two devices might play into V4L2's
> "subdev" concept.  Are they separate, independent sub devices of the
> ISC, or are they a single sub device.

subdev drivers are standalone drivers for, among others, i2c devices. The
top-level driver (atmel-isc + the device tree) is what pulls everything together.

This allows us to reuse such subdev drivers on other devices.

Regards,

	Hans

> 
> Any thoughts, intuition, pointers to existing code that addresses
> questions such as these, would be welcome.
> 
> Thanks.
> 
> --wpd
> 

Re: Looking for device driver advice

[Patrick Doyle]

Thank you Hans,
I can modify (or work with Atmel/Microchip to have modified) the
atmel-isc driver, so I that's not an issue.

With your feedback, I now have a target implementation to which I can aim.

For now, I'll be happy when I can get any image at all through my
pipeline... wish me luck! :-)

On a similar vein... the image is much higher resolution than I need
at the moment, so I will need to downsample it.

The SAMA5 has a downsampler built into its LCD engine.  Suppose I
wanted to treat that downsampler as an independent device and pass
image buffers through that downsampler (the LCD display output is
disabled in hardware when the device is configured in this mode)....
do you have any recommendations as to how I might structure a device
driver to do that.  At it's core, it would DMA a buffer from memory,
through the downsampler, and back into memory.  Is that something I
might also wire in as a pseudo-subdev of the ISC?  Or is there a
better abstraction for arbitrary image processing pipeline elements?

Oh yeah, and speaking about arbitrary image processing pipeline
elements, the Atmel ISC has a number of elements that could be enabled
(but are not currently supported by the driver).  Is there a model to
follow to enable features such as demosaicing, color space conversion,
gamma correction, 4:2:2 to 4:2:0 downsampling, etc...

All of this is on the list of things that I need to get working... by
next Tuesday ideally :-)
(No, it's not really "next Tuesday", but you get the idea.)

--wpd


On Wed, Apr 12, 2017 at 7:37 AM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> Hi Patrick,
>
> On 04/10/2017 10:13 PM, Patrick Doyle wrote:
>> I am looking for advice regarding the construction of a device driver
>> for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
>> MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
>> interface on a Soc (a Microchip/Atmel SAMAD2x device.)
>>
>> The Sony imager is controlled and configured via I2C, as is the
>> Toshiba converter.  I could write a single driver that configures both
>> devices and treats them as a single device that just happens to use 2
>> i2c addresses.  I could use the i2c_new_dummy() API to construct the
>> device abstraction for the second physical device at probe time for
>> the first physical device.
>>
>> Or I could do something smarter (or at least different), specifying
>> the two devices independently via my device tree file, perhaps linking
>> them together via "port" nodes.  Currently, I use the "port" node
>> concept to link an i2c imager to the Image System Controller (isc)
>> node in the SAMA5 device.  Perhaps that generalizes to a chain of
>> nodes linked together... I don't know.
>
> That would be the right solution. Unfortunately the atmel-isc.c driver
> (at least the version in the mainline kernel) only supports a single
> subdev device. At least, as far as I can see.
>
> What you have is a video pipeline of 2 subdevs and the atmel-isc as DMA
> engine:
>
> imx241 -> tc358748 -> atmel-isc
>
> connected in the device tree by ports.
>
> Looking at the code I think both subdev drivers would be loaded, but
> the atmel-isc driver would only call ops from the tc358748.
>
> The v4l2_subdev_call functions in atmel-isc should most likely be replaced
> by v4l2_device_call_all().
>
> But I don't have the tc358748 datasheet with the register information, so
> I am not sure if this is sufficient.
>
>> I'm also not sure how these two devices might play into V4L2's
>> "subdev" concept.  Are they separate, independent sub devices of the
>> ISC, or are they a single sub device.
>
> subdev drivers are standalone drivers for, among others, i2c devices. The
> top-level driver (atmel-isc + the device tree) is what pulls everything together.
>
> This allows us to reuse such subdev drivers on other devices.
>
> Regards,
>
>         Hans
>
>>
>> Any thoughts, intuition, pointers to existing code that addresses
>> questions such as these, would be welcome.
>>
>> Thanks.
>>
>> --wpd
>>
>

Re: Looking for device driver advice

[Hans Verkuil]

On 04/12/2017 03:13 PM, Patrick Doyle wrote:
> Thank you Hans,
> I can modify (or work with Atmel/Microchip to have modified) the
> atmel-isc driver, so I that's not an issue.
> 
> With your feedback, I now have a target implementation to which I can aim.
> 
> For now, I'll be happy when I can get any image at all through my
> pipeline... wish me luck! :-)
> 
> On a similar vein... the image is much higher resolution than I need
> at the moment, so I will need to downsample it.
> 
> The SAMA5 has a downsampler built into its LCD engine.  Suppose I
> wanted to treat that downsampler as an independent device and pass
> image buffers through that downsampler (the LCD display output is
> disabled in hardware when the device is configured in this mode)....
> do you have any recommendations as to how I might structure a device
> driver to do that.  At it's core, it would DMA a buffer from memory,
> through the downsampler, and back into memory.  Is that something I
> might also wire in as a pseudo-subdev of the ISC?  Or is there a
> better abstraction for arbitrary image processing pipeline elements?

I think this is out of scope of V4L2. Check with Atmel/Microchip.

> 
> Oh yeah, and speaking about arbitrary image processing pipeline
> elements, the Atmel ISC has a number of elements that could be enabled
> (but are not currently supported by the driver).  Is there a model to
> follow to enable features such as demosaicing, color space conversion,
> gamma correction, 4:2:2 to 4:2:0 downsampling, etc...

As far as I can see it already has support for colorspace conversion (I
presume you mean Bayer to YUV or RGB conversions) and ditto for 4:2:2 and
4:2:0. I see a gamma control as well.

Anyway, this is best discussed with your Atmel/Microchip contact.

Ah, I checked, this isn't upstream yet until kernel 4.12. It is in our
repo: https://git.linuxtv.org/media_tree.git/ in the master branch which
will feed into the mainline kernel.

> All of this is on the list of things that I need to get working... by
> next Tuesday ideally :-)
> (No, it's not really "next Tuesday", but you get the idea.)

Well, copying the atmel-isc code from our repo should help a lot :-)

Regards,

	Hans

> 
> --wpd
> 
> 
> On Wed, Apr 12, 2017 at 7:37 AM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> Hi Patrick,
>>
>> On 04/10/2017 10:13 PM, Patrick Doyle wrote:
>>> I am looking for advice regarding the construction of a device driver
>>> for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
>>> MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
>>> interface on a Soc (a Microchip/Atmel SAMAD2x device.)
>>>
>>> The Sony imager is controlled and configured via I2C, as is the
>>> Toshiba converter.  I could write a single driver that configures both
>>> devices and treats them as a single device that just happens to use 2
>>> i2c addresses.  I could use the i2c_new_dummy() API to construct the
>>> device abstraction for the second physical device at probe time for
>>> the first physical device.
>>>
>>> Or I could do something smarter (or at least different), specifying
>>> the two devices independently via my device tree file, perhaps linking
>>> them together via "port" nodes.  Currently, I use the "port" node
>>> concept to link an i2c imager to the Image System Controller (isc)
>>> node in the SAMA5 device.  Perhaps that generalizes to a chain of
>>> nodes linked together... I don't know.
>>
>> That would be the right solution. Unfortunately the atmel-isc.c driver
>> (at least the version in the mainline kernel) only supports a single
>> subdev device. At least, as far as I can see.
>>
>> What you have is a video pipeline of 2 subdevs and the atmel-isc as DMA
>> engine:
>>
>> imx241 -> tc358748 -> atmel-isc
>>
>> connected in the device tree by ports.
>>
>> Looking at the code I think both subdev drivers would be loaded, but
>> the atmel-isc driver would only call ops from the tc358748.
>>
>> The v4l2_subdev_call functions in atmel-isc should most likely be replaced
>> by v4l2_device_call_all().
>>
>> But I don't have the tc358748 datasheet with the register information, so
>> I am not sure if this is sufficient.
>>
>>> I'm also not sure how these two devices might play into V4L2's
>>> "subdev" concept.  Are they separate, independent sub devices of the
>>> ISC, or are they a single sub device.
>>
>> subdev drivers are standalone drivers for, among others, i2c devices. The
>> top-level driver (atmel-isc + the device tree) is what pulls everything together.
>>
>> This allows us to reuse such subdev drivers on other devices.
>>
>> Regards,
>>
>>         Hans
>>
>>>
>>> Any thoughts, intuition, pointers to existing code that addresses
>>> questions such as these, would be welcome.
>>>
>>> Thanks.
>>>
>>> --wpd
>>>
>>

Re: Looking for device driver advice

[Patrick Doyle]

Thank you again Hans.

On Wed, Apr 12, 2017 at 9:58 AM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 04/12/2017 03:13 PM, Patrick Doyle wrote:
>> The SAMA5 has a downsampler built into its LCD engine.  Suppose I
>> wanted to treat that downsampler as an independent device and pass
>> image buffers through that downsampler (the LCD display output is
>> disabled in hardware when the device is configured in this mode)....
>> do you have any recommendations as to how I might structure a device
>> driver to do that.  At it's core, it would DMA a buffer from memory,
>> through the downsampler, and back into memory.  Is that something I
>> might also wire in as a pseudo-subdev of the ISC?  Or is there a
>> better abstraction for arbitrary image processing pipeline elements?
>
> I think this is out of scope of V4L2. Check with Atmel/Microchip.

The V4L2 tie-in is in answering the question: Is there a standard V4L2
way to perform arbitrary video processing functions (such as
downsampling) with an arbitrary device.  If the answer is "no", that's
fine.  If the answer is, "yes, here is how Xilinx does it for their IP
modules", then I'll go look at some Xilinx stuff (which I'm going to
do anyway).

--wpd

Re: Looking for device driver advice

[Hans Verkuil]

On 04/12/2017 04:29 PM, Patrick Doyle wrote:
> Thank you again Hans.
> 
> On Wed, Apr 12, 2017 at 9:58 AM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> On 04/12/2017 03:13 PM, Patrick Doyle wrote:
>>> The SAMA5 has a downsampler built into its LCD engine.  Suppose I
>>> wanted to treat that downsampler as an independent device and pass
>>> image buffers through that downsampler (the LCD display output is
>>> disabled in hardware when the device is configured in this mode)....
>>> do you have any recommendations as to how I might structure a device
>>> driver to do that.  At it's core, it would DMA a buffer from memory,
>>> through the downsampler, and back into memory.  Is that something I
>>> might also wire in as a pseudo-subdev of the ISC?  Or is there a
>>> better abstraction for arbitrary image processing pipeline elements?
>>
>> I think this is out of scope of V4L2. Check with Atmel/Microchip.
> 
> The V4L2 tie-in is in answering the question: Is there a standard V4L2
> way to perform arbitrary video processing functions (such as
> downsampling) with an arbitrary device.  If the answer is "no", that's
> fine.  If the answer is, "yes, here is how Xilinx does it for their IP
> modules", then I'll go look at some Xilinx stuff (which I'm going to
> do anyway).

VIDIOC_S_FMT: the pixelformat you specify in the v4l2_pix_format struct
determines Bayer vs YUV 4:2:2 vs YUV 4:2:0 vs RGB.

So that triggers whatever conversion is needed to arrive at the desired
memory format.

https://www.linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/vidioc-g-fmt.html

Regards,

	Hans

Re: Looking for device driver advice

[Sakari Ailus]

Hi Hans and Patrick,

On Wed, Apr 12, 2017 at 01:37:33PM +0200, Hans Verkuil wrote:
> Hi Patrick,
> 
> On 04/10/2017 10:13 PM, Patrick Doyle wrote:
> > I am looking for advice regarding the construction of a device driver
> > for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
> > MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
> > interface on a Soc (a Microchip/Atmel SAMAD2x device.)
> > 
> > The Sony imager is controlled and configured via I2C, as is the
> > Toshiba converter.  I could write a single driver that configures both
> > devices and treats them as a single device that just happens to use 2
> > i2c addresses.  I could use the i2c_new_dummy() API to construct the
> > device abstraction for the second physical device at probe time for
> > the first physical device.
> > 
> > Or I could do something smarter (or at least different), specifying
> > the two devices independently via my device tree file, perhaps linking
> > them together via "port" nodes.  Currently, I use the "port" node
> > concept to link an i2c imager to the Image System Controller (isc)
> > node in the SAMA5 device.  Perhaps that generalizes to a chain of
> > nodes linked together... I don't know.
> 
> That would be the right solution. Unfortunately the atmel-isc.c driver
> (at least the version in the mainline kernel) only supports a single
> subdev device. At least, as far as I can see.

There have been multiple cases recently where the media pipeline can have
sub-devices controlled by more than two drivers. We need to have a common
approach on how we do handle such cases.

For instance, how is the entire DT graph parsed or when and how are the
device nodes created?

Parsing the graph should probably be initiated by the master driver but
instead implemented in the framework as it's a non-trivial task and common
to all such drivers. Another equestion is how do we best support this also
on existing drivers.

I actually have a small documentation patch on handling streaming control in
such cases as there are choices now to be made not thought about when the
sub-device ops were originally addeed. I'll cc you to that.

We do have a similar case currently in i.MX6, Nokia N9 (OMAP3) and on some
Renesas hardware unless I'm mistaken.

Cc Niklas.

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: Looking for device driver advice

[Niklas Söderlund]

Hi,

On 2017-04-16 13:51:21 +0300, Sakari Ailus wrote:
> Hi Hans and Patrick,
> 
> On Wed, Apr 12, 2017 at 01:37:33PM +0200, Hans Verkuil wrote:
> > Hi Patrick,
> > 
> > On 04/10/2017 10:13 PM, Patrick Doyle wrote:
> > > I am looking for advice regarding the construction of a device driver
> > > for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
> > > MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
> > > interface on a Soc (a Microchip/Atmel SAMAD2x device.)
> > > 
> > > The Sony imager is controlled and configured via I2C, as is the
> > > Toshiba converter.  I could write a single driver that configures both
> > > devices and treats them as a single device that just happens to use 2
> > > i2c addresses.  I could use the i2c_new_dummy() API to construct the
> > > device abstraction for the second physical device at probe time for
> > > the first physical device.
> > > 
> > > Or I could do something smarter (or at least different), specifying
> > > the two devices independently via my device tree file, perhaps linking
> > > them together via "port" nodes.  Currently, I use the "port" node
> > > concept to link an i2c imager to the Image System Controller (isc)
> > > node in the SAMA5 device.  Perhaps that generalizes to a chain of
> > > nodes linked together... I don't know.
> > 
> > That would be the right solution. Unfortunately the atmel-isc.c driver
> > (at least the version in the mainline kernel) only supports a single
> > subdev device. At least, as far as I can see.

I also think that two subdevices implemented in two separate drivers is 
the way to go. As it really is two different pieces of hardware,
right?

> 
> There have been multiple cases recently where the media pipeline can have
> sub-devices controlled by more than two drivers. We need to have a common
> approach on how we do handle such cases.

I agree that a common approach to the problem of when one subdevices can 
be controlled by more then one driver is needed. In this case however I 
think something else also needs to be defined. If I understand Hans and 
Patrick the issues is not that the hardware can be controlled by more 
then one driver. Instead it is that the atmel-isc.c driver only probes 
DT for one subdevice, so implementing it as more then one subdevices is 
problematic. If I misunderstand the problem please let me know.

If I understand the problem correctly it could be solved by modifying 
the atmel-isc.c driver to look for more then one subdevice in DT. But a 
common approach for drivers to find and bind arbitrary number of 
subdevices would be better, finding an approach that also solves the 
case where one subdevice can be used by more then one driver would be 
better still. If this common case also could cover the case where one DT 
node represents a driver which registers more then one subdevice which 
then can be used by different other drivers I would be very happy and a 
lot of my headaches would go away :-)

> 
> For instance, how is the entire DT graph parsed or when and how are the
> device nodes created?
> 
> Parsing the graph should probably be initiated by the master driver but
> instead implemented in the framework as it's a non-trivial task and common
> to all such drivers. Another equestion is how do we best support this also
> on existing drivers.

I agree that the master device probably should initiate the DT graph 
parsing and if possible there should be as much support as possible in 
the framework. One extra consideration here is that there might be more 
then one master device which uses the same subdevices. I have such cases 
today where different instances of the same driver use the same set of 
subdevices.

> 
> I actually have a small documentation patch on handling streaming control in
> such cases as there are choices now to be made not thought about when the
> sub-device ops were originally addeed. I'll cc you to that.
> 
> We do have a similar case currently in i.MX6, Nokia N9 (OMAP3) and on some
> Renesas hardware unless I'm mistaken.

Yes there are similar use-cases with Renesas Gen3, adding Kieran Bingham 
to CC as he hopefully will look into some of them.

> 
> Cc Niklas.

Thanks !

> 
> -- 
> Kind regards,
> 
> Sakari Ailus
> e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

-- 
Regards,
Niklas Söderlund

Re: Looking for device driver advice

[Sakari Ailus]

Hi Niklas,

On Sun, Apr 16, 2017 at 07:42:20PM +0200, Niklas Söderlund wrote:
> Hi,
> 
> On 2017-04-16 13:51:21 +0300, Sakari Ailus wrote:
> > Hi Hans and Patrick,
> > 
> > On Wed, Apr 12, 2017 at 01:37:33PM +0200, Hans Verkuil wrote:
> > > Hi Patrick,
> > > 
> > > On 04/10/2017 10:13 PM, Patrick Doyle wrote:
> > > > I am looking for advice regarding the construction of a device driver
> > > > for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
> > > > MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
> > > > interface on a Soc (a Microchip/Atmel SAMAD2x device.)
> > > > 
> > > > The Sony imager is controlled and configured via I2C, as is the
> > > > Toshiba converter.  I could write a single driver that configures both
> > > > devices and treats them as a single device that just happens to use 2
> > > > i2c addresses.  I could use the i2c_new_dummy() API to construct the
> > > > device abstraction for the second physical device at probe time for
> > > > the first physical device.
> > > > 
> > > > Or I could do something smarter (or at least different), specifying
> > > > the two devices independently via my device tree file, perhaps linking
> > > > them together via "port" nodes.  Currently, I use the "port" node
> > > > concept to link an i2c imager to the Image System Controller (isc)
> > > > node in the SAMA5 device.  Perhaps that generalizes to a chain of
> > > > nodes linked together... I don't know.
> > > 
> > > That would be the right solution. Unfortunately the atmel-isc.c driver
> > > (at least the version in the mainline kernel) only supports a single
> > > subdev device. At least, as far as I can see.
> 
> I also think that two subdevices implemented in two separate drivers is 
> the way to go. As it really is two different pieces of hardware,
> right?
> 
> > 
> > There have been multiple cases recently where the media pipeline can have
> > sub-devices controlled by more than two drivers. We need to have a common
> > approach on how we do handle such cases.
> 
> I agree that a common approach to the problem of when one subdevices can 
> be controlled by more then one driver is needed. In this case however I 
> think something else also needs to be defined. If I understand Hans and 
> Patrick the issues is not that the hardware can be controlled by more 
> then one driver. Instead it is that the atmel-isc.c driver only probes 
> DT for one subdevice, so implementing it as more then one subdevices is 
> problematic. If I misunderstand the problem please let me know.

The pipeline is (again, unless I'm mistaken):

sensor -> csi2-parallel converter -> SoC

So not very much unlike it's elsewhere.

> 
> If I understand the problem correctly it could be solved by modifying 
> the atmel-isc.c driver to look for more then one subdevice in DT. But a 
> common approach for drivers to find and bind arbitrary number of 
> subdevices would be better, finding an approach that also solves the 
> case where one subdevice can be used by more then one driver would be 
> better still. If this common case also could cover the case where one DT 
> node represents a driver which registers more then one subdevice which 
> then can be used by different other drivers I would be very happy and a 
> lot of my headaches would go away :-)
> 
> > 
> > For instance, how is the entire DT graph parsed or when and how are the
> > device nodes created?
> > 
> > Parsing the graph should probably be initiated by the master driver but
> > instead implemented in the framework as it's a non-trivial task and common
> > to all such drivers. Another equestion is how do we best support this also
> > on existing drivers.
> 
> I agree that the master device probably should initiate the DT graph 
> parsing and if possible there should be as much support as possible in 
> the framework. One extra consideration here is that there might be more 
> then one master device which uses the same subdevices. I have such cases 

Good point. I.e. you have two source pads that are connected to two
different master devices?

> today where different instances of the same driver use the same set of 
> subdevices.

Is this the intended state of matters? Or do I miss something? :-)

> 
> > 
> > I actually have a small documentation patch on handling streaming control in
> > such cases as there are choices now to be made not thought about when the
> > sub-device ops were originally addeed. I'll cc you to that.
> > 
> > We do have a similar case currently in i.MX6, Nokia N9 (OMAP3) and on some

I meant to say Nokia N900, not N9.

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: Looking for device driver advice

[Niklas Söderlund]

Hi Sakari,

On 2017-04-18 10:46:36 +0300, Sakari Ailus wrote:
> Hi Niklas,
> 
> On Sun, Apr 16, 2017 at 07:42:20PM +0200, Niklas Söderlund wrote:
> > Hi,
> > 
> > On 2017-04-16 13:51:21 +0300, Sakari Ailus wrote:
> > > Hi Hans and Patrick,
> > > 
> > > On Wed, Apr 12, 2017 at 01:37:33PM +0200, Hans Verkuil wrote:
> > > > Hi Patrick,
> > > > 
> > > > On 04/10/2017 10:13 PM, Patrick Doyle wrote:
> > > > > I am looking for advice regarding the construction of a device driver
> > > > > for a MIPI CSI2 imager (a Sony IMX241) that is connected to a
> > > > > MIPI<->Parallel converter (Toshiba TC358748) wired into a parallel
> > > > > interface on a Soc (a Microchip/Atmel SAMAD2x device.)
> > > > > 
> > > > > The Sony imager is controlled and configured via I2C, as is the
> > > > > Toshiba converter.  I could write a single driver that configures both
> > > > > devices and treats them as a single device that just happens to use 2
> > > > > i2c addresses.  I could use the i2c_new_dummy() API to construct the
> > > > > device abstraction for the second physical device at probe time for
> > > > > the first physical device.
> > > > > 
> > > > > Or I could do something smarter (or at least different), specifying
> > > > > the two devices independently via my device tree file, perhaps linking
> > > > > them together via "port" nodes.  Currently, I use the "port" node
> > > > > concept to link an i2c imager to the Image System Controller (isc)
> > > > > node in the SAMA5 device.  Perhaps that generalizes to a chain of
> > > > > nodes linked together... I don't know.
> > > > 
> > > > That would be the right solution. Unfortunately the atmel-isc.c driver
> > > > (at least the version in the mainline kernel) only supports a single
> > > > subdev device. At least, as far as I can see.
> > 
> > I also think that two subdevices implemented in two separate drivers is 
> > the way to go. As it really is two different pieces of hardware,
> > right?
> > 
> > > 
> > > There have been multiple cases recently where the media pipeline can have
> > > sub-devices controlled by more than two drivers. We need to have a common
> > > approach on how we do handle such cases.
> > 
> > I agree that a common approach to the problem of when one subdevices can 
> > be controlled by more then one driver is needed. In this case however I 
> > think something else also needs to be defined. If I understand Hans and 
> > Patrick the issues is not that the hardware can be controlled by more 
> > then one driver. Instead it is that the atmel-isc.c driver only probes 
> > DT for one subdevice, so implementing it as more then one subdevices is 
> > problematic. If I misunderstand the problem please let me know.
> 
> The pipeline is (again, unless I'm mistaken):
> 
> sensor -> csi2-parallel converter -> SoC
> 
> So not very much unlike it's elsewhere.
> 
> > 
> > If I understand the problem correctly it could be solved by modifying 
> > the atmel-isc.c driver to look for more then one subdevice in DT. But a 
> > common approach for drivers to find and bind arbitrary number of 
> > subdevices would be better, finding an approach that also solves the 
> > case where one subdevice can be used by more then one driver would be 
> > better still. If this common case also could cover the case where one DT 
> > node represents a driver which registers more then one subdevice which 
> > then can be used by different other drivers I would be very happy and a 
> > lot of my headaches would go away :-)
> > 
> > > 
> > > For instance, how is the entire DT graph parsed or when and how are the
> > > device nodes created?
> > > 
> > > Parsing the graph should probably be initiated by the master driver but
> > > instead implemented in the framework as it's a non-trivial task and common
> > > to all such drivers. Another equestion is how do we best support this also
> > > on existing drivers.
> > 
> > I agree that the master device probably should initiate the DT graph 
> > parsing and if possible there should be as much support as possible in 
> > the framework. One extra consideration here is that there might be more 
> > then one master device which uses the same subdevices. I have such cases 
> 
> Good point. I.e. you have two source pads that are connected to two
> different master devices?

No, I have one source pad which is connected to more then one master 
devices. The source pad carries CSI-2 streams so different master 
devices will receive different virtual channels, or better still 
different master devices can receive the same virtual channel but apply 
different scaling/conversions on the stream before it reaches the 
/dev/videoX node.

> 
> > today where different instances of the same driver use the same set of 
> > subdevices.
> 
> Is this the intended state of matters? Or do I miss something? :-)

Yes, it is the goal to be able to represent all hardware configurations 
in the media graph.

Currently the master devices is different instances of the same driver, 
rcar-vin. So the solution right now is that the first VIN creates a 
media device, finds all upstream subdevices and adds them to the media 
graph. Later instances of the rcar-vin driver will simply join the media 
device and this way gain access to the shard subdevices.

This works but creates a complex logic to share subdevices. If more then 
one master device could find the same subdevices I believe this logic 
could be simplified and maybe more functionality could be put in the 
V4L2 core.

> 
> > 
> > > 
> > > I actually have a small documentation patch on handling streaming control in
> > > such cases as there are choices now to be made not thought about when the
> > > sub-device ops were originally addeed. I'll cc you to that.
> > > 
> > > We do have a similar case currently in i.MX6, Nokia N9 (OMAP3) and on some
> 
> I meant to say Nokia N900, not N9.

Nice, I have a N900 :-)

> 
> -- 
> Kind regards,
> 
> Sakari Ailus
> e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

-- 
Regards,
Niklas Söderlund