Re: soc-camera and focuser vcm devices

Re: soc-camera and focuser vcm devices

[Guennadi Liakhovetski]

On Tue, 4 Nov 2014, Andrew Chew wrote:

> Hello, Guennadi
> 
> I was wondering if you can provide some advice as to how focuser vcm 
> devices would fit into the soc-camera framework.  We have a raw sensor 
> (an IMX219) with an AD5823 VCM (it's just a simple I2C device) to 
> provide variable focus.
> 
> I currently have the sensor and camera host driver instantiated via 
> device tree, following the guidelines in 
> Documentation/devicetree/bindings/media/video-interfaces.txt, by using 
> the of_graph stuff, and that all works fine.  However, I'm not sure what 
> the proper way is to associate a focuser with that particular image 
> pipeline, and I couldn't find any examples to draw from.

As discussed in an earlier thread [1], in principle soc-camera has support 
for attaching multiple subdevices to a camera-host (camera DMA engine) 
driver. In [2] you can see how I initially implemented DT for soc-camera, 
which also supported multiple subdevices. Beware, very old code. Some more 
comments are in [3].

Thanks
Guennadi

[1] http://www.spinics.net/lists/linux-sh/msg31436.html - see comment, 
following "Hm, I think, this isn't quite correct."

[2] http://marc.info/?l=linux-sh&m=134875489304837&w=1

[3] https://www.mail-archive.com/linux-media@vger.kernel.org/msg70514.html

> 
> Any help/guidance would be appreciated!
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information.  Any unauthorized review, use, disclosure or distribution
> is prohibited.  If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------
> 

RE: soc-camera and focuser vcm devices

[Andrew Chew]

> On Tue, 4 Nov 2014, Andrew Chew wrote:
> 
> > Hello, Guennadi
> >
> > I was wondering if you can provide some advice as to how focuser vcm
> > devices would fit into the soc-camera framework.  We have a raw sensor
> > (an IMX219) with an AD5823 VCM (it's just a simple I2C device) to
> > provide variable focus.
> >
> > I currently have the sensor and camera host driver instantiated via
> > device tree, following the guidelines in
> > Documentation/devicetree/bindings/media/video-interfaces.txt, by using
> > the of_graph stuff, and that all works fine.  However, I'm not sure
> > what the proper way is to associate a focuser with that particular
> > image pipeline, and I couldn't find any examples to draw from.
> 
> As discussed in an earlier thread [1], in principle soc-camera has support for
> attaching multiple subdevices to a camera-host (camera DMA engine) driver.
> In [2] you can see how I initially implemented DT for soc-camera, which also
> supported multiple subdevices. Beware, very old code. Some more
> comments are in [3].
> 
> Thanks
> Guennadi
> 
> [1] http://www.spinics.net/lists/linux-sh/msg31436.html - see comment,
> following "Hm, I think, this isn't quite correct."
> 
> [2] http://marc.info/?l=linux-sh&m=134875489304837&w=1
> 
> [3] https://www.mail-archive.com/linux-
> media@vger.kernel.org/msg70514.html

I think I get it, but just to clarify...currently, I'm going with the patch in [1], and I currently have multiple subdevs (sensors) hooked up and working with my soc-camera host driver.  [2] was an alternate implementation, right? (as in, I don't need it).

So I understand that I can just hook up the focuser and flash subdevs to the camera host via device tree, in the same exact way of using the of_graph stuff to hook up the sensor subdev.  My question is, is it then left up to the camera host driver to make sense of which subdevs do what?  Or should/will there be a common mechanism to bind ports on a camera host to a sensor, focuser, and flash under a common group which makes up an image pipeline?
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

RE: soc-camera and focuser vcm devices

[Guennadi Liakhovetski]

Hi Andrew,

On Fri, 7 Nov 2014, Andrew Chew wrote:

> > On Tue, 4 Nov 2014, Andrew Chew wrote:
> > 
> > > Hello, Guennadi
> > >
> > > I was wondering if you can provide some advice as to how focuser vcm
> > > devices would fit into the soc-camera framework.  We have a raw sensor
> > > (an IMX219) with an AD5823 VCM (it's just a simple I2C device) to
> > > provide variable focus.
> > >
> > > I currently have the sensor and camera host driver instantiated via
> > > device tree, following the guidelines in
> > > Documentation/devicetree/bindings/media/video-interfaces.txt, by using
> > > the of_graph stuff, and that all works fine.  However, I'm not sure
> > > what the proper way is to associate a focuser with that particular
> > > image pipeline, and I couldn't find any examples to draw from.
> > 
> > As discussed in an earlier thread [1], in principle soc-camera has support for
> > attaching multiple subdevices to a camera-host (camera DMA engine) driver.
> > In [2] you can see how I initially implemented DT for soc-camera, which also
> > supported multiple subdevices. Beware, very old code. Some more
> > comments are in [3].
> > 
> > Thanks
> > Guennadi
> > 
> > [1] http://www.spinics.net/lists/linux-sh/msg31436.html - see comment,
> > following "Hm, I think, this isn't quite correct."
> > 
> > [2] http://marc.info/?l=linux-sh&m=134875489304837&w=1
> > 
> > [3] https://www.mail-archive.com/linux-media@vger.kernel.org/msg70514.html
> 
> I think I get it, but just to clarify...currently, I'm going with the 
> patch in [1], and I currently have multiple subdevs (sensors) hooked up 
> and working with my soc-camera host driver.  [2] was an alternate 
> implementation, right? (as in, I don't need it).

No. [1] you don't need, it is already in the mainline, resp. its slightly 
updated version. I only quoted that thread for you above to read 
explanations about why that implementation didn't support multiple 
subdevices per host. To fix that support for multiple devices should be 
added to the current soc-camera mainline. To help with this task I quoted 
[2], which is my very early implementation attempt. [3] is just some more 
reading for general understanding.

> So I understand that I can just hook up the focuser and flash subdevs to 
> the camera host via device tree, in the same exact way of using the 
> of_graph stuff to hook up the sensor subdev.  My question is, is it then 
> left up to the camera host driver to make sense of which subdevs do 
> what?  Or should/will there be a common mechanism to bind ports on a 
> camera host to a sensor, focuser, and flash under a common group which 
> makes up an image pipeline?

Exactly, that's why you need multiple subdevices in groups, to have the 
whole pipeline probe only after all subdevices are available.

Thanks
Guennadi

RE: soc-camera and focuser vcm devices

[Andrew Chew]

> > I think I get it, but just to clarify...currently, I'm going with the
> > patch in [1], and I currently have multiple subdevs (sensors) hooked
> > up and working with my soc-camera host driver.  [2] was an alternate
> > implementation, right? (as in, I don't need it).
> 
> No. [1] you don't need, it is already in the mainline, resp. its slightly updated
> version. I only quoted that thread for you above to read explanations about
> why that implementation didn't support multiple subdevices per host. To fix
> that support for multiple devices should be added to the current soc-camera
> mainline. To help with this task I quoted [2], which is my very early
> implementation attempt. [3] is just some more reading for general
> understanding.
> 
> > So I understand that I can just hook up the focuser and flash subdevs
> > to the camera host via device tree, in the same exact way of using the
> > of_graph stuff to hook up the sensor subdev.  My question is, is it
> > then left up to the camera host driver to make sense of which subdevs
> > do what?  Or should/will there be a common mechanism to bind ports on
> > a camera host to a sensor, focuser, and flash under a common group
> > which makes up an image pipeline?
> 
> Exactly, that's why you need multiple subdevices in groups, to have the
> whole pipeline probe only after all subdevices are available.

Thanks, Guennadi.  I think I'm still misunderstanding something, since I basically took
[1], and am now (with a tiny modification of removing the hardcoded snippet of code
that explicitly prevented multiple subdevs from getting probed) able to instantiate
two sensors, both going through the camera host driver (i.e. I have /dev/video0 and
/dev/video1 both able to capture from each sensor, going through the same camera
host).

The part I don't understand is specifically how subdevs like focusers and flash
devices are supposed to work.  In examining the behavior of soc-camera more
closely after I add my focuser subdev, it seems that there is this general assumption
that all subdevs are image capture devices (because the subdev probe process
finishes around soc_camera_probe_finish(), which tries (and fails with the focuser)
to get supported capture formats.

But maybe I'm misunderstanding how focuser and flash devices are intended to be
Implemented?  I was under the impression that a focuser driver looks pretty much like
a sensor driver (as in, it eventually calls v4l2_i2c_subdev_init() and
v4l2_async_register_subdev().  In my case, the focuser's v4l2_async_register_subdev()
call fails because soc_camera_probe_finish() failed its soc_camera_init_user_formats().

Soc_camera_init_user_formats() calls into the camera host driver's get_formats(), which
tries to call the subdev's enum_mbus_fmt method.  Of course, the focuser subdev
doesn't implement this method, so it fails.  Even if I changed the camera host driver so
that it does something special with this subdev, the end result seems to me that we'll
get a /dev/video2 for the focuser, which isn't the behavior we want (we want the focuser
subdev to be available to the camera host, but it doesn't represent its own image
capture entity).

So what is the proper thing to do with the focuser driver to get it somehow aggregated
with a particular sensor subdev?  Or is this something that the soc-camera framework
doesn't define?  I don't see any example focuser drivers to go by, so this is all new to me.
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------