Userspace API for controlling the focus of the Surface Go [2] main/back-camera

Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Hans de Goede]

Hi All,

With my (and Dan's) kernel patch-series to enable the back camera on
the Surface Go shaping up (and hopefully going upstream soon),
the next step is to enable control of the focus lens for the back
camera.

The focus is controlled through a separate i2c-client which is
described by a 2nd I2cSerialBusV2 resource entry in the ACPI
device for the ov8865 sensor. By default the kernel only instantiates
an i2c-client for the first I2cSerialBusV2 resource entry for an
ACPI device, getting an i2c-client for the 2nd one is easy and
out of scope for this discussion.

The question which I have is, assuming we have the 2nd i2c-client
instantiated and we have a i2c-driver binding to it, how do we
represent the focus control to userspace.

I see 2 possible directions we can go here:

1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
list of the sensor. AFAIK we don't have infra for this atm, but
we could add some generic mechanism to do this to the v4l2-ctrls
core. IMHO from a userspace pov this is the cleanest, but at the
cost of some extra work / possible ugliness on the kernel side.

2. Register a separate v4l2_subdev for the focus-ctrl and in
some way provide information to userspace to which sensor this
belongs.

I believe that both are valid approaches. So before diving into
this I wonder what others are thinking about this.

Specific questions:

1. Hans Verkuil, what do you think about adding
support for another driver to inject ctrls into the ctrl
list of another v4l2(sub)dev ? Maybe something like this
already exists ? If not do you think this is feasible
and desirable to add ?

2. If we go with a separate v4l2_subdev, how do we communicate
to which sensor the focus-control belongs to userspace ?

Regards,

Hans

Re: Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Hans Verkuil]

On 25/10/2021 12:06, Hans de Goede wrote:
> Hi All,
> 
> With my (and Dan's) kernel patch-series to enable the back camera on
> the Surface Go shaping up (and hopefully going upstream soon),
> the next step is to enable control of the focus lens for the back
> camera.
> 
> The focus is controlled through a separate i2c-client which is
> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
> device for the ov8865 sensor. By default the kernel only instantiates
> an i2c-client for the first I2cSerialBusV2 resource entry for an
> ACPI device, getting an i2c-client for the 2nd one is easy and
> out of scope for this discussion.
> 
> The question which I have is, assuming we have the 2nd i2c-client
> instantiated and we have a i2c-driver binding to it, how do we
> represent the focus control to userspace.
> 
> I see 2 possible directions we can go here:
> 
> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
> list of the sensor. AFAIK we don't have infra for this atm, but
> we could add some generic mechanism to do this to the v4l2-ctrls
> core. IMHO from a userspace pov this is the cleanest, but at the
> cost of some extra work / possible ugliness on the kernel side.
> 
> 2. Register a separate v4l2_subdev for the focus-ctrl and in
> some way provide information to userspace to which sensor this
> belongs.
> 
> I believe that both are valid approaches. So before diving into
> this I wonder what others are thinking about this.
> 
> Specific questions:
> 
> 1. Hans Verkuil, what do you think about adding
> support for another driver to inject ctrls into the ctrl
> list of another v4l2(sub)dev ? Maybe something like this
> already exists ? If not do you think this is feasible
> and desirable to add ?
> 
> 2. If we go with a separate v4l2_subdev, how do we communicate
> to which sensor the focus-control belongs to userspace ?

What is the bridge driver that controls the sensor? I would need to
know a bit more about the architecture.

Is it MC-centric? Or is everything controlled through a video device?

In the latter case you want the video device to inherit the controls of
the sensor and the focus sub-devices, that's supported by the control
framework.

In the MC-centric case you probably want to have libcamera support that
can tie the focus subdev and the sensor subdev together.

v4l2_async_nf_parse_fwnode_sensor() supports linking LEDs or lens-focus
devices together with the sensor, so that's how a focus device can be
associated with a sensor at the ACPI/DT level. So support for 2) is
already available. A separate subdev is in my view certainly the correct
approach.

Regards,

	Hans

Re: Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Laurent Pinchart]

Hi Hans,

On Mon, Oct 25, 2021 at 12:25:26PM +0200, Hans Verkuil wrote:
> On 25/10/2021 12:06, Hans de Goede wrote:
> > Hi All,
> > 
> > With my (and Dan's) kernel patch-series to enable the back camera on
> > the Surface Go shaping up (and hopefully going upstream soon),
> > the next step is to enable control of the focus lens for the back
> > camera.
> > 
> > The focus is controlled through a separate i2c-client which is
> > described by a 2nd I2cSerialBusV2 resource entry in the ACPI
> > device for the ov8865 sensor. By default the kernel only instantiates
> > an i2c-client for the first I2cSerialBusV2 resource entry for an
> > ACPI device, getting an i2c-client for the 2nd one is easy and
> > out of scope for this discussion.
> > 
> > The question which I have is, assuming we have the 2nd i2c-client
> > instantiated and we have a i2c-driver binding to it, how do we
> > represent the focus control to userspace.
> > 
> > I see 2 possible directions we can go here:
> > 
> > 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
> > list of the sensor. AFAIK we don't have infra for this atm, but
> > we could add some generic mechanism to do this to the v4l2-ctrls
> > core. IMHO from a userspace pov this is the cleanest, but at the
> > cost of some extra work / possible ugliness on the kernel side.
> > 
> > 2. Register a separate v4l2_subdev for the focus-ctrl and in
> > some way provide information to userspace to which sensor this
> > belongs.
> > 
> > I believe that both are valid approaches. So before diving into
> > this I wonder what others are thinking about this.
> > 
> > Specific questions:
> > 
> > 1. Hans Verkuil, what do you think about adding
> > support for another driver to inject ctrls into the ctrl
> > list of another v4l2(sub)dev ? Maybe something like this
> > already exists ? If not do you think this is feasible
> > and desirable to add ?
> > 
> > 2. If we go with a separate v4l2_subdev, how do we communicate
> > to which sensor the focus-control belongs to userspace ?
> 
> What is the bridge driver that controls the sensor? I would need to
> know a bit more about the architecture.

Intel CIO2 (on a Windows-based machine with brain-dead ACPI design,
unfortunately).

> Is it MC-centric? Or is everything controlled through a video device?

MC-centric.

> In the latter case you want the video device to inherit the controls of
> the sensor and the focus sub-devices, that's supported by the control
> framework.
> 
> In the MC-centric case you probably want to have libcamera support that
> can tie the focus subdev and the sensor subdev together.

libcamera needs to associate the subdevs for the camera sensor and the
lens controller, so we need that information to be passed by the kernel
to userspace (there are two camera sensors in that machine).

> v4l2_async_nf_parse_fwnode_sensor() supports linking LEDs or lens-focus
> devices together with the sensor, so that's how a focus device can be
> associated with a sensor at the ACPI/DT level. So support for 2) is
> already available. A separate subdev is in my view certainly the correct
> approach.

That functions add subdev async notifier entries for the lens controller
and the flash controller, but as far as I know, we don't convey the
information to userspace.

-- 
Regards,

Laurent Pinchart

Re: Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Laurent Pinchart]

Hi Hans,

On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
> Hi All,
> 
> With my (and Dan's) kernel patch-series to enable the back camera on
> the Surface Go shaping up (and hopefully going upstream soon),
> the next step is to enable control of the focus lens for the back
> camera.
> 
> The focus is controlled through a separate i2c-client which is
> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
> device for the ov8865 sensor. By default the kernel only instantiates
> an i2c-client for the first I2cSerialBusV2 resource entry for an
> ACPI device, getting an i2c-client for the 2nd one is easy and
> out of scope for this discussion.
> 
> The question which I have is, assuming we have the 2nd i2c-client
> instantiated and we have a i2c-driver binding to it, how do we
> represent the focus control to userspace.
> 
> I see 2 possible directions we can go here:
> 
> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
> list of the sensor. AFAIK we don't have infra for this atm, but
> we could add some generic mechanism to do this to the v4l2-ctrls
> core. IMHO from a userspace pov this is the cleanest, but at the
> cost of some extra work / possible ugliness on the kernel side.
> 
> 2. Register a separate v4l2_subdev for the focus-ctrl and in
> some way provide information to userspace to which sensor this
> belongs.

The second approach is what V4L2 does already. We have a set of drivers
for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
drivers/media/i2c/).

> I believe that both are valid approaches. So before diving into
> this I wonder what others are thinking about this.
> 
> Specific questions:
> 
> 1. Hans Verkuil, what do you think about adding
> support for another driver to inject ctrls into the ctrl
> list of another v4l2(sub)dev ? Maybe something like this
> already exists ? If not do you think this is feasible
> and desirable to add ?
> 
> 2. If we go with a separate v4l2_subdev, how do we communicate
> to which sensor the focus-control belongs to userspace ?

The information was initially envisioned to be conveyed to userspace
through the media controller API, using the entity group ID to group the
camera sensor, lens controller and flash controller, but the
media_entity_desc.group_id field is now obsolete. No other mechanism
exist to replace that as far as I know, so we'll have to create
something. There have been some talks about using a special kind of link
to expose the relationship between the camera sensor and other
components.

-- 
Regards,

Laurent Pinchart

Re: Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Hans Verkuil]

On 25/10/2021 12:30, Laurent Pinchart wrote:
> Hi Hans,
> 
> On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
>> Hi All,
>>
>> With my (and Dan's) kernel patch-series to enable the back camera on
>> the Surface Go shaping up (and hopefully going upstream soon),
>> the next step is to enable control of the focus lens for the back
>> camera.
>>
>> The focus is controlled through a separate i2c-client which is
>> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
>> device for the ov8865 sensor. By default the kernel only instantiates
>> an i2c-client for the first I2cSerialBusV2 resource entry for an
>> ACPI device, getting an i2c-client for the 2nd one is easy and
>> out of scope for this discussion.
>>
>> The question which I have is, assuming we have the 2nd i2c-client
>> instantiated and we have a i2c-driver binding to it, how do we
>> represent the focus control to userspace.
>>
>> I see 2 possible directions we can go here:
>>
>> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
>> list of the sensor. AFAIK we don't have infra for this atm, but
>> we could add some generic mechanism to do this to the v4l2-ctrls
>> core. IMHO from a userspace pov this is the cleanest, but at the
>> cost of some extra work / possible ugliness on the kernel side.
>>
>> 2. Register a separate v4l2_subdev for the focus-ctrl and in
>> some way provide information to userspace to which sensor this
>> belongs.
> 
> The second approach is what V4L2 does already. We have a set of drivers
> for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
> drivers/media/i2c/).
> 
>> I believe that both are valid approaches. So before diving into
>> this I wonder what others are thinking about this.
>>
>> Specific questions:
>>
>> 1. Hans Verkuil, what do you think about adding
>> support for another driver to inject ctrls into the ctrl
>> list of another v4l2(sub)dev ? Maybe something like this
>> already exists ? If not do you think this is feasible
>> and desirable to add ?
>>
>> 2. If we go with a separate v4l2_subdev, how do we communicate
>> to which sensor the focus-control belongs to userspace ?
> 
> The information was initially envisioned to be conveyed to userspace
> through the media controller API, using the entity group ID to group the
> camera sensor, lens controller and flash controller, but the
> media_entity_desc.group_id field is now obsolete. No other mechanism
> exist to replace that as far as I know, so we'll have to create
> something. There have been some talks about using a special kind of link
> to expose the relationship between the camera sensor and other
> components.
> 

I thought this was implemented: there should be an interface link from the
sensor entity to the subdev for the flash or focus control.

To my knowledge, this is all available.

Regards,

	Hans

Re: Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Laurent Pinchart]

Hi Hans,

On Mon, Oct 25, 2021 at 12:47:05PM +0200, Hans Verkuil wrote:
> On 25/10/2021 12:30, Laurent Pinchart wrote:
> > On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
> >> Hi All,
> >>
> >> With my (and Dan's) kernel patch-series to enable the back camera on
> >> the Surface Go shaping up (and hopefully going upstream soon),
> >> the next step is to enable control of the focus lens for the back
> >> camera.
> >>
> >> The focus is controlled through a separate i2c-client which is
> >> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
> >> device for the ov8865 sensor. By default the kernel only instantiates
> >> an i2c-client for the first I2cSerialBusV2 resource entry for an
> >> ACPI device, getting an i2c-client for the 2nd one is easy and
> >> out of scope for this discussion.
> >>
> >> The question which I have is, assuming we have the 2nd i2c-client
> >> instantiated and we have a i2c-driver binding to it, how do we
> >> represent the focus control to userspace.
> >>
> >> I see 2 possible directions we can go here:
> >>
> >> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
> >> list of the sensor. AFAIK we don't have infra for this atm, but
> >> we could add some generic mechanism to do this to the v4l2-ctrls
> >> core. IMHO from a userspace pov this is the cleanest, but at the
> >> cost of some extra work / possible ugliness on the kernel side.
> >>
> >> 2. Register a separate v4l2_subdev for the focus-ctrl and in
> >> some way provide information to userspace to which sensor this
> >> belongs.
> > 
> > The second approach is what V4L2 does already. We have a set of drivers
> > for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
> > drivers/media/i2c/).
> > 
> >> I believe that both are valid approaches. So before diving into
> >> this I wonder what others are thinking about this.
> >>
> >> Specific questions:
> >>
> >> 1. Hans Verkuil, what do you think about adding
> >> support for another driver to inject ctrls into the ctrl
> >> list of another v4l2(sub)dev ? Maybe something like this
> >> already exists ? If not do you think this is feasible
> >> and desirable to add ?
> >>
> >> 2. If we go with a separate v4l2_subdev, how do we communicate
> >> to which sensor the focus-control belongs to userspace ?
> > 
> > The information was initially envisioned to be conveyed to userspace
> > through the media controller API, using the entity group ID to group the
> > camera sensor, lens controller and flash controller, but the
> > media_entity_desc.group_id field is now obsolete. No other mechanism
> > exist to replace that as far as I know, so we'll have to create
> > something. There have been some talks about using a special kind of link
> > to expose the relationship between the camera sensor and other
> > components.
> 
> I thought this was implemented: there should be an interface link from the
> sensor entity to the subdev for the flash or focus control.

Interface links are only meant to link entities to interfaces, not
entities to entities. media_create_intf_link() is only called for that
purpose today.

> To my knowledge, this is all available.

To my knowledge, it isn't :-) I'd be happy to be proven wrong though.

-- 
Regards,

Laurent Pinchart

Re: Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Hans Verkuil]

On 25/10/2021 12:55, Laurent Pinchart wrote:
> Hi Hans,
> 
> On Mon, Oct 25, 2021 at 12:47:05PM +0200, Hans Verkuil wrote:
>> On 25/10/2021 12:30, Laurent Pinchart wrote:
>>> On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
>>>> Hi All,
>>>>
>>>> With my (and Dan's) kernel patch-series to enable the back camera on
>>>> the Surface Go shaping up (and hopefully going upstream soon),
>>>> the next step is to enable control of the focus lens for the back
>>>> camera.
>>>>
>>>> The focus is controlled through a separate i2c-client which is
>>>> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
>>>> device for the ov8865 sensor. By default the kernel only instantiates
>>>> an i2c-client for the first I2cSerialBusV2 resource entry for an
>>>> ACPI device, getting an i2c-client for the 2nd one is easy and
>>>> out of scope for this discussion.
>>>>
>>>> The question which I have is, assuming we have the 2nd i2c-client
>>>> instantiated and we have a i2c-driver binding to it, how do we
>>>> represent the focus control to userspace.
>>>>
>>>> I see 2 possible directions we can go here:
>>>>
>>>> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
>>>> list of the sensor. AFAIK we don't have infra for this atm, but
>>>> we could add some generic mechanism to do this to the v4l2-ctrls
>>>> core. IMHO from a userspace pov this is the cleanest, but at the
>>>> cost of some extra work / possible ugliness on the kernel side.
>>>>
>>>> 2. Register a separate v4l2_subdev for the focus-ctrl and in
>>>> some way provide information to userspace to which sensor this
>>>> belongs.
>>>
>>> The second approach is what V4L2 does already. We have a set of drivers
>>> for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
>>> drivers/media/i2c/).
>>>
>>>> I believe that both are valid approaches. So before diving into
>>>> this I wonder what others are thinking about this.
>>>>
>>>> Specific questions:
>>>>
>>>> 1. Hans Verkuil, what do you think about adding
>>>> support for another driver to inject ctrls into the ctrl
>>>> list of another v4l2(sub)dev ? Maybe something like this
>>>> already exists ? If not do you think this is feasible
>>>> and desirable to add ?
>>>>
>>>> 2. If we go with a separate v4l2_subdev, how do we communicate
>>>> to which sensor the focus-control belongs to userspace ?
>>>
>>> The information was initially envisioned to be conveyed to userspace
>>> through the media controller API, using the entity group ID to group the
>>> camera sensor, lens controller and flash controller, but the
>>> media_entity_desc.group_id field is now obsolete. No other mechanism
>>> exist to replace that as far as I know, so we'll have to create
>>> something. There have been some talks about using a special kind of link
>>> to expose the relationship between the camera sensor and other
>>> components.
>>
>> I thought this was implemented: there should be an interface link from the
>> sensor entity to the subdev for the flash or focus control.
> 
> Interface links are only meant to link entities to interfaces, not
> entities to entities. media_create_intf_link() is only called for that
> purpose today.
> 
>> To my knowledge, this is all available.
> 
> To my knowledge, it isn't :-) I'd be happy to be proven wrong though.
> 

I think you are correct. The link is there on the DT-level, but it is not
discoverable in the media topology.

I have used the DT information in the past, but that was for a video-centric
driver. And the focus controls were nicely inherited by the video device.

Regards,

	Hans

Re: [libcamera-devel] Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Dave Stevenson]

Hi All.

On Mon, 25 Oct 2021 at 11:47, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 25/10/2021 12:30, Laurent Pinchart wrote:
> > Hi Hans,
> >
> > On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
> >> Hi All,
> >>
> >> With my (and Dan's) kernel patch-series to enable the back camera on
> >> the Surface Go shaping up (and hopefully going upstream soon),
> >> the next step is to enable control of the focus lens for the back
> >> camera.
> >>
> >> The focus is controlled through a separate i2c-client which is
> >> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
> >> device for the ov8865 sensor. By default the kernel only instantiates
> >> an i2c-client for the first I2cSerialBusV2 resource entry for an
> >> ACPI device, getting an i2c-client for the 2nd one is easy and
> >> out of scope for this discussion.
> >>
> >> The question which I have is, assuming we have the 2nd i2c-client
> >> instantiated and we have a i2c-driver binding to it, how do we
> >> represent the focus control to userspace.
> >>
> >> I see 2 possible directions we can go here:
> >>
> >> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
> >> list of the sensor. AFAIK we don't have infra for this atm, but
> >> we could add some generic mechanism to do this to the v4l2-ctrls
> >> core. IMHO from a userspace pov this is the cleanest, but at the
> >> cost of some extra work / possible ugliness on the kernel side.
> >>
> >> 2. Register a separate v4l2_subdev for the focus-ctrl and in
> >> some way provide information to userspace to which sensor this
> >> belongs.
> >
> > The second approach is what V4L2 does already. We have a set of drivers
> > for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
> > drivers/media/i2c/).
> >
> >> I believe that both are valid approaches. So before diving into
> >> this I wonder what others are thinking about this.
> >>
> >> Specific questions:
> >>
> >> 1. Hans Verkuil, what do you think about adding
> >> support for another driver to inject ctrls into the ctrl
> >> list of another v4l2(sub)dev ? Maybe something like this
> >> already exists ? If not do you think this is feasible
> >> and desirable to add ?
> >>
> >> 2. If we go with a separate v4l2_subdev, how do we communicate
> >> to which sensor the focus-control belongs to userspace ?
> >
> > The information was initially envisioned to be conveyed to userspace
> > through the media controller API, using the entity group ID to group the
> > camera sensor, lens controller and flash controller, but the
> > media_entity_desc.group_id field is now obsolete. No other mechanism
> > exist to replace that as far as I know, so we'll have to create
> > something. There have been some talks about using a special kind of link
> > to expose the relationship between the camera sensor and other
> > components.
> >
>
> I thought this was implemented: there should be an interface link from the
> sensor entity to the subdev for the flash or focus control.
>
> To my knowledge, this is all available.

We've been looking at focus and AF over the last few weeks, although
under device tree instead of ACPI.

With DT we have a lens driver bound to the sensor driver by giving the
sensor a lens-focus = <&vcm>; entry.
Media controller then reports it all under one entity, but it may be
true that we don't have a linking between the lens subdev and sensor.

pi@raspberrypi:~ $ media-ctl -p -d /dev/media2
Media controller API version 5.10.74

Media device information
------------------------
driver          unicam
model           unicam
serial
bus info        platform:fe801000.csi
hw revision     0x0
driver version  5.10.74

Device topology
- entity 1: imx477 10-001a (2 pads, 2 links)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev0
    pad0: Source
        [fmt:SRGGB12_1X12/4056x3040 field:none colorspace:unknown
xfer:none ycbcr:601 quantization:full-range
         crop.bounds:(8,16)/4056x3040
         crop:(8,16)/4056x3040]
        -> "unicam-image":0 [ENABLED,IMMUTABLE]
    pad1: Source
        [fmt:unknown/16384x1 field:none
         crop.bounds:(8,16)/4056x3040
         crop:(8,16)/4056x3040]
        -> "unicam-embedded":0 [ENABLED,IMMUTABLE]

- entity 4: ad5398 focus (0 pad, 0 link)
            type V4L2 subdev subtype Lens flags 0
            device node name /dev/v4l-subdev1

- entity 5: unicam-image (1 pad, 1 link)
            type Node subtype V4L flags 1
            device node name /dev/video0
    pad0: Sink
        <- "imx477 10-001a":0 [ENABLED,IMMUTABLE]

- entity 11: unicam-embedded (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
    pad0: Sink
        <- "imx477 10-001a":1 [ENABLED,IMMUTABLE]

The Pi may be slightly different from other platforms in that if you
enable the second CSI2 interface then it'll be a totally separate
/dev/media node, so we can view it as one overall entity even if not
identified as such.

The above was grabbed with an IMX477 with AF module. I haven't pushed
that configuration anywhere, but IMX135 with lens driver is at
https://github.com/raspberrypi/linux/pull/4612/

  Dave

Re: [libcamera-devel] Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Hans Verkuil]

On 25/10/2021 13:03, Dave Stevenson wrote:
> Hi All.
> 
> On Mon, 25 Oct 2021 at 11:47, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> On 25/10/2021 12:30, Laurent Pinchart wrote:
>>> Hi Hans,
>>>
>>> On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
>>>> Hi All,
>>>>
>>>> With my (and Dan's) kernel patch-series to enable the back camera on
>>>> the Surface Go shaping up (and hopefully going upstream soon),
>>>> the next step is to enable control of the focus lens for the back
>>>> camera.
>>>>
>>>> The focus is controlled through a separate i2c-client which is
>>>> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
>>>> device for the ov8865 sensor. By default the kernel only instantiates
>>>> an i2c-client for the first I2cSerialBusV2 resource entry for an
>>>> ACPI device, getting an i2c-client for the 2nd one is easy and
>>>> out of scope for this discussion.
>>>>
>>>> The question which I have is, assuming we have the 2nd i2c-client
>>>> instantiated and we have a i2c-driver binding to it, how do we
>>>> represent the focus control to userspace.
>>>>
>>>> I see 2 possible directions we can go here:
>>>>
>>>> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
>>>> list of the sensor. AFAIK we don't have infra for this atm, but
>>>> we could add some generic mechanism to do this to the v4l2-ctrls
>>>> core. IMHO from a userspace pov this is the cleanest, but at the
>>>> cost of some extra work / possible ugliness on the kernel side.
>>>>
>>>> 2. Register a separate v4l2_subdev for the focus-ctrl and in
>>>> some way provide information to userspace to which sensor this
>>>> belongs.
>>>
>>> The second approach is what V4L2 does already. We have a set of drivers
>>> for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
>>> drivers/media/i2c/).
>>>
>>>> I believe that both are valid approaches. So before diving into
>>>> this I wonder what others are thinking about this.
>>>>
>>>> Specific questions:
>>>>
>>>> 1. Hans Verkuil, what do you think about adding
>>>> support for another driver to inject ctrls into the ctrl
>>>> list of another v4l2(sub)dev ? Maybe something like this
>>>> already exists ? If not do you think this is feasible
>>>> and desirable to add ?
>>>>
>>>> 2. If we go with a separate v4l2_subdev, how do we communicate
>>>> to which sensor the focus-control belongs to userspace ?
>>>
>>> The information was initially envisioned to be conveyed to userspace
>>> through the media controller API, using the entity group ID to group the
>>> camera sensor, lens controller and flash controller, but the
>>> media_entity_desc.group_id field is now obsolete. No other mechanism
>>> exist to replace that as far as I know, so we'll have to create
>>> something. There have been some talks about using a special kind of link
>>> to expose the relationship between the camera sensor and other
>>> components.
>>>
>>
>> I thought this was implemented: there should be an interface link from the
>> sensor entity to the subdev for the flash or focus control.
>>
>> To my knowledge, this is all available.
> 
> We've been looking at focus and AF over the last few weeks, although
> under device tree instead of ACPI.
> 
> With DT we have a lens driver bound to the sensor driver by giving the
> sensor a lens-focus = <&vcm>; entry.
> Media controller then reports it all under one entity, but it may be
> true that we don't have a linking between the lens subdev and sensor.
> 
> pi@raspberrypi:~ $ media-ctl -p -d /dev/media2
> Media controller API version 5.10.74
> 
> Media device information
> ------------------------
> driver          unicam
> model           unicam
> serial
> bus info        platform:fe801000.csi
> hw revision     0x0
> driver version  5.10.74
> 
> Device topology
> - entity 1: imx477 10-001a (2 pads, 2 links)
>             type V4L2 subdev subtype Sensor flags 0
>             device node name /dev/v4l-subdev0
>     pad0: Source
>         [fmt:SRGGB12_1X12/4056x3040 field:none colorspace:unknown
> xfer:none ycbcr:601 quantization:full-range
>          crop.bounds:(8,16)/4056x3040
>          crop:(8,16)/4056x3040]
>         -> "unicam-image":0 [ENABLED,IMMUTABLE]
>     pad1: Source
>         [fmt:unknown/16384x1 field:none
>          crop.bounds:(8,16)/4056x3040
>          crop:(8,16)/4056x3040]
>         -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> 
> - entity 4: ad5398 focus (0 pad, 0 link)
>             type V4L2 subdev subtype Lens flags 0
>             device node name /dev/v4l-subdev1
> 
> - entity 5: unicam-image (1 pad, 1 link)
>             type Node subtype V4L flags 1
>             device node name /dev/video0
>     pad0: Sink
>         <- "imx477 10-001a":0 [ENABLED,IMMUTABLE]
> 
> - entity 11: unicam-embedded (1 pad, 1 link)
>              type Node subtype V4L flags 0
>              device node name /dev/video1
>     pad0: Sink
>         <- "imx477 10-001a":1 [ENABLED,IMMUTABLE]
> 
> The Pi may be slightly different from other platforms in that if you
> enable the second CSI2 interface then it'll be a totally separate
> /dev/media node, so we can view it as one overall entity even if not
> identified as such.
> 
> The above was grabbed with an IMX477 with AF module. I haven't pushed
> that configuration anywhere, but IMX135 with lens driver is at
> https://github.com/raspberrypi/linux/pull/4612/
> 

What is the output of:

v4l2-compliance -M /dev/media2 --verbose

That will also report any interface links.

Regards,

	Hans

Re: [libcamera-devel] Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Dave Stevenson]

Hi Hans

On Mon, 25 Oct 2021 at 12:10, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 25/10/2021 13:03, Dave Stevenson wrote:
> > Hi All.
> >
> > On Mon, 25 Oct 2021 at 11:47, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> >>
> >> On 25/10/2021 12:30, Laurent Pinchart wrote:
> >>> Hi Hans,
> >>>
> >>> On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
> >>>> Hi All,
> >>>>
> >>>> With my (and Dan's) kernel patch-series to enable the back camera on
> >>>> the Surface Go shaping up (and hopefully going upstream soon),
> >>>> the next step is to enable control of the focus lens for the back
> >>>> camera.
> >>>>
> >>>> The focus is controlled through a separate i2c-client which is
> >>>> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
> >>>> device for the ov8865 sensor. By default the kernel only instantiates
> >>>> an i2c-client for the first I2cSerialBusV2 resource entry for an
> >>>> ACPI device, getting an i2c-client for the 2nd one is easy and
> >>>> out of scope for this discussion.
> >>>>
> >>>> The question which I have is, assuming we have the 2nd i2c-client
> >>>> instantiated and we have a i2c-driver binding to it, how do we
> >>>> represent the focus control to userspace.
> >>>>
> >>>> I see 2 possible directions we can go here:
> >>>>
> >>>> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
> >>>> list of the sensor. AFAIK we don't have infra for this atm, but
> >>>> we could add some generic mechanism to do this to the v4l2-ctrls
> >>>> core. IMHO from a userspace pov this is the cleanest, but at the
> >>>> cost of some extra work / possible ugliness on the kernel side.
> >>>>
> >>>> 2. Register a separate v4l2_subdev for the focus-ctrl and in
> >>>> some way provide information to userspace to which sensor this
> >>>> belongs.
> >>>
> >>> The second approach is what V4L2 does already. We have a set of drivers
> >>> for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
> >>> drivers/media/i2c/).
> >>>
> >>>> I believe that both are valid approaches. So before diving into
> >>>> this I wonder what others are thinking about this.
> >>>>
> >>>> Specific questions:
> >>>>
> >>>> 1. Hans Verkuil, what do you think about adding
> >>>> support for another driver to inject ctrls into the ctrl
> >>>> list of another v4l2(sub)dev ? Maybe something like this
> >>>> already exists ? If not do you think this is feasible
> >>>> and desirable to add ?
> >>>>
> >>>> 2. If we go with a separate v4l2_subdev, how do we communicate
> >>>> to which sensor the focus-control belongs to userspace ?
> >>>
> >>> The information was initially envisioned to be conveyed to userspace
> >>> through the media controller API, using the entity group ID to group the
> >>> camera sensor, lens controller and flash controller, but the
> >>> media_entity_desc.group_id field is now obsolete. No other mechanism
> >>> exist to replace that as far as I know, so we'll have to create
> >>> something. There have been some talks about using a special kind of link
> >>> to expose the relationship between the camera sensor and other
> >>> components.
> >>>
> >>
> >> I thought this was implemented: there should be an interface link from the
> >> sensor entity to the subdev for the flash or focus control.
> >>
> >> To my knowledge, this is all available.
> >
> > We've been looking at focus and AF over the last few weeks, although
> > under device tree instead of ACPI.
> >
> > With DT we have a lens driver bound to the sensor driver by giving the
> > sensor a lens-focus = <&vcm>; entry.
> > Media controller then reports it all under one entity, but it may be
> > true that we don't have a linking between the lens subdev and sensor.
> >
> > pi@raspberrypi:~ $ media-ctl -p -d /dev/media2
> > Media controller API version 5.10.74
> >
> > Media device information
> > ------------------------
> > driver          unicam
> > model           unicam
> > serial
> > bus info        platform:fe801000.csi
> > hw revision     0x0
> > driver version  5.10.74
> >
> > Device topology
> > - entity 1: imx477 10-001a (2 pads, 2 links)
> >             type V4L2 subdev subtype Sensor flags 0
> >             device node name /dev/v4l-subdev0
> >     pad0: Source
> >         [fmt:SRGGB12_1X12/4056x3040 field:none colorspace:unknown
> > xfer:none ycbcr:601 quantization:full-range
> >          crop.bounds:(8,16)/4056x3040
> >          crop:(8,16)/4056x3040]
> >         -> "unicam-image":0 [ENABLED,IMMUTABLE]
> >     pad1: Source
> >         [fmt:unknown/16384x1 field:none
> >          crop.bounds:(8,16)/4056x3040
> >          crop:(8,16)/4056x3040]
> >         -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
> >
> > - entity 4: ad5398 focus (0 pad, 0 link)
> >             type V4L2 subdev subtype Lens flags 0
> >             device node name /dev/v4l-subdev1
> >
> > - entity 5: unicam-image (1 pad, 1 link)
> >             type Node subtype V4L flags 1
> >             device node name /dev/video0
> >     pad0: Sink
> >         <- "imx477 10-001a":0 [ENABLED,IMMUTABLE]
> >
> > - entity 11: unicam-embedded (1 pad, 1 link)
> >              type Node subtype V4L flags 0
> >              device node name /dev/video1
> >     pad0: Sink
> >         <- "imx477 10-001a":1 [ENABLED,IMMUTABLE]
> >
> > The Pi may be slightly different from other platforms in that if you
> > enable the second CSI2 interface then it'll be a totally separate
> > /dev/media node, so we can view it as one overall entity even if not
> > identified as such.
> >
> > The above was grabbed with an IMX477 with AF module. I haven't pushed
> > that configuration anywhere, but IMX135 with lens driver is at
> > https://github.com/raspberrypi/linux/pull/4612/
> >
>
> What is the output of:
>
> v4l2-compliance -M /dev/media2 --verbose
>
> That will also report any interface links.

pi@raspberrypi:~/nfs/v4l-utils/utils/v4l2-compliance $
./v4l2-compliance  -M /dev/media2 --verbose
v4l2-compliance 1.21.0-4859, 32 bits, 32-bit time_t
v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11

Compliance test for unicam device /dev/media2:

Media Driver Info:
    Driver name      : unicam
    Model            : unicam
    Serial           :
    Bus info         : platform:fe801000.csi
    Media version    : 5.10.74
    Hardware revision: 0x00000000 (0)
    Driver version   : 5.10.74

Required ioctls:
    test MEDIA_IOC_DEVICE_INFO: OK
    test invalid ioctls: OK

Allow for multiple opens:
    test second /dev/media2 open: OK
    test MEDIA_IOC_DEVICE_INFO: OK
    test for unlimited opens: OK

Media Controller ioctls:
        Entity: 0x00000001 (Name: 'imx477 10-001a', Function: Camera Sensor)
        Entity: 0x00000004 (Name: 'ad5398 focus', Function: Lens Controller)
        Entity: 0x00000005 (Name: 'unicam-image', Function: V4L2 I/O,
Flags: default)
        Entity: 0x0000000b (Name: 'unicam-embedded', Function: V4L2 I/O)
        Interface: 0x03000007 (Type: V4L Video, DevPath: /dev/video0)
        Interface: 0x0300000d (Type: V4L Video, DevPath: /dev/video1)
        Interface: 0x03000011 (Type: V4L Sub-Device, DevPath: /dev/v4l-subdev0)
        Interface: 0x03000013 (Type: V4L Sub-Device, DevPath: /dev/v4l-subdev1)
        Pad: 0x01000002 (0, imx477 10-001a, Source)
        Pad: 0x01000003 (1, imx477 10-001a, Source)
        Pad: 0x01000006 (0, unicam-image, Sink)
        Pad: 0x0100000c (0, unicam-embedded, Sink)
        Link: 0x02000008 (unicam-image to interface /dev/video0)
        Link: 0x02000009 (imx477 10-001a:0 -> unicam-image:0, Data,
Enabled, Immutable)
        Link: 0x0200000e (unicam-embedded to interface /dev/video1)
        Link: 0x0200000f (imx477 10-001a:1 -> unicam-embedded:0, Data,
Enabled, Immutable)
        Link: 0x02000012 (imx477 10-001a to interface /dev/v4l-subdev0)
        Link: 0x02000014 (ad5398 focus to interface /dev/v4l-subdev1)
    test MEDIA_IOC_G_TOPOLOGY: OK
    Entities: 4 Interfaces: 4 Pads: 4 Links: 6
        Entity: 0x00000001 (Name: 'imx477 10-001a', Type: Camera
Sensor, DevPath: /dev/v4l-subdev0)
        Entity: 0x00000004 (Name: 'ad5398 focus', Type: Lens
Controller, DevPath: /dev/v4l-subdev1)
        Entity: 0x00000005 (Name: 'unicam-image', Type: V4L2 I/O,
Flags: default, DevPath: /dev/video0)
        Entity: 0x0000000b (Name: 'unicam-embedded', Type: V4L2 I/O,
DevPath: /dev/video1)
    test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
    test MEDIA_IOC_SETUP_LINK: OK

Total for unicam device /dev/media2: 8, Succeeded: 8, Failed: 0, Warnings: 0

  Dave

Re: [libcamera-devel] Userspace API for controlling the focus of the Surface Go [2] main/back-camera

[Hans Verkuil]

On 25/10/2021 13:17, Dave Stevenson wrote:
> Hi Hans
> 
> On Mon, 25 Oct 2021 at 12:10, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> On 25/10/2021 13:03, Dave Stevenson wrote:
>>> Hi All.
>>>
>>> On Mon, 25 Oct 2021 at 11:47, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>>
>>>> On 25/10/2021 12:30, Laurent Pinchart wrote:
>>>>> Hi Hans,
>>>>>
>>>>> On Mon, Oct 25, 2021 at 12:06:30PM +0200, Hans de Goede wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> With my (and Dan's) kernel patch-series to enable the back camera on
>>>>>> the Surface Go shaping up (and hopefully going upstream soon),
>>>>>> the next step is to enable control of the focus lens for the back
>>>>>> camera.
>>>>>>
>>>>>> The focus is controlled through a separate i2c-client which is
>>>>>> described by a 2nd I2cSerialBusV2 resource entry in the ACPI
>>>>>> device for the ov8865 sensor. By default the kernel only instantiates
>>>>>> an i2c-client for the first I2cSerialBusV2 resource entry for an
>>>>>> ACPI device, getting an i2c-client for the 2nd one is easy and
>>>>>> out of scope for this discussion.
>>>>>>
>>>>>> The question which I have is, assuming we have the 2nd i2c-client
>>>>>> instantiated and we have a i2c-driver binding to it, how do we
>>>>>> represent the focus control to userspace.
>>>>>>
>>>>>> I see 2 possible directions we can go here:
>>>>>>
>>>>>> 1. Somehow inject an extra v4l2ctrl for this into the v4l2ctrl
>>>>>> list of the sensor. AFAIK we don't have infra for this atm, but
>>>>>> we could add some generic mechanism to do this to the v4l2-ctrls
>>>>>> core. IMHO from a userspace pov this is the cleanest, but at the
>>>>>> cost of some extra work / possible ugliness on the kernel side.
>>>>>>
>>>>>> 2. Register a separate v4l2_subdev for the focus-ctrl and in
>>>>>> some way provide information to userspace to which sensor this
>>>>>> belongs.
>>>>>
>>>>> The second approach is what V4L2 does already. We have a set of drivers
>>>>> for VCMs already (search for V4L2_CID_FOCUS_ABSOLUTE in
>>>>> drivers/media/i2c/).
>>>>>
>>>>>> I believe that both are valid approaches. So before diving into
>>>>>> this I wonder what others are thinking about this.
>>>>>>
>>>>>> Specific questions:
>>>>>>
>>>>>> 1. Hans Verkuil, what do you think about adding
>>>>>> support for another driver to inject ctrls into the ctrl
>>>>>> list of another v4l2(sub)dev ? Maybe something like this
>>>>>> already exists ? If not do you think this is feasible
>>>>>> and desirable to add ?
>>>>>>
>>>>>> 2. If we go with a separate v4l2_subdev, how do we communicate
>>>>>> to which sensor the focus-control belongs to userspace ?
>>>>>
>>>>> The information was initially envisioned to be conveyed to userspace
>>>>> through the media controller API, using the entity group ID to group the
>>>>> camera sensor, lens controller and flash controller, but the
>>>>> media_entity_desc.group_id field is now obsolete. No other mechanism
>>>>> exist to replace that as far as I know, so we'll have to create
>>>>> something. There have been some talks about using a special kind of link
>>>>> to expose the relationship between the camera sensor and other
>>>>> components.
>>>>>
>>>>
>>>> I thought this was implemented: there should be an interface link from the
>>>> sensor entity to the subdev for the flash or focus control.
>>>>
>>>> To my knowledge, this is all available.
>>>
>>> We've been looking at focus and AF over the last few weeks, although
>>> under device tree instead of ACPI.
>>>
>>> With DT we have a lens driver bound to the sensor driver by giving the
>>> sensor a lens-focus = <&vcm>; entry.
>>> Media controller then reports it all under one entity, but it may be
>>> true that we don't have a linking between the lens subdev and sensor.
>>>
>>> pi@raspberrypi:~ $ media-ctl -p -d /dev/media2
>>> Media controller API version 5.10.74
>>>
>>> Media device information
>>> ------------------------
>>> driver          unicam
>>> model           unicam
>>> serial
>>> bus info        platform:fe801000.csi
>>> hw revision     0x0
>>> driver version  5.10.74
>>>
>>> Device topology
>>> - entity 1: imx477 10-001a (2 pads, 2 links)
>>>             type V4L2 subdev subtype Sensor flags 0
>>>             device node name /dev/v4l-subdev0
>>>     pad0: Source
>>>         [fmt:SRGGB12_1X12/4056x3040 field:none colorspace:unknown
>>> xfer:none ycbcr:601 quantization:full-range
>>>          crop.bounds:(8,16)/4056x3040
>>>          crop:(8,16)/4056x3040]
>>>         -> "unicam-image":0 [ENABLED,IMMUTABLE]
>>>     pad1: Source
>>>         [fmt:unknown/16384x1 field:none
>>>          crop.bounds:(8,16)/4056x3040
>>>          crop:(8,16)/4056x3040]
>>>         -> "unicam-embedded":0 [ENABLED,IMMUTABLE]
>>>
>>> - entity 4: ad5398 focus (0 pad, 0 link)
>>>             type V4L2 subdev subtype Lens flags 0
>>>             device node name /dev/v4l-subdev1
>>>
>>> - entity 5: unicam-image (1 pad, 1 link)
>>>             type Node subtype V4L flags 1
>>>             device node name /dev/video0
>>>     pad0: Sink
>>>         <- "imx477 10-001a":0 [ENABLED,IMMUTABLE]
>>>
>>> - entity 11: unicam-embedded (1 pad, 1 link)
>>>              type Node subtype V4L flags 0
>>>              device node name /dev/video1
>>>     pad0: Sink
>>>         <- "imx477 10-001a":1 [ENABLED,IMMUTABLE]
>>>
>>> The Pi may be slightly different from other platforms in that if you
>>> enable the second CSI2 interface then it'll be a totally separate
>>> /dev/media node, so we can view it as one overall entity even if not
>>> identified as such.
>>>
>>> The above was grabbed with an IMX477 with AF module. I haven't pushed
>>> that configuration anywhere, but IMX135 with lens driver is at
>>> https://github.com/raspberrypi/linux/pull/4612/
>>>
>>
>> What is the output of:
>>
>> v4l2-compliance -M /dev/media2 --verbose
>>
>> That will also report any interface links.
> 
> pi@raspberrypi:~/nfs/v4l-utils/utils/v4l2-compliance $
> ./v4l2-compliance  -M /dev/media2 --verbose
> v4l2-compliance 1.21.0-4859, 32 bits, 32-bit time_t
> v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11
> 
> Compliance test for unicam device /dev/media2:
> 
> Media Driver Info:
>     Driver name      : unicam
>     Model            : unicam
>     Serial           :
>     Bus info         : platform:fe801000.csi
>     Media version    : 5.10.74
>     Hardware revision: 0x00000000 (0)
>     Driver version   : 5.10.74
> 
> Required ioctls:
>     test MEDIA_IOC_DEVICE_INFO: OK
>     test invalid ioctls: OK
> 
> Allow for multiple opens:
>     test second /dev/media2 open: OK
>     test MEDIA_IOC_DEVICE_INFO: OK
>     test for unlimited opens: OK
> 
> Media Controller ioctls:
>         Entity: 0x00000001 (Name: 'imx477 10-001a', Function: Camera Sensor)
>         Entity: 0x00000004 (Name: 'ad5398 focus', Function: Lens Controller)
>         Entity: 0x00000005 (Name: 'unicam-image', Function: V4L2 I/O,
> Flags: default)
>         Entity: 0x0000000b (Name: 'unicam-embedded', Function: V4L2 I/O)
>         Interface: 0x03000007 (Type: V4L Video, DevPath: /dev/video0)
>         Interface: 0x0300000d (Type: V4L Video, DevPath: /dev/video1)
>         Interface: 0x03000011 (Type: V4L Sub-Device, DevPath: /dev/v4l-subdev0)
>         Interface: 0x03000013 (Type: V4L Sub-Device, DevPath: /dev/v4l-subdev1)
>         Pad: 0x01000002 (0, imx477 10-001a, Source)
>         Pad: 0x01000003 (1, imx477 10-001a, Source)
>         Pad: 0x01000006 (0, unicam-image, Sink)
>         Pad: 0x0100000c (0, unicam-embedded, Sink)
>         Link: 0x02000008 (unicam-image to interface /dev/video0)
>         Link: 0x02000009 (imx477 10-001a:0 -> unicam-image:0, Data,
> Enabled, Immutable)
>         Link: 0x0200000e (unicam-embedded to interface /dev/video1)
>         Link: 0x0200000f (imx477 10-001a:1 -> unicam-embedded:0, Data,
> Enabled, Immutable)
>         Link: 0x02000012 (imx477 10-001a to interface /dev/v4l-subdev0)
>         Link: 0x02000014 (ad5398 focus to interface /dev/v4l-subdev1)
>     test MEDIA_IOC_G_TOPOLOGY: OK
>     Entities: 4 Interfaces: 4 Pads: 4 Links: 6
>         Entity: 0x00000001 (Name: 'imx477 10-001a', Type: Camera
> Sensor, DevPath: /dev/v4l-subdev0)
>         Entity: 0x00000004 (Name: 'ad5398 focus', Type: Lens
> Controller, DevPath: /dev/v4l-subdev1)
>         Entity: 0x00000005 (Name: 'unicam-image', Type: V4L2 I/O,
> Flags: default, DevPath: /dev/video0)
>         Entity: 0x0000000b (Name: 'unicam-embedded', Type: V4L2 I/O,
> DevPath: /dev/video1)
>     test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
>     test MEDIA_IOC_SETUP_LINK: OK
> 
> Total for unicam device /dev/media2: 8, Succeeded: 8, Failed: 0, Warnings: 0

Hmm, the ad5398 is just a stand-alone entity with a single interface link to
/dev/v4l-subdev1. The fact that it is related to imx477 is absent in the topology.

Laurent is right, support for this is missing in the MC.

Regards,

	Hans