[RFC] New class for low level sensors controls?

[RFC] New class for low level sensors controls?

[Sakari Ailus]

Hi all,

We are beginning to have raw bayer image sensor drivers in the mainline.
Typically such sensors are not controlled by general purpose applications
but e.g. require a camera control algorithm framework in user space. This
needs to be implemented in libv4l for general purpose applications to work
properly on this kind of hardware.

These sensors expose controls such as

- Per-component gain controls. Red, blue, green (blue) and green (red)
  gains.

- Link frequency. The frequency of the data link from the sensor to the
  bridge.

- Horizontal and vertical blanking.

None of these controls are suitable for use of general purpose applications
(let alone the end user!) but for the camera control algorithms.

We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
However, the controls in this class are relatively high level controls which
are suitable for end user. The algorithms in the libv4l or a webcam could
implement many of these controls whereas I see that only
V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.

My question is: would it make sense to create a new class of controls for
the low level sensor controls in a similar fashion we have a control class
for the flash controls?

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] New class for low level sensors controls?

[Laurent Pinchart]

Hi Sakari,

On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> Hi all,
> 
> We are beginning to have raw bayer image sensor drivers in the mainline.
> Typically such sensors are not controlled by general purpose applications
> but e.g. require a camera control algorithm framework in user space. This
> needs to be implemented in libv4l for general purpose applications to work
> properly on this kind of hardware.
> 
> These sensors expose controls such as
> 
> - Per-component gain controls. Red, blue, green (blue) and green (red)
>   gains.
>
> - Link frequency. The frequency of the data link from the sensor to the
>   bridge.
> 
> - Horizontal and vertical blanking.

Other controls often found in bayer sensors are black level compensation and 
test pattern.

> None of these controls are suitable for use of general purpose applications
> (let alone the end user!) but for the camera control algorithms.
> 
> We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> However, the controls in this class are relatively high level controls
> which are suitable for end user. The algorithms in the libv4l or a webcam
> could implement many of these controls whereas I see that only
> V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> 
> My question is: would it make sense to create a new class of controls for
> the low level sensor controls in a similar fashion we have a control class
> for the flash controls?

I think it would, but I'm not sure how we should name that class. 
V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be 
found there (digital gains, black leverl compensation, test pattern, ...) can 
also be found in ISPs or other hardware blocks.

-- 
Regards,

Laurent Pinchart

Re: [RFC] New class for low level sensors controls?

[Sakari Ailus]

On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> > Hi all,
> > 
> > We are beginning to have raw bayer image sensor drivers in the mainline.
> > Typically such sensors are not controlled by general purpose applications
> > but e.g. require a camera control algorithm framework in user space. This
> > needs to be implemented in libv4l for general purpose applications to work
> > properly on this kind of hardware.
> > 
> > These sensors expose controls such as
> > 
> > - Per-component gain controls. Red, blue, green (blue) and green (red)
> >   gains.
> >
> > - Link frequency. The frequency of the data link from the sensor to the
> >   bridge.
> > 
> > - Horizontal and vertical blanking.
> 
> Other controls often found in bayer sensors are black level compensation and 
> test pattern.
> 
> > None of these controls are suitable for use of general purpose applications
> > (let alone the end user!) but for the camera control algorithms.
> > 
> > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> > However, the controls in this class are relatively high level controls
> > which are suitable for end user. The algorithms in the libv4l or a webcam
> > could implement many of these controls whereas I see that only
> > V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> > 
> > My question is: would it make sense to create a new class of controls for
> > the low level sensor controls in a similar fashion we have a control class
> > for the flash controls?
> 
> I think it would, but I'm not sure how we should name that class. 
> V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be 
> found there (digital gains, black leverl compensation, test pattern, ...) can 
> also be found in ISPs or other hardware blocks.

I don't think ISPs typically implement test patterns. Do you know of any?

Should we separate controls which clearly apply to sensors only from the
rest?

For sensors only:

- Analog gain(s)
- Horizontal and vertical blanking
- Link frequency
- Test pattern

The following can be implemented also on ISPs:

- Per-component gains
- Black level compensation

Do we have more to add to the list?

If we keep the two the same class, I could propose the following names:

V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)
V4L2_CTRL_CLASS_SOURCE
V4L2_CTRL_CLASS_IMAGE_SOURCE

The last one would be a good name for the sensor control class, as far as I
understand some are using tuners with the OMAP 3 ISP these days. For the
another one, I propose V4L2_CTRL_CLASS_ISP.

Better names are always welcome. :-)

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] New class for low level sensors controls?

[Laurent Pinchart]

Hi Sakari,

On Tuesday 06 September 2011 14:22:27 Sakari Ailus wrote:
> On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> > > Hi all,
> > > 
> > > We are beginning to have raw bayer image sensor drivers in the
> > > mainline. Typically such sensors are not controlled by general purpose
> > > applications but e.g. require a camera control algorithm framework in
> > > user space. This needs to be implemented in libv4l for general purpose
> > > applications to work properly on this kind of hardware.
> > > 
> > > These sensors expose controls such as
> > > 
> > > - Per-component gain controls. Red, blue, green (blue) and green (red)
> > > 
> > >   gains.
> > > 
> > > - Link frequency. The frequency of the data link from the sensor to the
> > > 
> > >   bridge.
> > > 
> > > - Horizontal and vertical blanking.
> > 
> > Other controls often found in bayer sensors are black level compensation
> > and test pattern.
> > 
> > > None of these controls are suitable for use of general purpose
> > > applications (let alone the end user!) but for the camera control
> > > algorithms.
> > > 
> > > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera
> > > controls. However, the controls in this class are relatively high
> > > level controls which are suitable for end user. The algorithms in the
> > > libv4l or a webcam could implement many of these controls whereas I
> > > see that only V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw
> > > bayer sensors.
> > > 
> > > My question is: would it make sense to create a new class of controls
> > > for the low level sensor controls in a similar fashion we have a
> > > control class for the flash controls?
> > 
> > I think it would, but I'm not sure how we should name that class.
> > V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be
> > found there (digital gains, black leverl compensation, test pattern, ...)
> > can also be found in ISPs or other hardware blocks.
> 
> I don't think ISPs typically implement test patterns. Do you know of any?

Not from the top of my head, but I don't think it would be too uncommon.

> Should we separate controls which clearly apply to sensors only from the
> rest?
> 
> For sensors only:
> 
> - Analog gain(s)
> - Horizontal and vertical blanking
> - Link frequency
> - Test pattern
> 
> The following can be implemented also on ISPs:
> 
> - Per-component gains
> - Black level compensation
> 
> Do we have more to add to the list?

Not right now.

> If we keep the two the same class, I could propose the following names:
> 
> V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)
> V4L2_CTRL_CLASS_SOURCE
> V4L2_CTRL_CLASS_IMAGE_SOURCE
> 
> The last one would be a good name for the sensor control class, as far as I
> understand some are using tuners with the OMAP 3 ISP these days. For the
> another one, I propose V4L2_CTRL_CLASS_ISP.

The issue with ISP is that pretty much any digital-based control can fall into 
that class.

Maybe we should group controls by what they do, instead of the kind of 
component that implements them ?

-- 
Regards,

Laurent Pinchart

Re: [RFC] New class for low level sensors controls?

[Sakari Ailus]

On Thu, Sep 08, 2011 at 11:51:06AM +0200, Laurent Pinchart wrote:
> Hi Sakari,

Hi Laurent,

> On Tuesday 06 September 2011 14:22:27 Sakari Ailus wrote:
> > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> > > > Hi all,
> > > > 
> > > > We are beginning to have raw bayer image sensor drivers in the
> > > > mainline. Typically such sensors are not controlled by general purpose
> > > > applications but e.g. require a camera control algorithm framework in
> > > > user space. This needs to be implemented in libv4l for general purpose
> > > > applications to work properly on this kind of hardware.
> > > > 
> > > > These sensors expose controls such as
> > > > 
> > > > - Per-component gain controls. Red, blue, green (blue) and green (red)
> > > > 
> > > >   gains.
> > > > 
> > > > - Link frequency. The frequency of the data link from the sensor to the
> > > > 
> > > >   bridge.
> > > > 
> > > > - Horizontal and vertical blanking.
> > > 
> > > Other controls often found in bayer sensors are black level compensation
> > > and test pattern.
> > > 
> > > > None of these controls are suitable for use of general purpose
> > > > applications (let alone the end user!) but for the camera control
> > > > algorithms.
> > > > 
> > > > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera
> > > > controls. However, the controls in this class are relatively high
> > > > level controls which are suitable for end user. The algorithms in the
> > > > libv4l or a webcam could implement many of these controls whereas I
> > > > see that only V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw
> > > > bayer sensors.
> > > > 
> > > > My question is: would it make sense to create a new class of controls
> > > > for the low level sensor controls in a similar fashion we have a
> > > > control class for the flash controls?
> > > 
> > > I think it would, but I'm not sure how we should name that class.
> > > V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be
> > > found there (digital gains, black leverl compensation, test pattern, ...)
> > > can also be found in ISPs or other hardware blocks.
> > 
> > I don't think ISPs typically implement test patterns. Do you know of any?
> 
> Not from the top of my head, but I don't think it would be too uncommon.
> 
> > Should we separate controls which clearly apply to sensors only from the
> > rest?
> > 
> > For sensors only:
> > 
> > - Analog gain(s)
> > - Horizontal and vertical blanking
> > - Link frequency
> > - Test pattern
> > 
> > The following can be implemented also on ISPs:
> > 
> > - Per-component gains
> > - Black level compensation
> > 
> > Do we have more to add to the list?
> 
> Not right now.
> 
> > If we keep the two the same class, I could propose the following names:
> > 
> > V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)
> > V4L2_CTRL_CLASS_SOURCE
> > V4L2_CTRL_CLASS_IMAGE_SOURCE
> > 
> > The last one would be a good name for the sensor control class, as far as I
> > understand some are using tuners with the OMAP 3 ISP these days. For the
> > another one, I propose V4L2_CTRL_CLASS_ISP.
> 
> The issue with ISP is that pretty much any digital-based control can fall into 
> that class.
> 
> Maybe we should group controls by what they do, instead of the kind of 
> component that implements them ?

Sounds like a good idea. The two could fit into these classes:

- image capture control and
- digital image processing.

What do you think?

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] New class for low level sensors controls?

[Subash Patel]

Hi Sakari,

On 09/06/2011 05:52 PM, Sakari Ailus wrote:
> On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
>> Hi Sakari,
>>
>> On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
>>> Hi all,
>>>
>>> We are beginning to have raw bayer image sensor drivers in the mainline.
>>> Typically such sensors are not controlled by general purpose applications
>>> but e.g. require a camera control algorithm framework in user space. This
>>> needs to be implemented in libv4l for general purpose applications to work
>>> properly on this kind of hardware.
>>>
>>> These sensors expose controls such as
>>>
>>> - Per-component gain controls. Red, blue, green (blue) and green (red)
>>>    gains.
>>>
>>> - Link frequency. The frequency of the data link from the sensor to the
>>>    bridge.
>>>
>>> - Horizontal and vertical blanking.
>>
>> Other controls often found in bayer sensors are black level compensation and
>> test pattern.

Does all BAYER sensor allow the dark level compensation programming? I 
thought it must be auto dark level compensation, which is done by the 
sensor. The sensor detects the optical black value at start of each 
frame, and analog-to-digital conversion is shifted to compensate the 
dark level for that frame. Hence I am thinking if this should be a 
controllable feature.

>>
>>> None of these controls are suitable for use of general purpose applications
>>> (let alone the end user!) but for the camera control algorithms.
>>>
>>> We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
>>> However, the controls in this class are relatively high level controls
>>> which are suitable for end user. The algorithms in the libv4l or a webcam
>>> could implement many of these controls whereas I see that only
>>> V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
>>>
>>> My question is: would it make sense to create a new class of controls for
>>> the low level sensor controls in a similar fashion we have a control class
>>> for the flash controls?
>>
>> I think it would, but I'm not sure how we should name that class.
>> V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be
>> found there (digital gains, black leverl compensation, test pattern, ...) can
>> also be found in ISPs or other hardware blocks.
>
> I don't think ISPs typically implement test patterns. Do you know of any?
>
I know atleast two sensors (ov5642 and s5k4bafx) which have inbuilt ISP, 
programmed through i2c. They both have test patten generator. I think 
RAW(BAYER) sensors themselves cannot generate a test pattern without 
some h/w entity to convert RGGB into color bars in RGB or YUV.

> Should we separate controls which clearly apply to sensors only from the
> rest?
>
> For sensors only:
>
> - Analog gain(s)
> - Horizontal and vertical blanking
> - Link frequency
> - Test pattern

Where should the shutter operation be listed into? Also type (rolling, 
global) and method (manual, electronic) of shutter operation?

>
> The following can be implemented also on ISPs:
>
> - Per-component gains
> - Black level compensation
>
> Do we have more to add to the list?
>
> If we keep the two the same class, I could propose the following names:
>
> V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)

Instead of LL_CAMERA, wouldnt something like CAM_SENSOR_ARRAY would be 
more meaningful? We control the sensor array properties in this level.

> V4L2_CTRL_CLASS_SOURCE
> V4L2_CTRL_CLASS_IMAGE_SOURCE
>
> The last one would be a good name for the sensor control class, as far as I
> understand some are using tuners with the OMAP 3 ISP these days. For the
> another one, I propose V4L2_CTRL_CLASS_ISP.
>
> Better names are always welcome. :-)
>

Regards,
Subash

Re: [RFC] New class for low level sensors controls?

[Sakari Ailus]

Hi Subash,

Thanks for the comments.

On Thu, Sep 08, 2011 at 04:54:23PM +0530, Subash Patel wrote:
> Hi Sakari,
> 
> On 09/06/2011 05:52 PM, Sakari Ailus wrote:
> >On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> >>Hi Sakari,
> >>
> >>On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> >>>Hi all,
> >>>
> >>>We are beginning to have raw bayer image sensor drivers in the mainline.
> >>>Typically such sensors are not controlled by general purpose applications
> >>>but e.g. require a camera control algorithm framework in user space. This
> >>>needs to be implemented in libv4l for general purpose applications to work
> >>>properly on this kind of hardware.
> >>>
> >>>These sensors expose controls such as
> >>>
> >>>- Per-component gain controls. Red, blue, green (blue) and green (red)
> >>>   gains.
> >>>
> >>>- Link frequency. The frequency of the data link from the sensor to the
> >>>   bridge.
> >>>
> >>>- Horizontal and vertical blanking.
> >>
> >>Other controls often found in bayer sensors are black level compensation and
> >>test pattern.
> 
> Does all BAYER sensor allow the dark level compensation programming?

I'm not sure. I have always seen ISPs being used for that, not sensors.

> I thought it must be auto dark level compensation, which is done by
> the sensor. The sensor detects the optical black value at start of
> each frame, and analog-to-digital conversion is shifted to
> compensate the dark level for that frame. Hence I am thinking if
> this should be a controllable feature.

This is probably what smart sensors could do. If we have a raw bayer sensor
the computation of the optimal black level compensation could be done by
some of the controls algorithms run in the user space. Automatic exposure
probably?

It's also possible to have the value set statically as it is often
considered sensor model specific.

> >>
> >>>None of these controls are suitable for use of general purpose applications
> >>>(let alone the end user!) but for the camera control algorithms.
> >>>
> >>>We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> >>>However, the controls in this class are relatively high level controls
> >>>which are suitable for end user. The algorithms in the libv4l or a webcam
> >>>could implement many of these controls whereas I see that only
> >>>V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> >>>
> >>>My question is: would it make sense to create a new class of controls for
> >>>the low level sensor controls in a similar fashion we have a control class
> >>>for the flash controls?
> >>
> >>I think it would, but I'm not sure how we should name that class.
> >>V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be
> >>found there (digital gains, black leverl compensation, test pattern, ...) can
> >>also be found in ISPs or other hardware blocks.
> >
> >I don't think ISPs typically implement test patterns. Do you know of any?
> >
> I know atleast two sensors (ov5642 and s5k4bafx) which have inbuilt
> ISP, programmed through i2c. They both have test patten generator. I
> think RAW(BAYER) sensors themselves cannot generate a test pattern
> without some h/w entity to convert RGGB into color bars in RGB or
> YUV.

The sensors can produce test patterns using their native pixel format. The
conversion (if one is needed, likely not, since typically the raw data is of
primary interest in this case) can also be done using the ISP integrated to
the SoC.

> >Should we separate controls which clearly apply to sensors only from the
> >rest?
> >
> >For sensors only:
> >
> >- Analog gain(s)
> >- Horizontal and vertical blanking
> >- Link frequency
> >- Test pattern
> 
> Where should the shutter operation be listed into? Also type
> (rolling, global) and method (manual, electronic) of shutter
> operation?

I'm in favour of Laurent's suggestion to classify these controls based on
what do they control, not where they are implemented in, and naming the
class accordingly.

In my reply to him, these controls would be part of the image capture
control class.

> >
> >The following can be implemented also on ISPs:
> >
> >- Per-component gains
> >- Black level compensation
> >
> >Do we have more to add to the list?
> >
> >If we keep the two the same class, I could propose the following names:
> >
> >V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)
> 
> Instead of LL_CAMERA, wouldnt something like CAM_SENSOR_ARRAY would
> be more meaningful? We control the sensor array properties in this
> level.
> 
> >V4L2_CTRL_CLASS_SOURCE
> >V4L2_CTRL_CLASS_IMAGE_SOURCE
> >
> >The last one would be a good name for the sensor control class, as far as I
> >understand some are using tuners with the OMAP 3 ISP these days. For the
> >another one, I propose V4L2_CTRL_CLASS_ISP.
> >
> >Better names are always welcome. :-)
> >
> 
> Regards,
> Subash

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] New class for low level sensors controls?

[Guennadi Liakhovetski]

On Tue, 6 Sep 2011, Sakari Ailus wrote:

> On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > Hi Sakari,
> > 
> > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> > > Hi all,
> > > 
> > > We are beginning to have raw bayer image sensor drivers in the mainline.

Well, we've been "beginning" since several years now;-) But why only 
Bayer? Don't we want to have the same controls available on sensors, 
sending processed  YUV / RGB data, or for those, capable of both Bayer and 
processed?

> > > Typically such sensors are not controlled by general purpose applications
> > > but e.g. require a camera control algorithm framework in user space. This
> > > needs to be implemented in libv4l for general purpose applications to work
> > > properly on this kind of hardware.
> > > 
> > > These sensors expose controls such as
> > > 
> > > - Per-component gain controls. Red, blue, green (blue) and green (red)
> > >   gains.
> > >
> > > - Link frequency. The frequency of the data link from the sensor to the
> > >   bridge.
> > > 
> > > - Horizontal and vertical blanking.
> > 
> > Other controls often found in bayer sensors are black level compensation and 
> > test pattern.

May I suggest a couple more:

(1) snapshot mode (I really badly want this one, please;-))
(2) flash controls (yes, I know we already have V4L2_CTRL_CLASS_FLASH, I 
    just have the impression, that these controls are mostly meant for 
    either pure software implementations, or for external controllers, I 
    think it should also be possible to have them exported by a normal 
    sensor driver, but wasn't really sure. So, wanted to point out to this 
    possibility once again.)
(3) AEC / AGC regions
(4) stereo (3D anyone?;)) - no, don't think we need it now...

> > > None of these controls are suitable for use of general purpose applications
> > > (let alone the end user!) but for the camera control algorithms.
> > > 
> > > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> > > However, the controls in this class are relatively high level controls
> > > which are suitable for end user. The algorithms in the libv4l or a webcam
> > > could implement many of these controls whereas I see that only
> > > V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> > > 
> > > My question is: would it make sense to create a new class of controls for
> > > the low level sensor controls in a similar fashion we have a control class
> > > for the flash controls?
> > 
> > I think it would, but I'm not sure how we should name that class. 
> > V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be 
> > found there (digital gains, black leverl compensation, test pattern, ...) can 
> > also be found in ISPs or other hardware blocks.
> 
> I don't think ISPs typically implement test patterns. Do you know of any?

Yes, i.MX31.

> Should we separate controls which clearly apply to sensors only from the
> rest?
> 
> For sensors only:
> 
> - Analog gain(s)
> - Horizontal and vertical blanking
> - Link frequency
> - Test pattern
> 
> The following can be implemented also on ISPs:
> 
> - Per-component gains
> - Black level compensation
> 
> Do we have more to add to the list?
> 
> If we keep the two the same class, I could propose the following names:
> 
> V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)
> V4L2_CTRL_CLASS_SOURCE
> V4L2_CTRL_CLASS_IMAGE_SOURCE
> 
> The last one would be a good name for the sensor control class, as far as I
> understand some are using tuners with the OMAP 3 ISP these days. For the
> another one, I propose V4L2_CTRL_CLASS_ISP.
> 
> Better names are always welcome. :-)

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFC] New class for low level sensors controls?

[Sakari Ailus]

On Thu, Sep 08, 2011 at 02:35:54PM +0200, Guennadi Liakhovetski wrote:
> On Tue, 6 Sep 2011, Sakari Ailus wrote:
> 
> > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > Hi Sakari,
> > > 
> > > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> > > > Hi all,
> > > > 
> > > > We are beginning to have raw bayer image sensor drivers in the mainline.
> 
> Well, we've been "beginning" since several years now;-) But why only 
> Bayer? Don't we want to have the same controls available on sensors, 
> sending processed  YUV / RGB data, or for those, capable of both Bayer and 
> processed?

I agree. Many smart sensors just don't offer those controls, at least not
for the purpose of using host side white balance algorithm, so on smart
sensors those typically are already covered.

My wording was bad; what I meant to say was that that these controls are not
standardised.

> > > > Typically such sensors are not controlled by general purpose applications
> > > > but e.g. require a camera control algorithm framework in user space. This
> > > > needs to be implemented in libv4l for general purpose applications to work
> > > > properly on this kind of hardware.
> > > > 
> > > > These sensors expose controls such as
> > > > 
> > > > - Per-component gain controls. Red, blue, green (blue) and green (red)
> > > >   gains.
> > > >
> > > > - Link frequency. The frequency of the data link from the sensor to the
> > > >   bridge.
> > > > 
> > > > - Horizontal and vertical blanking.
> > > 
> > > Other controls often found in bayer sensors are black level compensation and 
> > > test pattern.
> 
> May I suggest a couple more:
> 
> (1) snapshot mode (I really badly want this one, please;-))

Sounds good.

> (2) flash controls (yes, I know we already have V4L2_CTRL_CLASS_FLASH, I 
>     just have the impression, that these controls are mostly meant for 
>     either pure software implementations, or for external controllers, I 
>     think it should also be possible to have them exported by a normal 
>     sensor driver, but wasn't really sure. So, wanted to point out to this 
>     possibility once again.)

There was a lengthy discussion during the review process. Do you see that
something would be missing from the current interface?

I vaguely remember that you had a sensor that expected to be controlled by
the flash controller considering the flash timing, rather than the other way
around.

> (3) AEC / AGC regions

This might get tricky. I wonder how we could eventually standardise these
ones.

> (4) stereo (3D anyone?;)) - no, don't think we need it now...

I wonder what is needed to support this on V4L2. It might also depend on how
the hardware will look like.

> > > > None of these controls are suitable for use of general purpose applications
> > > > (let alone the end user!) but for the camera control algorithms.
> > > > 
> > > > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> > > > However, the controls in this class are relatively high level controls
> > > > which are suitable for end user. The algorithms in the libv4l or a webcam
> > > > could implement many of these controls whereas I see that only
> > > > V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> > > > 
> > > > My question is: would it make sense to create a new class of controls for
> > > > the low level sensor controls in a similar fashion we have a control class
> > > > for the flash controls?
> > > 
> > > I think it would, but I'm not sure how we should name that class. 
> > > V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be 
> > > found there (digital gains, black leverl compensation, test pattern, ...) can 
> > > also be found in ISPs or other hardware blocks.
> > 
> > I don't think ISPs typically implement test patterns. Do you know of any?
> 
> Yes, i.MX31.

Good to know that.

But you already know I'm with Laurent in believing that the controls should
be classified based on what they do, not where they are implemented. :-)

> > Should we separate controls which clearly apply to sensors only from the
> > rest?
> > 
> > For sensors only:
> > 
> > - Analog gain(s)
> > - Horizontal and vertical blanking
> > - Link frequency
> > - Test pattern
> > 
> > The following can be implemented also on ISPs:
> > 
> > - Per-component gains
> > - Black level compensation
> > 
> > Do we have more to add to the list?
> > 
> > If we keep the two the same class, I could propose the following names:
> > 
> > V4L2_CTRL_CLASS_LL_CAMERA (for low level camera)
> > V4L2_CTRL_CLASS_SOURCE
> > V4L2_CTRL_CLASS_IMAGE_SOURCE
> > 
> > The last one would be a good name for the sensor control class, as far as I
> > understand some are using tuners with the OMAP 3 ISP these days. For the
> > another one, I propose V4L2_CTRL_CLASS_ISP.
> > 
> > Better names are always welcome. :-)
> 
> Thanks
> Guennadi
> ---
> Guennadi Liakhovetski, Ph.D.
> Freelance Open-Source Software Developer
> http://www.open-technology.de/

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] New class for low level sensors controls?

[Guennadi Liakhovetski]

On Thu, 8 Sep 2011, Sakari Ailus wrote:

> On Thu, Sep 08, 2011 at 02:35:54PM +0200, Guennadi Liakhovetski wrote:
> > On Tue, 6 Sep 2011, Sakari Ailus wrote:
> > 
> > > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > > Hi Sakari,
> > > > 
> > > > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:

[snip]

> > > > > Typically such sensors are not controlled by general purpose applications
> > > > > but e.g. require a camera control algorithm framework in user space. This
> > > > > needs to be implemented in libv4l for general purpose applications to work
> > > > > properly on this kind of hardware.
> > > > > 
> > > > > These sensors expose controls such as
> > > > > 
> > > > > - Per-component gain controls. Red, blue, green (blue) and green (red)
> > > > >   gains.
> > > > >
> > > > > - Link frequency. The frequency of the data link from the sensor to the
> > > > >   bridge.
> > > > > 
> > > > > - Horizontal and vertical blanking.
> > > > 
> > > > Other controls often found in bayer sensors are black level compensation and 
> > > > test pattern.
> > 
> > May I suggest a couple more:
> > 
> > (1) snapshot mode (I really badly want this one, please;-))
> 
> Sounds good.
> 
> > (2) flash controls (yes, I know we already have V4L2_CTRL_CLASS_FLASH, I 
> >     just have the impression, that these controls are mostly meant for 
> >     either pure software implementations, or for external controllers, I 
> >     think it should also be possible to have them exported by a normal 
> >     sensor driver, but wasn't really sure. So, wanted to point out to this 
> >     possibility once again.)
> 
> There was a lengthy discussion during the review process. Do you see that
> something would be missing from the current interface?

I didn't try to systemise them. Just having a look at a couple of 
datasheets I see a few more parameters, like flash strobe delay, etc. But 
I don't think we should just add as many of them as we can just for the 
sake of them. I think, as soon as someone comes with a new requirement, 
we'll think of a way to implement it. 

> I vaguely remember that you had a sensor that expected to be controlled by
> the flash controller considering the flash timing, rather than the other way
> around.

Maybe:-) But again - let's wait until someone needs that functionality.

> > (3) AEC / AGC regions
> 
> This might get tricky. I wonder how we could eventually standardise these
> ones.
> 
> > (4) stereo (3D anyone?;)) - no, don't think we need it now...
> 
> I wonder what is needed to support this on V4L2. It might also depend on how
> the hardware will look like.

The sensor, that I looked at (mt9v022) has one bit to turn stereo mode on 
or off, and one more bit to set master or slave.

> > > > > None of these controls are suitable for use of general purpose applications
> > > > > (let alone the end user!) but for the camera control algorithms.
> > > > > 
> > > > > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> > > > > However, the controls in this class are relatively high level controls
> > > > > which are suitable for end user. The algorithms in the libv4l or a webcam
> > > > > could implement many of these controls whereas I see that only
> > > > > V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> > > > > 
> > > > > My question is: would it make sense to create a new class of controls for
> > > > > the low level sensor controls in a similar fashion we have a control class
> > > > > for the flash controls?
> > > > 
> > > > I think it would, but I'm not sure how we should name that class. 
> > > > V4L2_CTRL_CLASS_SENSOR is tempting, but many of the controls that will be 
> > > > found there (digital gains, black leverl compensation, test pattern, ...) can 
> > > > also be found in ISPs or other hardware blocks.
> > > 
> > > I don't think ISPs typically implement test patterns. Do you know of any?
> > 
> > Yes, i.MX31.
> 
> Good to know that.
> 
> But you already know I'm with Laurent in believing that the controls should
> be classified based on what they do, not where they are implemented. :-)

Sure, I agree too, that was just FYI.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFC] New class for low level sensors controls?

[Laurent Pinchart]

Hi Guennadi,

On Thursday 08 September 2011 14:35:54 Guennadi Liakhovetski wrote:
> On Tue, 6 Sep 2011, Sakari Ailus wrote:
> > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:

[snip]

> > > > Typically such sensors are not controlled by general purpose
> > > > applications but e.g. require a camera control algorithm framework
> > > > in user space. This needs to be implemented in libv4l for general
> > > > purpose applications to work properly on this kind of hardware.
> > > > 
> > > > These sensors expose controls such as
> > > > 
> > > > - Per-component gain controls. Red, blue, green (blue) and green
> > > > (red) gains.
> > > > 
> > > > - Link frequency. The frequency of the data link from the sensor to
> > > > the bridge.
> > > > 
> > > > - Horizontal and vertical blanking.
> > > 
> > > Other controls often found in bayer sensors are black level
> > > compensation and test pattern.
> 
> May I suggest a couple more:
> 
> (1) snapshot mode (I really badly want this one, please;-))

What do you mean exactly by snapshot mode ? Is that just external trigger, or 
does it cover more features than that ?

> (2) flash controls (yes, I know we already have V4L2_CTRL_CLASS_FLASH, I
>     just have the impression, that these controls are mostly meant for
>     either pure software implementations, or for external controllers, I
>     think it should also be possible to have them exported by a normal
>     sensor driver, but wasn't really sure. So, wanted to point out to this
>     possibility once again.)

As Sakari told you in his answer, we can add new controls to the flash class.

> (3) AEC / AGC regions

This will get tricky. I'm tempted to propose the idea of v4l2_rect controls 
and control arrays again :-)

> (4) stereo (3D anyone?;)) - no, don't think we need it now...

-- 
Regards,

Laurent Pinchart

Re: [RFC] New class for low level sensors controls?

[Laurent Pinchart]

Hi Sakari,

On Thursday 08 September 2011 13:44:28 Sakari Ailus wrote:
> On Thu, Sep 08, 2011 at 04:54:23PM +0530, Subash Patel wrote:
> > On 09/06/2011 05:52 PM, Sakari Ailus wrote:
> > > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > >
> > > > Other controls often found in bayer sensors are black level
> > > > compensation and test pattern.
> > 
> > Does all BAYER sensor allow the dark level compensation programming?
> 
> I'm not sure. I have always seen ISPs being used for that, not sensors.
> 
> > I thought it must be auto dark level compensation, which is done by
> > the sensor. The sensor detects the optical black value at start of
> > each frame, and analog-to-digital conversion is shifted to
> > compensate the dark level for that frame. Hence I am thinking if
> > this should be a controllable feature.
> 
> This is probably what smart sensors could do. If we have a raw bayer sensor
> the computation of the optimal black level compensation could be done by
> some of the controls algorithms run in the user space. Automatic exposure
> probably?

Many "non-smart" raw bayer sensors implement both manual and automatic black 
level compensation. In the first case the user programs a value to be 
subtracted from the pixels (whether that's done in the analog or digital 
domain might be sensor-specific), and in the second case the sensor computes a 
mean black level value based on black lines (optically unexposed) at the top 
of the image.

-- 
Regards,

Laurent Pinchart

Re: [RFC] New class for low level sensors controls?

[Sakari Ailus]

On Tue, Sep 13, 2011 at 12:33:58PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Thursday 08 September 2011 13:44:28 Sakari Ailus wrote:
> > On Thu, Sep 08, 2011 at 04:54:23PM +0530, Subash Patel wrote:
> > > On 09/06/2011 05:52 PM, Sakari Ailus wrote:
> > > > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > > >
> > > > > Other controls often found in bayer sensors are black level
> > > > > compensation and test pattern.
> > > 
> > > Does all BAYER sensor allow the dark level compensation programming?
> > 
> > I'm not sure. I have always seen ISPs being used for that, not sensors.
> > 
> > > I thought it must be auto dark level compensation, which is done by
> > > the sensor. The sensor detects the optical black value at start of
> > > each frame, and analog-to-digital conversion is shifted to
> > > compensate the dark level for that frame. Hence I am thinking if
> > > this should be a controllable feature.
> > 
> > This is probably what smart sensors could do. If we have a raw bayer sensor
> > the computation of the optimal black level compensation could be done by
> > some of the controls algorithms run in the user space. Automatic exposure
> > probably?
> 
> Many "non-smart" raw bayer sensors implement both manual and automatic black 
> level compensation. In the first case the user programs a value to be 
> subtracted from the pixels (whether that's done in the analog or digital 
> domain might be sensor-specific), and in the second case the sensor computes a 
> mean black level value based on black lines (optically unexposed) at the top 
> of the image.

Sounds like two controls to me, right?

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] New class for low level sensors controls?

[Laurent Pinchart]

Hi Sakari,

On Tuesday 13 September 2011 14:00:36 Sakari Ailus wrote:
> On Tue, Sep 13, 2011 at 12:33:58PM +0200, Laurent Pinchart wrote:
> > On Thursday 08 September 2011 13:44:28 Sakari Ailus wrote:
> > > On Thu, Sep 08, 2011 at 04:54:23PM +0530, Subash Patel wrote:
> > > > On 09/06/2011 05:52 PM, Sakari Ailus wrote:
> > > > > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > > > > Other controls often found in bayer sensors are black level
> > > > > > compensation and test pattern.
> > > > 
> > > > Does all BAYER sensor allow the dark level compensation programming?
> > > 
> > > I'm not sure. I have always seen ISPs being used for that, not sensors.
> > > 
> > > > I thought it must be auto dark level compensation, which is done by
> > > > the sensor. The sensor detects the optical black value at start of
> > > > each frame, and analog-to-digital conversion is shifted to
> > > > compensate the dark level for that frame. Hence I am thinking if
> > > > this should be a controllable feature.
> > > 
> > > This is probably what smart sensors could do. If we have a raw bayer
> > > sensor the computation of the optimal black level compensation could
> > > be done by some of the controls algorithms run in the user space.
> > > Automatic exposure probably?
> > 
> > Many "non-smart" raw bayer sensors implement both manual and automatic
> > black level compensation. In the first case the user programs a value to
> > be subtracted from the pixels (whether that's done in the analog or
> > digital domain might be sensor-specific), and in the second case the
> > sensor computes a mean black level value based on black lines (optically
> > unexposed) at the top of the image.
> 
> Sounds like two controls to me, right?

At least two, an auto control and a manual value control. Additional controls 
(such as the number of lines on which to compute the black level average 
automatically for instance, or a read-only control to report the computed 
average) will probably be added later.

-- 
Regards,

Laurent Pinchart

Re: [RFC] New class for low level sensors controls?

[Guennadi Liakhovetski]

On Tue, 13 Sep 2011, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Thursday 08 September 2011 14:35:54 Guennadi Liakhovetski wrote:
> > On Tue, 6 Sep 2011, Sakari Ailus wrote:
> > > On Tue, Sep 06, 2011 at 01:41:11PM +0200, Laurent Pinchart wrote:
> > > > On Tuesday 06 September 2011 13:36:53 Sakari Ailus wrote:
> 
> [snip]
> 
> > > > > Typically such sensors are not controlled by general purpose
> > > > > applications but e.g. require a camera control algorithm framework
> > > > > in user space. This needs to be implemented in libv4l for general
> > > > > purpose applications to work properly on this kind of hardware.
> > > > > 
> > > > > These sensors expose controls such as
> > > > > 
> > > > > - Per-component gain controls. Red, blue, green (blue) and green
> > > > > (red) gains.
> > > > > 
> > > > > - Link frequency. The frequency of the data link from the sensor to
> > > > > the bridge.
> > > > > 
> > > > > - Horizontal and vertical blanking.
> > > > 
> > > > Other controls often found in bayer sensors are black level
> > > > compensation and test pattern.
> > 
> > May I suggest a couple more:
> > 
> > (1) snapshot mode (I really badly want this one, please;-))
> 
> What do you mean exactly by snapshot mode ? Is that just external trigger, or 
> does it cover more features than that ?

I mean flipping the "snapshot" bit in the respective sensor configuration 
register. There are variants, of course. On some sensors there's just one 
such bit and all it does is enable the flash strobe. On others it also 
switches the configuration registers to immediately start recording a 
different frame size. On some other sensors, I think, there are more than 
2 sets of such configurations, then you need more than one control to 
switch between them. On yet other sensors you can program, how many frames 
you want to take in this mode (before switching back or before halting). I 
think, as a minimum we would expect from this control: switch flash strobe 
on & switch to the snapshot frame size, driver implementation details 
depend on the hardware capabilities, of course.

Thanks
Guennadi

> 
> > (2) flash controls (yes, I know we already have V4L2_CTRL_CLASS_FLASH, I
> >     just have the impression, that these controls are mostly meant for
> >     either pure software implementations, or for external controllers, I
> >     think it should also be possible to have them exported by a normal
> >     sensor driver, but wasn't really sure. So, wanted to point out to this
> >     possibility once again.)
> 
> As Sakari told you in his answer, we can add new controls to the flash class.
> 
> > (3) AEC / AGC regions
> 
> This will get tricky. I'm tempted to propose the idea of v4l2_rect controls 
> and control arrays again :-)
> 
> > (4) stereo (3D anyone?;)) - no, don't think we need it now...
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFC] New class for low level sensors controls?

[Hans Verkuil]

On Tuesday, September 06, 2011 13:36:53 Sakari Ailus wrote:
> Hi all,
> 
> We are beginning to have raw bayer image sensor drivers in the mainline.
> Typically such sensors are not controlled by general purpose applications
> but e.g. require a camera control algorithm framework in user space. This
> needs to be implemented in libv4l for general purpose applications to work
> properly on this kind of hardware.
> 
> These sensors expose controls such as
> 
> - Per-component gain controls. Red, blue, green (blue) and green (red)
>   gains.
> 
> - Link frequency. The frequency of the data link from the sensor to the
>   bridge.
> 
> - Horizontal and vertical blanking.
> 
> None of these controls are suitable for use of general purpose applications
> (let alone the end user!) but for the camera control algorithms.
> 
> We have a control class called V4L2_CTRL_CLASS_CAMERA for camera controls.
> However, the controls in this class are relatively high level controls which
> are suitable for end user. The algorithms in the libv4l or a webcam could
> implement many of these controls whereas I see that only
> V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> 
> My question is: would it make sense to create a new class of controls for
> the low level sensor controls in a similar fashion we have a control class
> for the flash controls?

I'm plowing through all the mails on this list that piled up during the last
3-4 weeks, so my reply is a bit late :-)

I don't believe a new class is the right approach. If such controls are part
of a sub-device, then they can be marked 'private', which means they are only
accessible through the subdev device node.

For low-level controls that are part of the bridge chip there is currently no
suitable way of hiding them.

In that case the best approach would be to add a new control flag called
V4L2_CTRL_FLAG_HIDE (or _INVISIBLE, or _LOW_LEVEL, or something similar).
Applications can filter such controls and not show them.

I've toyed with this idea before, but of course it has the disadvantage of
requiring application support.

One alternative to this is to let QUERYCTRL skip such hidden controls unless
instructed otherwise (e.g. by adding a V4L2_CTRL_FLAG_SHOW_ALL to the control's
id). But I think that's getting a bit too complex.

I think adding a 'HIDE' flag of some sort is a good idea. It's simple and does
the job.

Regards,

	Hans

Re: [RFC] New class for low level sensors controls?

[Laurent Pinchart]

Hi Hans,

On Monday 26 September 2011 11:51:05 Hans Verkuil wrote:
> On Tuesday, September 06, 2011 13:36:53 Sakari Ailus wrote:
> > Hi all,
> > 
> > We are beginning to have raw bayer image sensor drivers in the mainline.
> > Typically such sensors are not controlled by general purpose applications
> > but e.g. require a camera control algorithm framework in user space. This
> > needs to be implemented in libv4l for general purpose applications to
> > work properly on this kind of hardware.
> > 
> > These sensors expose controls such as
> > 
> > - Per-component gain controls. Red, blue, green (blue) and green (red)
> > 
> >   gains.
> > 
> > - Link frequency. The frequency of the data link from the sensor to the
> > 
> >   bridge.
> > 
> > - Horizontal and vertical blanking.
> > 
> > None of these controls are suitable for use of general purpose
> > applications (let alone the end user!) but for the camera control
> > algorithms.
> > 
> > We have a control class called V4L2_CTRL_CLASS_CAMERA for camera
> > controls. However, the controls in this class are relatively high level
> > controls which are suitable for end user. The algorithms in the libv4l
> > or a webcam could implement many of these controls whereas I see that
> > only
> > V4L2_CID_EXPOSURE_ABSOLUTE might be implemented by raw bayer sensors.
> > 
> > My question is: would it make sense to create a new class of controls for
> > the low level sensor controls in a similar fashion we have a control
> > class for the flash controls?
> 
> I'm plowing through all the mails on this list that piled up during the
> last 3-4 weeks, so my reply is a bit late :-)
> 
> I don't believe a new class is the right approach. If such controls are
> part of a sub-device, then they can be marked 'private', which means they
> are only accessible through the subdev device node.

We're not trying to hide controls, but to standardize low-level sensor 
controls that are common across many sensors. They don't really belong to any 
of the existing classes, hence the idea of creating a new class for them.

> For low-level controls that are part of the bridge chip there is currently
> no suitable way of hiding them.
> 
> In that case the best approach would be to add a new control flag called
> V4L2_CTRL_FLAG_HIDE (or _INVISIBLE, or _LOW_LEVEL, or something similar).
> Applications can filter such controls and not show them.
> 
> I've toyed with this idea before, but of course it has the disadvantage of
> requiring application support.
> 
> One alternative to this is to let QUERYCTRL skip such hidden controls
> unless instructed otherwise (e.g. by adding a V4L2_CTRL_FLAG_SHOW_ALL to
> the control's id). But I think that's getting a bit too complex.
> 
> I think adding a 'HIDE' flag of some sort is a good idea. It's simple and
> does the job.

-- 
Regards,

Laurent Pinchart