[RFC v4] V4L2 API for flash devices

[RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Hi,

This is a fourth proposal for an interface for controlling flash devices
on the V4L2/v4l2_subdev APIs.

I want to thank everyone who have participated to the development of the
flash interface.

Comments and questions are very, very welcome as always.


Changes since v3 [12]
=====================

- V4L2_CID_FLASH_STROBE changed to button control,
V4L2_CID_FLASH_STROBE_STOP button control added,
V4L2_CID_FLASH_STROBE_STATUS boolean control added.

- No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.

- V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.

- V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and
V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY.

- Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
extensions".


Changes since v2 [9]
====================

- Rearranged proposed controls. V4L2_CID_FLASH_LED_MODE is now the first
control.

- Added an open question on naming of indicator and torch controls.

- V4L2_CID_FLASH_STROBE_MODE renamed to V4L2_CID_FLASH_STROBE_WHENCE.
V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE renamed to
V4L2_CID_FLASH_EXTERNAL_STROBE_MODE.

- Removed CID_ from V4L2_CID_FLASH_EXTERNAL_STROBE_MODE values.

- Added a new use case based on [11]: Synchronised LED flash (hardware
strobe, timed exposure).

- Added section on possible future extensions.

- Complemented the open question on units.


Changes since v1 [7]
====================

- V4L2_FLASH_STROBE_MODE_EXT_STROBE renamed to
V4L2_FLASH_STROBE_MODE_EXTERNAL.

- V4L2_CID_FLASH_STROBE control changed from button to bool.

- Removed suggestion of adding V4L2_CID_FLASH_DURATION.
V4L2_CID_FLASH_TIMEOUT is used as hardware timeout.

- Added control access info (ro/rw).

- V4L2_FLASH_MODE_NONE added, V4L2_FLASH_LED_MODE_FLASH no longer forced
as 1 in enum.

- Bits use (1 << x) instead of 0x00... format.

- Added an open question on flash LED mode controls.

- Added an open question on a new control:
V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE.

- Added an open question on control units.


Scope
=====

This RFC is focused mostly on the ADP1653 [1] and similar chips [2, 3]
which provides following functionality. [2, 3] mostly differ on the
available faults --- for example, there are faults also for the
indicator LED.

- High power LED output (flash or torch modes)
- Low power indicator LED output (a.k.a. privacy light)
- Programmable flash timeout
- Software and hardware strobe
- Fault detection
	- Overvoltage
	- Overtemperature
	- Short circuit
	- Timeout
- Programmable current (both high-power and indicator LEDs)

If anyone else is aware of hardware which significantly differs from
these and does not get served well under the proposed interface, please
tell about it.

This RFC does NOT address the synchronisation of the flash to a given
frame since this task is typically performed by the sensor through a
strobe signal. The host does not have enough information for this ---
exact timing information on the exposure of the sensor pixel array. In
this case the flash synchronisation is visible to the flash controller
as the hardware strobe originating from the sensor.

Flash synchronisation requires

1) flash control capability from the sensor including a strobe output,
2) strobe input in the flash controller,
3) (optionally) ability to program sensor parameters at given frame,
such as flash strobe, and
4) ability to read back metadata produced by the sensor related to a
given frame. This should include whether the frame is exposed with
flash, i.e. the sensor's flash strobe output.

Since we have little examples of both in terms of hardware support,
which is in practice required, it was decided to postpone the interface
specification for now. [6]

Xenon flash controllers exist but I don't have a specific example of
those. Typically the interface is quite simple. Gpio pins for charge and
strobe. The length of the strobe signal determines the strength of the
flash pulse. The strobe is controlled by the sensor as for LED flash if
it is hardware based.

See "Possible future extensions" section below for more.


Known use cases
===============

The use case listed below concentrate on using a flash in a mobile
device, for example in a mobile phone. The use cases could be somewhat
different in devices the primary use of which is camera.

Unsynchronised LED flash (software strobe)
------------------------------------------

Unsynchronised LED flash is controlled directly by the host as the
sensor. The flash must be enabled by the host before the exposure of the
image starts and disabled once it ends. The host is fully responsible
for the timing of the flash.

Example of such device: Nokia N900.

Synchronised LED flash (hardware strobe)
----------------------------------------

The synchronised LED flash is pre-programmed by the host (power and
timeout) but controlled by the sensor through a strobe signal from the
sensor to the flash.

The sensor controls the flash duration and timing. This control
typically must be programmed to the sensor, and specifying an interface
for this is out of scope of this RFC.

The LED flash controllers we know of can function in both synchronised
and unsynchronised modes.

LED flash as torch
------------------

LED flash may be used as torch in conjunction with another use case
involving camera or individually. [4]

Synchronised xenon flash
------------------------

The synchronised xenon flash is controlled more closely by the sensor
than the LED flash. There is no separate intensity control for the xenon
flash as its intensity is determined by the length of the strobe pulse.
Several consecutive strobe pluses are possible but this needs to be
still controlled by the sensor.

Synchronised xenon flash (timed exposure)
-----------------------------------------

The same as above, but the flash is triggered around the end of the
exposure. [11] This also probably requires a sensor which can do
synchronous exposure, i.e. can start (and stop) the exposure of all
lines at the same time.


Proposed interface
==================

The flash, either LED or xenon, does not require large amounts of data
to control it. There are parameters to control it but they are
independent and assumably some hardware would only support some subsets
of the functionality available somewhere else. Thus V4L2 controls seem
an ideal way to support flash controllers.

A separate control class is reserved for the flash controls. It is
called V4L2_CTRL_CLASS_FLASH.

Type of the control; type of flash is in parentheses after the control.


	V4L2_CID_FLASH_LED_MODE (menu; rw; LED)

enum v4l2_flash_led_mode {
	V4L2_FLASH_LED_MODE_NONE,
	V4L2_FLASH_LED_MODE_FLASH,
	V4L2_FLASH_LED_MODE_TORCH,
};


	V4L2_CID_FLASH_STROBE (button; wo; LED, xenon)

Strobe the flash using software strobe from the host, typically over I2C
or a GPIO. The flash is NOT synchronised to sensor pixel are exposure
since the command is given asynchronously. Alternatively, if the flash
controller is a master in the system, the sensor exposure may be
triggered based on software strobe.


	V4L2_CID_FLASH_STROBE_STOP (button; wo; LED)

This control may be used to shut down the strobe immediately.


	V4L2_CID_FLASH_STROBE_STATUS (boolean; ro; LED)

This contol may be used to query the status of the strobe (on/off).


	V4L2_CID_FLASH_STROBE_MODE (menu; rw; LED)

Use hardware or software strobe. If hardware strobe is selected, the
flash controller is a slave in the system where the sensor produces the
strobe signal to the flash.

In this case the flash controller setup is limited to programming strobe
timeout and power (LED flash) and the sensor controls the timing and
length of the strobe.

enum v4l2_flash_strobe_whence {
	V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
	V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
};


	V4L2_CID_FLASH_TIMEOUT (integer; rw; LED)

The flash controller provides timeout functionality to shut down the led
in case the host fails to do that. For hardware strobe, this is the
maximum amount of time the flash should stay on, and the purpose of the
setting is to prevent the LED from catching fire.

For software strobe, the setting may be used to limit the length of the
strobe using a hardware watchdog. The granularity of the timeout in [1,
2, 3] is very coarse.

A standard unit such as ms or µs should be used.


	V4L2_CID_FLASH_INTENSITY (integer; rw; LED)

Intensity of the flash in hardware specific units. The LED flash
controller provides current to the LED but the actual luminous power is
dictated by the LED connected to the controller.


	V4L2_CID_FLASH_TORCH_INTENSITY (integer; rw; LED)

Intensity of the flash in hardware specific units.


	V4L2_CID_FLASH_INDICATOR_INTENSITY (integer; rw; LED)

Intensity of the indicator light in hardware specific units.


	V4L2_CID_FLASH_FAULT (bit field; ro; LED)

This is a bitmask containing the fault information for the flash. This
assumes the proposed V4L2 bit mask controls [5]; otherwise this would
likely need to be a set of controls.

#define V4L2_FLASH_FAULT_OVER_VOLTAGE		(1 << 0)
#define V4L2_FLASH_FAULT_TIMEOUT		(1 << 1)
#define V4L2_FLASH_FAULT_OVER_TEMPERATURE	(1 << 2)
#define V4L2_FLASH_FAULT_SHORT_CIRCUIT		(1 << 3)

Several faults may occur at single occasion. The ADP1653 is able to
inform the user a fault has occurred, so a V4L2 control event (proposed
earlier) could be used for that.

These faults are supported by the ADP1653. More faults may be added as
support for more chips require that. In some other hardware faults are
available for indicator led as well.


	V4L2_CID_FLASH_CHARGE (bool; rw; xenon)

Charge control for the xenon flash. Enable or disable charging.


	V4L2_CID_FLASH_READY (bool; ro; xenon, LED)

Flash is ready to strobe. On xenon flash this tells the capacitor has
been charged, on LED flash it's that the LED is no longer too hot.

The implementation on LED flash may be modelling the temperature
behaviour of the LED in the driver (or elsewhere, e.g. library or board
code) if the hardware does not provide direct temperature information
from the LED.

A V4L2 control event should be produced whenever the flash becomes ready.


Open questions
==============

1. Flash LED mode control
-------------------------

Should the flash led mode control be a single control or a set of
controls? A single control would make it easier for application to
choose between different modes, but on the other hand limits future
extensibility if there would have to be further splitting of the modes. [8]

3. Units
--------

Which units should e.g. V4L2_CID_FLASH_TIMEOUT be using? It'd be very
useful to have standard unit on this control, like ms or µs.

Some controls, like V4L2_CID_FLASH_INTENSITY, can't easily use a
standard unit. The luminous output depends on the LED connected (you
could use an incandescent lightbulb too, I suppose?) to the flash
controller. These controls typically control the current supplied by the
flash controller.

The timing controls on the sensor could use pixels (or lines) since
these are the units the sensor uses itself. Converting between pixels
(or lines) and SI units is trivial since the pixel clock is known in the
user space.

The flash chip, on the other hand, has no knowledge of sensor pixel
clock, so it should use SI units. Also, the timing control currently in
the flash chip itself is very coarse.

PROPOSAL: If the component has internal timing which is known elsewhere,
such the sensor, the controls should use these units. Otherwise,
standard SI units should be used. In the sensor's case, this could be µs.


Possible future extensions
==========================

1. Indicator faults
-------------------

Some chips do support these. As they are part of the same register (at
least on some chips [10]), they could be part of V4L2_CID_FLASH_FAULT
control.

2. Strobe output control on sensor
----------------------------------

Sensor requires strobe output control to trigger hardware strobe (on
next possible frame).

3. Sensor metadata on frames
----------------------------

It'd be useful to be able to read back sensor metadata. If the flash is
strobed (on sensor hardware) while streaming, it's difficult to know
otherwise which frame in the stream has been exposed with flash.

4. Timing of strobe
-------------------

Hardware strobe timing relative to frame start. The length of the strobe
should be timeable as well.

5. Flash type control
---------------------

Probably it'd be good to be able to tell which kind of flash we have
(xenon/LED). This should be added no later than we have a xenon flash
driver.

V4L2_CID_FLASH_TYPE

6. V4L2_CID_FLASH_EXTERNAL_STROBE_MODE
--------------------------------------

The implementation of this control is postponed until we have a driver
which implements this functionality.

	V4L2_CID_FLASH_EXTERNAL_STROBE_MODE (bool; rw; xenon, LED)

Whether the flash controller considers external strobe as edge, when the
only limit of the strobe is the timeout on flash controller, or level,
when the flash strobe will last as long as the strobe signal, or as long
until the timeout expires.

enum v4l2_flash_external_strobe_mode {
	V4L2_FLASH_EXTERNAL_STROBE_MODE_LEVEL,
	V4L2_FLASH_EXTERNAL_STROBE_MODE_EDGE,
};

Should V4L2_CID_FLASH_EXTERNAL_STROBE_MODE be renamed? Are _EDGE and
_LEVEL menu values enough to describe the functionality?


References
==========

[1] ADP1653 datasheet.
http://www.analog.com/static/imported-files/data_sheets/ADP1653.pdf

[2] LM3555. http://www.national.com/mpf/LM/LM3555.html#Overview

[3] AS3645 product brief.
http://www.austriamicrosystems.com/eng/Products/Lighting-Management/Camera-Flash-LED-Drivers/AS3645

[4] Flashlight applet for the N900.
http://maemo.org/downloads/product/Maemo5/flashlight-applet/

[5] V4L2 Warszaw brainstorming meeting notes, day 2.
http://www.retiisi.org.uk/v4l2/v4l2-brainstorming-warsaw-2011-03/notes/day%202%20(SGz6LU2esk).html

[6] V4L2 Warszaw brainstorming meeting notes, day 3.
http://www.retiisi.org.uk/v4l2/v4l2-brainstorming-warsaw-2011-03/notes/day%203%20(RhoYa0X9D7).html

[7] [RFC] V4L2 flash API for flash devices.
http://www.spinics.net/lists/linux-media/msg30725.html

[8] http://www.spinics.net/lists/linux-media/msg30794.html

[9] [RFC v2] V4L2 flash API for flash devices.
http://www.spinics.net/lists/linux-media/msg31135.html

[10] AS3645 datasheet.
http://www.cdiweb.com/datasheets/austriamicro/AS3645_Datasheet_v1-6.pdf

[11] Andrew's use cases for flash.
http://www.spinics.net/lists/linux-media/msg31363.html

[12] [RFC v3] V4L2 flash API for flash devices.
http://www.spinics.net/lists/linux-media/msg31425.html


Cheers,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Hans Verkuil]

On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
> Hi,
> 
> This is a fourth proposal for an interface for controlling flash devices
> on the V4L2/v4l2_subdev APIs.
> 
> I want to thank everyone who have participated to the development of the
> flash interface.
> 
> Comments and questions are very, very welcome as always.
> 
> 
> Changes since v3 [12]
> =====================
> 
> - V4L2_CID_FLASH_STROBE changed to button control,
> V4L2_CID_FLASH_STROBE_STOP button control added,
> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
> 
> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
> 
> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
> 
> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and
> V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY.
> 
> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
> extensions".
> 
> 
> Changes since v2 [9]
> ====================
> 
> - Rearranged proposed controls. V4L2_CID_FLASH_LED_MODE is now the first
> control.
> 
> - Added an open question on naming of indicator and torch controls.
> 
> - V4L2_CID_FLASH_STROBE_MODE renamed to V4L2_CID_FLASH_STROBE_WHENCE.
> V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE renamed to
> V4L2_CID_FLASH_EXTERNAL_STROBE_MODE.
> 
> - Removed CID_ from V4L2_CID_FLASH_EXTERNAL_STROBE_MODE values.
> 
> - Added a new use case based on [11]: Synchronised LED flash (hardware
> strobe, timed exposure).
> 
> - Added section on possible future extensions.
> 
> - Complemented the open question on units.
> 
> 
> Changes since v1 [7]
> ====================
> 
> - V4L2_FLASH_STROBE_MODE_EXT_STROBE renamed to
> V4L2_FLASH_STROBE_MODE_EXTERNAL.
> 
> - V4L2_CID_FLASH_STROBE control changed from button to bool.
> 
> - Removed suggestion of adding V4L2_CID_FLASH_DURATION.
> V4L2_CID_FLASH_TIMEOUT is used as hardware timeout.
> 
> - Added control access info (ro/rw).
> 
> - V4L2_FLASH_MODE_NONE added, V4L2_FLASH_LED_MODE_FLASH no longer forced
> as 1 in enum.
> 
> - Bits use (1 << x) instead of 0x00... format.
> 
> - Added an open question on flash LED mode controls.
> 
> - Added an open question on a new control:
> V4L2_CID_FLASH_EXTERNAL_STROBE_WHENCE.
> 
> - Added an open question on control units.
> 
> 
> Scope
> =====
> 
> This RFC is focused mostly on the ADP1653 [1] and similar chips [2, 3]
> which provides following functionality. [2, 3] mostly differ on the
> available faults --- for example, there are faults also for the
> indicator LED.
> 
> - High power LED output (flash or torch modes)
> - Low power indicator LED output (a.k.a. privacy light)
> - Programmable flash timeout
> - Software and hardware strobe
> - Fault detection
> 	- Overvoltage
> 	- Overtemperature
> 	- Short circuit
> 	- Timeout
> - Programmable current (both high-power and indicator LEDs)
> 
> If anyone else is aware of hardware which significantly differs from
> these and does not get served well under the proposed interface, please
> tell about it.
> 
> This RFC does NOT address the synchronisation of the flash to a given
> frame since this task is typically performed by the sensor through a
> strobe signal. The host does not have enough information for this ---
> exact timing information on the exposure of the sensor pixel array. In
> this case the flash synchronisation is visible to the flash controller
> as the hardware strobe originating from the sensor.
> 
> Flash synchronisation requires
> 
> 1) flash control capability from the sensor including a strobe output,
> 2) strobe input in the flash controller,
> 3) (optionally) ability to program sensor parameters at given frame,
> such as flash strobe, and
> 4) ability to read back metadata produced by the sensor related to a
> given frame. This should include whether the frame is exposed with
> flash, i.e. the sensor's flash strobe output.
> 
> Since we have little examples of both in terms of hardware support,
> which is in practice required, it was decided to postpone the interface
> specification for now. [6]
> 
> Xenon flash controllers exist but I don't have a specific example of
> those. Typically the interface is quite simple. Gpio pins for charge and
> strobe. The length of the strobe signal determines the strength of the
> flash pulse. The strobe is controlled by the sensor as for LED flash if
> it is hardware based.
> 
> See "Possible future extensions" section below for more.
> 
> 
> Known use cases
> ===============
> 
> The use case listed below concentrate on using a flash in a mobile
> device, for example in a mobile phone. The use cases could be somewhat
> different in devices the primary use of which is camera.
> 
> Unsynchronised LED flash (software strobe)
> ------------------------------------------
> 
> Unsynchronised LED flash is controlled directly by the host as the
> sensor. The flash must be enabled by the host before the exposure of the
> image starts and disabled once it ends. The host is fully responsible
> for the timing of the flash.
> 
> Example of such device: Nokia N900.
> 
> Synchronised LED flash (hardware strobe)
> ----------------------------------------
> 
> The synchronised LED flash is pre-programmed by the host (power and
> timeout) but controlled by the sensor through a strobe signal from the
> sensor to the flash.
> 
> The sensor controls the flash duration and timing. This control
> typically must be programmed to the sensor, and specifying an interface
> for this is out of scope of this RFC.
> 
> The LED flash controllers we know of can function in both synchronised
> and unsynchronised modes.
> 
> LED flash as torch
> ------------------
> 
> LED flash may be used as torch in conjunction with another use case
> involving camera or individually. [4]
> 
> Synchronised xenon flash
> ------------------------
> 
> The synchronised xenon flash is controlled more closely by the sensor
> than the LED flash. There is no separate intensity control for the xenon
> flash as its intensity is determined by the length of the strobe pulse.
> Several consecutive strobe pluses are possible but this needs to be
> still controlled by the sensor.
> 
> Synchronised xenon flash (timed exposure)
> -----------------------------------------
> 
> The same as above, but the flash is triggered around the end of the
> exposure. [11] This also probably requires a sensor which can do
> synchronous exposure, i.e. can start (and stop) the exposure of all
> lines at the same time.
> 
> 
> Proposed interface
> ==================
> 
> The flash, either LED or xenon, does not require large amounts of data
> to control it. There are parameters to control it but they are
> independent and assumably some hardware would only support some subsets
> of the functionality available somewhere else. Thus V4L2 controls seem
> an ideal way to support flash controllers.
> 
> A separate control class is reserved for the flash controls. It is
> called V4L2_CTRL_CLASS_FLASH.
> 
> Type of the control; type of flash is in parentheses after the control.
> 
> 
> 	V4L2_CID_FLASH_LED_MODE (menu; rw; LED)
> 
> enum v4l2_flash_led_mode {
> 	V4L2_FLASH_LED_MODE_NONE,
> 	V4L2_FLASH_LED_MODE_FLASH,
> 	V4L2_FLASH_LED_MODE_TORCH,
> };
> 
> 
> 	V4L2_CID_FLASH_STROBE (button; wo; LED, xenon)
> 
> Strobe the flash using software strobe from the host, typically over I2C
> or a GPIO. The flash is NOT synchronised to sensor pixel are exposure
> since the command is given asynchronously. Alternatively, if the flash
> controller is a master in the system, the sensor exposure may be
> triggered based on software strobe.

I would like a comment here that the driver will always stop the strobe
to prevent hardware damage, but that STROBE_STOP can be called to shut
down the strobe earlier.

Also, how to protect against nastiness like this:

struct v4l2_control ctrl = { V4L2_CID_FLASH_STROBE, 0 };

for (;;)
	ioctl(fd, VIDIOC_S_CTRL, &ctrl);

Return -EBUSY as long as either a strobe is in progress or if it is unsafe to
start a new one?

> 
> 
> 	V4L2_CID_FLASH_STROBE_STOP (button; wo; LED)
> 
> This control may be used to shut down the strobe immediately.
> 
> 
> 	V4L2_CID_FLASH_STROBE_STATUS (boolean; ro; LED)
> 
> This contol may be used to query the status of the strobe (on/off).
> 
> 
> 	V4L2_CID_FLASH_STROBE_MODE (menu; rw; LED)
> 
> Use hardware or software strobe. If hardware strobe is selected, the
> flash controller is a slave in the system where the sensor produces the
> strobe signal to the flash.
> 
> In this case the flash controller setup is limited to programming strobe
> timeout and power (LED flash) and the sensor controls the timing and
> length of the strobe.
> 
> enum v4l2_flash_strobe_whence {
> 	V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
> 	V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
> };

Perhaps use 'type' instead of 'whence'? English isn't my native language,
but it sounds pretty archaic to me.

> 
> 
> 	V4L2_CID_FLASH_TIMEOUT (integer; rw; LED)
> 
> The flash controller provides timeout functionality to shut down the led
> in case the host fails to do that. For hardware strobe, this is the
> maximum amount of time the flash should stay on, and the purpose of the
> setting is to prevent the LED from catching fire.

It is a bit ambiguous. I assume you mean to say that the application can
set the timeout to any value less than the maximum of the control (which
is the maximum allowed by the hardware).
 
> For software strobe, the setting may be used to limit the length of the
> strobe using a hardware watchdog. The granularity of the timeout in [1,
> 2, 3] is very coarse.
> 
> A standard unit such as ms or µs should be used.
> 
> 
> 	V4L2_CID_FLASH_INTENSITY (integer; rw; LED)
> 
> Intensity of the flash in hardware specific units. The LED flash
> controller provides current to the LED but the actual luminous power is
> dictated by the LED connected to the controller.
> 
> 
> 	V4L2_CID_FLASH_TORCH_INTENSITY (integer; rw; LED)
> 
> Intensity of the flash in hardware specific units.
> 
> 
> 	V4L2_CID_FLASH_INDICATOR_INTENSITY (integer; rw; LED)
> 
> Intensity of the indicator light in hardware specific units.
> 
> 
> 	V4L2_CID_FLASH_FAULT (bit field; ro; LED)
> 
> This is a bitmask containing the fault information for the flash. This
> assumes the proposed V4L2 bit mask controls [5]; otherwise this would
> likely need to be a set of controls.
> 
> #define V4L2_FLASH_FAULT_OVER_VOLTAGE		(1 << 0)
> #define V4L2_FLASH_FAULT_TIMEOUT		(1 << 1)
> #define V4L2_FLASH_FAULT_OVER_TEMPERATURE	(1 << 2)
> #define V4L2_FLASH_FAULT_SHORT_CIRCUIT		(1 << 3)
> 
> Several faults may occur at single occasion. The ADP1653 is able to
> inform the user a fault has occurred, so a V4L2 control event (proposed
> earlier) could be used for that.
> 
> These faults are supported by the ADP1653. More faults may be added as
> support for more chips require that. In some other hardware faults are
> available for indicator led as well.
> 
> 
> 	V4L2_CID_FLASH_CHARGE (bool; rw; xenon)
> 
> Charge control for the xenon flash. Enable or disable charging.
> 
> 
> 	V4L2_CID_FLASH_READY (bool; ro; xenon, LED)
> 
> Flash is ready to strobe. On xenon flash this tells the capacitor has
> been charged, on LED flash it's that the LED is no longer too hot.
> 
> The implementation on LED flash may be modelling the temperature
> behaviour of the LED in the driver (or elsewhere, e.g. library or board
> code) if the hardware does not provide direct temperature information
> from the LED.
> 
> A V4L2 control event should be produced whenever the flash becomes ready.
> 
> 
> Open questions
> ==============
> 
> 1. Flash LED mode control
> -------------------------
> 
> Should the flash led mode control be a single control or a set of
> controls? A single control would make it easier for application to
> choose between different modes, but on the other hand limits future
> extensibility if there would have to be further splitting of the modes. [8]

I think I like the current proposal best.

> 3. Units
> --------
> 
> Which units should e.g. V4L2_CID_FLASH_TIMEOUT be using? It'd be very
> useful to have standard unit on this control, like ms or µs.

This should be standardized. The question is how flexible we want this.
I would definitely go for µs or perhaps even ns. But what should the
maximum duration be? Are there extreme cases where you can have a really
long flash in special types of photography? Perhaps this should be a 64-bit
number?

> Some controls, like V4L2_CID_FLASH_INTENSITY, can't easily use a
> standard unit. The luminous output depends on the LED connected (you
> could use an incandescent lightbulb too, I suppose?) to the flash
> controller. These controls typically control the current supplied by the
> flash controller.
> 
> The timing controls on the sensor could use pixels (or lines) since
> these are the units the sensor uses itself. Converting between pixels
> (or lines) and SI units is trivial since the pixel clock is known in the
> user space.
> 
> The flash chip, on the other hand, has no knowledge of sensor pixel
> clock, so it should use SI units. Also, the timing control currently in
> the flash chip itself is very coarse.
> 
> PROPOSAL: If the component has internal timing which is known elsewhere,
> such the sensor, the controls should use these units. Otherwise,
> standard SI units should be used. In the sensor's case, this could be µs.

Not sure about that. I would go for SI units at all time.

> 
> 
> Possible future extensions
> ==========================
> 
> 1. Indicator faults
> -------------------
> 
> Some chips do support these. As they are part of the same register (at
> least on some chips [10]), they could be part of V4L2_CID_FLASH_FAULT
> control.
> 
> 2. Strobe output control on sensor
> ----------------------------------
> 
> Sensor requires strobe output control to trigger hardware strobe (on
> next possible frame).
> 
> 3. Sensor metadata on frames
> ----------------------------
> 
> It'd be useful to be able to read back sensor metadata. If the flash is
> strobed (on sensor hardware) while streaming, it's difficult to know
> otherwise which frame in the stream has been exposed with flash.

I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer flag?
That way userspace can tell if that particular frame was taken with flash.

> 4. Timing of strobe
> -------------------
> 
> Hardware strobe timing relative to frame start. The length of the strobe
> should be timeable as well.
> 
> 5. Flash type control
> ---------------------
> 
> Probably it'd be good to be able to tell which kind of flash we have
> (xenon/LED). This should be added no later than we have a xenon flash
> driver.
> 
> V4L2_CID_FLASH_TYPE
> 
> 6. V4L2_CID_FLASH_EXTERNAL_STROBE_MODE
> --------------------------------------
> 
> The implementation of this control is postponed until we have a driver
> which implements this functionality.
> 
> 	V4L2_CID_FLASH_EXTERNAL_STROBE_MODE (bool; rw; xenon, LED)
> 
> Whether the flash controller considers external strobe as edge, when the
> only limit of the strobe is the timeout on flash controller, or level,
> when the flash strobe will last as long as the strobe signal, or as long
> until the timeout expires.
> 
> enum v4l2_flash_external_strobe_mode {
> 	V4L2_FLASH_EXTERNAL_STROBE_MODE_LEVEL,
> 	V4L2_FLASH_EXTERNAL_STROBE_MODE_EDGE,
> };
> 
> Should V4L2_CID_FLASH_EXTERNAL_STROBE_MODE be renamed? Are _EDGE and
> _LEVEL menu values enough to describe the functionality?
> 
> 
> References
> ==========
> 
> [1] ADP1653 datasheet.
> http://www.analog.com/static/imported-files/data_sheets/ADP1653.pdf
> 
> [2] LM3555. http://www.national.com/mpf/LM/LM3555.html#Overview
> 
> [3] AS3645 product brief.
> http://www.austriamicrosystems.com/eng/Products/Lighting-Management/Camera-Flash-LED-Drivers/AS3645
> 
> [4] Flashlight applet for the N900.
> http://maemo.org/downloads/product/Maemo5/flashlight-applet/
> 
> [5] V4L2 Warszaw brainstorming meeting notes, day 2.
> http://www.retiisi.org.uk/v4l2/v4l2-brainstorming-warsaw-2011-03/notes/day%202%20(SGz6LU2esk).html
> 
> [6] V4L2 Warszaw brainstorming meeting notes, day 3.
> http://www.retiisi.org.uk/v4l2/v4l2-brainstorming-warsaw-2011-03/notes/day%203%20(RhoYa0X9D7).html
> 
> [7] [RFC] V4L2 flash API for flash devices.
> http://www.spinics.net/lists/linux-media/msg30725.html
> 
> [8] http://www.spinics.net/lists/linux-media/msg30794.html
> 
> [9] [RFC v2] V4L2 flash API for flash devices.
> http://www.spinics.net/lists/linux-media/msg31135.html
> 
> [10] AS3645 datasheet.
> http://www.cdiweb.com/datasheets/austriamicro/AS3645_Datasheet_v1-6.pdf
> 
> [11] Andrew's use cases for flash.
> http://www.spinics.net/lists/linux-media/msg31363.html
> 
> [12] [RFC v3] V4L2 flash API for flash devices.
> http://www.spinics.net/lists/linux-media/msg31425.html
> 
> 
> Cheers,
> 
> 

I think this is starting to look good.

Regards,

	Hans

Re: [RFC v4] V4L2 API for flash devices

[Sylwester Nawrocki]

Hi Hans,

On 05/07/2011 02:46 PM, Hans Verkuil wrote:
> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>> Hi,
>>
>> This is a fourth proposal for an interface for controlling flash devices
>> on the V4L2/v4l2_subdev APIs.
>>
>> I want to thank everyone who have participated to the development of the
>> flash interface.
>>
>> Comments and questions are very, very welcome as always.
>>
>>
>> Changes since v3 [12]
>> =====================
>>
>> - V4L2_CID_FLASH_STROBE changed to button control,
>> V4L2_CID_FLASH_STROBE_STOP button control added,
>> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
>>
>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
>>
>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
>>
>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and
>> V4L2_CID_INDICATOR_INTENSITY renamed to V4L2_CID_FLASH_INDICATOR_INTENSITY.
>>
>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
>> extensions".
>>

[snip]

>>
>> 3. Sensor metadata on frames
>> ----------------------------
>>
>> It'd be useful to be able to read back sensor metadata. If the flash is
>> strobed (on sensor hardware) while streaming, it's difficult to know
>> otherwise which frame in the stream has been exposed with flash.
>
> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer flag?
> That way userspace can tell if that particular frame was taken with flash.

This looks more as a workaround for the problem rather than a good long 
term solution. It might be tempting to use the buffer flags which seem
to be be more or less intended for buffer control.
I'd like much more to see a buffer flags to be used to indicate whether
an additional plane of (meta)data is carried by the buffer.
There seem to be many more parameters, than a single flag indicating 
whether the frame has been exposed with flash or not, needed to be 
carried over to user space.
But then we might need some standard format of the meta data, perhaps
control id/value pairs and possibly a per plane configurable memory
type.

Also as Sakari indicated some sensors adopt custom meta data formats
so maybe we need to introduce standard fourcc like IDs for meta data
formats? I am not sure whether it is possible to create common 
description of an image meta data that fits all H/W.

--
Regards,
Sylwester

Re: [RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Hans Verkuil wrote:
[clip]
>> 	V4L2_CID_FLASH_STROBE (button; wo; LED, xenon)
>>
>> Strobe the flash using software strobe from the host, typically over I2C
>> or a GPIO. The flash is NOT synchronised to sensor pixel are exposure
>> since the command is given asynchronously. Alternatively, if the flash
>> controller is a master in the system, the sensor exposure may be
>> triggered based on software strobe.
> 
> I would like a comment here that the driver will always stop the strobe
> to prevent hardware damage, but that STROBE_STOP can be called to shut
> down the strobe earlier.
> 
> Also, how to protect against nastiness like this:
> 
> struct v4l2_control ctrl = { V4L2_CID_FLASH_STROBE, 0 };
> 
> for (;;)
> 	ioctl(fd, VIDIOC_S_CTRL, &ctrl);
> 
> Return -EBUSY as long as either a strobe is in progress or if it is unsafe to
> start a new one?

This is not implemented in the current adp1653 driver but yes, it should
be done. However, it isn't simple to implement in a way that it would
still allow the optimal use of the flash.

I'll add that to the todo list for the driver.

Modelling of the flash LED cooldown period should be relatively
independent of the board, actual LED and the flash chip, so this might
make a nice library in the future.

>>
>>
>> 	V4L2_CID_FLASH_STROBE_STOP (button; wo; LED)
>>
>> This control may be used to shut down the strobe immediately.
>>
>>
>> 	V4L2_CID_FLASH_STROBE_STATUS (boolean; ro; LED)
>>
>> This contol may be used to query the status of the strobe (on/off).
>>
>>
>> 	V4L2_CID_FLASH_STROBE_MODE (menu; rw; LED)
>>
>> Use hardware or software strobe. If hardware strobe is selected, the
>> flash controller is a slave in the system where the sensor produces the
>> strobe signal to the flash.
>>
>> In this case the flash controller setup is limited to programming strobe
>> timeout and power (LED flash) and the sensor controls the timing and
>> length of the strobe.
>>
>> enum v4l2_flash_strobe_whence {
>> 	V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
>> 	V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
>> };
> 
> Perhaps use 'type' instead of 'whence'? English isn't my native language,
> but it sounds pretty archaic to me.

Neither it is mine. I actually forgot to change the name of the enum to
v4l2_flash_strobe_mode so it would match the control.

What do you think of that?

>>
>>
>> 	V4L2_CID_FLASH_TIMEOUT (integer; rw; LED)
>>
>> The flash controller provides timeout functionality to shut down the led
>> in case the host fails to do that. For hardware strobe, this is the
>> maximum amount of time the flash should stay on, and the purpose of the
>> setting is to prevent the LED from catching fire.
> 
> It is a bit ambiguous. I assume you mean to say that the application can
> set the timeout to any value less than the maximum of the control (which
> is the maximum allowed by the hardware).

Yes; that's what I intended. I'll attempt to formulate it better in the
forthcoming patches. :-)

[clip]

>> Open questions
>> ==============
>>
>> 1. Flash LED mode control
>> -------------------------
>>
>> Should the flash led mode control be a single control or a set of
>> controls? A single control would make it easier for application to
>> choose between different modes, but on the other hand limits future
>> extensibility if there would have to be further splitting of the modes. [8]
> 
> I think I like the current proposal best.

I agree.

>> 3. Units
>> --------
>>
>> Which units should e.g. V4L2_CID_FLASH_TIMEOUT be using? It'd be very
>> useful to have standard unit on this control, like ms or µs.
> 
> This should be standardized. The question is how flexible we want this.
> I would definitely go for µs or perhaps even ns. But what should the
> maximum duration be? Are there extreme cases where you can have a really
> long flash in special types of photography? Perhaps this should be a 64-bit
> number?

On xenon flashes the strobe strength is controlled by strobe duration,
as far as I understand. I think it would be better to hide that under
the V4L2_CID_FLASH_INTENSITY, though.

The timeout on adp1653 the timeout is really coarse, there are 16
different values for it.

On other chips more precise control could be possible.

>> Some controls, like V4L2_CID_FLASH_INTENSITY, can't easily use a
>> standard unit. The luminous output depends on the LED connected (you
>> could use an incandescent lightbulb too, I suppose?) to the flash
>> controller. These controls typically control the current supplied by the
>> flash controller.
>>
>> The timing controls on the sensor could use pixels (or lines) since
>> these are the units the sensor uses itself. Converting between pixels
>> (or lines) and SI units is trivial since the pixel clock is known in the
>> user space.
>>
>> The flash chip, on the other hand, has no knowledge of sensor pixel
>> clock, so it should use SI units. Also, the timing control currently in
>> the flash chip itself is very coarse.
>>
>> PROPOSAL: If the component has internal timing which is known elsewhere,
>> such the sensor, the controls should use these units. Otherwise,
>> standard SI units should be used. In the sensor's case, this could be µs.
> 
> Not sure about that. I would go for SI units at all time.

Sometimes that's not possible.

The LED flash current on the adp1653 is known but I don't think that's
an universal property of LED flash controllers.

Which brings up another question: we have essentially the same control
but sometimes depending on the chip, the unit could be different. It
would be good to be able to tell this to the user space.

I think it would be nice to be able to attach unit and prefix (or
10^exponent) to the control. VIDIOC_QUERYCTRL would be the natural place
for passing such information to user space.

However, there are just two reserved fields left in struct
v4l2_queryctrl. Do you think it would be good use of those fields to add
such information there? Four bytes would definitely be enough for the
two together, perhaps even less.

>>
>>
>> Possible future extensions
>> ==========================
>>
>> 1. Indicator faults
>> -------------------
>>
>> Some chips do support these. As they are part of the same register (at
>> least on some chips [10]), they could be part of V4L2_CID_FLASH_FAULT
>> control.
>>
>> 2. Strobe output control on sensor
>> ----------------------------------
>>
>> Sensor requires strobe output control to trigger hardware strobe (on
>> next possible frame).
>>
>> 3. Sensor metadata on frames
>> ----------------------------
>>
>> It'd be useful to be able to read back sensor metadata. If the flash is
>> strobed (on sensor hardware) while streaming, it's difficult to know
>> otherwise which frame in the stream has been exposed with flash.
> 
> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer flag?
> That way userspace can tell if that particular frame was taken with flash.

That would be one way, yes.

But whether a video buffer had flash exposure is just one property of
the buffer; there are several such properties, for example exposure time
and gain.

I think this would be best resolved as part of the rest of the metadata
passing to user space.

>> 4. Timing of strobe
>> -------------------
>>
>> Hardware strobe timing relative to frame start. The length of the strobe
>> should be timeable as well.
>>
>> 5. Flash type control
>> ---------------------
>>
>> Probably it'd be good to be able to tell which kind of flash we have
>> (xenon/LED). This should be added no later than we have a xenon flash
>> driver.
>>
>> V4L2_CID_FLASH_TYPE
>>
>> 6. V4L2_CID_FLASH_EXTERNAL_STROBE_MODE
>> --------------------------------------
>>
>> The implementation of this control is postponed until we have a driver
>> which implements this functionality.
>>
>> 	V4L2_CID_FLASH_EXTERNAL_STROBE_MODE (bool; rw; xenon, LED)
>>
>> Whether the flash controller considers external strobe as edge, when the
>> only limit of the strobe is the timeout on flash controller, or level,
>> when the flash strobe will last as long as the strobe signal, or as long
>> until the timeout expires.
>>
>> enum v4l2_flash_external_strobe_mode {
>> 	V4L2_FLASH_EXTERNAL_STROBE_MODE_LEVEL,
>> 	V4L2_FLASH_EXTERNAL_STROBE_MODE_EDGE,
>> };
>>
>> Should V4L2_CID_FLASH_EXTERNAL_STROBE_MODE be renamed? Are _EDGE and
>> _LEVEL menu values enough to describe the functionality?
>>
>>
>> References
>> ==========
>>
>> [1] ADP1653 datasheet.
>> http://www.analog.com/static/imported-files/data_sheets/ADP1653.pdf
>>
>> [2] LM3555. http://www.national.com/mpf/LM/LM3555.html#Overview
>>
>> [3] AS3645 product brief.
>> http://www.austriamicrosystems.com/eng/Products/Lighting-Management/Camera-Flash-LED-Drivers/AS3645
>>
>> [4] Flashlight applet for the N900.
>> http://maemo.org/downloads/product/Maemo5/flashlight-applet/
>>
>> [5] V4L2 Warszaw brainstorming meeting notes, day 2.
>> http://www.retiisi.org.uk/v4l2/v4l2-brainstorming-warsaw-2011-03/notes/day%202%20(SGz6LU2esk).html
>>
>> [6] V4L2 Warszaw brainstorming meeting notes, day 3.
>> http://www.retiisi.org.uk/v4l2/v4l2-brainstorming-warsaw-2011-03/notes/day%203%20(RhoYa0X9D7).html
>>
>> [7] [RFC] V4L2 flash API for flash devices.
>> http://www.spinics.net/lists/linux-media/msg30725.html
>>
>> [8] http://www.spinics.net/lists/linux-media/msg30794.html
>>
>> [9] [RFC v2] V4L2 flash API for flash devices.
>> http://www.spinics.net/lists/linux-media/msg31135.html
>>
>> [10] AS3645 datasheet.
>> http://www.cdiweb.com/datasheets/austriamicro/AS3645_Datasheet_v1-6.pdf
>>
>> [11] Andrew's use cases for flash.
>> http://www.spinics.net/lists/linux-media/msg31363.html
>>
>> [12] [RFC v3] V4L2 flash API for flash devices.
>> http://www.spinics.net/lists/linux-media/msg31425.html
>>
>>
>> Cheers,
>>
>>
> 
> I think this is starting to look good.

Thanks! :-)

Kidn regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Sylwester Nawrocki wrote:
> Hi Hans,

Hi Sylwester, Hans,

> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>> Hi,
>>>
>>> This is a fourth proposal for an interface for controlling flash devices
>>> on the V4L2/v4l2_subdev APIs.
>>>
>>> I want to thank everyone who have participated to the development of the
>>> flash interface.
>>>
>>> Comments and questions are very, very welcome as always.
>>>
>>>
>>> Changes since v3 [12]
>>> =====================
>>>
>>> - V4L2_CID_FLASH_STROBE changed to button control,
>>> V4L2_CID_FLASH_STROBE_STOP button control added,
>>> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
>>>
>>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
>>>
>>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
>>>
>>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and
>>> V4L2_CID_INDICATOR_INTENSITY renamed to
>>> V4L2_CID_FLASH_INDICATOR_INTENSITY.
>>>
>>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
>>> extensions".
>>>
> 
> [snip]
> 
>>>
>>> 3. Sensor metadata on frames
>>> ----------------------------
>>>
>>> It'd be useful to be able to read back sensor metadata. If the flash is
>>> strobed (on sensor hardware) while streaming, it's difficult to know
>>> otherwise which frame in the stream has been exposed with flash.
>>
>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
>> flag?
>> That way userspace can tell if that particular frame was taken with
>> flash.
> 
> This looks more as a workaround for the problem rather than a good long
> term solution. It might be tempting to use the buffer flags which seem
> to be be more or less intended for buffer control.
> I'd like much more to see a buffer flags to be used to indicate whether
> an additional plane of (meta)data is carried by the buffer.
> There seem to be many more parameters, than a single flag indicating
> whether the frame has been exposed with flash or not, needed to be
> carried over to user space.
> But then we might need some standard format of the meta data, perhaps
> control id/value pairs and possibly a per plane configurable memory
> type.

There are multiple possible approaches for this.

For sensors where metadata is register-value pairs, that is, essentially
V4L2 control values, I think this should be parsed by the sensor driver.
The ISP (camera bridge) driver does receive the data so it'd have to
"ask for help" from the sensor driver.

As discussed previously, using V4L2 control events shouldn't probably be
the way to go, but I think it's obvious that this is _somehow_ bound to
controls, at least control ids.

> Also as Sakari indicated some sensors adopt custom meta data formats
> so maybe we need to introduce standard fourcc like IDs for meta data
> formats? I am not sure whether it is possible to create common
> description of an image meta data that fits all H/W.

I'm not sure either since I know of only one example. That example, i.e.
register-value pairs, should be something that I'd assume _some_ other
hardware uses as well, but there could exist also hardware which
doesn't. This solution might not work on that hardware.

If there is metadata which does not associate to V4L2 controls (or
ioctls), either existing or not yet existing, then that probably should
be parsed by the driver. On the other hand, I can't think of metadata
that wouldn't fall under this right now. :-)

Do you know more examples of sensor produced metadata than SMIA++?

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Sylwester Nawrocki]

Hi Sakari,

On 05/09/2011 12:11 AM, Sakari Ailus wrote:
> Sylwester Nawrocki wrote:
>> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
>>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>> Hi,
>>>>
>>>> This is a fourth proposal for an interface for controlling flash devices
>>>> on the V4L2/v4l2_subdev APIs.
>>>>
>>>> I want to thank everyone who have participated to the development of the
>>>> flash interface.
>>>>
>>>> Comments and questions are very, very welcome as always.
>>>>
>>>>
>>>> Changes since v3 [12]
>>>> =====================
>>>>
>>>> - V4L2_CID_FLASH_STROBE changed to button control,
>>>> V4L2_CID_FLASH_STROBE_STOP button control added,
>>>> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
>>>>
>>>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
>>>>
>>>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
>>>>
>>>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and
>>>> V4L2_CID_INDICATOR_INTENSITY renamed to
>>>> V4L2_CID_FLASH_INDICATOR_INTENSITY.
>>>>
>>>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
>>>> extensions".
>>>>
>>
>> [snip]
>>
>>>>
>>>> 3. Sensor metadata on frames
>>>> ----------------------------
>>>>
>>>> It'd be useful to be able to read back sensor metadata. If the flash is
>>>> strobed (on sensor hardware) while streaming, it's difficult to know
>>>> otherwise which frame in the stream has been exposed with flash.
>>>
>>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
>>> flag?
>>> That way userspace can tell if that particular frame was taken with
>>> flash.
>>
>> This looks more as a workaround for the problem rather than a good long
>> term solution. It might be tempting to use the buffer flags which seem
>> to be be more or less intended for buffer control.
>> I'd like much more to see a buffer flags to be used to indicate whether
>> an additional plane of (meta)data is carried by the buffer.
>> There seem to be many more parameters, than a single flag indicating
>> whether the frame has been exposed with flash or not, needed to be
>> carried over to user space.
>> But then we might need some standard format of the meta data, perhaps
>> control id/value pairs and possibly a per plane configurable memory
>> type.
> 
> There are multiple possible approaches for this.
> 
> For sensors where metadata is register-value pairs, that is, essentially
> V4L2 control values, I think this should be parsed by the sensor driver.
> The ISP (camera bridge) driver does receive the data so it'd have to
> "ask for help" from the sensor driver.

I am inclined to let the ISP drivers parse the data but on the other hand
it might be difficult to access same DMA buffers in kernel _and_ user space.

> 
> As discussed previously, using V4L2 control events shouldn't probably be
> the way to go, but I think it's obvious that this is _somehow_ bound to
> controls, at least control ids.
> 
>> Also as Sakari indicated some sensors adopt custom meta data formats
>> so maybe we need to introduce standard fourcc like IDs for meta data
>> formats? I am not sure whether it is possible to create common
>> description of an image meta data that fits all H/W.
> 
> I'm not sure either since I know of only one example. That example, i.e.
> register-value pairs, should be something that I'd assume _some_ other
> hardware uses as well, but there could exist also hardware which
> doesn't. This solution might not work on that hardware.

Of course it's hard to find a silver bullet for a hardware we do not know ;)

> 
> If there is metadata which does not associate to V4L2 controls (or
> ioctls), either existing or not yet existing, then that probably should
> be parsed by the driver. On the other hand, I can't think of metadata
> that wouldn't fall under this right now. :-)

Some metadata are arrays of length specific to a given attribute,
I wonder how to support that with v4l2 controls ?

> 
> Do you know more examples of sensor produced metadata than SMIA++?

The only metadata I've had a bit experience with was regular EXIF tags
which could be retrieved from ISP through I2C bus.
These were mostly fixed point arithmetic numbers in [32-bit numerator/
32-bit denominator] form carrying exposure time, shutter speed, aperture,
brightness, flash, etc. information. The tags could be read from ISP after
it buffered a frame in its memory and processed it.
In case of a JPEG image format the tags can be embedded into the main
image file. But the image processors not always supported that so we used
to have an ioctl for the purpose of retrieving the metadata in user space.
In some cases it is desired to read data directly from the driver rather
than parsing a relatively large buffer.
It would be good to have a uniform interface for passing such data to
applications. I think in that particular use case a control id/value pair
sequences would do.

--
Regards,
Sylwester 

Re: [RFC v4] V4L2 API for flash devices

[Andy Walls]

On Sat, 2011-05-07 at 14:46 +0200, Hans Verkuil wrote:
> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
> >
> > 
> > enum v4l2_flash_strobe_whence {
> > 	V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
> > 	V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
> > };
> 
> Perhaps use 'type' instead of 'whence'? English isn't my native language,
> but it sounds pretty archaic to me.

"SOURCE" is better than "WHENCE" here.


"whence" is certainly very formal and used very little.  "whence" likely
still gets some use in English, simply because a terse synonym doesn't
exist.

The problem with using whence is that many English speakers won't know
its correct definition.

	"whence" means "from what source, origin, or place"

In your use here, the implicit "from" in the definition of whence is
essential.  However, most (American) English speakers that I know think
"whence" simply means "where".

Regards,
Andy

Re: [RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Andy Walls wrote:
> On Sat, 2011-05-07 at 14:46 +0200, Hans Verkuil wrote:
>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>
>>>
>>> enum v4l2_flash_strobe_whence {
>>> 	V4L2_FLASH_STROBE_WHENCE_SOFTWARE,
>>> 	V4L2_FLASH_STROBE_WHENCE_EXTERNAL,
>>> };
>>
>> Perhaps use 'type' instead of 'whence'? English isn't my native language,
>> but it sounds pretty archaic to me.
> 
> "SOURCE" is better than "WHENCE" here.
> 
> 
> "whence" is certainly very formal and used very little.  "whence" likely
> still gets some use in English, simply because a terse synonym doesn't
> exist.
> 
> The problem with using whence is that many English speakers won't know
> its correct definition.
> 
> 	"whence" means "from what source, origin, or place"
> 
> In your use here, the implicit "from" in the definition of whence is
> essential.  However, most (American) English speakers that I know think
> "whence" simply means "where".

Thanks for the feedback, Andy!

WHENCE has since changed to MODE (or at least should have been), but I
think SOURCE is even better. I'll switch to that.

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Sylwester Nawrocki wrote:
> Hi Sakari,

Hi Sylwester,

> On 05/09/2011 12:11 AM, Sakari Ailus wrote:
>> Sylwester Nawrocki wrote:
>>> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
>>>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>>> Hi,
>>>>>
>>>>> This is a fourth proposal for an interface for controlling flash devices
>>>>> on the V4L2/v4l2_subdev APIs.
>>>>>
>>>>> I want to thank everyone who have participated to the development of the
>>>>> flash interface.
>>>>>
>>>>> Comments and questions are very, very welcome as always.
>>>>>
>>>>>
>>>>> Changes since v3 [12]
>>>>> =====================
>>>>>
>>>>> - V4L2_CID_FLASH_STROBE changed to button control,
>>>>> V4L2_CID_FLASH_STROBE_STOP button control added,
>>>>> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
>>>>>
>>>>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
>>>>>
>>>>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
>>>>>
>>>>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY and
>>>>> V4L2_CID_INDICATOR_INTENSITY renamed to
>>>>> V4L2_CID_FLASH_INDICATOR_INTENSITY.
>>>>>
>>>>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
>>>>> extensions".
>>>>>
>>>
>>> [snip]
>>>
>>>>>
>>>>> 3. Sensor metadata on frames
>>>>> ----------------------------
>>>>>
>>>>> It'd be useful to be able to read back sensor metadata. If the flash is
>>>>> strobed (on sensor hardware) while streaming, it's difficult to know
>>>>> otherwise which frame in the stream has been exposed with flash.
>>>>
>>>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
>>>> flag?
>>>> That way userspace can tell if that particular frame was taken with
>>>> flash.
>>>
>>> This looks more as a workaround for the problem rather than a good long
>>> term solution. It might be tempting to use the buffer flags which seem
>>> to be be more or less intended for buffer control.
>>> I'd like much more to see a buffer flags to be used to indicate whether
>>> an additional plane of (meta)data is carried by the buffer.
>>> There seem to be many more parameters, than a single flag indicating
>>> whether the frame has been exposed with flash or not, needed to be
>>> carried over to user space.
>>> But then we might need some standard format of the meta data, perhaps
>>> control id/value pairs and possibly a per plane configurable memory
>>> type.
>>
>> There are multiple possible approaches for this.
>>
>> For sensors where metadata is register-value pairs, that is, essentially
>> V4L2 control values, I think this should be parsed by the sensor driver.
>> The ISP (camera bridge) driver does receive the data so it'd have to
>> "ask for help" from the sensor driver.
> 
> I am inclined to let the ISP drivers parse the data but on the other hand
> it might be difficult to access same DMA buffers in kernel _and_ user space.

This is just about mapping the buffer to both kernel and user spaces. If
the ISP has an iommu the kernel mapping might already exist if it comes
from vmalloc().

>> As discussed previously, using V4L2 control events shouldn't probably be
>> the way to go, but I think it's obvious that this is _somehow_ bound to
>> controls, at least control ids.
>>
>>> Also as Sakari indicated some sensors adopt custom meta data formats
>>> so maybe we need to introduce standard fourcc like IDs for meta data
>>> formats? I am not sure whether it is possible to create common
>>> description of an image meta data that fits all H/W.
>>
>> I'm not sure either since I know of only one example. That example, i.e.
>> register-value pairs, should be something that I'd assume _some_ other
>> hardware uses as well, but there could exist also hardware which
>> doesn't. This solution might not work on that hardware.
> 
> Of course it's hard to find a silver bullet for a hardware we do not know ;)
> 
>>
>> If there is metadata which does not associate to V4L2 controls (or
>> ioctls), either existing or not yet existing, then that probably should
>> be parsed by the driver. On the other hand, I can't think of metadata
>> that wouldn't fall under this right now. :-)
> 
> Some metadata are arrays of length specific to a given attribute,
> I wonder how to support that with v4l2 controls ?

Is the metadata something which really isn't associated to any V4L2
control? Are we now talking about a sensor which is more complex than a
regular raw bayer sensor?

>> Do you know more examples of sensor produced metadata than SMIA++?
> 
> The only metadata I've had a bit experience with was regular EXIF tags
> which could be retrieved from ISP through I2C bus.

That obviously won't map to V4L2 controls.

This should very likely be just passed to user space as-is as
different... plane?

In some cases it's time critical to pass data to user space that
otherwise could be associated with a video buffer. I wonder if this case
is time critical or not.

> These were mostly fixed point arithmetic numbers in [32-bit numerator/
> 32-bit denominator] form carrying exposure time, shutter speed, aperture,
> brightness, flash, etc. information. The tags could be read from ISP after
> it buffered a frame in its memory and processed it.
> In case of a JPEG image format the tags can be embedded into the main
> image file. But the image processors not always supported that so we used
> to have an ioctl for the purpose of retrieving the metadata in user space.
> In some cases it is desired to read data directly from the driver rather
> than parsing a relatively large buffer.
> It would be good to have a uniform interface for passing such data to
> applications. I think in that particular use case a control id/value pair
> sequences would do.

Do you think this is "control id" or non-control id, and whether the
value is the same data type than the V4L2 control would be? That would
match to what I'm aware of, too.

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Laurent Pinchart]

Hi Sakari,

On Tuesday 17 May 2011 22:34:36 Sakari Ailus wrote:
> Sylwester Nawrocki wrote:
> > On 05/09/2011 12:11 AM, Sakari Ailus wrote:
> >> Sylwester Nawrocki wrote:
> >>> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
> >>>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
> >>>>> Hi,
> >>>>> 
> >>>>> This is a fourth proposal for an interface for controlling flash
> >>>>> devices on the V4L2/v4l2_subdev APIs.
> >>>>> 
> >>>>> I want to thank everyone who have participated to the development of
> >>>>> the flash interface.
> >>>>> 
> >>>>> Comments and questions are very, very welcome as always.
> >>>>> 
> >>>>> 
> >>>>> Changes since v3 [12]
> >>>>> =====================
> >>>>> 
> >>>>> - V4L2_CID_FLASH_STROBE changed to button control,
> >>>>> V4L2_CID_FLASH_STROBE_STOP button control added,
> >>>>> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
> >>>>> 
> >>>>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
> >>>>> 
> >>>>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
> >>>>> 
> >>>>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY
> >>>>> and V4L2_CID_INDICATOR_INTENSITY renamed to
> >>>>> V4L2_CID_FLASH_INDICATOR_INTENSITY.
> >>>>> 
> >>>>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
> >>>>> extensions".
> >>> 
> >>> [snip]
> >>> 
> >>>>> 3. Sensor metadata on frames
> >>>>> ----------------------------
> >>>>> 
> >>>>> It'd be useful to be able to read back sensor metadata. If the flash
> >>>>> is strobed (on sensor hardware) while streaming, it's difficult to
> >>>>> know otherwise which frame in the stream has been exposed with
> >>>>> flash.
> >>>> 
> >>>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
> >>>> flag?
> >>>> That way userspace can tell if that particular frame was taken with
> >>>> flash.
> >>> 
> >>> This looks more as a workaround for the problem rather than a good long
> >>> term solution. It might be tempting to use the buffer flags which seem
> >>> to be be more or less intended for buffer control.
> >>> I'd like much more to see a buffer flags to be used to indicate whether
> >>> an additional plane of (meta)data is carried by the buffer.
> >>> There seem to be many more parameters, than a single flag indicating
> >>> whether the frame has been exposed with flash or not, needed to be
> >>> carried over to user space.
> >>> But then we might need some standard format of the meta data, perhaps
> >>> control id/value pairs and possibly a per plane configurable memory
> >>> type.
> >> 
> >> There are multiple possible approaches for this.
> >> 
> >> For sensors where metadata is register-value pairs, that is, essentially
> >> V4L2 control values, I think this should be parsed by the sensor driver.
> >> The ISP (camera bridge) driver does receive the data so it'd have to
> >> "ask for help" from the sensor driver.
> > 
> > I am inclined to let the ISP drivers parse the data but on the other hand
> > it might be difficult to access same DMA buffers in kernel _and_ user
> > space.
> 
> This is just about mapping the buffer to both kernel and user spaces. If
> the ISP has an iommu the kernel mapping might already exist if it comes
> from vmalloc().

And we're also trying to get rid of that mapping to facilitate cache 
management. Any API we create for metadata parsing will need to take potential 
cache-related performances issues into account at the design stage.

> >> As discussed previously, using V4L2 control events shouldn't probably be
> >> the way to go, but I think it's obvious that this is _somehow_ bound to
> >> controls, at least control ids.
> >> 
> >>> Also as Sakari indicated some sensors adopt custom meta data formats
> >>> so maybe we need to introduce standard fourcc like IDs for meta data
> >>> formats? I am not sure whether it is possible to create common
> >>> description of an image meta data that fits all H/W.
> >> 
> >> I'm not sure either since I know of only one example. That example, i.e.
> >> register-value pairs, should be something that I'd assume _some_ other
> >> hardware uses as well, but there could exist also hardware which
> >> doesn't. This solution might not work on that hardware.
> > 
> > Of course it's hard to find a silver bullet for a hardware we do not know
> > ;)
> > 
> >> If there is metadata which does not associate to V4L2 controls (or
> >> ioctls), either existing or not yet existing, then that probably should
> >> be parsed by the driver. On the other hand, I can't think of metadata
> >> that wouldn't fall under this right now. :-)
> > 
> > Some metadata are arrays of length specific to a given attribute,
> > I wonder how to support that with v4l2 controls ?
> 
> Is the metadata something which really isn't associated to any V4L2
> control? Are we now talking about a sensor which is more complex than a
> regular raw bayer sensor?
> 
> >> Do you know more examples of sensor produced metadata than SMIA++?
> > 
> > The only metadata I've had a bit experience with was regular EXIF tags
> > which could be retrieved from ISP through I2C bus.
> 
> That obviously won't map to V4L2 controls.
> 
> This should very likely be just passed to user space as-is as
> different... plane?
> 
> In some cases it's time critical to pass data to user space that
> otherwise could be associated with a video buffer. I wonder if this case
> is time critical or not.
> 
> > These were mostly fixed point arithmetic numbers in [32-bit numerator/
> > 32-bit denominator] form carrying exposure time, shutter speed, aperture,
> > brightness, flash, etc. information. The tags could be read from ISP
> > after it buffered a frame in its memory and processed it.
> > In case of a JPEG image format the tags can be embedded into the main
> > image file. But the image processors not always supported that so we used
> > to have an ioctl for the purpose of retrieving the metadata in user
> > space. In some cases it is desired to read data directly from the driver
> > rather than parsing a relatively large buffer.
> > It would be good to have a uniform interface for passing such data to
> > applications. I think in that particular use case a control id/value pair
> > sequences would do.
> 
> Do you think this is "control id" or non-control id, and whether the
> value is the same data type than the V4L2 control would be? That would
> match to what I'm aware of, too.

-- 
Regards,

Laurent Pinchart

Re: [RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Laurent Pinchart wrote:
> Hi Sakari,

Hi Laurent,

> On Tuesday 17 May 2011 22:34:36 Sakari Ailus wrote:
>> Sylwester Nawrocki wrote:
>>> On 05/09/2011 12:11 AM, Sakari Ailus wrote:
>>>> Sylwester Nawrocki wrote:
>>>>> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
>>>>>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> This is a fourth proposal for an interface for controlling flash
>>>>>>> devices on the V4L2/v4l2_subdev APIs.
>>>>>>>
>>>>>>> I want to thank everyone who have participated to the development of
>>>>>>> the flash interface.
>>>>>>>
>>>>>>> Comments and questions are very, very welcome as always.
>>>>>>>
>>>>>>>
>>>>>>> Changes since v3 [12]
>>>>>>> =====================
>>>>>>>
>>>>>>> - V4L2_CID_FLASH_STROBE changed to button control,
>>>>>>> V4L2_CID_FLASH_STROBE_STOP button control added,
>>>>>>> V4L2_CID_FLASH_STROBE_STATUS boolean control added.
>>>>>>>
>>>>>>> - No reason to say xenon flashes can't use V4L2_CID_FLASH_STROBE.
>>>>>>>
>>>>>>> - V4L2_CID_FLASH_STROBE_WHENCE changed to V4L2_CID_FLASH_STROBE_MODE.
>>>>>>>
>>>>>>> - V4L2_CID_TORCH_INTENSITY renamed to V4L2_CID_FLASH_TORCH_INTENSITY
>>>>>>> and V4L2_CID_INDICATOR_INTENSITY renamed to
>>>>>>> V4L2_CID_FLASH_INDICATOR_INTENSITY.
>>>>>>>
>>>>>>> - Moved V4L2_CID_FLASH_EXTERNAL_STROBE_MODE under "Possible future
>>>>>>> extensions".
>>>>>
>>>>> [snip]
>>>>>
>>>>>>> 3. Sensor metadata on frames
>>>>>>> ----------------------------
>>>>>>>
>>>>>>> It'd be useful to be able to read back sensor metadata. If the flash
>>>>>>> is strobed (on sensor hardware) while streaming, it's difficult to
>>>>>>> know otherwise which frame in the stream has been exposed with
>>>>>>> flash.
>>>>>>
>>>>>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
>>>>>> flag?
>>>>>> That way userspace can tell if that particular frame was taken with
>>>>>> flash.
>>>>>
>>>>> This looks more as a workaround for the problem rather than a good long
>>>>> term solution. It might be tempting to use the buffer flags which seem
>>>>> to be be more or less intended for buffer control.
>>>>> I'd like much more to see a buffer flags to be used to indicate whether
>>>>> an additional plane of (meta)data is carried by the buffer.
>>>>> There seem to be many more parameters, than a single flag indicating
>>>>> whether the frame has been exposed with flash or not, needed to be
>>>>> carried over to user space.
>>>>> But then we might need some standard format of the meta data, perhaps
>>>>> control id/value pairs and possibly a per plane configurable memory
>>>>> type.
>>>>
>>>> There are multiple possible approaches for this.
>>>>
>>>> For sensors where metadata is register-value pairs, that is, essentially
>>>> V4L2 control values, I think this should be parsed by the sensor driver.
>>>> The ISP (camera bridge) driver does receive the data so it'd have to
>>>> "ask for help" from the sensor driver.
>>>
>>> I am inclined to let the ISP drivers parse the data but on the other hand
>>> it might be difficult to access same DMA buffers in kernel _and_ user
>>> space.
>>
>> This is just about mapping the buffer to both kernel and user spaces. If
>> the ISP has an iommu the kernel mapping might already exist if it comes
>> from vmalloc().
> 
> And we're also trying to get rid of that mapping to facilitate cache 
> management. Any API we create for metadata parsing will need to take potential 
> cache-related performances issues into account at the design stage.

In this case, it's not necessary to map this memory to user space at all
so the kernel mapping would be the only one.

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Sylwester Nawrocki]

Hi Sakari,

On 05/17/2011 10:34 PM, Sakari Ailus wrote:
> Sylwester Nawrocki wrote:
>> On 05/09/2011 12:11 AM, Sakari Ailus wrote:
>>> Sylwester Nawrocki wrote:
>>>> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
>>>>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>>>> Hi,
>>>>>>
>>>>>> This is a fourth proposal for an interface for controlling flash devices
>>>>>> on the V4L2/v4l2_subdev APIs.
...
>>>>>> 3. Sensor metadata on frames
>>>>>> ----------------------------
>>>>>>
>>>>>> It'd be useful to be able to read back sensor metadata. If the flash is
>>>>>> strobed (on sensor hardware) while streaming, it's difficult to know
>>>>>> otherwise which frame in the stream has been exposed with flash.
>>>>>
>>>>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
>>>>> flag?
>>>>> That way userspace can tell if that particular frame was taken with
>>>>> flash.
>>>>
>>>> This looks more as a workaround for the problem rather than a good long
>>>> term solution. It might be tempting to use the buffer flags which seem
>>>> to be be more or less intended for buffer control.
>>>> I'd like much more to see a buffer flags to be used to indicate whether
>>>> an additional plane of (meta)data is carried by the buffer.
>>>> There seem to be many more parameters, than a single flag indicating
>>>> whether the frame has been exposed with flash or not, needed to be
>>>> carried over to user space.
>>>> But then we might need some standard format of the meta data, perhaps
>>>> control id/value pairs and possibly a per plane configurable memory
>>>> type.
>>>
>>> There are multiple possible approaches for this.
>>>
>>> For sensors where metadata is register-value pairs, that is, essentially
>>> V4L2 control values, I think this should be parsed by the sensor driver.
>>> The ISP (camera bridge) driver does receive the data so it'd have to
>>> "ask for help" from the sensor driver.
>>
>> I am inclined to let the ISP drivers parse the data but on the other hand
>> it might be difficult to access same DMA buffers in kernel _and_ user space.
> 
> This is just about mapping the buffer to both kernel and user spaces. If
> the ISP has an iommu the kernel mapping might already exist if it comes
> from vmalloc().

Yes, I know. I was thinking of possibly required different mapping attributes
for kernel and user space and the problems on ARM with that.
Also as metadata is supposed to occupy only small part of a frame buffer perhaps
only one page or so could be mapped in kernel space.
I'm referring here mainly to SMIA++ method of storing metadata.

In case of sensors I used to work with it wouldn't be necessary to touch the
main frame buffers as the metadata is transmitted out of band.

> 
>>> As discussed previously, using V4L2 control events shouldn't probably be
>>> the way to go, but I think it's obvious that this is _somehow_ bound to
>>> controls, at least control ids.
>>>
>>>> Also as Sakari indicated some sensors adopt custom meta data formats
>>>> so maybe we need to introduce standard fourcc like IDs for meta data
>>>> formats? I am not sure whether it is possible to create common
>>>> description of an image meta data that fits all H/W.
>>>
>>> I'm not sure either since I know of only one example. That example, i.e.
>>> register-value pairs, should be something that I'd assume _some_ other
>>> hardware uses as well, but there could exist also hardware which
>>> doesn't. This solution might not work on that hardware.
>>
>> Of course it's hard to find a silver bullet for a hardware we do not know ;)
>>
>>>
>>> If there is metadata which does not associate to V4L2 controls (or
>>> ioctls), either existing or not yet existing, then that probably should
>>> be parsed by the driver. On the other hand, I can't think of metadata
>>> that wouldn't fall under this right now. :-)
>>
>> Some metadata are arrays of length specific to a given attribute,
>> I wonder how to support that with v4l2 controls ?
> 
> Is the metadata something which really isn't associated to any V4L2
> control? Are we now talking about a sensor which is more complex than a
> regular raw bayer sensor?

I referred to tags defined in EXIF standard, as you may know every tag
is basically an array of specific data type. For most tag types the array
length is 1 though.
Similar problem is solved by the V4L extended string controls API.

And yes this kind of tags are mostly produced by more powerful ISPs,
with, for instance, 3A or distortion corrections implemented in their firmware.
So this is a little bit different situation than a raw sensor with OMAP3 ISP.

> 
>>> Do you know more examples of sensor produced metadata than SMIA++?
>>
>> The only metadata I've had a bit experience with was regular EXIF tags
>> which could be retrieved from ISP through I2C bus.
> 
> That obviously won't map to V4L2 controls.
> 
> This should very likely be just passed to user space as-is as
> different... plane?

Yes, but I am trying to assess whether it's possible to create some
generic tag data structure so such plane contains an array of such 
data structures and application would know how to handle that. 
Independently of the underlying H/W.

> 
> In some cases it's time critical to pass data to user space that
> otherwise could be associated with a video buffer. I wonder if this case
> is time critical or not.

No, it's rather no time critical. The frame data is buffered inside ISP,
even in case of multi-frame capture mode.
But, of course, we also need to consider a time critical case.

> 
>> These were mostly fixed point arithmetic numbers in [32-bit numerator/
>> 32-bit denominator] form carrying exposure time, shutter speed, aperture,
>> brightness, flash, etc. information. The tags could be read from ISP after
>> it buffered a frame in its memory and processed it.
>> In case of a JPEG image format the tags can be embedded into the main
>> image file. But the image processors not always supported that so we used
>> to have an ioctl for the purpose of retrieving the metadata in user space.
>> In some cases it is desired to read data directly from the driver rather
>> than parsing a relatively large buffer.
>> It would be good to have a uniform interface for passing such data to
>> applications. I think in that particular use case a control id/value pair
>> sequences would do.
> 
> Do you think this is "control id" or non-control id, and whether the
> value is the same data type than the V4L2 control would be? That would
> match to what I'm aware of, too.

Whatever id ;) I started with controls because they have defined data types.
A list of attributes in my case could be mapped to control ids.
But data types are different, the values are represented by rational numbers.

But perhaps we could define a generic list of attributes for metadata, together
with a generic tag data structure ?

I'm not yet even sure if it would be acceptable to interpret a data plane in user
space as an array of some type of data structure. 
 
--
Regards,
Sylwester

Re: [RFC v4] V4L2 API for flash devices

[Sakari Ailus]

Sylwester Nawrocki wrote:
> Hi Sakari,

Hi Sylwester,

> On 05/17/2011 10:34 PM, Sakari Ailus wrote:
>> Sylwester Nawrocki wrote:
>>> On 05/09/2011 12:11 AM, Sakari Ailus wrote:
>>>> Sylwester Nawrocki wrote:
>>>>> On 05/07/2011 02:46 PM, Hans Verkuil wrote:
>>>>>> On Thursday, May 05, 2011 20:49:21 Sakari Ailus wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> This is a fourth proposal for an interface for controlling flash devices
>>>>>>> on the V4L2/v4l2_subdev APIs.
> ...
>>>>>>> 3. Sensor metadata on frames
>>>>>>> ----------------------------
>>>>>>>
>>>>>>> It'd be useful to be able to read back sensor metadata. If the flash is
>>>>>>> strobed (on sensor hardware) while streaming, it's difficult to know
>>>>>>> otherwise which frame in the stream has been exposed with flash.
>>>>>>
>>>>>> I wonder if it would make sense to have a V4L2_BUF_FLAG_FLASH buffer
>>>>>> flag?
>>>>>> That way userspace can tell if that particular frame was taken with
>>>>>> flash.
>>>>>
>>>>> This looks more as a workaround for the problem rather than a good long
>>>>> term solution. It might be tempting to use the buffer flags which seem
>>>>> to be be more or less intended for buffer control.
>>>>> I'd like much more to see a buffer flags to be used to indicate whether
>>>>> an additional plane of (meta)data is carried by the buffer.
>>>>> There seem to be many more parameters, than a single flag indicating
>>>>> whether the frame has been exposed with flash or not, needed to be
>>>>> carried over to user space.
>>>>> But then we might need some standard format of the meta data, perhaps
>>>>> control id/value pairs and possibly a per plane configurable memory
>>>>> type.
>>>>
>>>> There are multiple possible approaches for this.
>>>>
>>>> For sensors where metadata is register-value pairs, that is, essentially
>>>> V4L2 control values, I think this should be parsed by the sensor driver.
>>>> The ISP (camera bridge) driver does receive the data so it'd have to
>>>> "ask for help" from the sensor driver.
>>>
>>> I am inclined to let the ISP drivers parse the data but on the other hand
>>> it might be difficult to access same DMA buffers in kernel _and_ user space.
>>
>> This is just about mapping the buffer to both kernel and user spaces. If
>> the ISP has an iommu the kernel mapping might already exist if it comes
>> from vmalloc().
> 
> Yes, I know. I was thinking of possibly required different mapping attributes
> for kernel and user space and the problems on ARM with that.

As I replied to Laurent, this is not an issue since the memory wouldn't
be mapped to user space. The user space would get the result which has
been interpreted by the sensor driver.

> Also as metadata is supposed to occupy only small part of a frame buffer perhaps
> only one page or so could be mapped in kernel space.
> I'm referring here mainly to SMIA++ method of storing metadata.

Yes, I agree.

> In case of sensors I used to work with it wouldn't be necessary to touch the
> main frame buffers as the metadata is transmitted out of band.

This depends on link a little, but on CSI-2 this is a separate channel.
In other cases the metadata is part of the frame --- but I don't know
anyone using metadata in such a case, so it _might_ be possible to just
ignore this use case.

Another alternative would be to pass binary blobs to user space and let
a library to interpret the data. This might be the way to go at least
with some more exotic cases.

>>
>>>> As discussed previously, using V4L2 control events shouldn't probably be
>>>> the way to go, but I think it's obvious that this is _somehow_ bound to
>>>> controls, at least control ids.
>>>>
>>>>> Also as Sakari indicated some sensors adopt custom meta data formats
>>>>> so maybe we need to introduce standard fourcc like IDs for meta data
>>>>> formats? I am not sure whether it is possible to create common
>>>>> description of an image meta data that fits all H/W.
>>>>
>>>> I'm not sure either since I know of only one example. That example, i.e.
>>>> register-value pairs, should be something that I'd assume _some_ other
>>>> hardware uses as well, but there could exist also hardware which
>>>> doesn't. This solution might not work on that hardware.
>>>
>>> Of course it's hard to find a silver bullet for a hardware we do not know ;)
>>>
>>>>
>>>> If there is metadata which does not associate to V4L2 controls (or
>>>> ioctls), either existing or not yet existing, then that probably should
>>>> be parsed by the driver. On the other hand, I can't think of metadata
>>>> that wouldn't fall under this right now. :-)
>>>
>>> Some metadata are arrays of length specific to a given attribute,
>>> I wonder how to support that with v4l2 controls ?
>>
>> Is the metadata something which really isn't associated to any V4L2
>> control? Are we now talking about a sensor which is more complex than a
>> regular raw bayer sensor?
> 
> I referred to tags defined in EXIF standard, as you may know every tag
> is basically an array of specific data type. For most tag types the array
> length is 1 though.
> Similar problem is solved by the V4L extended string controls API.
> 
> And yes this kind of tags are mostly produced by more powerful ISPs,
> with, for instance, 3A or distortion corrections implemented in their firmware.
> So this is a little bit different situation than a raw sensor with OMAP3 ISP.

True.

>>>> Do you know more examples of sensor produced metadata than SMIA++?
>>>
>>> The only metadata I've had a bit experience with was regular EXIF tags
>>> which could be retrieved from ISP through I2C bus.
>>
>> That obviously won't map to V4L2 controls.
>>
>> This should very likely be just passed to user space as-is as
>> different... plane?
> 
> Yes, but I am trying to assess whether it's possible to create some
> generic tag data structure so such plane contains an array of such 
> data structures and application would know how to handle that. 
> Independently of the underlying H/W.

There's one issue I see --- the kernel drivers do know how to interpret
the data, but then again the user space must be able to tell which part
of the data the user space is interested in. In many cases I'd assume
it's just one or two variables within a 8k (or so) metadata block.

I think it would be ideal if the sensor driver would be able to
interpret the data (or parts of it) and pass them to the user space. But
I don't know how generic this is, taken your example of rational numbers
used in metadata below.

>> In some cases it's time critical to pass data to user space that
>> otherwise could be associated with a video buffer. I wonder if this case
>> is time critical or not.
> 
> No, it's rather no time critical. The frame data is buffered inside ISP,
> even in case of multi-frame capture mode.
> But, of course, we also need to consider a time critical case.

I think in a time critical case, there must be an event to tell that the
data has arrived, or alternatively, if it's passed to user space as
such, a new video node for that. The set of buffers is different from
the image buffers, so a new video buffer queue is required.

>>> These were mostly fixed point arithmetic numbers in [32-bit numerator/
>>> 32-bit denominator] form carrying exposure time, shutter speed, aperture,
>>> brightness, flash, etc. information. The tags could be read from ISP after
>>> it buffered a frame in its memory and processed it.
>>> In case of a JPEG image format the tags can be embedded into the main
>>> image file. But the image processors not always supported that so we used
>>> to have an ioctl for the purpose of retrieving the metadata in user space.
>>> In some cases it is desired to read data directly from the driver rather
>>> than parsing a relatively large buffer.
>>> It would be good to have a uniform interface for passing such data to
>>> applications. I think in that particular use case a control id/value pair
>>> sequences would do.
>>
>> Do you think this is "control id" or non-control id, and whether the
>> value is the same data type than the V4L2 control would be? That would
>> match to what I'm aware of, too.
> 
> Whatever id ;) I started with controls because they have defined data types.
> A list of attributes in my case could be mapped to control ids.
> But data types are different, the values are represented by rational numbers.

What do they represent? They should be in the same units the sensor is
programmed with, but somehow I feel the sensor is trying to be smarter
than it should be.

> But perhaps we could define a generic list of attributes for metadata, together
> with a generic tag data structure ?

There are a few buts:

- The data can be something up to kilobytes in size. We don't want to
parse all of it per every frame unless it's really necessary. Most often
the user is interested in just a few items while it might like to get
all of it sometimes. This could be probably resolved relatively easily,
though.

- Which formats are your rational numbers in? A kernel interface can't
really have floating point numbers, so there would need to be a sane way
to pass these to user space.

- What kind of other types of metadata we could run into? Would it fit
to this interface?

> I'm not yet even sure if it would be acceptable to interpret a data plane in user
> space as an array of some type of data structure. 

If the data structure is known to user space, then it's definitely fine.

As an alternative, I'm thinking of a library which would be generic but
its interface would not be a kernel/user space interface which will
never change. My worry is that with two examples (or did you have
more?), can we design an interface generic enough for hopefully
everyone? My knowledge of sensor metadata is limited to just these
examples. The answer may well be yes, though.

Kind regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC v4] V4L2 API for flash devices

[Mauro Carvalho Chehab]

Em 19-05-2011 05:12, Sakari Ailus escreveu:
> Sylwester Nawrocki wrote:

>>>> These were mostly fixed point arithmetic numbers in [32-bit numerator/
>>>> 32-bit denominator] form carrying exposure time, shutter speed, aperture,
>>>> brightness, flash, etc. information. The tags could be read from ISP after
>>>> it buffered a frame in its memory and processed it.
>>>> In case of a JPEG image format the tags can be embedded into the main
>>>> image file. But the image processors not always supported that so we used
>>>> to have an ioctl for the purpose of retrieving the metadata in user space.
>>>> In some cases it is desired to read data directly from the driver rather
>>>> than parsing a relatively large buffer.
>>>> It would be good to have a uniform interface for passing such data to
>>>> applications. I think in that particular use case a control id/value pair
>>>> sequences would do.
 
> - Which formats are your rational numbers in? A kernel interface can't
> really have floating point numbers, so there would need to be a sane way
> to pass these to user space.

The V4L2 API has support for rational numbers. The frame rate is specified as a
rational number. There's a struct for that:

struct v4l2_fract {
	__u32   numerator;
	__u32   denominator;
};

Cheers,
Mauro