[RFC] Timestamps and V4L2

[RFC] Timestamps and V4L2

[Sakari Ailus]

Hi all,


This RFC intends to summarise and further the recent discussion on
linux-media regarding the proposed changes of timestamping V4L2 buffers.


The problem
===========

The V4L2 has long used realtime timestamps (such as
clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
handing them over to the user. This has been found problematic in
associating the video buffers with data from other sources: realtime clock
may jump around due to daylight saving time, for example, and ALSA
(audio-video synchronisation is a common use case) user space API does not
provide the user with realtime timestamps, but instead uses monotonic time
(i.e. clock_gettime(CLOCK_MONOTONIC, ...)).

This is especially an issue in embedded systems where video recording is a
common use case. Drivers typically used in such systems have silently
switched to use monotonic timestamps. While against the spec, this is
necessary for those systems to operate properly.

In general, realtime timestamps are seen of little use in other than
debugging purposes, but monotonic timestamps are fine for that as well. It's
still possible that an application I'm not aware of uses them in a peculiar
way that would be adversely affected by changing to monotonic timestamps.
Nevertheless, we're not supposed to break the API (or ABI). It'd be also
very important for the application to know what kind of timestamps are
provided by the device.


Requirements, wishes and constraints
====================================

Now that it seems to be about the time to fix these issues, it's worth
looking a little bit to the future to anticipate the coming changes to be
able to accommodate them better later on.

- The new default should be monotonic. As the monotonic timestamps are seen
to be the most useful, they should be made the default.

- timeval vs. timespec. The two structs can be used to store timestamp
information. They are not compatible with each other. It's a little bit
uncertain what's the case with all the architectures but it looks like the
timespec fits into the space of timeval in all cases. If timespec is
considered to be used somewhere the compatibility must be ensured. Timespec
is better than timeval since timespec has more precision and it's the same
struct that's used everywhere else in the V4L2 API: timespec does not need
conversion to timespec in the user space.

struct timespec {
        __kernel_time_t tv_sec;                 /* seconds */
        long            tv_nsec;                /* nanoseconds */
};

struct timeval {
        __kernel_time_t         tv_sec;         /* seconds */
        __kernel_suseconds_t    tv_usec;        /* microseconds */
};

To be able to use timespec, the user would have to most likely explicitly
choose to do that.

- Users should know what kind of timestamps the device produces. This
includes existing and future kernels. What should be considered are
uninformed porting drivers back and forth across kernel versions and
out-of-date kernel header files.

- Device-dependent timestamps. Some devices such as the uvcvideo ones
produce device-dependent timestamps for synchronising video and audio, both
produced by the same physical hardware device. For uvcvideo these timestamps
are unsigned 32-bit integers.

- There's also another clock, Linux-specific raw monotonic clock (as in
clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
cases than the regular monotonic clock. The difference is that the raw
monotonic clock is free from the NTP adjustments. It would be nice for the
user to be able to choose the clock used for timestamps. This is especially
important for device-dependent timestamps: not all applications can be
expected to be able to use them.

- The field adjacent to timestamp, timecode, is 128 bits wide, and not used
by a single driver. This field could be re-used.


Possible solutions
==================

Not all of the solutions below that have been proposed are mutually
exclusive. That's also what's making the choice difficult: the ultimate
solution to the issue of timestamping may involve several of these --- or
possibly something better that's not on the list.


Use of timespec
---------------

If we can conclude timespec will always fit into the size of timeval (or
timecode) we could use timespec instead. The solution should still make
the use of timespec explicit to the user space. This seems to conflict with
the idea of making monotonic timestamps the default: the default can't be
anything incompatible with timeval, and at the same time it's the most
important that the monotonic timestamps are timespec.


Kernel version as indicator of timestamp
----------------------------------------

Conversion of drivers to use monotonic timestamp is trivial, so the
conversion could be done once and for all drivers. The kernel version could
be used to indicate the type of the timestamp.

If this approach is taken care must be taken when new drivers are
integrated: developers sometimes use old kernels for development and might
also use an old driver for guidance on timestamps, thus using real-time
timestamps when monotonic timestamps should be used. This approach has an
advantage over the capability flag below: which is that we don't populate
the interface with essentially dead definitions.


Capability flag for monotonic timestamps
----------------------------------------

A capability flag can be used to tell whether the timestamp is monotonic.
However, it's not extensible cleanly to provide selectable timestamps. These
are not features that are needed right now, though.

The upside of this option is ease of implementation and use, but it's not
extensible. Also we're left with a flag that's set for all drivers: in the
end it provides no information to the user and is only noise in the spec.


Control for timestamp type
--------------------------

Using a control to tell the type of the timestamp is extensible but not as
easy to implement than the capability flag: each and every device would get
an additional control. The value should likely be also file handle specific,
and we do not have file handle specific controls yet.

In the meantime the control could be read-only, and later made read-write
when the timestamp type can be made selectable. Much of he work of
timestamping can be done by the framework: drivers can use a single helper
function and need to create one extra standard control.

Should the control also have an effect on the types of the timestamps in
V4L2 events? Likely yes.


Device-dependent timestamp
--------------------------

Should we agree on selectable timestamps, the existing timestamp field (or a
union with another field of different type) could be used for the
device-dependent timestamps. Alternatively we can choose to re-use the
existing timecode field.

At the moment there's no known use case for passing device-dependent
timestamps at the same time with monotonic timestamps.


Now what?
=========

Almost as many options have been presented as there were opinions, but we
need to agree to have a single one. My personal leaning is on using a
control for the purpose as it is the most flexible alternative. I'd still
need to see an implementation of that but it doesn't seem that difficult,
especially when it's read-only. And even for read-write control the vast
majority of the work can be done by the V4L2 framework.

Questions, comments and opinions are very, very welcome.


Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [RFC] Timestamps and V4L2

[Rémi Denis-Courmont]

Le jeudi 20 septembre 2012 23:21:22, Sakari Ailus a écrit :
> The problem
> ===========
> 
> The V4L2 has long used realtime timestamps (such as
> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> handing them over to the user. This has been found problematic in
> associating the video buffers with data from other sources: realtime clock
> may jump around due to daylight saving time, for example, and ALSA
> (audio-video synchronisation is a common use case) user space API does not
> provide the user with realtime timestamps, but instead uses monotonic time
> (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> 
> This is especially an issue in embedded systems where video recording is a
> common use case. Drivers typically used in such systems have silently
> switched to use monotonic timestamps. While against the spec, this is
> necessary for those systems to operate properly.
> 
> In general, realtime timestamps are seen of little use in other than
> debugging purposes, but monotonic timestamps are fine for that as well.
> It's still possible that an application I'm not aware of uses them in a
> peculiar way that would be adversely affected by changing to monotonic
> timestamps. Nevertheless, we're not supposed to break the API (or ABI).

You cannot break something that never worked. Aside from debugging, the main 
use of timestamp is to synchronize the video signal with something else and/or 
to detect drift (which is more or less the same thing anyway).

The current timestamp infos are _not_ usable. gstreamer has some fuzzy logic 
to try to figure out what it really means. VLC just ignores it and generates 
its own imprecise timestamp.

> It'd be also very important for the application to know what kind of
> timestamps are provided by the device.

I would rather expect the HAL to convert the timestamp to something standard 
than have the application deal with an ever expanding list of types of 
timestamp that need to be supported.

In other words, either V4L2 converts the timestamp to a type set by the 
application, or there is only one type of timestamp at all (CLOCK_MONOTONIC).
Anything else *will* fail.

> Requirements, wishes and constraints
> ====================================
> 
> Now that it seems to be about the time to fix these issues, it's worth
> looking a little bit to the future to anticipate the coming changes to be
> able to accommodate them better later on.
> 
> - The new default should be monotonic. As the monotonic timestamps are seen
> to be the most useful, they should be made the default.

> - timeval vs. timespec.
> (...)

> - Users should know what kind of timestamps the device produces. This
> includes existing and future kernels. What should be considered are
> uninformed porting drivers back and forth across kernel versions and
> out-of-date kernel header files.
> 
> - Device-dependent timestamps. Some devices such as the uvcvideo ones
> produce device-dependent timestamps for synchronising video and audio, both
> produced by the same physical hardware device. For uvcvideo these
> timestamps are unsigned 32-bit integers.
> 
> - There's also another clock, Linux-specific raw monotonic clock (as in
> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> cases than the regular monotonic clock. The difference is that the raw
> monotonic clock is free from the NTP adjustments. It would be nice for the
> user to be able to choose the clock used for timestamps. This is especially
> important for device-dependent timestamps: not all applications can be
> expected to be able to use them.

Then you need to create a new VIODIC_S_CLOCK control. If the control fails, 
then the application knows that it cannot trust the timestamp. So there is no 
need for a version number neither a flag.

This does seem overkill though. You'll need to keep a clockid_t for each open 
V4L2 file handle.

> - The field adjacent to timestamp, timecode, is 128 bits wide, and not used
> by a single driver. This field could be re-used.
> 
> 
> Possible solutions
> ==================
> 
> Not all of the solutions below that have been proposed are mutually
> exclusive. That's also what's making the choice difficult: the ultimate
> solution to the issue of timestamping may involve several of these --- or
> possibly something better that's not on the list.
> 
> 
> Use of timespec
> ---------------
> 
> If we can conclude timespec will always fit into the size of timeval (or
> timecode) we could use timespec instead. The solution should still make
> the use of timespec explicit to the user space. This seems to conflict with
> the idea of making monotonic timestamps the default: the default can't be
> anything incompatible with timeval, and at the same time it's the most
> important that the monotonic timestamps are timespec.

That is totally irrelevant. If the application knows what type of timestamp is 
used, it knows whether to use timespec or timeval, presumably timespec. If the 
default is changed to CLOCK_MONOTONIC, compatibility is broken, even if 
timeval is retained.

That being said, I think microsecond precision will be enough for longer than 
the foreseeable future. We are not even measuring frame rates in kHz yet. Even 
audio sampling rate is not reaching the MHz.

So I don't really care. Also, many applications will anyway convert the field 
to some internal timestamp representation.

> Kernel version as indicator of timestamp
> ----------------------------------------
> 
> Conversion of drivers to use monotonic timestamp is trivial, so the
> conversion could be done once and for all drivers. The kernel version could
> be used to indicate the type of the timestamp.

Was there never a V4L driver exposing a version number higher than 3 << 16? If 
there was, the version number cannot be used for comparison and this is moot.

> If this approach is taken care must be taken when new drivers are
> integrated: developers sometimes use old kernels for development and might
> also use an old driver for guidance on timestamps, thus using real-time
> timestamps when monotonic timestamps should be used.

The quality insurance problem will exist no matter what solution you pick. In 
particular, the same problem exists with the capability flag, if it ends up 
being set by the core blindly for all drivers. Also, with control solution, 
the driver could ignore the file handle value and blindly set the timestamp.

> Capability flag for monotonic timestamps
> ----------------------------------------
> 
> A capability flag can be used to tell whether the timestamp is monotonic.
> However, it's not extensible cleanly to provide selectable timestamps.
> These are not features that are needed right now, though.

> Control for timestamp type
> --------------------------
> 
> Using a control to tell the type of the timestamp is extensible but not as
> easy to implement than the capability flag: each and every device would get
> an additional control. The value should likely be also file handle
> specific, and we do not have file handle specific controls yet.
> 
> In the meantime the control could be read-only, and later made read-write
> when the timestamp type can be made selectable. Much of he work of
> timestamping can be done by the framework: drivers can use a single helper
> function and need to create one extra standard control.
> 
> Should the control also have an effect on the types of the timestamps in
> V4L2 events? Likely yes.
> 
> 
> Device-dependent timestamp
> --------------------------
> 
> Should we agree on selectable timestamps, the existing timestamp field (or
> a union with another field of different type) could be used for the
> device-dependent timestamps. Alternatively we can choose to re-use the
> existing timecode field.
> 
> At the moment there's no known use case for passing device-dependent
> timestamps at the same time with monotonic timestamps.

-- 
Rémi Denis-Courmont
http://www.remlab.net/

Re: [RFC] Timestamps and V4L2

[Christian Gmeiner]

2012/9/20 Sakari Ailus <sakari.ailus@iki.fi>:
> Hi all,
>
>
> This RFC intends to summarise and further the recent discussion on
> linux-media regarding the proposed changes of timestamping V4L2 buffers.
>
>
> The problem
> ===========
>
> The V4L2 has long used realtime timestamps (such as
> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> handing them over to the user. This has been found problematic in
> associating the video buffers with data from other sources: realtime clock
> may jump around due to daylight saving time, for example, and ALSA
> (audio-video synchronisation is a common use case) user space API does not
> provide the user with realtime timestamps, but instead uses monotonic time
> (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
>
> This is especially an issue in embedded systems where video recording is a
> common use case. Drivers typically used in such systems have silently
> switched to use monotonic timestamps. While against the spec, this is
> necessary for those systems to operate properly.
>
> In general, realtime timestamps are seen of little use in other than
> debugging purposes, but monotonic timestamps are fine for that as well. It's
> still possible that an application I'm not aware of uses them in a peculiar
> way that would be adversely affected by changing to monotonic timestamps.
> Nevertheless, we're not supposed to break the API (or ABI). It'd be also
> very important for the application to know what kind of timestamps are
> provided by the device.
>
>
> Requirements, wishes and constraints
> ====================================
>
> Now that it seems to be about the time to fix these issues, it's worth
> looking a little bit to the future to anticipate the coming changes to be
> able to accommodate them better later on.
>
> - The new default should be monotonic. As the monotonic timestamps are seen
> to be the most useful, they should be made the default.
>
> - timeval vs. timespec. The two structs can be used to store timestamp
> information. They are not compatible with each other. It's a little bit
> uncertain what's the case with all the architectures but it looks like the
> timespec fits into the space of timeval in all cases. If timespec is
> considered to be used somewhere the compatibility must be ensured. Timespec
> is better than timeval since timespec has more precision and it's the same
> struct that's used everywhere else in the V4L2 API: timespec does not need
> conversion to timespec in the user space.
>
> struct timespec {
>         __kernel_time_t tv_sec;                 /* seconds */
>         long            tv_nsec;                /* nanoseconds */
> };
>
> struct timeval {
>         __kernel_time_t         tv_sec;         /* seconds */
>         __kernel_suseconds_t    tv_usec;        /* microseconds */
> };
>
> To be able to use timespec, the user would have to most likely explicitly
> choose to do that.
>
> - Users should know what kind of timestamps the device produces. This
> includes existing and future kernels. What should be considered are
> uninformed porting drivers back and forth across kernel versions and
> out-of-date kernel header files.
>
> - Device-dependent timestamps. Some devices such as the uvcvideo ones
> produce device-dependent timestamps for synchronising video and audio, both
> produced by the same physical hardware device. For uvcvideo these timestamps
> are unsigned 32-bit integers.


What about pure output devices like old-school mpeg2 cards? The timestamps for
audio and video are pure PTS values.

>
> - There's also another clock, Linux-specific raw monotonic clock (as in
> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> cases than the regular monotonic clock. The difference is that the raw
> monotonic clock is free from the NTP adjustments. It would be nice for the
> user to be able to choose the clock used for timestamps. This is especially
> important for device-dependent timestamps: not all applications can be
> expected to be able to use them.
>
> - The field adjacent to timestamp, timecode, is 128 bits wide, and not used
> by a single driver. This field could be re-used.
>
>
> Possible solutions
> ==================
>
> Not all of the solutions below that have been proposed are mutually
> exclusive. That's also what's making the choice difficult: the ultimate
> solution to the issue of timestamping may involve several of these --- or
> possibly something better that's not on the list.
>
>
> Use of timespec
> ---------------
>
> If we can conclude timespec will always fit into the size of timeval (or
> timecode) we could use timespec instead. The solution should still make
> the use of timespec explicit to the user space. This seems to conflict with
> the idea of making monotonic timestamps the default: the default can't be
> anything incompatible with timeval, and at the same time it's the most
> important that the monotonic timestamps are timespec.
>
>
> Kernel version as indicator of timestamp
> ----------------------------------------
>
> Conversion of drivers to use monotonic timestamp is trivial, so the
> conversion could be done once and for all drivers. The kernel version could
> be used to indicate the type of the timestamp.
>
> If this approach is taken care must be taken when new drivers are
> integrated: developers sometimes use old kernels for development and might
> also use an old driver for guidance on timestamps, thus using real-time
> timestamps when monotonic timestamps should be used. This approach has an
> advantage over the capability flag below: which is that we don't populate
> the interface with essentially dead definitions.
>
>
> Capability flag for monotonic timestamps
> ----------------------------------------
>
> A capability flag can be used to tell whether the timestamp is monotonic.
> However, it's not extensible cleanly to provide selectable timestamps. These
> are not features that are needed right now, though.
>
> The upside of this option is ease of implementation and use, but it's not
> extensible. Also we're left with a flag that's set for all drivers: in the
> end it provides no information to the user and is only noise in the spec.
>
>
> Control for timestamp type
> --------------------------
>
> Using a control to tell the type of the timestamp is extensible but not as
> easy to implement than the capability flag: each and every device would get
> an additional control. The value should likely be also file handle specific,
> and we do not have file handle specific controls yet.
>
> In the meantime the control could be read-only, and later made read-write
> when the timestamp type can be made selectable. Much of he work of
> timestamping can be done by the framework: drivers can use a single helper
> function and need to create one extra standard control.
>
> Should the control also have an effect on the types of the timestamps in
> V4L2 events? Likely yes.
>
>
> Device-dependent timestamp
> --------------------------
>
> Should we agree on selectable timestamps, the existing timestamp field (or a
> union with another field of different type) could be used for the
> device-dependent timestamps. Alternatively we can choose to re-use the
> existing timecode field.
>
> At the moment there's no known use case for passing device-dependent
> timestamps at the same time with monotonic timestamps.
>
>
> Now what?
> =========
>
> Almost as many options have been presented as there were opinions, but we
> need to agree to have a single one. My personal leaning is on using a
> control for the purpose as it is the most flexible alternative. I'd still
> need to see an implementation of that but it doesn't seem that difficult,
> especially when it's read-only. And even for read-write control the vast
> majority of the work can be done by the V4L2 framework.
>
> Questions, comments and opinions are very, very welcome.
>
>

---
Christian Gmeiner, MSc

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> Hi all,
> 
> 
> This RFC intends to summarise and further the recent discussion on
> linux-media regarding the proposed changes of timestamping V4L2 buffers.
> 
> 
> The problem
> ===========
> 
> The V4L2 has long used realtime timestamps (such as
> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> handing them over to the user. This has been found problematic in
> associating the video buffers with data from other sources: realtime clock
> may jump around due to daylight saving time, for example, and ALSA
> (audio-video synchronisation is a common use case) user space API does not
> provide the user with realtime timestamps, but instead uses monotonic time
> (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> 
> This is especially an issue in embedded systems where video recording is a
> common use case. Drivers typically used in such systems have silently
> switched to use monotonic timestamps. While against the spec, this is
> necessary for those systems to operate properly.
> 
> In general, realtime timestamps are seen of little use in other than
> debugging purposes, but monotonic timestamps are fine for that as well. It's
> still possible that an application I'm not aware of uses them in a peculiar
> way that would be adversely affected by changing to monotonic timestamps.
> Nevertheless, we're not supposed to break the API (or ABI). It'd be also
> very important for the application to know what kind of timestamps are
> provided by the device.
> 
> 
> Requirements, wishes and constraints
> ====================================
> 
> Now that it seems to be about the time to fix these issues, it's worth
> looking a little bit to the future to anticipate the coming changes to be
> able to accommodate them better later on.
> 
> - The new default should be monotonic. As the monotonic timestamps are seen
> to be the most useful, they should be made the default.
> 
> - timeval vs. timespec. The two structs can be used to store timestamp
> information. They are not compatible with each other. It's a little bit
> uncertain what's the case with all the architectures but it looks like the
> timespec fits into the space of timeval in all cases. If timespec is
> considered to be used somewhere the compatibility must be ensured. Timespec
> is better than timeval since timespec has more precision and it's the same
> struct that's used everywhere else in the V4L2 API: timespec does not need
> conversion to timespec in the user space.
> 
> struct timespec {
>         __kernel_time_t tv_sec;                 /* seconds */
>         long            tv_nsec;                /* nanoseconds */
> };
> 
> struct timeval {
>         __kernel_time_t         tv_sec;         /* seconds */
>         __kernel_suseconds_t    tv_usec;        /* microseconds */
> };
> 
> To be able to use timespec, the user would have to most likely explicitly
> choose to do that.
> 
> - Users should know what kind of timestamps the device produces. This
> includes existing and future kernels. What should be considered are
> uninformed porting drivers back and forth across kernel versions and
> out-of-date kernel header files.
> 
> - Device-dependent timestamps. Some devices such as the uvcvideo ones
> produce device-dependent timestamps for synchronising video and audio, both
> produced by the same physical hardware device. For uvcvideo these timestamps
> are unsigned 32-bit integers.
> 
> - There's also another clock, Linux-specific raw monotonic clock (as in
> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> cases than the regular monotonic clock. The difference is that the raw
> monotonic clock is free from the NTP adjustments. It would be nice for the
> user to be able to choose the clock used for timestamps. This is especially
> important for device-dependent timestamps: not all applications can be
> expected to be able to use them.
> 
> - The field adjacent to timestamp, timecode, is 128 bits wide, and not used
> by a single driver. This field could be re-used.
> 
> 
> Possible solutions
> ==================
> 
> Not all of the solutions below that have been proposed are mutually
> exclusive. That's also what's making the choice difficult: the ultimate
> solution to the issue of timestamping may involve several of these --- or
> possibly something better that's not on the list.
> 
> 
> Use of timespec
> ---------------
> 
> If we can conclude timespec will always fit into the size of timeval (or
> timecode) we could use timespec instead. The solution should still make
> the use of timespec explicit to the user space. This seems to conflict with
> the idea of making monotonic timestamps the default: the default can't be
> anything incompatible with timeval, and at the same time it's the most
> important that the monotonic timestamps are timespec.

We have to keep timeval. Changing this will break the ABI. I see absolutely
no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and that's
far, far removed from ns precisions. Should we ever have to support high-speed
cameras running at 60000 Hz, then we'll talk again.

For me this is a non-issue.

> Kernel version as indicator of timestamp
> ----------------------------------------
> 
> Conversion of drivers to use monotonic timestamp is trivial, so the
> conversion could be done once and for all drivers. The kernel version could
> be used to indicate the type of the timestamp.
> 
> If this approach is taken care must be taken when new drivers are
> integrated: developers sometimes use old kernels for development and might
> also use an old driver for guidance on timestamps, thus using real-time
> timestamps when monotonic timestamps should be used.

More importantly, this also fails when users use out-of-tree drivers.

> This approach has an
> advantage over the capability flag below: which is that we don't populate
> the interface with essentially dead definitions.

Using a kernel version to decide whether some feature is available or not is
IMHO something of a last resort. It's very application unfriendly.

> 
> Capability flag for monotonic timestamps
> ----------------------------------------
> 
> A capability flag can be used to tell whether the timestamp is monotonic.
> However, it's not extensible cleanly to provide selectable timestamps. These
> are not features that are needed right now, though.
> 
> The upside of this option is ease of implementation and use, but it's not
> extensible. Also we're left with a flag that's set for all drivers: in the
> end it provides no information to the user and is only noise in the spec.
> 
> 
> Control for timestamp type
> --------------------------
> 
> Using a control to tell the type of the timestamp is extensible but not as
> easy to implement than the capability flag: each and every device would get
> an additional control. The value should likely be also file handle specific,
> and we do not have file handle specific controls yet.

Yes, we do. You can make per-file handle controls. M2M devices need that.

I'm not sure why this would be filehandle specific, BTW.

> In the meantime the control could be read-only, and later made read-write
> when the timestamp type can be made selectable. Much of he work of
> timestamping can be done by the framework: drivers can use a single helper
> function and need to create one extra standard control.
> 
> Should the control also have an effect on the types of the timestamps in
> V4L2 events? Likely yes.

You are missing one other option:

Using v4l2_buffer flags to report the clock
-------------------------------------------

By defining flags like this:

V4L2_BUF_FLAG_CLOCK_MASK	0x7000
/* Possible Clocks */
V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know */
V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000

you could tell the application which clock is used.

This does allow for more clocks to be added in the future and clock selection
would then be done by a control or possibly an ioctl. For now there are no
plans to do such things, so this flag should be sufficient. And it can be
implemented very efficiently. It works with existing drivers as well, since
they will report CLOCK_UNKNOWN.

I am very much in favor of this approach.

> 
> 
> Device-dependent timestamp
> --------------------------
> 
> Should we agree on selectable timestamps, the existing timestamp field (or a
> union with another field of different type) could be used for the
> device-dependent timestamps.

No. Device timestamps should get their own field. You want to be able to relate
device timestamps with the monotonic timestamps, so you need both.

> Alternatively we can choose to re-use the
> existing timecode field.
> 
> At the moment there's no known use case for passing device-dependent
> timestamps at the same time with monotonic timestamps.

Well, the use case is there, but there is no driver support. The device
timestamps should be 64 bits to accomodate things like PTS and DTS from
MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
or perhaps 4 u32 fields.

> 
> 
> Now what?
> =========
> 
> Almost as many options have been presented as there were opinions, but we
> need to agree to have a single one. My personal leaning is on using a
> control for the purpose as it is the most flexible alternative. I'd still
> need to see an implementation of that but it doesn't seem that difficult,
> especially when it's read-only. And even for read-write control the vast
> majority of the work can be done by the V4L2 framework.
> 
> Questions, comments and opinions are very, very welcome.

Regards,

	Hans

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Hans,

Thanks for the comments.

Hans Verkuil wrote:
> On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
>> Hi all,
>>
>>
>> This RFC intends to summarise and further the recent discussion on
>> linux-media regarding the proposed changes of timestamping V4L2 buffers.
>>
>>
>> The problem
>> ===========
>>
>> The V4L2 has long used realtime timestamps (such as
>> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
>> handing them over to the user. This has been found problematic in
>> associating the video buffers with data from other sources: realtime clock
>> may jump around due to daylight saving time, for example, and ALSA
>> (audio-video synchronisation is a common use case) user space API does not
>> provide the user with realtime timestamps, but instead uses monotonic time
>> (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
>>
>> This is especially an issue in embedded systems where video recording is a
>> common use case. Drivers typically used in such systems have silently
>> switched to use monotonic timestamps. While against the spec, this is
>> necessary for those systems to operate properly.
>>
>> In general, realtime timestamps are seen of little use in other than
>> debugging purposes, but monotonic timestamps are fine for that as well. It's
>> still possible that an application I'm not aware of uses them in a peculiar
>> way that would be adversely affected by changing to monotonic timestamps.
>> Nevertheless, we're not supposed to break the API (or ABI). It'd be also
>> very important for the application to know what kind of timestamps are
>> provided by the device.
>>
>>
>> Requirements, wishes and constraints
>> ====================================
>>
>> Now that it seems to be about the time to fix these issues, it's worth
>> looking a little bit to the future to anticipate the coming changes to be
>> able to accommodate them better later on.
>>
>> - The new default should be monotonic. As the monotonic timestamps are seen
>> to be the most useful, they should be made the default.
>>
>> - timeval vs. timespec. The two structs can be used to store timestamp
>> information. They are not compatible with each other. It's a little bit
>> uncertain what's the case with all the architectures but it looks like the
>> timespec fits into the space of timeval in all cases. If timespec is
>> considered to be used somewhere the compatibility must be ensured. Timespec
>> is better than timeval since timespec has more precision and it's the same
>> struct that's used everywhere else in the V4L2 API: timespec does not need
>> conversion to timespec in the user space.
>>
>> struct timespec {
>>          __kernel_time_t tv_sec;                 /* seconds */
>>          long            tv_nsec;                /* nanoseconds */
>> };
>>
>> struct timeval {
>>          __kernel_time_t         tv_sec;         /* seconds */
>>          __kernel_suseconds_t    tv_usec;        /* microseconds */
>> };
>>
>> To be able to use timespec, the user would have to most likely explicitly
>> choose to do that.
>>
>> - Users should know what kind of timestamps the device produces. This
>> includes existing and future kernels. What should be considered are
>> uninformed porting drivers back and forth across kernel versions and
>> out-of-date kernel header files.
>>
>> - Device-dependent timestamps. Some devices such as the uvcvideo ones
>> produce device-dependent timestamps for synchronising video and audio, both
>> produced by the same physical hardware device. For uvcvideo these timestamps
>> are unsigned 32-bit integers.
>>
>> - There's also another clock, Linux-specific raw monotonic clock (as in
>> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
>> cases than the regular monotonic clock. The difference is that the raw
>> monotonic clock is free from the NTP adjustments. It would be nice for the
>> user to be able to choose the clock used for timestamps. This is especially
>> important for device-dependent timestamps: not all applications can be
>> expected to be able to use them.
>>
>> - The field adjacent to timestamp, timecode, is 128 bits wide, and not used
>> by a single driver. This field could be re-used.
>>
>>
>> Possible solutions
>> ==================
>>
>> Not all of the solutions below that have been proposed are mutually
>> exclusive. That's also what's making the choice difficult: the ultimate
>> solution to the issue of timestamping may involve several of these --- or
>> possibly something better that's not on the list.
>>
>>
>> Use of timespec
>> ---------------
>>
>> If we can conclude timespec will always fit into the size of timeval (or
>> timecode) we could use timespec instead. The solution should still make
>> the use of timespec explicit to the user space. This seems to conflict with
>> the idea of making monotonic timestamps the default: the default can't be
>> anything incompatible with timeval, and at the same time it's the most
>> important that the monotonic timestamps are timespec.
>
> We have to keep timeval. Changing this will break the ABI. I see absolutely
> no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and that's
> far, far removed from ns precisions. Should we ever have to support high-speed
> cameras running at 60000 Hz, then we'll talk again.
>
> For me this is a non-issue.
>
>> Kernel version as indicator of timestamp
>> ----------------------------------------
>>
>> Conversion of drivers to use monotonic timestamp is trivial, so the
>> conversion could be done once and for all drivers. The kernel version could
>> be used to indicate the type of the timestamp.
>>
>> If this approach is taken care must be taken when new drivers are
>> integrated: developers sometimes use old kernels for development and might
>> also use an old driver for guidance on timestamps, thus using real-time
>> timestamps when monotonic timestamps should be used.
>
> More importantly, this also fails when users use out-of-tree drivers.

Could you mention some examples what we could be breaking in particular?

>> This approach has an
>> advantage over the capability flag below: which is that we don't populate
>> the interface with essentially dead definitions.
>
> Using a kernel version to decide whether some feature is available or not is
> IMHO something of a last resort. It's very application unfriendly.

Could be, but that's a passing pain. We're going to live with the flags 
for the foreseeable future, whether we need them or not.

>>
>> Capability flag for monotonic timestamps
>> ----------------------------------------
>>
>> A capability flag can be used to tell whether the timestamp is monotonic.
>> However, it's not extensible cleanly to provide selectable timestamps. These
>> are not features that are needed right now, though.
>>
>> The upside of this option is ease of implementation and use, but it's not
>> extensible. Also we're left with a flag that's set for all drivers: in the
>> end it provides no information to the user and is only noise in the spec.
>>
>>
>> Control for timestamp type
>> --------------------------
>>
>> Using a control to tell the type of the timestamp is extensible but not as
>> easy to implement than the capability flag: each and every device would get
>> an additional control. The value should likely be also file handle specific,
>> and we do not have file handle specific controls yet.
>
> Yes, we do. You can make per-file handle controls. M2M devices need that.

Thanks for correcting me.

> I'm not sure why this would be filehandle specific, BTW.

Good point. I thought that as other properties of the buffers are 
specific to file handles, including format when using CREATE_BUFS, it'd 
make sense to make the timestamp source file-handle specific as well.

What do you think?

>> In the meantime the control could be read-only, and later made read-write
>> when the timestamp type can be made selectable. Much of he work of
>> timestamping can be done by the framework: drivers can use a single helper
>> function and need to create one extra standard control.
>>
>> Should the control also have an effect on the types of the timestamps in
>> V4L2 events? Likely yes.
>
> You are missing one other option:
>
> Using v4l2_buffer flags to report the clock
> -------------------------------------------
>
> By defining flags like this:
>
> V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> /* Possible Clocks */
> V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know */
> V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
>
> you could tell the application which clock is used.
>
> This does allow for more clocks to be added in the future and clock selection
> would then be done by a control or possibly an ioctl. For now there are no
> plans to do such things, so this flag should be sufficient. And it can be
> implemented very efficiently. It works with existing drivers as well, since
> they will report CLOCK_UNKNOWN.
>
> I am very much in favor of this approach.

Thanks for adding this. I knew I was forgetting something but didn't 
remember what --- I swear it was unintentional! :-)

If we'd add more clocks without providing an ability to choose the clock 
from the user space, how would the clock be selected? It certainly isn't 
the driver's job, nor I think it should be system-specific either 
(platform data on embedded systems).

It's up to the application and its needs. That would suggest we should 
always provide monotonic timestamps to applications (besides a potential 
driver-specific timestamp), and for that purpose the capability flag --- 
I admit I disliked the idea at first --- is enough.

What comes to buffer flags, the application would also have to receive 
the first buffer from the device to even know what kind of timestamps 
the device uses, or at least call QUERYBUF. And in principle the flag 
should be checked on every buffer, unless we also specify the flag is 
the same for all buffers. And at certain point this will stop to make 
any sense...

A capability flag is cleaner solution from this perspective, and it can 
be amended by a control (or an ioctl) later on: the flag can be 
disregarded by applications whenever the control is present. If the 
application doesn't know about the control it can still rely on the 
flag. (I think this would be less clean than to go for the control right 
from the beginning, but better IMO.)

>>
>>
>> Device-dependent timestamp
>> --------------------------
>>
>> Should we agree on selectable timestamps, the existing timestamp field (or a
>> union with another field of different type) could be used for the
>> device-dependent timestamps.
>
> No. Device timestamps should get their own field. You want to be able to relate
> device timestamps with the monotonic timestamps, so you need both.
>
>> Alternatively we can choose to re-use the
>> existing timecode field.
>>
>> At the moment there's no known use case for passing device-dependent
>> timestamps at the same time with monotonic timestamps.
>
> Well, the use case is there, but there is no driver support. The device
> timestamps should be 64 bits to accomodate things like PTS and DTS from
> MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
> or perhaps 4 u32 fields.

That should be an union for different kinds (or rather types) of 
device-dependent timestamps. On uvcvideo I think this is u32, not u64. 
We should be also able to tell what kind device dependent timestamp 
there is --- should buffer flags be used for that as well?

Kind regards,

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [RFC] Timestamps and V4L2

[Sylwester Nawrocki]

On 09/22/2012 02:38 PM, Sakari Ailus wrote:
>> You are missing one other option:
>>
>> Using v4l2_buffer flags to report the clock
>> -------------------------------------------
>>
>> By defining flags like this:
>>
>> V4L2_BUF_FLAG_CLOCK_MASK 0x7000
>> /* Possible Clocks */
>> V4L2_BUF_FLAG_CLOCK_UNKNOWN 0x0000 /* system or monotonic, we don't 
>> know */
>> V4L2_BUF_FLAG_CLOCK_MONOTONIC 0x1000
>>
>> you could tell the application which clock is used.
>>
>> This does allow for more clocks to be added in the future and clock 
>> selection
>> would then be done by a control or possibly an ioctl. For now there 
>> are no
>> plans to do such things, so this flag should be sufficient. And it can be
>> implemented very efficiently. It works with existing drivers as well, 
>> since
>> they will report CLOCK_UNKNOWN.
>>
>> I am very much in favor of this approach.

+1

I think I like this idea best, it's relatively simple (even with adding
support for reporting flags in VIDIOC_QUERYBUF) for the purpose.

If we ever need the clock selection API I would vote for an IOCTL.
The controls API is a bad choice for something such fundamental as
type of clock for buffer timestamping IMHO. Let's stop making the
controls API a dumping ground for almost everything in V4L2! ;)

> Thanks for adding this. I knew I was forgetting something but didn't 
> remember what --- I swear it was unintentional! :-)
> 
> If we'd add more clocks without providing an ability to choose the clock 
> from the user space, how would the clock be selected? It certainly isn't 
> the driver's job, nor I think it should be system-specific either 
> (platform data on embedded systems).
> 
> It's up to the application and its needs. That would suggest we should 
> always provide monotonic timestamps to applications (besides a potential 
> driver-specific timestamp), and for that purpose the capability flag --- 
> I admit I disliked the idea at first --- is enough.
> 
> What comes to buffer flags, the application would also have to receive 
> the first buffer from the device to even know what kind of timestamps 
> the device uses, or at least call QUERYBUF. And in principle the flag 
> should be checked on every buffer, unless we also specify the flag is 
> the same for all buffers. And at certain point this will stop to make 
> any sense...

Good point. Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
timestamps type only for the time they are being called. Not per buffer,
per device. And applications would be checking the flags any time they
want to find out what is the buffer timestamp type. Or every time if it
don't have full control over the device (S/G_PRIORITY).

> A capability flag is cleaner solution from this perspective, and it can 
> be amended by a control (or an ioctl) later on: the flag can be 
> disregarded by applications whenever the control is present. If the 
> application doesn't know about the control it can still rely on the 
> flag. (I think this would be less clean than to go for the control right 
> from the beginning, but better IMO.)

But with the capability flag we would only be able to report one type of
clock, right ?

>>> Device-dependent timestamp
>>> --------------------------
>>>
>>> Should we agree on selectable timestamps, the existing timestamp 
>>> field (or a
>>> union with another field of different type) could be used for the
>>> device-dependent timestamps.
>>
>> No. Device timestamps should get their own field. You want to be able 
>> to relate
>> device timestamps with the monotonic timestamps, so you need both.
>>
>>> Alternatively we can choose to re-use the
>>> existing timecode field.
>>>
>>> At the moment there's no known use case for passing device-dependent
>>> timestamps at the same time with monotonic timestamps.
>>
>> Well, the use case is there, but there is no driver support. The device
>> timestamps should be 64 bits to accomodate things like PTS and DTS from
>> MPEG streams. Since timecode is 128 bits we might want to use two u64 
>> fields
>> or perhaps 4 u32 fields.
> 
> That should be an union for different kinds (or rather types) of 
> device-dependent timestamps. On uvcvideo I think this is u32, not u64. 
> We should be also able to tell what kind device dependent timestamp 
> there is --- should buffer flags be used for that as well?

Timecode has 'type' and 'flags' fields, couldn't it be accommodated for 
reporting device-dependant timestamps as well ?

--

Regards,
Sylwester

Re: [RFC] Timestamps and V4L2

[Daniel Glöckner]

On Sat, Sep 22, 2012 at 07:12:52PM +0200, Sylwester Nawrocki wrote:
> If we ever need the clock selection API I would vote for an IOCTL.
> The controls API is a bad choice for something such fundamental as
> type of clock for buffer timestamping IMHO. Let's stop making the
> controls API a dumping ground for almost everything in V4L2! ;)

> Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
> timestamps type only for the time they are being called. Not per buffer,
> per device. And applications would be checking the flags any time they
> want to find out what is the buffer timestamp type. Or every time if it
> don't have full control over the device (S/G_PRIORITY).

I'm all for adding an IOCTL, but if we think about adding a
VIDIOC_S_TIMESTAMP_TYPE in the future, we might as well add a
VIDIOC_G_TIMESTAMP_TYPE right now. Old drivers will return ENOSYS,
so the application knows it will have to guess the type (or take own
timestamps).

I can't imagine anything useful coming from an app that has to process
timestamps that change their source every now and then and I seriously
doubt anyone will go to such an extent that they check the timestamp
type on every buffer. If they don't set their priority high enough to
prevent others from changing the timestamp type, they also run the
risk of someone else changing the image format. It should be enough to
forbid changing the timestamp type while I/O is in progress, as it is
done for VIDIOC_S_FMT.

  Daniel

Re: [RFC] Timestamps and V4L2

[Sylwester Nawrocki]

On 09/22/2012 10:28 PM, Daniel Glöckner wrote:
> On Sat, Sep 22, 2012 at 07:12:52PM +0200, Sylwester Nawrocki wrote:
>> If we ever need the clock selection API I would vote for an IOCTL.
>> The controls API is a bad choice for something such fundamental as
>> type of clock for buffer timestamping IMHO. Let's stop making the
>> controls API a dumping ground for almost everything in V4L2! ;)
> 
>> Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
>> timestamps type only for the time they are being called. Not per buffer,
>> per device. And applications would be checking the flags any time they
>> want to find out what is the buffer timestamp type. Or every time if it
>> don't have full control over the device (S/G_PRIORITY).
> 
> I'm all for adding an IOCTL, but if we think about adding a
> VIDIOC_S_TIMESTAMP_TYPE in the future, we might as well add a
> VIDIOC_G_TIMESTAMP_TYPE right now. Old drivers will return ENOSYS,
> so the application knows it will have to guess the type (or take own
> timestamps).

Hmm, would it make sense to design a single ioctl that would allow
getting and setting the clock type, e.g. VIDIOC_CLOCK/TIMESTAMP_TYPE ?

> I can't imagine anything useful coming from an app that has to process
> timestamps that change their source every now and then and I seriously
> doubt anyone will go to such an extent that they check the timestamp
> type on every buffer. If they don't set their priority high enough to
> prevent others from changing the timestamp type, they also run the
> risk of someone else changing the image format. It should be enough to
> forbid changing the timestamp type while I/O is in progress, as it is
> done for VIDIOC_S_FMT.

I agree, but mem-to-mem devices can have multiple logically independent,
"concurrent" streams active. If the clock type is per device it might 
not be that straightforward...


Regards,
Sylwester

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Sylwester,

On Sunday 23 September 2012 20:40:36 Sylwester Nawrocki wrote:
> On 09/22/2012 10:28 PM, Daniel Glöckner wrote:
> > On Sat, Sep 22, 2012 at 07:12:52PM +0200, Sylwester Nawrocki wrote:
> >> If we ever need the clock selection API I would vote for an IOCTL.
> >> The controls API is a bad choice for something such fundamental as
> >> type of clock for buffer timestamping IMHO. Let's stop making the
> >> controls API a dumping ground for almost everything in V4L2! ;)
> >> 
> >> Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
> >> timestamps type only for the time they are being called. Not per buffer,
> >> per device. And applications would be checking the flags any time they
> >> want to find out what is the buffer timestamp type. Or every time if it
> >> don't have full control over the device (S/G_PRIORITY).
> > 
> > I'm all for adding an IOCTL, but if we think about adding a
> > VIDIOC_S_TIMESTAMP_TYPE in the future, we might as well add a
> > VIDIOC_G_TIMESTAMP_TYPE right now. Old drivers will return ENOSYS,
> > so the application knows it will have to guess the type (or take own
> > timestamps).
> 
> Hmm, would it make sense to design a single ioctl that would allow
> getting and setting the clock type, e.g. VIDIOC_CLOCK/TIMESTAMP_TYPE ?
> 
> > I can't imagine anything useful coming from an app that has to process
> > timestamps that change their source every now and then and I seriously
> > doubt anyone will go to such an extent that they check the timestamp
> > type on every buffer. If they don't set their priority high enough to
> > prevent others from changing the timestamp type, they also run the
> > risk of someone else changing the image format. It should be enough to
> > forbid changing the timestamp type while I/O is in progress, as it is
> > done for VIDIOC_S_FMT.
> 
> I agree, but mem-to-mem devices can have multiple logically independent,
> "concurrent" streams active. If the clock type is per device it might
> not be that straightforward...

Does the clock type need to be selectable for mem-to-mem devices ? Do device-
specific timestamps make sense there ?

-- 
Regards,

Laurent Pinchart

RE: Re: [RFC] Timestamps and V4L2

[Kamil Debski]

Hi,

Sorry for such a long absence on the mailing list after the Mini summit,
I was out of office. I see that the discussion about timestamps has
already started, so I would like to add some comments. Especially
about mem-to-mem devices.

> Subject: Re: [RFC] Timestamps and V4L2
> Date: Tue, 25 Sep 2012 02:35:47 +0200
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Hi Sylwester,
> 
> On Sunday 23 September 2012 20:40:36 Sylwester Nawrocki wrote:
> > On 09/22/2012 10:28 PM, Daniel Glöckner wrote:
> > > On Sat, Sep 22, 2012 at 07:12:52PM +0200, Sylwester Nawrocki wrote:
> > >> If we ever need the clock selection API I would vote for an IOCTL.
> > >> The controls API is a bad choice for something such fundamental as
> > >> type of clock for buffer timestamping IMHO. Let's stop making the
> > >> controls API a dumping ground for almost everything in V4L2! ;)
> > >>
> > >> Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
> > >> timestamps type only for the time they are being called. Not per
buffer,
> > >> per device. And applications would be checking the flags any time they
> > >> want to find out what is the buffer timestamp type. Or every time if it
> > >> don't have full control over the device (S/G_PRIORITY).
> > >
> > > I'm all for adding an IOCTL, but if we think about adding a
> > > VIDIOC_S_TIMESTAMP_TYPE in the future, we might as well add a
> > > VIDIOC_G_TIMESTAMP_TYPE right now. Old drivers will return ENOSYS,
> > > so the application knows it will have to guess the type (or take own
> > > timestamps).
> >
> > Hmm, would it make sense to design a single ioctl that would allow
> > getting and setting the clock type, e.g. VIDIOC_CLOCK/TIMESTAMP_TYPE ?
> >
> > > I can't imagine anything useful coming from an app that has to process
> > > timestamps that change their source every now and then and I seriously
> > > doubt anyone will go to such an extent that they check the timestamp
> > > type on every buffer. If they don't set their priority high enough to
> > > prevent others from changing the timestamp type, they also run the
> > > risk of someone else changing the image format. It should be enough to
> > > forbid changing the timestamp type while I/O is in progress, as it is
> > > done for VIDIOC_S_FMT.
> >
> > I agree, but mem-to-mem devices can have multiple logically independent,
> > "concurrent" streams active. If the clock type is per device it might
> > not be that straightforward...
> 
> Does the clock type need to be selectable for mem-to-mem devices ? Do
device-
> specific timestamps make sense there ?

I think that device-specific (and device assigned) timestamp does not make
much
sense for m2m devices. The solution of my preference is copying the timestamp
and timecode structures from the OUTPUT buffer to the CAPTURE buffer resulting
from processing of the said OUTPUT buffer.

Let's analyze an m2m video codec (decoder):
1) The processed CAPTURE buffers can be dequeued in a different order then the
OUTPUT buffers that were used to generate them. The OUTPUT buffers are
supplied
in decoding order and are returned in display order.
2) One OUTPUT buffer can generate multiple CAPTURE buffers (two frames in one
compressed buffer)
3) A single CAPTURE buffer can be generated from multiple OUTPUT buffers (for
example in the slice mode, where each OUTPUT buffer contains only part of the
frame)

So when the contents of the timestamp/timecode are copied we get:
- a way to identify related OUTPUT and CAPTURE buffers (important)
- in case 2 we get multiple CAPTURE buffers with the same timestamp/timecode
  and incrementing sequence number
- in case 3 we get an CAPTURE buffer with the value of timestamp/timecode of
the
  last buffer OUTPUT used to generate that buffer
- an increasing sequence number as it is simply incremented for every dequeued
  CAPTURE buffer (not affect by copying the timestamp, but I think it's worth
  to mention this)

For simpler m2m devices - such as FIMC and G2D it still makes sense, but is
not
that necessary, as they return buffers CAPTURE in the same order as OTUPUT
buffers, which they were generated from.

Best wishes,
--
Kamil Debski
Linux Platform Group
Samsung Poland R&D Center

Re: [RFC] Timestamps and V4L2

[Sylwester Nawrocki]

Hi Laurent,

On 09/25/2012 02:35 AM, Laurent Pinchart wrote:
> Does the clock type need to be selectable for mem-to-mem devices ? Do device-
> specific timestamps make sense there ?

I'd like to clarify one thing here, i.e. if we select device-specific
timestamps how should the v4l2_buffer::timestamp field behave ?

Are these two things exclusive ? Or should v4l2_buffer::timestamp be 
valid even if device-specific timestamps are enabled ?

With regards to your question, I think device-specific timestamps make
sense for mem-to-mem devices. Maybe not for the very simple ones, that
process buffers 1-to-1, but codecs may need it. I was told the Exynos/
S5P Multi Format Codec device has some register the timestamps could
be read from, but it's currently not used by the s5p-mfc driver. Kamil
might provide more details on that.

I guess if capture and output devices can have their timestamping clocks
selectable it should be also possible for mem-to-mem devices.

--

Regards,
Sylwester

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Sylwester,

On Thursday 27 September 2012 00:30:49 Sylwester Nawrocki wrote:
> On 09/25/2012 02:35 AM, Laurent Pinchart wrote:
> > Does the clock type need to be selectable for mem-to-mem devices ? Do
> > device- specific timestamps make sense there ?
> 
> I'd like to clarify one thing here, i.e. if we select device-specific
> timestamps how should the v4l2_buffer::timestamp field behave ?
> 
> Are these two things exclusive ? Or should v4l2_buffer::timestamp be
> valid even if device-specific timestamps are enabled ?

That's a very good question. The use cases I have in mind don't need both at 
the same time. The point of device-specific timestamps is to get a precise 
timestamp corresponding to the frame capture time, instead of the frame 
transfer time. They need to be correlated with system timestamps, but for that 
we need device-specific APIs to pass correlation information to userspace. 
Passing a "transfer time" system timestamp along with the device timestamp 
would be useless, as there would be no good correlation between the two.

> With regards to your question, I think device-specific timestamps make
> sense for mem-to-mem devices. Maybe not for the very simple ones, that
> process buffers 1-to-1, but codecs may need it. I was told the Exynos/
> S5P Multi Format Codec device has some register the timestamps could
> be read from, but it's currently not used by the s5p-mfc driver. Kamil
> might provide more details on that.

What kind of timestamps are they ?

> I guess if capture and output devices can have their timestamping clocks
> selectable it should be also possible for mem-to-mem devices.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Sylwester,

On Sat, Sep 22, 2012 at 07:12:52PM +0200, Sylwester Nawrocki wrote:
> On 09/22/2012 02:38 PM, Sakari Ailus wrote:
> >> You are missing one other option:
> >>
> >> Using v4l2_buffer flags to report the clock
> >> -------------------------------------------
> >>
> >> By defining flags like this:
> >>
> >> V4L2_BUF_FLAG_CLOCK_MASK 0x7000
> >> /* Possible Clocks */
> >> V4L2_BUF_FLAG_CLOCK_UNKNOWN 0x0000 /* system or monotonic, we don't 
> >> know */
> >> V4L2_BUF_FLAG_CLOCK_MONOTONIC 0x1000
> >>
> >> you could tell the application which clock is used.
> >>
> >> This does allow for more clocks to be added in the future and clock 
> >> selection
> >> would then be done by a control or possibly an ioctl. For now there 
> >> are no
> >> plans to do such things, so this flag should be sufficient. And it can be
> >> implemented very efficiently. It works with existing drivers as well, 
> >> since
> >> they will report CLOCK_UNKNOWN.
> >>
> >> I am very much in favor of this approach.
> 
> +1
> 
> I think I like this idea best, it's relatively simple (even with adding
> support for reporting flags in VIDIOC_QUERYBUF) for the purpose.
> 
> If we ever need the clock selection API I would vote for an IOCTL.
> The controls API is a bad choice for something such fundamental as
> type of clock for buffer timestamping IMHO. Let's stop making the
> controls API a dumping ground for almost everything in V4L2! ;)

Why would the control API be worse than an IOCTL for choosing the type of
the timestamp? The control API after all has functionality for exactly for
this: this is an obvious menu control.

What comes to the nature of things that can be configured using controls and
what can be done using IOCTLs I see no difference. It's just a mechanism.
That's what traditional Unix APIs do in general: provide mechanism, not a
policy.

> > Thanks for adding this. I knew I was forgetting something but didn't 
> > remember what --- I swear it was unintentional! :-)
> > 
> > If we'd add more clocks without providing an ability to choose the clock 
> > from the user space, how would the clock be selected? It certainly isn't 
> > the driver's job, nor I think it should be system-specific either 
> > (platform data on embedded systems).
> > 
> > It's up to the application and its needs. That would suggest we should 
> > always provide monotonic timestamps to applications (besides a potential 
> > driver-specific timestamp), and for that purpose the capability flag --- 
> > I admit I disliked the idea at first --- is enough.
> > 
> > What comes to buffer flags, the application would also have to receive 
> > the first buffer from the device to even know what kind of timestamps 
> > the device uses, or at least call QUERYBUF. And in principle the flag 
> > should be checked on every buffer, unless we also specify the flag is 
> > the same for all buffers. And at certain point this will stop to make 
> > any sense...
> 
> Good point. Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
> timestamps type only for the time they are being called. Not per buffer,
> per device. And applications would be checking the flags any time they
> want to find out what is the buffer timestamp type. Or every time if it
> don't have full control over the device (S/G_PRIORITY).

I think we should have valid timestamp flags for every buffer. What I meant
to say was that we should say in the definition that the flags will be the
same for every buffer --- that will hold until (or if) we provide a
mechanism for timestamp source selecton. In that canse the flags will
reflect the user-selected timestamp.

> > A capability flag is cleaner solution from this perspective, and it can 
> > be amended by a control (or an ioctl) later on: the flag can be 
> > disregarded by applications whenever the control is present. If the 
> > application doesn't know about the control it can still rely on the 
> > flag. (I think this would be less clean than to go for the control right 
> > from the beginning, but better IMO.)
> 
> But with the capability flag we would only be able to report one type of
> clock, right ?

That's true but doesn't that apply to any other non-application-selectable
timestamp source, apart from the device dependent timestamps?

> >>> Device-dependent timestamp
> >>> --------------------------
> >>>
> >>> Should we agree on selectable timestamps, the existing timestamp 
> >>> field (or a
> >>> union with another field of different type) could be used for the
> >>> device-dependent timestamps.
> >>
> >> No. Device timestamps should get their own field. You want to be able 
> >> to relate
> >> device timestamps with the monotonic timestamps, so you need both.
> >>
> >>> Alternatively we can choose to re-use the
> >>> existing timecode field.
> >>>
> >>> At the moment there's no known use case for passing device-dependent
> >>> timestamps at the same time with monotonic timestamps.
> >>
> >> Well, the use case is there, but there is no driver support. The device
> >> timestamps should be 64 bits to accomodate things like PTS and DTS from
> >> MPEG streams. Since timecode is 128 bits we might want to use two u64 
> >> fields
> >> or perhaps 4 u32 fields.
> > 
> > That should be an union for different kinds (or rather types) of 
> > device-dependent timestamps. On uvcvideo I think this is u32, not u64. 
> > We should be also able to tell what kind device dependent timestamp 
> > there is --- should buffer flags be used for that as well?
> 
> Timecode has 'type' and 'flags' fields, couldn't it be accommodated for 
> reporting device-dependant timestamps as well ?

The whole timdecode field should be removed as it no longer is used. At
least part of that will then be used for the device specific timestamp.

Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [RFC] Timestamps and V4L2

[Rémi Denis-Courmont]

Le dimanche 23 septembre 2012 14:43:42, Sakari Ailus a écrit :
> > I think I like this idea best, it's relatively simple (even with adding
> > support for reporting flags in VIDIOC_QUERYBUF) for the purpose.
> > 
> > If we ever need the clock selection API I would vote for an IOCTL.
> > The controls API is a bad choice for something such fundamental as
> > type of clock for buffer timestamping IMHO. Let's stop making the
> > controls API a dumping ground for almost everything in V4L2! ;)
> 
> Why would the control API be worse than an IOCTL for choosing the type of
> the timestamp? The control API after all has functionality for exactly for
> this: this is an obvious menu control.
> 
> What comes to the nature of things that can be configured using controls
> and what can be done using IOCTLs I see no difference. It's just a
> mechanism. That's what traditional Unix APIs do in general: provide
> mechanism, not a policy.

Seriously? Timestamp is _not_ a controllable hardware feature like brightness 
or flash. Controls are meant to build user interface controls for interaction 
with the user. Timestamp is _not_ something the user should control directly. 
The application should figure out what it gets and what it needs.

Or why do you use STREAMON/STREAMOFF instead of a STREAM boolean control, eh?

-- 
Rémi Denis-Courmont
http://www.remlab.net/

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Mon September 24 2012 22:11:50 Rémi Denis-Courmont wrote:
> Le dimanche 23 septembre 2012 14:43:42, Sakari Ailus a écrit :
> > > I think I like this idea best, it's relatively simple (even with adding
> > > support for reporting flags in VIDIOC_QUERYBUF) for the purpose.
> > > 
> > > If we ever need the clock selection API I would vote for an IOCTL.
> > > The controls API is a bad choice for something such fundamental as
> > > type of clock for buffer timestamping IMHO. Let's stop making the
> > > controls API a dumping ground for almost everything in V4L2! ;)
> > 
> > Why would the control API be worse than an IOCTL for choosing the type of
> > the timestamp? The control API after all has functionality for exactly for
> > this: this is an obvious menu control.
> > 
> > What comes to the nature of things that can be configured using controls
> > and what can be done using IOCTLs I see no difference. It's just a
> > mechanism. That's what traditional Unix APIs do in general: provide
> > mechanism, not a policy.
> 
> Seriously? Timestamp is _not_ a controllable hardware feature like brightness 
> or flash. Controls are meant to build user interface controls for interaction 
> with the user. Timestamp is _not_ something the user should control directly. 
> The application should figure out what it gets and what it needs.

Exactly. I agree completely.

Regards,

	Hans

> 
> Or why do you use STREAMON/STREAMOFF instead of a STREAM boolean control, eh?
> 
> 

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Sylwester,

On Saturday 22 September 2012 19:12:52 Sylwester Nawrocki wrote:
> On 09/22/2012 02:38 PM, Sakari Ailus wrote:
> >> You are missing one other option:
> >> 
> >> Using v4l2_buffer flags to report the clock
> >> -------------------------------------------
> >> 
> >> By defining flags like this:
> >> 
> >> V4L2_BUF_FLAG_CLOCK_MASK 0x7000
> >> /* Possible Clocks */
> >> V4L2_BUF_FLAG_CLOCK_UNKNOWN 0x0000 /* system or monotonic, we don't
> >> know */
> >> V4L2_BUF_FLAG_CLOCK_MONOTONIC 0x1000
> >> 
> >> you could tell the application which clock is used.
> >> 
> >> This does allow for more clocks to be added in the future and clock
> >> selection would then be done by a control or possibly an ioctl. For now
> >> there are no plans to do such things, so this flag should be sufficient.
> >> And it can be implemented very efficiently. It works with existing
> >> drivers as well, since they will report CLOCK_UNKNOWN.
> >> 
> >> I am very much in favor of this approach.
> 
> +1
> 
> I think I like this idea best, it's relatively simple (even with adding
> support for reporting flags in VIDIOC_QUERYBUF) for the purpose.
> 
> If we ever need the clock selection API I would vote for an IOCTL.
> The controls API is a bad choice for something such fundamental as
> type of clock for buffer timestamping IMHO. Let's stop making the
> controls API a dumping ground for almost everything in V4L2! ;)

What's wrong in using the control API in this case ? :-)

> > Thanks for adding this. I knew I was forgetting something but didn't
> > remember what --- I swear it was unintentional! :-)
> > 
> > If we'd add more clocks without providing an ability to choose the clock
> > from the user space, how would the clock be selected? It certainly isn't
> > the driver's job, nor I think it should be system-specific either
> > (platform data on embedded systems).
> > 
> > It's up to the application and its needs. That would suggest we should
> > always provide monotonic timestamps to applications (besides a potential
> > driver-specific timestamp), and for that purpose the capability flag ---
> > I admit I disliked the idea at first --- is enough.
> > 
> > What comes to buffer flags, the application would also have to receive
> > the first buffer from the device to even know what kind of timestamps
> > the device uses, or at least call QUERYBUF. And in principle the flag
> > should be checked on every buffer, unless we also specify the flag is
> > the same for all buffers. And at certain point this will stop to make
> > any sense...
> 
> Good point. Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
> timestamps type only for the time they are being called. Not per buffer,
> per device. And applications would be checking the flags any time they
> want to find out what is the buffer timestamp type. Or every time if it
> don't have full control over the device (S/G_PRIORITY).
> 
> > A capability flag is cleaner solution from this perspective, and it can
> > be amended by a control (or an ioctl) later on: the flag can be
> > disregarded by applications whenever the control is present. If the
> > application doesn't know about the control it can still rely on the
> > flag. (I think this would be less clean than to go for the control right
> > from the beginning, but better IMO.)
> 
> But with the capability flag we would only be able to report one type of
> clock, right ?

That's correct. The capability flag could mean "I support the clock selection 
API and default to a monotonic timestamp" though.

> >>> Device-dependent timestamp
> >>> --------------------------
> >>> 
> >>> Should we agree on selectable timestamps, the existing timestamp field
> >>> (or a union with another field of different type) could be used for the
> >>> device-dependent timestamps.
> >> 
> >> No. Device timestamps should get their own field. You want to be able
> >> to relate device timestamps with the monotonic timestamps, so you need
> >> both.
> >> 
> >>> Alternatively we can choose to re-use the existing timecode field.
> >>> 
> >>> At the moment there's no known use case for passing device-dependent
> >>> timestamps at the same time with monotonic timestamps.
> >> 
> >> Well, the use case is there, but there is no driver support. The device
> >> timestamps should be 64 bits to accomodate things like PTS and DTS from
> >> MPEG streams. Since timecode is 128 bits we might want to use two u64
> >> fields or perhaps 4 u32 fields.
> > 
> > That should be an union for different kinds (or rather types) of
> > device-dependent timestamps. On uvcvideo I think this is u32, not u64.
> > We should be also able to tell what kind device dependent timestamp
> > there is --- should buffer flags be used for that as well?
> 
> Timecode has 'type' and 'flags' fields, couldn't it be accommodated for
> reporting device-dependant timestamps as well ?

The timecode field is free for reuse, so we can definitely use it for device-
specific timestamps.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Sylwester Nawrocki]

Hi Laurent,

On 09/25/2012 02:34 AM, Laurent Pinchart wrote:
> On Saturday 22 September 2012 19:12:52 Sylwester Nawrocki wrote:
>> On 09/22/2012 02:38 PM, Sakari Ailus wrote:
>>>> You are missing one other option:
>>>>
>>>> Using v4l2_buffer flags to report the clock
>>>> -------------------------------------------
>>>>
>>>> By defining flags like this:
>>>>
>>>> V4L2_BUF_FLAG_CLOCK_MASK 0x7000
>>>> /* Possible Clocks */
>>>> V4L2_BUF_FLAG_CLOCK_UNKNOWN 0x0000 /* system or monotonic, we don't
>>>> know */
>>>> V4L2_BUF_FLAG_CLOCK_MONOTONIC 0x1000
>>>>
>>>> you could tell the application which clock is used.
>>>>
>>>> This does allow for more clocks to be added in the future and clock
>>>> selection would then be done by a control or possibly an ioctl. For now
>>>> there are no plans to do such things, so this flag should be sufficient.
>>>> And it can be implemented very efficiently. It works with existing
>>>> drivers as well, since they will report CLOCK_UNKNOWN.
>>>>
>>>> I am very much in favor of this approach.
>>
>> +1
>>
>> I think I like this idea best, it's relatively simple (even with adding
>> support for reporting flags in VIDIOC_QUERYBUF) for the purpose.
>>
>> If we ever need the clock selection API I would vote for an IOCTL.
>> The controls API is a bad choice for something such fundamental as
>> type of clock for buffer timestamping IMHO. Let's stop making the
>> controls API a dumping ground for almost everything in V4L2! ;)
> 
> What's wrong in using the control API in this case ? :-)

It just feels like not the right tool for the job. I don't think
timestamps are something the user is supposed to play with directly.
IMHO it's better to keep controls for purposes they were originally
designed as much as possible, i.e. for tweaking some parameters of
a device. The controls are already quite complex and probably not so
easy to grasp from a user perspective, even without things like
timestamps added there.

>>> Thanks for adding this. I knew I was forgetting something but didn't
>>> remember what --- I swear it was unintentional! :-)
>>>
>>> If we'd add more clocks without providing an ability to choose the clock
>>> from the user space, how would the clock be selected? It certainly isn't
>>> the driver's job, nor I think it should be system-specific either
>>> (platform data on embedded systems).
>>>
>>> It's up to the application and its needs. That would suggest we should
>>> always provide monotonic timestamps to applications (besides a potential
>>> driver-specific timestamp), and for that purpose the capability flag ---
>>> I admit I disliked the idea at first --- is enough.
>>>
>>> What comes to buffer flags, the application would also have to receive
>>> the first buffer from the device to even know what kind of timestamps
>>> the device uses, or at least call QUERYBUF. And in principle the flag
>>> should be checked on every buffer, unless we also specify the flag is
>>> the same for all buffers. And at certain point this will stop to make
>>> any sense...
>>
>> Good point. Perhaps VIDIOC_QUERYBUF and VIDIOC_DQBUF should be reporting
>> timestamps type only for the time they are being called. Not per buffer,
>> per device. And applications would be checking the flags any time they
>> want to find out what is the buffer timestamp type. Or every time if it
>> don't have full control over the device (S/G_PRIORITY).
>>
>>> A capability flag is cleaner solution from this perspective, and it can
>>> be amended by a control (or an ioctl) later on: the flag can be
>>> disregarded by applications whenever the control is present. If the
>>> application doesn't know about the control it can still rely on the
>>> flag. (I think this would be less clean than to go for the control right
>>> from the beginning, but better IMO.)
>>
>> But with the capability flag we would only be able to report one type of
>> clock, right ?
> 
> That's correct. The capability flag could mean "I support the clock selection
> API and default to a monotonic timestamp" though.

OK, sounds good.

>>>>> Device-dependent timestamp
>>>>> --------------------------
>>>>>
>>>>> Should we agree on selectable timestamps, the existing timestamp field
>>>>> (or a union with another field of different type) could be used for the
>>>>> device-dependent timestamps.
>>>>
>>>> No. Device timestamps should get their own field. You want to be able
>>>> to relate device timestamps with the monotonic timestamps, so you need
>>>> both.
>>>>
>>>>> Alternatively we can choose to re-use the existing timecode field.
>>>>>
>>>>> At the moment there's no known use case for passing device-dependent
>>>>> timestamps at the same time with monotonic timestamps.
>>>>
>>>> Well, the use case is there, but there is no driver support. The device
>>>> timestamps should be 64 bits to accomodate things like PTS and DTS from
>>>> MPEG streams. Since timecode is 128 bits we might want to use two u64
>>>> fields or perhaps 4 u32 fields.
>>>
>>> That should be an union for different kinds (or rather types) of
>>> device-dependent timestamps. On uvcvideo I think this is u32, not u64.
>>> We should be also able to tell what kind device dependent timestamp
>>> there is --- should buffer flags be used for that as well?
>>
>> Timecode has 'type' and 'flags' fields, couldn't it be accommodated for
>> reporting device-dependant timestamps as well ?
> 
> The timecode field is free for reuse, so we can definitely use it for device-
> specific timestamps.

All right, I didn't realize then we could just completely reuse this field.
How it would be re-designed is another topic for (long) discussions. :)

--

Regards,
Sylwester

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Sat September 22 2012 14:38:07 Sakari Ailus wrote:
> Hi Hans,
> 
> Thanks for the comments.
> 
> Hans Verkuil wrote:
> > On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> >> Hi all,
> >>
> >>
> >> This RFC intends to summarise and further the recent discussion on
> >> linux-media regarding the proposed changes of timestamping V4L2 buffers.
> >>
> >>
> >> The problem
> >> ===========
> >>
> >> The V4L2 has long used realtime timestamps (such as
> >> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> >> handing them over to the user. This has been found problematic in
> >> associating the video buffers with data from other sources: realtime clock
> >> may jump around due to daylight saving time, for example, and ALSA
> >> (audio-video synchronisation is a common use case) user space API does not
> >> provide the user with realtime timestamps, but instead uses monotonic time
> >> (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> >>
> >> This is especially an issue in embedded systems where video recording is a
> >> common use case. Drivers typically used in such systems have silently
> >> switched to use monotonic timestamps. While against the spec, this is
> >> necessary for those systems to operate properly.
> >>
> >> In general, realtime timestamps are seen of little use in other than
> >> debugging purposes, but monotonic timestamps are fine for that as well. It's
> >> still possible that an application I'm not aware of uses them in a peculiar
> >> way that would be adversely affected by changing to monotonic timestamps.
> >> Nevertheless, we're not supposed to break the API (or ABI). It'd be also
> >> very important for the application to know what kind of timestamps are
> >> provided by the device.
> >>
> >>
> >> Requirements, wishes and constraints
> >> ====================================
> >>
> >> Now that it seems to be about the time to fix these issues, it's worth
> >> looking a little bit to the future to anticipate the coming changes to be
> >> able to accommodate them better later on.
> >>
> >> - The new default should be monotonic. As the monotonic timestamps are seen
> >> to be the most useful, they should be made the default.
> >>
> >> - timeval vs. timespec. The two structs can be used to store timestamp
> >> information. They are not compatible with each other. It's a little bit
> >> uncertain what's the case with all the architectures but it looks like the
> >> timespec fits into the space of timeval in all cases. If timespec is
> >> considered to be used somewhere the compatibility must be ensured. Timespec
> >> is better than timeval since timespec has more precision and it's the same
> >> struct that's used everywhere else in the V4L2 API: timespec does not need
> >> conversion to timespec in the user space.
> >>
> >> struct timespec {
> >>          __kernel_time_t tv_sec;                 /* seconds */
> >>          long            tv_nsec;                /* nanoseconds */
> >> };
> >>
> >> struct timeval {
> >>          __kernel_time_t         tv_sec;         /* seconds */
> >>          __kernel_suseconds_t    tv_usec;        /* microseconds */
> >> };
> >>
> >> To be able to use timespec, the user would have to most likely explicitly
> >> choose to do that.
> >>
> >> - Users should know what kind of timestamps the device produces. This
> >> includes existing and future kernels. What should be considered are
> >> uninformed porting drivers back and forth across kernel versions and
> >> out-of-date kernel header files.
> >>
> >> - Device-dependent timestamps. Some devices such as the uvcvideo ones
> >> produce device-dependent timestamps for synchronising video and audio, both
> >> produced by the same physical hardware device. For uvcvideo these timestamps
> >> are unsigned 32-bit integers.
> >>
> >> - There's also another clock, Linux-specific raw monotonic clock (as in
> >> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> >> cases than the regular monotonic clock. The difference is that the raw
> >> monotonic clock is free from the NTP adjustments. It would be nice for the
> >> user to be able to choose the clock used for timestamps. This is especially
> >> important for device-dependent timestamps: not all applications can be
> >> expected to be able to use them.
> >>
> >> - The field adjacent to timestamp, timecode, is 128 bits wide, and not used
> >> by a single driver. This field could be re-used.
> >>
> >>
> >> Possible solutions
> >> ==================
> >>
> >> Not all of the solutions below that have been proposed are mutually
> >> exclusive. That's also what's making the choice difficult: the ultimate
> >> solution to the issue of timestamping may involve several of these --- or
> >> possibly something better that's not on the list.
> >>
> >>
> >> Use of timespec
> >> ---------------
> >>
> >> If we can conclude timespec will always fit into the size of timeval (or
> >> timecode) we could use timespec instead. The solution should still make
> >> the use of timespec explicit to the user space. This seems to conflict with
> >> the idea of making monotonic timestamps the default: the default can't be
> >> anything incompatible with timeval, and at the same time it's the most
> >> important that the monotonic timestamps are timespec.
> >
> > We have to keep timeval. Changing this will break the ABI. I see absolutely
> > no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and that's
> > far, far removed from ns precisions. Should we ever have to support high-speed
> > cameras running at 60000 Hz, then we'll talk again.
> >
> > For me this is a non-issue.
> >
> >> Kernel version as indicator of timestamp
> >> ----------------------------------------
> >>
> >> Conversion of drivers to use monotonic timestamp is trivial, so the
> >> conversion could be done once and for all drivers. The kernel version could
> >> be used to indicate the type of the timestamp.
> >>
> >> If this approach is taken care must be taken when new drivers are
> >> integrated: developers sometimes use old kernels for development and might
> >> also use an old driver for guidance on timestamps, thus using real-time
> >> timestamps when monotonic timestamps should be used.
> >
> > More importantly, this also fails when users use out-of-tree drivers.
> 
> Could you mention some examples what we could be breaking in particular?

The version reported in QUERYCAP by out-of-tree drivers is unreliable. It has
no relationship with a kernel version in general. Actually, even inside the
kernel I've come across drivers that still fill in the version number themselves.

> >> This approach has an
> >> advantage over the capability flag below: which is that we don't populate
> >> the interface with essentially dead definitions.
> >
> > Using a kernel version to decide whether some feature is available or not is
> > IMHO something of a last resort. It's very application unfriendly.
> 
> Could be, but that's a passing pain. We're going to live with the flags 
> for the foreseeable future, whether we need them or not.

Never underestimate the glacial speed (or lack of speed) with which applications
adapt to new features.

> 
> >>
> >> Capability flag for monotonic timestamps
> >> ----------------------------------------
> >>
> >> A capability flag can be used to tell whether the timestamp is monotonic.
> >> However, it's not extensible cleanly to provide selectable timestamps. These
> >> are not features that are needed right now, though.
> >>
> >> The upside of this option is ease of implementation and use, but it's not
> >> extensible. Also we're left with a flag that's set for all drivers: in the
> >> end it provides no information to the user and is only noise in the spec.
> >>
> >>
> >> Control for timestamp type
> >> --------------------------
> >>
> >> Using a control to tell the type of the timestamp is extensible but not as
> >> easy to implement than the capability flag: each and every device would get
> >> an additional control. The value should likely be also file handle specific,
> >> and we do not have file handle specific controls yet.
> >
> > Yes, we do. You can make per-file handle controls. M2M devices need that.
> 
> Thanks for correcting me.
> 
> > I'm not sure why this would be filehandle specific, BTW.
> 
> Good point. I thought that as other properties of the buffers are 
> specific to file handles, including format when using CREATE_BUFS, it'd 
> make sense to make the timestamp source file-handle specific as well.
> 
> What do you think?

I don't think it makes sense to have different streams from the same device
use different clocks.

> >> In the meantime the control could be read-only, and later made read-write
> >> when the timestamp type can be made selectable. Much of he work of
> >> timestamping can be done by the framework: drivers can use a single helper
> >> function and need to create one extra standard control.
> >>
> >> Should the control also have an effect on the types of the timestamps in
> >> V4L2 events? Likely yes.
> >
> > You are missing one other option:
> >
> > Using v4l2_buffer flags to report the clock
> > -------------------------------------------
> >
> > By defining flags like this:
> >
> > V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> > /* Possible Clocks */
> > V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know */
> > V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> >
> > you could tell the application which clock is used.
> >
> > This does allow for more clocks to be added in the future and clock selection
> > would then be done by a control or possibly an ioctl. For now there are no
> > plans to do such things, so this flag should be sufficient. And it can be
> > implemented very efficiently. It works with existing drivers as well, since
> > they will report CLOCK_UNKNOWN.
> >
> > I am very much in favor of this approach.
> 
> Thanks for adding this. I knew I was forgetting something but didn't 
> remember what --- I swear it was unintentional! :-)
> 
> If we'd add more clocks without providing an ability to choose the clock 
> from the user space, how would the clock be selected? It certainly isn't 
> the driver's job, nor I think it should be system-specific either 
> (platform data on embedded systems).

IF a driver supports more than one clock (which I really don't see happening
anytime soon), then we either need a control to select the clock or an ioctl.
And something as well to enumerate the available clocks. I'm leaning towards
ioctls, but I think this should be decided if we ever get an actual use-case
for this.

> It's up to the application and its needs. That would suggest we should 
> always provide monotonic timestamps to applications (besides a potential 
> driver-specific timestamp), and for that purpose the capability flag --- 
> I admit I disliked the idea at first --- is enough.
> 
> What comes to buffer flags, the application would also have to receive 
> the first buffer from the device to even know what kind of timestamps 
> the device uses, or at least call QUERYBUF. And in principle the flag 
> should be checked on every buffer, unless we also specify the flag is 
> the same for all buffers. And at certain point this will stop to make 
> any sense...

It should definitely be the same for all buffers. And since apps will
typically call querybuf anyway I don't see this as a problem. These
clocks are also specific to the streaming I/O API, so reporting this as
part of that API makes sense to me as well.

> A capability flag is cleaner solution from this perspective, and it can 
> be amended by a control (or an ioctl) later on: the flag can be 
> disregarded by applications whenever the control is present.

Yuck.

> If the 
> application doesn't know about the control it can still rely on the 
> flag. (I think this would be less clean than to go for the control right 
> from the beginning, but better IMO.)
> 
> >>
> >>
> >> Device-dependent timestamp
> >> --------------------------
> >>
> >> Should we agree on selectable timestamps, the existing timestamp field (or a
> >> union with another field of different type) could be used for the
> >> device-dependent timestamps.
> >
> > No. Device timestamps should get their own field. You want to be able to relate
> > device timestamps with the monotonic timestamps, so you need both.
> >
> >> Alternatively we can choose to re-use the
> >> existing timecode field.
> >>
> >> At the moment there's no known use case for passing device-dependent
> >> timestamps at the same time with monotonic timestamps.
> >
> > Well, the use case is there, but there is no driver support. The device
> > timestamps should be 64 bits to accomodate things like PTS and DTS from
> > MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
> > or perhaps 4 u32 fields.
> 
> That should be an union for different kinds (or rather types) of 
> device-dependent timestamps. On uvcvideo I think this is u32, not u64. 
> We should be also able to tell what kind device dependent timestamp 
> there is --- should buffer flags be used for that as well?

That's definitely part of the buffer flags. The presence of timecode is already
signalled using that. And not every buffer may have device timestamps (that
depends on the hardware), so you have to signal it through the buffer flags.

An anonymous union might be best with the buffer flags signalling the type of
the union. What I don't know is how to specify the type. Shall we just specify
the type of the union (e.g. 4 u32 fields or 2 u64 fields) and leave the
interpretation of those fields up to the application based on the driver name?
Or shall the type act more like a fourcc in that it also uniquely identifies
the interpretation of the timestamps?

Or should all device timestamps be converted to a timespec by the driver?

Answers on a postcard.

Regards,

	Hans

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Hans,

Hans Verkuil wrote:
> On Sat September 22 2012 14:38:07 Sakari Ailus wrote:
>> Hi Hans,
>>
>> Thanks for the comments.
>>
>> Hans Verkuil wrote:
>>> On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
...
>>>>
>>>> Capability flag for monotonic timestamps
>>>> ----------------------------------------
>>>>
>>>> A capability flag can be used to tell whether the timestamp is monotonic.
>>>> However, it's not extensible cleanly to provide selectable timestamps. These
>>>> are not features that are needed right now, though.
>>>>
>>>> The upside of this option is ease of implementation and use, but it's not
>>>> extensible. Also we're left with a flag that's set for all drivers: in the
>>>> end it provides no information to the user and is only noise in the spec.
>>>>
>>>>
>>>> Control for timestamp type
>>>> --------------------------
>>>>
>>>> Using a control to tell the type of the timestamp is extensible but not as
>>>> easy to implement than the capability flag: each and every device would get
>>>> an additional control. The value should likely be also file handle specific,
>>>> and we do not have file handle specific controls yet.
>>>
>>> Yes, we do. You can make per-file handle controls. M2M devices need that.
>>
>> Thanks for correcting me.
>>
>>> I'm not sure why this would be filehandle specific, BTW.
>>
>> Good point. I thought that as other properties of the buffers are
>> specific to file handles, including format when using CREATE_BUFS, it'd
>> make sense to make the timestamp source file-handle specific as well.
>>
>> What do you think?
>
> I don't think it makes sense to have different streams from the same device
> use different clocks.
>
>>>> In the meantime the control could be read-only, and later made read-write
>>>> when the timestamp type can be made selectable. Much of he work of
>>>> timestamping can be done by the framework: drivers can use a single helper
>>>> function and need to create one extra standard control.
>>>>
>>>> Should the control also have an effect on the types of the timestamps in
>>>> V4L2 events? Likely yes.
>>>
>>> You are missing one other option:
>>>
>>> Using v4l2_buffer flags to report the clock
>>> -------------------------------------------
>>>
>>> By defining flags like this:
>>>
>>> V4L2_BUF_FLAG_CLOCK_MASK	0x7000
>>> /* Possible Clocks */
>>> V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know */
>>> V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
>>>
>>> you could tell the application which clock is used.
>>>
>>> This does allow for more clocks to be added in the future and clock selection
>>> would then be done by a control or possibly an ioctl. For now there are no
>>> plans to do such things, so this flag should be sufficient. And it can be
>>> implemented very efficiently. It works with existing drivers as well, since
>>> they will report CLOCK_UNKNOWN.
>>>
>>> I am very much in favor of this approach.
>>
>> Thanks for adding this. I knew I was forgetting something but didn't
>> remember what --- I swear it was unintentional! :-)
>>
>> If we'd add more clocks without providing an ability to choose the clock
>> from the user space, how would the clock be selected? It certainly isn't
>> the driver's job, nor I think it should be system-specific either
>> (platform data on embedded systems).
>
> IF a driver supports more than one clock (which I really don't see happening
> anytime soon), then we either need a control to select the clock or an ioctl.

The timestamps can be taken from any standard clock using a helper 
function. Right now drivers call do_gettimeofday() or ktime_get_ts(), 
it's the same whether the function was called e.g. v4l2_get_ts(). The 
driver doesn't even need to know which timer is being used to make that 
timestamp. It used to be the realtime clock and soon it's the monotonic 
clock. So I really don't see why the timestamp source should depend on 
the driver. It rather depends on what the user wants.

That said, I also don't see use for other clocks _right now_ than the 
monotonic one.

> And something as well to enumerate the available clocks. I'm leaning towards
> ioctls, but I think this should be decided if we ever get an actual use-case
> for this.

I think it should be a control --- I see no reason to add new IOCTLs 
when controls are equally, or even better, fit for the same job. But 
let's decide this only when the functionality is needed.

>> It's up to the application and its needs. That would suggest we should
>> always provide monotonic timestamps to applications (besides a potential
>> driver-specific timestamp), and for that purpose the capability flag ---
>> I admit I disliked the idea at first --- is enough.
>>
>> What comes to buffer flags, the application would also have to receive
>> the first buffer from the device to even know what kind of timestamps
>> the device uses, or at least call QUERYBUF. And in principle the flag
>> should be checked on every buffer, unless we also specify the flag is
>> the same for all buffers. And at certain point this will stop to make
>> any sense...
>
> It should definitely be the same for all buffers. And since apps will
> typically call querybuf anyway I don't see this as a problem. These
> clocks are also specific to the streaming I/O API, so reporting this as
> part of that API makes sense to me as well.
 >
>> A capability flag is cleaner solution from this perspective, and it can
>> be amended by a control (or an ioctl) later on: the flag can be
>> disregarded by applications whenever the control is present.
>
> Yuck.

What's so bad in it then? The user doesn't need to start dealing with 
buffers whenever (s)he needs to know what kind of timestamps the device 
provides.

Timestamps are also used on the event interface (and also on subdevs), 
which currently uses monotonic timestamps (at least it was intended to 
do so, it's missing from the documentation). The timestamps on V4L2 and 
V4L2 subdev must match to be useful for the application, so we should 
not allow using other clock than monotonic before we truly make the 
timestamp source selectable by the application.

A flash subdev, for example, could produce events while the driver 
producing timestamps for the video buffers doesn't know the type of the 
timestamp used by the flash driver. One still must be able to compare 
the two in the user space to make use of them.

I'm ok-ish with using video buffer flags to tell the type of the 
timestamp, but those flags might not be useful (like the capability 
flag) if the timestamp source is made selectable. What I still don't 
like is that you have to start dealing with buffers before you know what 
kind of timestamps you're getting.

I'd like to have Laurent's and Remi's opinion before proceeding to the 
implementation.

...

>>>> Device-dependent timestamp
>>>> --------------------------
>>>>
>>>> Should we agree on selectable timestamps, the existing timestamp field (or a
>>>> union with another field of different type) could be used for the
>>>> device-dependent timestamps.
>>>
>>> No. Device timestamps should get their own field. You want to be able to relate
>>> device timestamps with the monotonic timestamps, so you need both.
>>>
>>>> Alternatively we can choose to re-use the
>>>> existing timecode field.
>>>>
>>>> At the moment there's no known use case for passing device-dependent
>>>> timestamps at the same time with monotonic timestamps.
>>>
>>> Well, the use case is there, but there is no driver support. The device
>>> timestamps should be 64 bits to accomodate things like PTS and DTS from
>>> MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
>>> or perhaps 4 u32 fields.
>>
>> That should be an union for different kinds (or rather types) of
>> device-dependent timestamps. On uvcvideo I think this is u32, not u64.
>> We should be also able to tell what kind device dependent timestamp
>> there is --- should buffer flags be used for that as well?
>
> That's definitely part of the buffer flags. The presence of timecode is already
> signalled using that. And not every buffer may have device timestamps (that
> depends on the hardware), so you have to signal it through the buffer flags.
>
> An anonymous union might be best with the buffer flags signalling the type of
> the union. What I don't know is how to specify the type. Shall we just specify
> the type of the union (e.g. 4 u32 fields or 2 u64 fields) and leave the
> interpretation of those fields up to the application based on the driver name?
> Or shall the type act more like a fourcc in that it also uniquely identifies
> the interpretation of the timestamps?

I think it should uniquely identify it, at least if the MPEG timestamps 
produced by some devices are relevant for video buffers. In this case it 
wouldn't make much sense for the applications having to know what are 
the drivers producing such timestamps. And this might not be the only 
case like that.

> Or should all device timestamps be converted to a timespec by the driver?

Good question.

> Answers on a postcard.

Who are you going to send it? ;-)

Cheers,

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Sun September 23 2012 15:07:24 Sakari Ailus wrote:
> Hi Hans,
> 
> Hans Verkuil wrote:
> > On Sat September 22 2012 14:38:07 Sakari Ailus wrote:
> >> Hi Hans,
> >>
> >> Thanks for the comments.
> >>
> >> Hans Verkuil wrote:
> >>> On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> ...
> >>>>
> >>>> Capability flag for monotonic timestamps
> >>>> ----------------------------------------
> >>>>
> >>>> A capability flag can be used to tell whether the timestamp is monotonic.
> >>>> However, it's not extensible cleanly to provide selectable timestamps. These
> >>>> are not features that are needed right now, though.
> >>>>
> >>>> The upside of this option is ease of implementation and use, but it's not
> >>>> extensible. Also we're left with a flag that's set for all drivers: in the
> >>>> end it provides no information to the user and is only noise in the spec.
> >>>>
> >>>>
> >>>> Control for timestamp type
> >>>> --------------------------
> >>>>
> >>>> Using a control to tell the type of the timestamp is extensible but not as
> >>>> easy to implement than the capability flag: each and every device would get
> >>>> an additional control. The value should likely be also file handle specific,
> >>>> and we do not have file handle specific controls yet.
> >>>
> >>> Yes, we do. You can make per-file handle controls. M2M devices need that.
> >>
> >> Thanks for correcting me.
> >>
> >>> I'm not sure why this would be filehandle specific, BTW.
> >>
> >> Good point. I thought that as other properties of the buffers are
> >> specific to file handles, including format when using CREATE_BUFS, it'd
> >> make sense to make the timestamp source file-handle specific as well.
> >>
> >> What do you think?
> >
> > I don't think it makes sense to have different streams from the same device
> > use different clocks.
> >
> >>>> In the meantime the control could be read-only, and later made read-write
> >>>> when the timestamp type can be made selectable. Much of he work of
> >>>> timestamping can be done by the framework: drivers can use a single helper
> >>>> function and need to create one extra standard control.
> >>>>
> >>>> Should the control also have an effect on the types of the timestamps in
> >>>> V4L2 events? Likely yes.
> >>>
> >>> You are missing one other option:
> >>>
> >>> Using v4l2_buffer flags to report the clock
> >>> -------------------------------------------
> >>>
> >>> By defining flags like this:
> >>>
> >>> V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> >>> /* Possible Clocks */
> >>> V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know */
> >>> V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> >>>
> >>> you could tell the application which clock is used.
> >>>
> >>> This does allow for more clocks to be added in the future and clock selection
> >>> would then be done by a control or possibly an ioctl. For now there are no
> >>> plans to do such things, so this flag should be sufficient. And it can be
> >>> implemented very efficiently. It works with existing drivers as well, since
> >>> they will report CLOCK_UNKNOWN.
> >>>
> >>> I am very much in favor of this approach.
> >>
> >> Thanks for adding this. I knew I was forgetting something but didn't
> >> remember what --- I swear it was unintentional! :-)
> >>
> >> If we'd add more clocks without providing an ability to choose the clock
> >> from the user space, how would the clock be selected? It certainly isn't
> >> the driver's job, nor I think it should be system-specific either
> >> (platform data on embedded systems).
> >
> > IF a driver supports more than one clock (which I really don't see happening
> > anytime soon), then we either need a control to select the clock or an ioctl.
> 
> The timestamps can be taken from any standard clock using a helper 
> function. Right now drivers call do_gettimeofday() or ktime_get_ts(), 
> it's the same whether the function was called e.g. v4l2_get_ts(). The 
> driver doesn't even need to know which timer is being used to make that 
> timestamp. It used to be the realtime clock and soon it's the monotonic 
> clock. So I really don't see why the timestamp source should depend on 
> the driver. It rather depends on what the user wants.
> 
> That said, I also don't see use for other clocks _right now_ than the 
> monotonic one.
> 
> > And something as well to enumerate the available clocks. I'm leaning towards
> > ioctls, but I think this should be decided if we ever get an actual use-case
> > for this.
> 
> I think it should be a control --- I see no reason to add new IOCTLs 
> when controls are equally, or even better, fit for the same job. But 
> let's decide this only when the functionality is needed.

Yes, please. I think it is premature to talk about how to select a clock,
since I doubt we will ever need it. We just need a way to tell the app which
clock is being used (which in practice is either UNKNOWN or MONOTONIC), and
keep the option of adding other clocks open.

For M2M devices it is possible to select different clocks per-filehandle,
so that precludes having a QUERYCAP capability (that was a good point that
Sylwester made). Adding a clock mask to the buffer flags does exactly what
we need IMHO.

> >> It's up to the application and its needs. That would suggest we should
> >> always provide monotonic timestamps to applications (besides a potential
> >> driver-specific timestamp), and for that purpose the capability flag ---
> >> I admit I disliked the idea at first --- is enough.
> >>
> >> What comes to buffer flags, the application would also have to receive
> >> the first buffer from the device to even know what kind of timestamps
> >> the device uses, or at least call QUERYBUF. And in principle the flag
> >> should be checked on every buffer, unless we also specify the flag is
> >> the same for all buffers. And at certain point this will stop to make
> >> any sense...
> >
> > It should definitely be the same for all buffers. And since apps will
> > typically call querybuf anyway I don't see this as a problem. These
> > clocks are also specific to the streaming I/O API, so reporting this as
> > part of that API makes sense to me as well.
>  >
> >> A capability flag is cleaner solution from this perspective, and it can
> >> be amended by a control (or an ioctl) later on: the flag can be
> >> disregarded by applications whenever the control is present.
> >
> > Yuck.
> 
> What's so bad in it then? The user doesn't need to start dealing with 
> buffers whenever (s)he needs to know what kind of timestamps the device 
> provides.
> 
> Timestamps are also used on the event interface (and also on subdevs), 
> which currently uses monotonic timestamps (at least it was intended to 
> do so, it's missing from the documentation). The timestamps on V4L2 and 
> V4L2 subdev must match to be useful for the application, so we should 
> not allow using other clock than monotonic before we truly make the 
> timestamp source selectable by the application.
> 
> A flash subdev, for example, could produce events while the driver 
> producing timestamps for the video buffers doesn't know the type of the 
> timestamp used by the flash driver. One still must be able to compare 
> the two in the user space to make use of them.
> 
> I'm ok-ish with using video buffer flags to tell the type of the 
> timestamp, but those flags might not be useful (like the capability 
> flag) if the timestamp source is made selectable. What I still don't 
> like is that you have to start dealing with buffers before you know what 
> kind of timestamps you're getting.

If in the future we want to be able to select clock sources, then that
will automatically also provide a way of querying the current clock source
outside of the buffer flags. In the meantime, having the clock source as
part of the buffer flags solves the current problem with a minimum of fuss,
and even allows for switching clock sources on the fly should we ever want
that (highly unlikely, I admit).

> I'd like to have Laurent's and Remi's opinion before proceeding to the 
> implementation.
> 
> ...
> 
> >>>> Device-dependent timestamp
> >>>> --------------------------
> >>>>
> >>>> Should we agree on selectable timestamps, the existing timestamp field (or a
> >>>> union with another field of different type) could be used for the
> >>>> device-dependent timestamps.
> >>>
> >>> No. Device timestamps should get their own field. You want to be able to relate
> >>> device timestamps with the monotonic timestamps, so you need both.
> >>>
> >>>> Alternatively we can choose to re-use the
> >>>> existing timecode field.
> >>>>
> >>>> At the moment there's no known use case for passing device-dependent
> >>>> timestamps at the same time with monotonic timestamps.
> >>>
> >>> Well, the use case is there, but there is no driver support. The device
> >>> timestamps should be 64 bits to accomodate things like PTS and DTS from
> >>> MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
> >>> or perhaps 4 u32 fields.
> >>
> >> That should be an union for different kinds (or rather types) of
> >> device-dependent timestamps. On uvcvideo I think this is u32, not u64.
> >> We should be also able to tell what kind device dependent timestamp
> >> there is --- should buffer flags be used for that as well?
> >
> > That's definitely part of the buffer flags. The presence of timecode is already
> > signalled using that. And not every buffer may have device timestamps (that
> > depends on the hardware), so you have to signal it through the buffer flags.
> >
> > An anonymous union might be best with the buffer flags signalling the type of
> > the union. What I don't know is how to specify the type. Shall we just specify
> > the type of the union (e.g. 4 u32 fields or 2 u64 fields) and leave the
> > interpretation of those fields up to the application based on the driver name?
> > Or shall the type act more like a fourcc in that it also uniquely identifies
> > the interpretation of the timestamps?
> 
> I think it should uniquely identify it, at least if the MPEG timestamps 
> produced by some devices are relevant for video buffers. In this case it 
> wouldn't make much sense for the applications having to know what are 
> the drivers producing such timestamps. And this might not be the only 
> case like that.
> 
> > Or should all device timestamps be converted to a timespec by the driver?
> 
> Good question.

I'm leaning towards this, actually. What's the point of a device timestamp if
you don't know what the units are? And if you do, then you can convert it.

> 
> > Answers on a postcard.
> 
> Who are you going to send it? ;-)

What's your address? :-)

Regards,

	Hans

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Hans,

On Sunday 23 September 2012 11:18:45 Hans Verkuil wrote:
> On Sat September 22 2012 14:38:07 Sakari Ailus wrote:
> > Hans Verkuil wrote:
> > > On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> > >> Hi all,
> > >> 
> > >> 
> > >> This RFC intends to summarise and further the recent discussion on
> > >> linux-media regarding the proposed changes of timestamping V4L2
> > >> buffers.
> > >> 
> > >> 
> > >> The problem
> > >> ===========
> > >> 
> > >> The V4L2 has long used realtime timestamps (such as
> > >> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> > >> handing them over to the user. This has been found problematic in
> > >> associating the video buffers with data from other sources: realtime
> > >> clock may jump around due to daylight saving time, for example, and
> > >> ALSA (audio-video synchronisation is a common use case) user space API
> > >> does not provide the user with realtime timestamps, but instead uses
> > >> monotonic time (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> > >> 
> > >> This is especially an issue in embedded systems where video recording
> > >> is a common use case. Drivers typically used in such systems have
> > >> silently switched to use monotonic timestamps. While against the spec,
> > >> this is necessary for those systems to operate properly.
> > >> 
> > >> In general, realtime timestamps are seen of little use in other than
> > >> debugging purposes, but monotonic timestamps are fine for that as well.
> > >> It's still possible that an application I'm not aware of uses them in
> > >> a peculiar way that would be adversely affected by changing to
> > >> monotonic timestamps. Nevertheless, we're not supposed to break the
> > >> API (or ABI). It'd be also very important for the application to know
> > >> what kind of timestamps are provided by the device.
> > >> 
> > >> 
> > >> Requirements, wishes and constraints
> > >> ====================================
> > >> 
> > >> Now that it seems to be about the time to fix these issues, it's worth
> > >> looking a little bit to the future to anticipate the coming changes to
> > >> be able to accommodate them better later on.
> > >> 
> > >> - The new default should be monotonic. As the monotonic timestamps are
> > >> seen to be the most useful, they should be made the default.
> > >> 
> > >> - timeval vs. timespec. The two structs can be used to store timestamp
> > >> information. They are not compatible with each other. It's a little bit
> > >> uncertain what's the case with all the architectures but it looks like
> > >> the timespec fits into the space of timeval in all cases. If timespec
> > >> is considered to be used somewhere the compatibility must be ensured.
> > >> Timespec is better than timeval since timespec has more precision and
> > >> it's the same struct that's used everywhere else in the V4L2 API:
> > >> timespec does not need conversion to timespec in the user space.
> > >> 
> > >> struct timespec {
> > >>          __kernel_time_t tv_sec;                 /* seconds */
> > >>          long            tv_nsec;                /* nanoseconds */
> > >> };
> > >> 
> > >> struct timeval {
> > >>          __kernel_time_t         tv_sec;         /* seconds */
> > >>          __kernel_suseconds_t    tv_usec;        /* microseconds */
> > >> };
> > >> 
> > >> To be able to use timespec, the user would have to most likely
> > >> explicitly choose to do that.
> > >> 
> > >> - Users should know what kind of timestamps the device produces. This
> > >> includes existing and future kernels. What should be considered are
> > >> uninformed porting drivers back and forth across kernel versions and
> > >> out-of-date kernel header files.
> > >> 
> > >> - Device-dependent timestamps. Some devices such as the uvcvideo ones
> > >> produce device-dependent timestamps for synchronising video and audio,
> > >> both produced by the same physical hardware device. For uvcvideo these
> > >> timestamps are unsigned 32-bit integers.
> > >> 
> > >> - There's also another clock, Linux-specific raw monotonic clock (as in
> > >> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some
> > >> use cases than the regular monotonic clock. The difference is that the
> > >> raw monotonic clock is free from the NTP adjustments. It would be nice
> > >> for the user to be able to choose the clock used for timestamps. This
> > >> is especially important for device-dependent timestamps: not all
> > >> applications can be expected to be able to use them.
> > >> 
> > >> - The field adjacent to timestamp, timecode, is 128 bits wide, and not
> > >> used by a single driver. This field could be re-used.
> > >> 
> > >> 
> > >> Possible solutions
> > >> ==================
> > >> 
> > >> Not all of the solutions below that have been proposed are mutually
> > >> exclusive. That's also what's making the choice difficult: the ultimate
> > >> solution to the issue of timestamping may involve several of these ---
> > >> or possibly something better that's not on the list.
> > >> 
> > >> 
> > >> Use of timespec
> > >> ---------------
> > >> 
> > >> If we can conclude timespec will always fit into the size of timeval
> > >> (or timecode) we could use timespec instead. The solution should still
> > >> make the use of timespec explicit to the user space. This seems to
> > >> conflict with the idea of making monotonic timestamps the default: the
> > >> default can't be anything incompatible with timeval, and at the same
> > >> time it's the most important that the monotonic timestamps are
> > >> timespec.
> > > 
> > > We have to keep timeval. Changing this will break the ABI. I see
> > > absolutely no reason to use timespec for video. At 60 Hz a frame takes
> > > 16.67 ms, and that's far, far removed from ns precisions. Should we ever
> > > have to support high-speed cameras running at 60000 Hz, then we'll talk
> > > again.
> > > 
> > > For me this is a non-issue.
> > > 
> > >> Kernel version as indicator of timestamp
> > >> ----------------------------------------
> > >> 
> > >> Conversion of drivers to use monotonic timestamp is trivial, so the
> > >> conversion could be done once and for all drivers. The kernel version
> > >> could be used to indicate the type of the timestamp.
> > >> 
> > >> If this approach is taken care must be taken when new drivers are
> > >> integrated: developers sometimes use old kernels for development and
> > >> might also use an old driver for guidance on timestamps, thus using
> > >> real-time timestamps when monotonic timestamps should be used.
> > > 
> > > More importantly, this also fails when users use out-of-tree drivers.
> > 
> > Could you mention some examples what we could be breaking in particular?
> 
> The version reported in QUERYCAP by out-of-tree drivers is unreliable. It
> has no relationship with a kernel version in general. Actually, even inside
> the kernel I've come across drivers that still fill in the version number
> themselves.
>
> > >> This approach has an advantage over the capability flag below: which is
> > >> that we don't populate the interface with essentially dead definitions.
> > > 
> > > Using a kernel version to decide whether some feature is available or
> > > not is IMHO something of a last resort. It's very application
> > > unfriendly.
> > 
> > Could be, but that's a passing pain. We're going to live with the flags
> > for the foreseeable future, whether we need them or not.
> 
> Never underestimate the glacial speed (or lack of speed) with which
> applications adapt to new features.
> 
> > >> Capability flag for monotonic timestamps
> > >> ----------------------------------------
> > >> 
> > >> A capability flag can be used to tell whether the timestamp is
> > >> monotonic. However, it's not extensible cleanly to provide selectable
> > >> timestamps. These are not features that are needed right now, though.
> > >> 
> > >> The upside of this option is ease of implementation and use, but it's
> > >> not extensible. Also we're left with a flag that's set for all drivers:
> > >> in the end it provides no information to the user and is only noise in
> > >> the spec.
> > >> 
> > >> 
> > >> Control for timestamp type
> > >> --------------------------
> > >> 
> > >> Using a control to tell the type of the timestamp is extensible but not
> > >> as easy to implement than the capability flag: each and every device
> > >> would get an additional control. The value should likely be also file
> > >> handle specific, and we do not have file handle specific controls yet.
> > > 
> > > Yes, we do. You can make per-file handle controls. M2M devices need
> > > that.
> > 
> > Thanks for correcting me.
> > 
> > > I'm not sure why this would be filehandle specific, BTW.
> > 
> > Good point. I thought that as other properties of the buffers are
> > specific to file handles, including format when using CREATE_BUFS, it'd
> > make sense to make the timestamp source file-handle specific as well.
> > 
> > What do you think?
> 
> I don't think it makes sense to have different streams from the same device
> use different clocks.

Not on different streams, but on the same stream for two consecutive 
applications. If application A selects a device-specific timestamps, 
application B that is not aware of timestamp selection should get standard 
timestamps when it streams from the device.

> > >> In the meantime the control could be read-only, and later made
> > >> read-write when the timestamp type can be made selectable. Much of he
> > >> work of timestamping can be done by the framework: drivers can use a
> > >> single helper function and need to create one extra standard control.
> > >> 
> > >> Should the control also have an effect on the types of the timestamps
> > >> in V4L2 events? Likely yes.
> > > 
> > > You are missing one other option:
> > > 
> > > Using v4l2_buffer flags to report the clock
> > > -------------------------------------------
> > > 
> > > By defining flags like this:
> > > 
> > > V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> > > /* Possible Clocks */
> > > V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't
> > > know */ V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> > > 
> > > you could tell the application which clock is used.
> > > 
> > > This does allow for more clocks to be added in the future and clock
> > > selection would then be done by a control or possibly an ioctl. For now
> > > there are no plans to do such things, so this flag should be
> > > sufficient. And it can be implemented very efficiently. It works with
> > > existing drivers as well, since they will report CLOCK_UNKNOWN.
> > > 
> > > I am very much in favor of this approach.
> > 
> > Thanks for adding this. I knew I was forgetting something but didn't
> > remember what --- I swear it was unintentional! :-)
> > 
> > If we'd add more clocks without providing an ability to choose the clock
> > from the user space, how would the clock be selected? It certainly isn't
> > the driver's job, nor I think it should be system-specific either
> > (platform data on embedded systems).
> 
> IF a driver supports more than one clock (which I really don't see happening
> anytime soon),

uvcvideo will support both PTS and monotonic system time (and possibly raw 
monotonic system time).

> then we either need a control to select the clock or an ioctl. And something
> as well to enumerate the available clocks. I'm leaning towards ioctls, but I
> think this should be decided if we ever get an actual use-case for this.

Wouldn't a control be easier ? The control API already provides everything we 
need.

> > It's up to the application and its needs. That would suggest we should
> > always provide monotonic timestamps to applications (besides a potential
> > driver-specific timestamp), and for that purpose the capability flag ---
> > I admit I disliked the idea at first --- is enough.
> > 
> > What comes to buffer flags, the application would also have to receive
> > the first buffer from the device to even know what kind of timestamps
> > the device uses, or at least call QUERYBUF. And in principle the flag
> > should be checked on every buffer, unless we also specify the flag is
> > the same for all buffers. And at certain point this will stop to make
> > any sense...
> 
> It should definitely be the same for all buffers. And since apps will
> typically call querybuf anyway I don't see this as a problem. These
> clocks are also specific to the streaming I/O API, so reporting this as
> part of that API makes sense to me as well.
> 
> > A capability flag is cleaner solution from this perspective, and it can
> > be amended by a control (or an ioctl) later on: the flag can be
> > disregarded by applications whenever the control is present.
> 
> Yuck.
> 
> > If the application doesn't know about the control it can still rely on the
> > flag. (I think this would be less clean than to go for the control right
> > from the beginning, but better IMO.)
> > 
> > >> Device-dependent timestamp
> > >> --------------------------
> > >> 
> > >> Should we agree on selectable timestamps, the existing timestamp field
> > >> (or a union with another field of different type) could be used for
> > >> the device-dependent timestamps.
> > > 
> > > No. Device timestamps should get their own field. You want to be able to
> > > relate device timestamps with the monotonic timestamps, so you need
> > > both.
> > >
> > >> Alternatively we can choose to re-use the existing timecode field.
> > >> 
> > >> At the moment there's no known use case for passing device-dependent
> > >> timestamps at the same time with monotonic timestamps.
> > > 
> > > Well, the use case is there, but there is no driver support. The device
> > > timestamps should be 64 bits to accomodate things like PTS and DTS from
> > > MPEG streams. Since timecode is 128 bits we might want to use two u64
> > > fields or perhaps 4 u32 fields.
> > 
> > That should be an union for different kinds (or rather types) of
> > device-dependent timestamps. On uvcvideo I think this is u32, not u64.
> > We should be also able to tell what kind device dependent timestamp
> > there is --- should buffer flags be used for that as well?
> 
> That's definitely part of the buffer flags. The presence of timecode is
> already signalled using that. And not every buffer may have device
> timestamps (that depends on the hardware), so you have to signal it through
> the buffer flags.
> 
> An anonymous union might be best with the buffer flags signalling the type
> of the union. What I don't know is how to specify the type. Shall we just
> specify the type of the union (e.g. 4 u32 fields or 2 u64 fields) and leave
> the interpretation of those fields up to the application based on the
> driver name? Or shall the type act more like a fourcc in that it also
> uniquely identifies the interpretation of the timestamps?
> 
> Or should all device timestamps be converted to a timespec by the driver?

Conversion to a timespec requires floating point operation to be accurate in 
the general case. Implementing proper conversion algorithms requires userspace 
code.

> Answers on a postcard.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

On Friday 21 September 2012 11:33:24 Hans Verkuil wrote:
> On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> > Hi all,
> > 
> > 
> > This RFC intends to summarise and further the recent discussion on
> > linux-media regarding the proposed changes of timestamping V4L2 buffers.
> > 
> > 
> > The problem
> > ===========
> > 
> > The V4L2 has long used realtime timestamps (such as
> > clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> > handing them over to the user. This has been found problematic in
> > associating the video buffers with data from other sources: realtime clock
> > may jump around due to daylight saving time, for example, and ALSA
> > (audio-video synchronisation is a common use case) user space API does not
> > provide the user with realtime timestamps, but instead uses monotonic time
> > (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> > 
> > This is especially an issue in embedded systems where video recording is a
> > common use case. Drivers typically used in such systems have silently
> > switched to use monotonic timestamps. While against the spec, this is
> > necessary for those systems to operate properly.
> > 
> > In general, realtime timestamps are seen of little use in other than
> > debugging purposes, but monotonic timestamps are fine for that as well.
> > It's still possible that an application I'm not aware of uses them in a
> > peculiar way that would be adversely affected by changing to monotonic
> > timestamps. Nevertheless, we're not supposed to break the API (or ABI).
> > It'd be also very important for the application to know what kind of
> > timestamps are provided by the device.
> > 
> > 
> > Requirements, wishes and constraints
> > ====================================
> > 
> > Now that it seems to be about the time to fix these issues, it's worth
> > looking a little bit to the future to anticipate the coming changes to be
> > able to accommodate them better later on.
> > 
> > - The new default should be monotonic. As the monotonic timestamps are
> > seen
> > to be the most useful, they should be made the default.
> > 
> > - timeval vs. timespec. The two structs can be used to store timestamp
> > information. They are not compatible with each other. It's a little bit
> > uncertain what's the case with all the architectures but it looks like the
> > timespec fits into the space of timeval in all cases. If timespec is
> > considered to be used somewhere the compatibility must be ensured.
> > Timespec
> > is better than timeval since timespec has more precision and it's the same
> > struct that's used everywhere else in the V4L2 API: timespec does not need
> > conversion to timespec in the user space.
> > 
> > struct timespec {
> > 
> >         __kernel_time_t tv_sec;                 /* seconds */
> >         long            tv_nsec;                /* nanoseconds */
> > 
> > };
> > 
> > struct timeval {
> > 
> >         __kernel_time_t         tv_sec;         /* seconds */
> >         __kernel_suseconds_t    tv_usec;        /* microseconds */
> > 
> > };
> > 
> > To be able to use timespec, the user would have to most likely explicitly
> > choose to do that.
> > 
> > - Users should know what kind of timestamps the device produces. This
> > includes existing and future kernels. What should be considered are
> > uninformed porting drivers back and forth across kernel versions and
> > out-of-date kernel header files.
> > 
> > - Device-dependent timestamps. Some devices such as the uvcvideo ones
> > produce device-dependent timestamps for synchronising video and audio,
> > both
> > produced by the same physical hardware device. For uvcvideo these
> > timestamps are unsigned 32-bit integers.
> > 
> > - There's also another clock, Linux-specific raw monotonic clock (as in
> > clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> > cases than the regular monotonic clock. The difference is that the raw
> > monotonic clock is free from the NTP adjustments. It would be nice for the
> > user to be able to choose the clock used for timestamps. This is
> > especially
> > important for device-dependent timestamps: not all applications can be
> > expected to be able to use them.
> > 
> > - The field adjacent to timestamp, timecode, is 128 bits wide, and not
> > used
> > by a single driver. This field could be re-used.
> > 
> > 
> > Possible solutions
> > ==================
> > 
> > Not all of the solutions below that have been proposed are mutually
> > exclusive. That's also what's making the choice difficult: the ultimate
> > solution to the issue of timestamping may involve several of these --- or
> > possibly something better that's not on the list.
> > 
> > 
> > Use of timespec
> > ---------------
> > 
> > If we can conclude timespec will always fit into the size of timeval (or
> > timecode) we could use timespec instead. The solution should still make
> > the use of timespec explicit to the user space. This seems to conflict
> > with
> > the idea of making monotonic timestamps the default: the default can't be
> > anything incompatible with timeval, and at the same time it's the most
> > important that the monotonic timestamps are timespec.
> 
> We have to keep timeval. Changing this will break the ABI. I see absolutely
> no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and
> that's far, far removed from ns precisions. Should we ever have to support
> high-speed cameras running at 60000 Hz, then we'll talk again.
> 
> For me this is a non-issue.
> 
> > Kernel version as indicator of timestamp
> > ----------------------------------------
> > 
> > Conversion of drivers to use monotonic timestamp is trivial, so the
> > conversion could be done once and for all drivers. The kernel version
> > could
> > be used to indicate the type of the timestamp.
> > 
> > If this approach is taken care must be taken when new drivers are
> > integrated: developers sometimes use old kernels for development and might
> > also use an old driver for guidance on timestamps, thus using real-time
> > timestamps when monotonic timestamps should be used.
> 
> More importantly, this also fails when users use out-of-tree drivers.
> 
> > This approach has an
> > advantage over the capability flag below: which is that we don't populate
> > the interface with essentially dead definitions.
> 
> Using a kernel version to decide whether some feature is available or not is
> IMHO something of a last resort. It's very application unfriendly.
> > Capability flag for monotonic timestamps
> > ----------------------------------------
> > 
> > A capability flag can be used to tell whether the timestamp is monotonic.
> > However, it's not extensible cleanly to provide selectable timestamps.
> > These are not features that are needed right now, though.
> > 
> > The upside of this option is ease of implementation and use, but it's not
> > extensible. Also we're left with a flag that's set for all drivers: in the
> > end it provides no information to the user and is only noise in the spec.
> > 
> > 
> > Control for timestamp type
> > --------------------------
> > 
> > Using a control to tell the type of the timestamp is extensible but not as
> > easy to implement than the capability flag: each and every device would
> > get
> > an additional control. The value should likely be also file handle
> > specific, and we do not have file handle specific controls yet.
> 
> Yes, we do. You can make per-file handle controls. M2M devices need that.
> 
> I'm not sure why this would be filehandle specific, BTW.
> 
> > In the meantime the control could be read-only, and later made read-write
> > when the timestamp type can be made selectable. Much of he work of
> > timestamping can be done by the framework: drivers can use a single helper
> > function and need to create one extra standard control.
> > 
> > Should the control also have an effect on the types of the timestamps in
> > V4L2 events? Likely yes.
> 
> You are missing one other option:
> 
> Using v4l2_buffer flags to report the clock
> -------------------------------------------
> 
> By defining flags like this:
> 
> V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> /* Possible Clocks */
> V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know 
*/
> V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> 
> you could tell the application which clock is used.
> 
> This does allow for more clocks to be added in the future and clock
> selection would then be done by a control or possibly an ioctl. For now
> there are no plans to do such things, so this flag should be sufficient.
> And it can be implemented very efficiently. It works with existing drivers
> as well, since they will report CLOCK_UNKNOWN.
> 
> I am very much in favor of this approach.
> 
> > Device-dependent timestamp
> > --------------------------
> > 
> > Should we agree on selectable timestamps, the existing timestamp field (or
> > a union with another field of different type) could be used for the
> > device-dependent timestamps.
> 
> No. Device timestamps should get their own field. You want to be able to
> relate device timestamps with the monotonic timestamps, so you need both.
> > Alternatively we can choose to re-use the
> > existing timecode field.
> > 
> > At the moment there's no known use case for passing device-dependent
> > timestamps at the same time with monotonic timestamps.
> 
> Well, the use case is there, but there is no driver support. The device
> timestamps should be 64 bits to accomodate things like PTS and DTS from
> MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
> or perhaps 4 u32 fields.
> 
> > Now what?
> > =========
> > 
> > Almost as many options have been presented as there were opinions, but we
> > need to agree to have a single one. My personal leaning is on using a
> > control for the purpose as it is the most flexible alternative. I'd still
> > need to see an implementation of that but it doesn't seem that difficult,
> > especially when it's read-only. And even for read-write control the vast
> > majority of the work can be done by the V4L2 framework.
> > 
> > Questions, comments and opinions are very, very welcome.
> 
> Regards,
> 
> 	Hans
-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Hans,

On Friday 21 September 2012 11:33:24 Hans Verkuil wrote:
> On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> > Hi all,
> > 
> > 
> > This RFC intends to summarise and further the recent discussion on
> > linux-media regarding the proposed changes of timestamping V4L2 buffers.
> > 
> > 
> > The problem
> > ===========
> > 
> > The V4L2 has long used realtime timestamps (such as
> > clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> > handing them over to the user. This has been found problematic in
> > associating the video buffers with data from other sources: realtime clock
> > may jump around due to daylight saving time, for example, and ALSA
> > (audio-video synchronisation is a common use case) user space API does not
> > provide the user with realtime timestamps, but instead uses monotonic time
> > (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> > 
> > This is especially an issue in embedded systems where video recording is a
> > common use case. Drivers typically used in such systems have silently
> > switched to use monotonic timestamps. While against the spec, this is
> > necessary for those systems to operate properly.
> > 
> > In general, realtime timestamps are seen of little use in other than
> > debugging purposes, but monotonic timestamps are fine for that as well.
> > It's still possible that an application I'm not aware of uses them in a
> > peculiar way that would be adversely affected by changing to monotonic
> > timestamps. Nevertheless, we're not supposed to break the API (or ABI).
> > It'd be also very important for the application to know what kind of
> > timestamps are provided by the device.
> > 
> > 
> > Requirements, wishes and constraints
> > ====================================
> > 
> > Now that it seems to be about the time to fix these issues, it's worth
> > looking a little bit to the future to anticipate the coming changes to be
> > able to accommodate them better later on.
> > 
> > - The new default should be monotonic. As the monotonic timestamps are
> > seen
> > to be the most useful, they should be made the default.
> > 
> > - timeval vs. timespec. The two structs can be used to store timestamp
> > information. They are not compatible with each other. It's a little bit
> > uncertain what's the case with all the architectures but it looks like the
> > timespec fits into the space of timeval in all cases. If timespec is
> > considered to be used somewhere the compatibility must be ensured.
> > Timespec
> > is better than timeval since timespec has more precision and it's the same
> > struct that's used everywhere else in the V4L2 API: timespec does not need
> > conversion to timespec in the user space.
> > 
> > struct timespec {
> > 
> >         __kernel_time_t tv_sec;                 /* seconds */
> >         long            tv_nsec;                /* nanoseconds */
> > 
> > };
> > 
> > struct timeval {
> > 
> >         __kernel_time_t         tv_sec;         /* seconds */
> >         __kernel_suseconds_t    tv_usec;        /* microseconds */
> > 
> > };
> > 
> > To be able to use timespec, the user would have to most likely explicitly
> > choose to do that.
> > 
> > - Users should know what kind of timestamps the device produces. This
> > includes existing and future kernels. What should be considered are
> > uninformed porting drivers back and forth across kernel versions and
> > out-of-date kernel header files.
> > 
> > - Device-dependent timestamps. Some devices such as the uvcvideo ones
> > produce device-dependent timestamps for synchronising video and audio,
> > both
> > produced by the same physical hardware device. For uvcvideo these
> > timestamps are unsigned 32-bit integers.
> > 
> > - There's also another clock, Linux-specific raw monotonic clock (as in
> > clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> > cases than the regular monotonic clock. The difference is that the raw
> > monotonic clock is free from the NTP adjustments. It would be nice for the
> > user to be able to choose the clock used for timestamps. This is
> > especially
> > important for device-dependent timestamps: not all applications can be
> > expected to be able to use them.
> > 
> > - The field adjacent to timestamp, timecode, is 128 bits wide, and not
> > used
> > by a single driver. This field could be re-used.
> > 
> > 
> > Possible solutions
> > ==================
> > 
> > Not all of the solutions below that have been proposed are mutually
> > exclusive. That's also what's making the choice difficult: the ultimate
> > solution to the issue of timestamping may involve several of these --- or
> > possibly something better that's not on the list.
> > 
> > 
> > Use of timespec
> > ---------------
> > 
> > If we can conclude timespec will always fit into the size of timeval (or
> > timecode) we could use timespec instead. The solution should still make
> > the use of timespec explicit to the user space. This seems to conflict
> > with
> > the idea of making monotonic timestamps the default: the default can't be
> > anything incompatible with timeval, and at the same time it's the most
> > important that the monotonic timestamps are timespec.
> 
> We have to keep timeval. Changing this will break the ABI. I see absolutely
> no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and
> that's far, far removed from ns precisions. Should we ever have to support
> high-speed cameras running at 60000 Hz, then we'll talk again.
> 
> For me this is a non-issue.

It's not an issue per-se, but we should favor timespec over timeval in the 
solution we come up with if possible without any drawback.

> > Kernel version as indicator of timestamp
> > ----------------------------------------
> > 
> > Conversion of drivers to use monotonic timestamp is trivial, so the
> > conversion could be done once and for all drivers. The kernel version
> > could be used to indicate the type of the timestamp.
> > 
> > If this approach is taken care must be taken when new drivers are
> > integrated: developers sometimes use old kernels for development and might
> > also use an old driver for guidance on timestamps, thus using real-time
> > timestamps when monotonic timestamps should be used.
> 
> More importantly, this also fails when users use out-of-tree drivers.

We have enough trouble with in-tree drivers without considering out-of-tree 
drivers.

> > This approach has an advantage over the capability flag below: which is
> > that we don't populate the interface with essentially dead definitions.
> 
> Using a kernel version to decide whether some feature is available or not is
> IMHO something of a last resort. It's very application unfriendly.

It's pretty easy to use for applications. It has the advantage of not 
polluting the API with flags that will always be set, but isn't very explicit.

> > Capability flag for monotonic timestamps
> > ----------------------------------------
> > 
> > A capability flag can be used to tell whether the timestamp is monotonic.
> > However, it's not extensible cleanly to provide selectable timestamps.
> > These are not features that are needed right now, though.
> > 
> > The upside of this option is ease of implementation and use, but it's not
> > extensible. Also we're left with a flag that's set for all drivers: in the
> > end it provides no information to the user and is only noise in the spec.
> > 
> > 
> > Control for timestamp type
> > --------------------------
> > 
> > Using a control to tell the type of the timestamp is extensible but not as
> > easy to implement than the capability flag: each and every device would
> > get an additional control. The value should likely be also file handle
> > specific, and we do not have file handle specific controls yet.
> 
> Yes, we do. You can make per-file handle controls. M2M devices need that.
> 
> I'm not sure why this would be filehandle specific, BTW.

Because an application that selects a non-default clock must not break a later 
application that wouldn't be clock-aware.

> > In the meantime the control could be read-only, and later made read-write
> > when the timestamp type can be made selectable. Much of he work of
> > timestamping can be done by the framework: drivers can use a single helper
> > function and need to create one extra standard control.
> > 
> > Should the control also have an effect on the types of the timestamps in
> > V4L2 events? Likely yes.
> 
> You are missing one other option:
> 
> Using v4l2_buffer flags to report the clock
> -------------------------------------------
> 
> By defining flags like this:
> 
> V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> /* Possible Clocks */
> V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know 
*/
> V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> 
> you could tell the application which clock is used.
> 
> This does allow for more clocks to be added in the future and clock
> selection would then be done by a control or possibly an ioctl.

Clock selection could also be done by setting the buffer flag at QBUF time.

> For now there are no plans to do such things, so this flag should be
> sufficient. And it can be implemented very efficiently. It works with
> existing drivers as well, since they will report CLOCK_UNKNOWN.
> 
> I am very much in favor of this approach.
> 
> > Device-dependent timestamp
> > --------------------------
> > 
> > Should we agree on selectable timestamps, the existing timestamp field (or
> > a union with another field of different type) could be used for the
> > device-dependent timestamps.
> 
> No. Device timestamps should get their own field. You want to be able to
> relate device timestamps with the monotonic timestamps, so you need both.
>
> > Alternatively we can choose to re-use the existing timecode field.
> > 
> > At the moment there's no known use case for passing device-dependent
> > timestamps at the same time with monotonic timestamps.
> 
> Well, the use case is there, but there is no driver support. The device
> timestamps should be 64 bits to accomodate things like PTS and DTS from
> MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
> or perhaps 4 u32 fields.

Why would an application need both the PTS and the system timestamp ?

> > Now what?
> > =========
> > 
> > Almost as many options have been presented as there were opinions, but we
> > need to agree to have a single one. My personal leaning is on using a
> > control for the purpose as it is the most flexible alternative. I'd still
> > need to see an implementation of that but it doesn't seem that difficult,
> > especially when it's read-only. And even for read-write control the vast
> > majority of the work can be done by the V4L2 framework.
> > 
> > Questions, comments and opinions are very, very welcome.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
> Hi Hans,
> 
> On Friday 21 September 2012 11:33:24 Hans Verkuil wrote:
> > On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> > > Hi all,
> > > 
> > > 
> > > This RFC intends to summarise and further the recent discussion on
> > > linux-media regarding the proposed changes of timestamping V4L2 buffers.
> > > 
> > > 
> > > The problem
> > > ===========
> > > 
> > > The V4L2 has long used realtime timestamps (such as
> > > clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> > > handing them over to the user. This has been found problematic in
> > > associating the video buffers with data from other sources: realtime clock
> > > may jump around due to daylight saving time, for example, and ALSA
> > > (audio-video synchronisation is a common use case) user space API does not
> > > provide the user with realtime timestamps, but instead uses monotonic time
> > > (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> > > 
> > > This is especially an issue in embedded systems where video recording is a
> > > common use case. Drivers typically used in such systems have silently
> > > switched to use monotonic timestamps. While against the spec, this is
> > > necessary for those systems to operate properly.
> > > 
> > > In general, realtime timestamps are seen of little use in other than
> > > debugging purposes, but monotonic timestamps are fine for that as well.
> > > It's still possible that an application I'm not aware of uses them in a
> > > peculiar way that would be adversely affected by changing to monotonic
> > > timestamps. Nevertheless, we're not supposed to break the API (or ABI).
> > > It'd be also very important for the application to know what kind of
> > > timestamps are provided by the device.
> > > 
> > > 
> > > Requirements, wishes and constraints
> > > ====================================
> > > 
> > > Now that it seems to be about the time to fix these issues, it's worth
> > > looking a little bit to the future to anticipate the coming changes to be
> > > able to accommodate them better later on.
> > > 
> > > - The new default should be monotonic. As the monotonic timestamps are
> > > seen
> > > to be the most useful, they should be made the default.
> > > 
> > > - timeval vs. timespec. The two structs can be used to store timestamp
> > > information. They are not compatible with each other. It's a little bit
> > > uncertain what's the case with all the architectures but it looks like the
> > > timespec fits into the space of timeval in all cases. If timespec is
> > > considered to be used somewhere the compatibility must be ensured.
> > > Timespec
> > > is better than timeval since timespec has more precision and it's the same
> > > struct that's used everywhere else in the V4L2 API: timespec does not need
> > > conversion to timespec in the user space.
> > > 
> > > struct timespec {
> > > 
> > >         __kernel_time_t tv_sec;                 /* seconds */
> > >         long            tv_nsec;                /* nanoseconds */
> > > 
> > > };
> > > 
> > > struct timeval {
> > > 
> > >         __kernel_time_t         tv_sec;         /* seconds */
> > >         __kernel_suseconds_t    tv_usec;        /* microseconds */
> > > 
> > > };
> > > 
> > > To be able to use timespec, the user would have to most likely explicitly
> > > choose to do that.
> > > 
> > > - Users should know what kind of timestamps the device produces. This
> > > includes existing and future kernels. What should be considered are
> > > uninformed porting drivers back and forth across kernel versions and
> > > out-of-date kernel header files.
> > > 
> > > - Device-dependent timestamps. Some devices such as the uvcvideo ones
> > > produce device-dependent timestamps for synchronising video and audio,
> > > both
> > > produced by the same physical hardware device. For uvcvideo these
> > > timestamps are unsigned 32-bit integers.
> > > 
> > > - There's also another clock, Linux-specific raw monotonic clock (as in
> > > clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> > > cases than the regular monotonic clock. The difference is that the raw
> > > monotonic clock is free from the NTP adjustments. It would be nice for the
> > > user to be able to choose the clock used for timestamps. This is
> > > especially
> > > important for device-dependent timestamps: not all applications can be
> > > expected to be able to use them.
> > > 
> > > - The field adjacent to timestamp, timecode, is 128 bits wide, and not
> > > used
> > > by a single driver. This field could be re-used.
> > > 
> > > 
> > > Possible solutions
> > > ==================
> > > 
> > > Not all of the solutions below that have been proposed are mutually
> > > exclusive. That's also what's making the choice difficult: the ultimate
> > > solution to the issue of timestamping may involve several of these --- or
> > > possibly something better that's not on the list.
> > > 
> > > 
> > > Use of timespec
> > > ---------------
> > > 
> > > If we can conclude timespec will always fit into the size of timeval (or
> > > timecode) we could use timespec instead. The solution should still make
> > > the use of timespec explicit to the user space. This seems to conflict
> > > with
> > > the idea of making monotonic timestamps the default: the default can't be
> > > anything incompatible with timeval, and at the same time it's the most
> > > important that the monotonic timestamps are timespec.
> > 
> > We have to keep timeval. Changing this will break the ABI. I see absolutely
> > no reason to use timespec for video. At 60 Hz a frame takes 16.67 ms, and
> > that's far, far removed from ns precisions. Should we ever have to support
> > high-speed cameras running at 60000 Hz, then we'll talk again.
> > 
> > For me this is a non-issue.
> 
> It's not an issue per-se, but we should favor timespec over timeval in the 
> solution we come up with if possible without any drawback.
> 
> > > Kernel version as indicator of timestamp
> > > ----------------------------------------
> > > 
> > > Conversion of drivers to use monotonic timestamp is trivial, so the
> > > conversion could be done once and for all drivers. The kernel version
> > > could be used to indicate the type of the timestamp.
> > > 
> > > If this approach is taken care must be taken when new drivers are
> > > integrated: developers sometimes use old kernels for development and might
> > > also use an old driver for guidance on timestamps, thus using real-time
> > > timestamps when monotonic timestamps should be used.
> > 
> > More importantly, this also fails when users use out-of-tree drivers.
> 
> We have enough trouble with in-tree drivers without considering out-of-tree 
> drivers.

Actually, the same problem still occurs with some in-tree drivers as well. And
some distros do distribute out-of-tree drivers (the V4L2 loopback driver is
actually part of at least one distro). It's just not 100% reliable.

> > > This approach has an advantage over the capability flag below: which is
> > > that we don't populate the interface with essentially dead definitions.
> > 
> > Using a kernel version to decide whether some feature is available or not is
> > IMHO something of a last resort. It's very application unfriendly.
> 
> It's pretty easy to use for applications. It has the advantage of not 
> polluting the API with flags that will always be set, but isn't very explicit.
> 
> > > Capability flag for monotonic timestamps
> > > ----------------------------------------
> > > 
> > > A capability flag can be used to tell whether the timestamp is monotonic.
> > > However, it's not extensible cleanly to provide selectable timestamps.
> > > These are not features that are needed right now, though.
> > > 
> > > The upside of this option is ease of implementation and use, but it's not
> > > extensible. Also we're left with a flag that's set for all drivers: in the
> > > end it provides no information to the user and is only noise in the spec.
> > > 
> > > 
> > > Control for timestamp type
> > > --------------------------
> > > 
> > > Using a control to tell the type of the timestamp is extensible but not as
> > > easy to implement than the capability flag: each and every device would
> > > get an additional control. The value should likely be also file handle
> > > specific, and we do not have file handle specific controls yet.
> > 
> > Yes, we do. You can make per-file handle controls. M2M devices need that.
> > 
> > I'm not sure why this would be filehandle specific, BTW.
> 
> Because an application that selects a non-default clock must not break a later 
> application that wouldn't be clock-aware.
> 
> > > In the meantime the control could be read-only, and later made read-write
> > > when the timestamp type can be made selectable. Much of he work of
> > > timestamping can be done by the framework: drivers can use a single helper
> > > function and need to create one extra standard control.
> > > 
> > > Should the control also have an effect on the types of the timestamps in
> > > V4L2 events? Likely yes.
> > 
> > You are missing one other option:
> > 
> > Using v4l2_buffer flags to report the clock
> > -------------------------------------------
> > 
> > By defining flags like this:
> > 
> > V4L2_BUF_FLAG_CLOCK_MASK	0x7000
> > /* Possible Clocks */
> > V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know 
> */
> > V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> > 
> > you could tell the application which clock is used.
> > 
> > This does allow for more clocks to be added in the future and clock
> > selection would then be done by a control or possibly an ioctl.
> 
> Clock selection could also be done by setting the buffer flag at QBUF time.

True. Not a bad idea, actually. You would have to specify that setting the
clock to 0 (UNKNOWN) or any other unsupported clock, then that will be mapped
to MONOTONIC for newer kernels, but that's no problem.

It has the advantage of not requiring any controls, ioctls, etc. The only
disadvantage is that you can't check if a particular clock is actually
supported. Although I guess you could do a QBUF followed by QUERYBUF to check
the clock bits. But you can't change to a different clock for that buffer
afterwards (at least, not until it is dequeued).

> > For now there are no plans to do such things, so this flag should be
> > sufficient. And it can be implemented very efficiently. It works with
> > existing drivers as well, since they will report CLOCK_UNKNOWN.
> > 
> > I am very much in favor of this approach.
> > 
> > > Device-dependent timestamp
> > > --------------------------
> > > 
> > > Should we agree on selectable timestamps, the existing timestamp field (or
> > > a union with another field of different type) could be used for the
> > > device-dependent timestamps.
> > 
> > No. Device timestamps should get their own field. You want to be able to
> > relate device timestamps with the monotonic timestamps, so you need both.
> >
> > > Alternatively we can choose to re-use the existing timecode field.
> > > 
> > > At the moment there's no known use case for passing device-dependent
> > > timestamps at the same time with monotonic timestamps.
> > 
> > Well, the use case is there, but there is no driver support. The device
> > timestamps should be 64 bits to accomodate things like PTS and DTS from
> > MPEG streams. Since timecode is 128 bits we might want to use two u64 fields
> > or perhaps 4 u32 fields.
> 
> Why would an application need both the PTS and the system timestamp ?

The PTS may not always be available. Particularly for program streams there
doesn't have to be a 1-1 mapping between PTS and the buffer. Some buffers
may contain none or two or more PTSes. Although in such cases it may be
debatable whether the API should provide the PTS anyway, this is more for the
application to deal with.

BTW, I think we should also fix the description of the timestamp in the spec.
Currently it says:

"For input streams this is the system time (as returned by the gettimeofday()
function) when the first data byte was captured. For output streams the data
will not be displayed before this time, secondary to the nominal frame rate
determined by the current video standard in enqueued order. Applications can
for example zero this field to display frames as soon as possible. The driver
stores the time at which the first data byte was actually sent out in the
timestamp field. This permits applications to monitor the drift between the
video and system clock."

To my knowledge all capture drivers set the timestamp to the time the *last*
data byte was captured, not the first.

And there are no output drivers able to handle a non-zero timestamp. And the
output drivers also set the timestamp to the time the *last* data byte was
sent out.

I think the spec should be updated to reflect this.

Regards,

	Hans

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Hans,

On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
> On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
> > On Friday 21 September 2012 11:33:24 Hans Verkuil wrote:
> > > On Thu September 20 2012 22:21:22 Sakari Ailus wrote:
> > > > Hi all,
> > > > 
> > > > This RFC intends to summarise and further the recent discussion on
> > > > linux-media regarding the proposed changes of timestamping V4L2
> > > > buffers.
> > > > 
> > > > The problem
> > > > ===========
> > > > 
> > > > The V4L2 has long used realtime timestamps (such as
> > > > clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> > > > handing them over to the user. This has been found problematic in
> > > > associating the video buffers with data from other sources: realtime
> > > > clock may jump around due to daylight saving time, for example, and
> > > > ALSA (audio-video synchronisation is a common use case) user space API
> > > > does not provide the user with realtime timestamps, but instead uses
> > > > monotonic time (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
> > > > 
> > > > This is especially an issue in embedded systems where video recording
> > > > is a common use case. Drivers typically used in such systems have
> > > > silently switched to use monotonic timestamps. While against the spec,
> > > > this is necessary for those systems to operate properly.
> > > > 
> > > > In general, realtime timestamps are seen of little use in other than
> > > > debugging purposes, but monotonic timestamps are fine for that as
> > > > well. It's still possible that an application I'm not aware of uses
> > > > them in a peculiar way that would be adversely affected by changing to
> > > > monotonic timestamps. Nevertheless, we're not supposed to break the
> > > > API (or ABI). It'd be also very important for the application to know
> > > > what kind of timestamps are provided by the device.
> > > > 
> > > > 
> > > > Requirements, wishes and constraints
> > > > ====================================
> > > > 
> > > > Now that it seems to be about the time to fix these issues, it's worth
> > > > looking a little bit to the future to anticipate the coming changes to
> > > > be able to accommodate them better later on.
> > > > 
> > > > - The new default should be monotonic. As the monotonic timestamps are
> > > > seen to be the most useful, they should be made the default.
> > > > 
> > > > - timeval vs. timespec. The two structs can be used to store timestamp
> > > > information. They are not compatible with each other. It's a little
> > > > bit uncertain what's the case with all the architectures but it looks
> > > > like the timespec fits into the space of timeval in all cases. If
> > > > timespec is considered to be used somewhere the compatibility must be
> > > > ensured. Timespec is better than timeval since timespec has more
> > > > precision and it's the same struct that's used everywhere else in the
> > > > V4L2 API: timespec does not need conversion to timespec in the user
> > > > space.
> > > > 
> > > > struct timespec {
> > > > 
> > > >         __kernel_time_t tv_sec;                 /* seconds */
> > > >         long            tv_nsec;                /* nanoseconds */
> > > > 
> > > > };
> > > > 
> > > > struct timeval {
> > > > 
> > > >         __kernel_time_t         tv_sec;         /* seconds */
> > > >         __kernel_suseconds_t    tv_usec;        /* microseconds */
> > > > 
> > > > };
> > > > 
> > > > To be able to use timespec, the user would have to most likely
> > > > explicitly choose to do that.
> > > > 
> > > > - Users should know what kind of timestamps the device produces. This
> > > > includes existing and future kernels. What should be considered are
> > > > uninformed porting drivers back and forth across kernel versions and
> > > > out-of-date kernel header files.
> > > > 
> > > > - Device-dependent timestamps. Some devices such as the uvcvideo ones
> > > > produce device-dependent timestamps for synchronising video and audio,
> > > > both produced by the same physical hardware device. For uvcvideo these
> > > > timestamps are unsigned 32-bit integers.
> > > > 
> > > > - There's also another clock, Linux-specific raw monotonic clock (as
> > > > in clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in
> > > > some use cases than the regular monotonic clock. The difference is
> > > > that the raw monotonic clock is free from the NTP adjustments. It
> > > > would be nice for the user to be able to choose the clock used for
> > > > timestamps. This is especially important for device-dependent
> > > > timestamps: not all applications can be expected to be able to use
> > > > them.
> > > > 
> > > > - The field adjacent to timestamp, timecode, is 128 bits wide, and not
> > > > used by a single driver. This field could be re-used.
> > > > 
> > > > 
> > > > Possible solutions
> > > > ==================
> > > > 
> > > > Not all of the solutions below that have been proposed are mutually
> > > > exclusive. That's also what's making the choice difficult: the
> > > > ultimate solution to the issue of timestamping may involve several of
> > > > these --- or possibly something better that's not on the list.
> > > > 
> > > > 
> > > > Use of timespec
> > > > ---------------
> > > > 
> > > > If we can conclude timespec will always fit into the size of timeval
> > > > (or timecode) we could use timespec instead. The solution should still
> > > > make the use of timespec explicit to the user space. This seems to
> > > > conflict with the idea of making monotonic timestamps the default: the
> > > > default can't be anything incompatible with timeval, and at the same
> > > > time it's the most important that the monotonic timestamps are
> > > > timespec.
> > > 
> > > We have to keep timeval. Changing this will break the ABI. I see
> > > absolutely no reason to use timespec for video. At 60 Hz a frame takes
> > > 16.67 ms, and that's far, far removed from ns precisions. Should we ever
> > > have to support high-speed cameras running at 60000 Hz, then we'll talk
> > > again.
> > > 
> > > For me this is a non-issue.
> > 
> > It's not an issue per-se, but we should favor timespec over timeval in the
> > solution we come up with if possible without any drawback.
> > 
> > > > Kernel version as indicator of timestamp
> > > > ----------------------------------------
> > > > 
> > > > Conversion of drivers to use monotonic timestamp is trivial, so the
> > > > conversion could be done once and for all drivers. The kernel version
> > > > could be used to indicate the type of the timestamp.
> > > > 
> > > > If this approach is taken care must be taken when new drivers are
> > > > integrated: developers sometimes use old kernels for development and
> > > > might also use an old driver for guidance on timestamps, thus using
> > > > real-time timestamps when monotonic timestamps should be used.
> > > 
> > > More importantly, this also fails when users use out-of-tree drivers.
> > 
> > We have enough trouble with in-tree drivers without considering
> > out-of-tree drivers.
> 
> Actually, the same problem still occurs with some in-tree drivers as well.
> And some distros do distribute out-of-tree drivers (the V4L2 loopback
> driver is actually part of at least one distro). It's just not 100%
> reliable.

Distros should be responsible for handling that properly ;-) I agree that we 
don't live in a perfect world though.

> > > > This approach has an advantage over the capability flag below: which
> > > > is that we don't populate the interface with essentially dead
> > > > definitions.
> > > 
> > > Using a kernel version to decide whether some feature is available or
> > > not is IMHO something of a last resort. It's very application
> > > unfriendly.
> >
> > It's pretty easy to use for applications. It has the advantage of not
> > polluting the API with flags that will always be set, but isn't very
> > explicit.
> >
> > > > Capability flag for monotonic timestamps
> > > > ----------------------------------------
> > > > 
> > > > A capability flag can be used to tell whether the timestamp is
> > > > monotonic. However, it's not extensible cleanly to provide selectable
> > > > timestamps. These are not features that are needed right now, though.
> > > > 
> > > > The upside of this option is ease of implementation and use, but it's
> > > > not extensible. Also we're left with a flag that's set for all
> > > > drivers: in the end it provides no information to the user and is only
> > > > noise in the spec.
> > > > 
> > > > 
> > > > Control for timestamp type
> > > > --------------------------
> > > > 
> > > > Using a control to tell the type of the timestamp is extensible but
> > > > not as easy to implement than the capability flag: each and every
> > > > device would get an additional control. The value should likely be
> > > > also file handle specific, and we do not have file handle specific
> > > > controls yet.
> > > 
> > > Yes, we do. You can make per-file handle controls. M2M devices need
> > > that.
> > > 
> > > I'm not sure why this would be filehandle specific, BTW.
> > 
> > Because an application that selects a non-default clock must not break a
> > later application that wouldn't be clock-aware.
> >
> > > > In the meantime the control could be read-only, and later made
> > > > read-write when the timestamp type can be made selectable. Much of he
> > > > work of timestamping can be done by the framework: drivers can use a
> > > > single helper function and need to create one extra standard control.
> > > > 
> > > > Should the control also have an effect on the types of the timestamps
> > > > in V4L2 events? Likely yes.
> > > 
> > > You are missing one other option:
> > > 
> > > Using v4l2_buffer flags to report the clock
> > > -------------------------------------------
> > > 
> > > By defining flags like this:
> > > 
> > > V4L2_BUF_FLAG_CLOCK_MASK	0x7000		/* Possible Clocks */
> > > V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000	/* system or monotonic, we don't
> > > know */
> > > V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
> > > 
> > > you could tell the application which clock is used.
> > > 
> > > This does allow for more clocks to be added in the future and clock
> > > selection would then be done by a control or possibly an ioctl.
> > 
> > Clock selection could also be done by setting the buffer flag at QBUF
> > time.
> 
> True. Not a bad idea, actually. You would have to specify that setting the
> clock to 0 (UNKNOWN) or any other unsupported clock, then that will be
> mapped to MONOTONIC for newer kernels, but that's no problem.
> 
> It has the advantage of not requiring any controls, ioctls, etc. The only
> disadvantage is that you can't check if a particular clock is actually
> supported. Although I guess you could do a QBUF followed by QUERYBUF to
> check the clock bits. But you can't change to a different clock for that
> buffer afterwards (at least, not until it is dequeued).

My main concern with this approach is indeed that applications have no way to 
know beforehand what clocks are supported. That's why I would prefer a 
control, especially if we want to mandate that the selected clock must be the 
same for all buffers and must not change during streaming.

Does anyone have a use case for changing the clock during streaming, or can we 
consider that the clock source will not change during streaming and will be 
the same for all buffers ?

> > > For now there are no plans to do such things, so this flag should be
> > > sufficient. And it can be implemented very efficiently. It works with
> > > existing drivers as well, since they will report CLOCK_UNKNOWN.
> > > 
> > > I am very much in favor of this approach.
> > > 
> > > > Device-dependent timestamp
> > > > --------------------------
> > > > 
> > > > Should we agree on selectable timestamps, the existing timestamp field
> > > > (or a union with another field of different type) could be used for
> > > > the device-dependent timestamps.
> > > 
> > > No. Device timestamps should get their own field. You want to be able to
> > > relate device timestamps with the monotonic timestamps, so you need
> > > both.
> > > 
> > > > Alternatively we can choose to re-use the existing timecode field.
> > > > 
> > > > At the moment there's no known use case for passing device-dependent
> > > > timestamps at the same time with monotonic timestamps.
> > > 
> > > Well, the use case is there, but there is no driver support. The device
> > > timestamps should be 64 bits to accomodate things like PTS and DTS from
> > > MPEG streams. Since timecode is 128 bits we might want to use two u64
> > > fields or perhaps 4 u32 fields.
> > 
> > Why would an application need both the PTS and the system timestamp ?
> 
> The PTS may not always be available. Particularly for program streams there
> doesn't have to be a 1-1 mapping between PTS and the buffer. Some buffers
> may contain none or two or more PTSes. Although in such cases it may be
> debatable whether the API should provide the PTS anyway, this is more for
> the application to deal with.

If the PTS is imbedded in the stream I'm not sure it makes sense to provide it 
in the buffers, especially if the way to provide it would be to parse the 
stream in the driver.

> BTW, I think we should also fix the description of the timestamp in the
> spec. Currently it says:
> 
> "For input streams this is the system time (as returned by the
> gettimeofday() function) when the first data byte was captured. For output
> streams the data will not be displayed before this time, secondary to the
> nominal frame rate determined by the current video standard in enqueued
> order. Applications can for example zero this field to display frames as
> soon as possible. The driver stores the time at which the first data byte
> was actually sent out in the timestamp field. This permits applications to
> monitor the drift between the video and system clock."
> 
> To my knowledge all capture drivers set the timestamp to the time the *last*
> data byte was captured, not the first.

The uvcvideo driver uses the time the first image packet is received :-) Most 
other drivers use the time the last byte was *received*, not captured. That's 
a very important difference, as it influences audio/video synchronization. 
Providing the time at which the first byte was captured is better than the 
time the last byte was captured in my opinion. Unfortunately when images are 
transferred by DMA it's often impossible to get any meaningful timestamp.

> And there are no output drivers able to handle a non-zero timestamp. And the
> output drivers also set the timestamp to the time the *last* data byte was
> sent out.
> 
> I think the spec should be updated to reflect this.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Tue 25 September 2012 12:48:01 Laurent Pinchart wrote:
> Hi Hans,
> 
> On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
> > On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
> > BTW, I think we should also fix the description of the timestamp in the
> > spec. Currently it says:
> > 
> > "For input streams this is the system time (as returned by the
> > gettimeofday() function) when the first data byte was captured. For output
> > streams the data will not be displayed before this time, secondary to the
> > nominal frame rate determined by the current video standard in enqueued
> > order. Applications can for example zero this field to display frames as
> > soon as possible. The driver stores the time at which the first data byte
> > was actually sent out in the timestamp field. This permits applications to
> > monitor the drift between the video and system clock."
> > 
> > To my knowledge all capture drivers set the timestamp to the time the *last*
> > data byte was captured, not the first.
> 
> The uvcvideo driver uses the time the first image packet is received :-) Most 
> other drivers use the time the last byte was *received*, not captured.

Unless the hardware buffers more than a few lines there is very little
difference between the time the last byte was received and when it was captured.

But you are correct, it is typically the time the last byte was received.

Should we signal this as well? First vs last byte? Or shall we standardize?

BTW, the human mind is amazingly tolerant when it comes to A/V synchronization.
Audio can be up to 50 ms ahead of the video and up to I believe 120 ms lagging
behind the video before most people will notice. So being off by one frame won't
be noticable at all.

Regards,

	Hans

> That's 
> a very important difference, as it influences audio/video synchronization. 
> Providing the time at which the first byte was captured is better than the 
> time the last byte was captured in my opinion. Unfortunately when images are 
> transferred by DMA it's often impossible to get any meaningful timestamp.
> 
> > And there are no output drivers able to handle a non-zero timestamp. And the
> > output drivers also set the timestamp to the time the *last* data byte was
> > sent out.
> > 
> > I think the spec should be updated to reflect this.
> 
> 

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Hans,

On Tuesday 25 September 2012 12:54:34 Hans Verkuil wrote:
> On Tue 25 September 2012 12:48:01 Laurent Pinchart wrote:
> > On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
> > > On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
> > > BTW, I think we should also fix the description of the timestamp in the
> > > spec. Currently it says:
> > > 
> > > "For input streams this is the system time (as returned by the
> > > gettimeofday() function) when the first data byte was captured. For
> > > output streams the data will not be displayed before this time,
> > > secondary to the nominal frame rate determined by the current video
> > > standard in enqueued order. Applications can for example zero this field
> > > to display frames as soon as possible. The driver stores the time at
> > > which the first data byte was actually sent out in the timestamp field.
> > > This permits applications to monitor the drift between the video and
> > > system clock."
> > > 
> > > To my knowledge all capture drivers set the timestamp to the time the
> > > *last* data byte was captured, not the first.
> > 
> > The uvcvideo driver uses the time the first image packet is received :-)
> > Most other drivers use the time the last byte was *received*, not
> > captured.
>
> Unless the hardware buffers more than a few lines there is very little
> difference between the time the last byte was received and when it was
> captured.

It won't differ much, but if we want to change the spec to reflect the 
reality, then we should be as precise as possible.
 
> But you are correct, it is typically the time the last byte was received.
> 
> Should we signal this as well? First vs last byte? Or shall we standardize?

Good question. On one hand forcing drivers to report the timestamp of the last 
captured byte when they can report the first is a step back, on the other hand 
I'm not sure if it would be worth it to report what the device does exactly. 
This could all fit in a couple of new clock-related ioctls though. I wasn't a 
big fan of ioctls instead of a control for clock source selection, but if we 
start to shove more information in there ioctls begin to make sense.

> BTW, the human mind is amazingly tolerant when it comes to A/V
> synchronization. Audio can be up to 50 ms ahead of the video and up to I
> believe 120 ms lagging behind the video before most people will notice. So
> being off by one frame won't be noticable at all.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Hans,

Hans Verkuil wrote:
> On Tue 25 September 2012 12:48:01 Laurent Pinchart wrote:
>> Hi Hans,
>>
>> On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
>>> On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
>>> BTW, I think we should also fix the description of the timestamp in the
>>> spec. Currently it says:
>>>
>>> "For input streams this is the system time (as returned by the
>>> gettimeofday() function) when the first data byte was captured. For output
>>> streams the data will not be displayed before this time, secondary to the
>>> nominal frame rate determined by the current video standard in enqueued
>>> order. Applications can for example zero this field to display frames as
>>> soon as possible. The driver stores the time at which the first data byte
>>> was actually sent out in the timestamp field. This permits applications to
>>> monitor the drift between the video and system clock."
>>>
>>> To my knowledge all capture drivers set the timestamp to the time the *last*
>>> data byte was captured, not the first.
>>
>> The uvcvideo driver uses the time the first image packet is received :-) Most 
>> other drivers use the time the last byte was *received*, not captured.
> 
> Unless the hardware buffers more than a few lines there is very little
> difference between the time the last byte was received and when it was captured.
> 
> But you are correct, it is typically the time the last byte was received.
> 
> Should we signal this as well? First vs last byte? Or shall we standardize?

My personal opinion would be to change the spec to say what almost every
driver does: it's the timestamp from the moment the last pixel has been
received. We have the frame sync event for telling when the frame starts
btw. The same event could be used for signalling whenever a given line
starts. I don't see frame end fitting to that quite as nicely but I
guess it could be possible.

> BTW, the human mind is amazingly tolerant when it comes to A/V synchronization.
> Audio can be up to 50 ms ahead of the video and up to I believe 120 ms lagging
> behind the video before most people will notice. So being off by one frame won't
> be noticable at all.

I wonder if this is what most DVD players do. What they do is not
pretty. The difference could be more, though. ;-)

Cheers,

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Sakari,

On Tuesday 25 September 2012 23:12:00 Sakari Ailus wrote:
> Hans Verkuil wrote:
> > On Tue 25 September 2012 12:48:01 Laurent Pinchart wrote:
> >> On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
> >>> On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
> >>> BTW, I think we should also fix the description of the timestamp in the
> >>> spec. Currently it says:
> >>> 
> >>> "For input streams this is the system time (as returned by the
> >>> gettimeofday() function) when the first data byte was captured. For
> >>> output streams the data will not be displayed before this time,
> >>> secondary to the nominal frame rate determined by the current video
> >>> standard in enqueued order. Applications can for example zero this field
> >>> to display frames as soon as possible. The driver stores the time at
> >>> which the first data byte was actually sent out in the timestamp field.
> >>> This permits applications to monitor the drift between the video and
> >>> system clock."
> >>> 
> >>> To my knowledge all capture drivers set the timestamp to the time the
> >>> *last* data byte was captured, not the first.
> >> 
> >> The uvcvideo driver uses the time the first image packet is received :-)
> >> Most other drivers use the time the last byte was *received*, not
> >> captured.
> >
> > Unless the hardware buffers more than a few lines there is very little
> > difference between the time the last byte was received and when it was
> > captured.
> > 
> > But you are correct, it is typically the time the last byte was received.
> > 
> > Should we signal this as well? First vs last byte? Or shall we
> > standardize?
> 
> My personal opinion would be to change the spec to say what almost every
> driver does: it's the timestamp from the moment the last pixel has been
> received. We have the frame sync event for telling when the frame starts
> btw. The same event could be used for signalling whenever a given line
> starts. I don't see frame end fitting to that quite as nicely but I
> guess it could be possible.

The uvcvideo driver can timestamp the buffers with the system time at which 
the first packet in the frame is received, but has no way to generate a frame 
start event: the frame start event should correspond to the time the frame 
starts, not to the time the first packet in the frame is received. That 
information isn't available to the driver.

> > BTW, the human mind is amazingly tolerant when it comes to A/V
> > synchronization. Audio can be up to 50 ms ahead of the video and up to I
> > believe 120 ms lagging behind the video before most people will notice.
> > So being off by one frame won't be noticable at all.
> 
> I wonder if this is what most DVD players do. What they do is not
> pretty. The difference could be more, though. ;-)

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Laurent,
Laurent Pinchart wrote:
> On Tuesday 25 September 2012 23:12:00 Sakari Ailus wrote:
>> Hans Verkuil wrote:
>>> On Tue 25 September 2012 12:48:01 Laurent Pinchart wrote:
>>>> On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
>>>>> On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
>>>>> BTW, I think we should also fix the description of the timestamp in the
>>>>> spec. Currently it says:
>>>>>
>>>>> "For input streams this is the system time (as returned by the
>>>>> gettimeofday() function) when the first data byte was captured. For
>>>>> output streams the data will not be displayed before this time,
>>>>> secondary to the nominal frame rate determined by the current video
>>>>> standard in enqueued order. Applications can for example zero this field
>>>>> to display frames as soon as possible. The driver stores the time at
>>>>> which the first data byte was actually sent out in the timestamp field.
>>>>> This permits applications to monitor the drift between the video and
>>>>> system clock."
>>>>>
>>>>> To my knowledge all capture drivers set the timestamp to the time the
>>>>> *last* data byte was captured, not the first.
>>>>
>>>> The uvcvideo driver uses the time the first image packet is received :-)
>>>> Most other drivers use the time the last byte was *received*, not
>>>> captured.
>>>
>>> Unless the hardware buffers more than a few lines there is very little
>>> difference between the time the last byte was received and when it was
>>> captured.
>>>
>>> But you are correct, it is typically the time the last byte was received.
>>>
>>> Should we signal this as well? First vs last byte? Or shall we
>>> standardize?
>>
>> My personal opinion would be to change the spec to say what almost every
>> driver does: it's the timestamp from the moment the last pixel has been
>> received. We have the frame sync event for telling when the frame starts
>> btw. The same event could be used for signalling whenever a given line
>> starts. I don't see frame end fitting to that quite as nicely but I
>> guess it could be possible.
>
> The uvcvideo driver can timestamp the buffers with the system time at which
> the first packet in the frame is received, but has no way to generate a frame
> start event: the frame start event should correspond to the time the frame
> starts, not to the time the first packet in the frame is received. That
> information isn't available to the driver.

Aren't the two about equal, apart from the possible delays caused by the 
USB bus? The spec says about the frame sync event that it's "Triggered 
immediately when the reception of a frame has begun."

Cheers,

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Sakari,

On Wednesday 26 September 2012 22:17:22 Sakari Ailus wrote:
> Laurent Pinchart wrote:
> > On Tuesday 25 September 2012 23:12:00 Sakari Ailus wrote:
> >> Hans Verkuil wrote:
> >>> On Tue 25 September 2012 12:48:01 Laurent Pinchart wrote:
> >>>> On Tuesday 25 September 2012 08:47:45 Hans Verkuil wrote:
> >>>>> On Tue September 25 2012 02:00:55 Laurent Pinchart wrote:
> >>>>> BTW, I think we should also fix the description of the timestamp in
> >>>>> the spec. Currently it says:
> >>>>> 
> >>>>> "For input streams this is the system time (as returned by the
> >>>>> gettimeofday() function) when the first data byte was captured. For
> >>>>> output streams the data will not be displayed before this time,
> >>>>> secondary to the nominal frame rate determined by the current video
> >>>>> standard in enqueued order. Applications can for example zero this
> >>>>> field to display frames as soon as possible. The driver stores the
> >>>>> time at which the first data byte was actually sent out in the
> >>>>> timestamp field. This permits applications to monitor the drift
> >>>>> between the video and system clock."
> >>>>> 
> >>>>> To my knowledge all capture drivers set the timestamp to the time the
> >>>>> *last* data byte was captured, not the first.
> >>>> 
> >>>> The uvcvideo driver uses the time the first image packet is received
> >>>> :-)
> >>>> Most other drivers use the time the last byte was *received*, not
> >>>> captured.
> >>> 
> >>> Unless the hardware buffers more than a few lines there is very little
> >>> difference between the time the last byte was received and when it was
> >>> captured.
> >>> 
> >>> But you are correct, it is typically the time the last byte was
> >>> received.
> >>> 
> >>> Should we signal this as well? First vs last byte? Or shall we
> >>> standardize?
> >> 
> >> My personal opinion would be to change the spec to say what almost every
> >> driver does: it's the timestamp from the moment the last pixel has been
> >> received. We have the frame sync event for telling when the frame starts
> >> btw. The same event could be used for signalling whenever a given line
> >> starts. I don't see frame end fitting to that quite as nicely but I
> >> guess it could be possible.
> > 
> > The uvcvideo driver can timestamp the buffers with the system time at
> > which the first packet in the frame is received, but has no way to
> > generate a frame start event: the frame start event should correspond to
> > the time the frame starts, not to the time the first packet in the frame
> > is received. That information isn't available to the driver.
> 
> Aren't the two about equal, apart from the possible delays caused by the
> USB bus?

Apart from the possible delays caused by buffering on the device side, by 
transfers and by interrupt latency. For some applications that can matter. 
That's why we need support for device timestamps in the first place.

> The spec says about the frame sync event that it's "Triggered immediately
> when the reception of a frame has begun."

Then the OMAP3 ISP driver got it wrong, as we send the event when the hardware 
detects a vertical sync pulse :-)

We will never be able to implement the exact same behaviour for all devices. 
USB drivers typically don't receive VS events from the devices but can 
generate an event when they receive the first packet of a frame. On the other 
hand, DMA-based devices (PCI, SoC) usually only get notified of the end of the 
transfer (as opposed to the beginning of the transfer), and often (?) support 
generating an interrupt on VS detection. I think the best we can do is 
document in the spec that those different behaviours exist. We could add a 
capability flag to inform applications of the exact behaviour, but I don't 
think that's really worth it.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Hans and Laurent,

Hans Verkuil wrote:
...
>>> Using v4l2_buffer flags to report the clock
>>> -------------------------------------------
>>>
>>> By defining flags like this:
>>>
>>> V4L2_BUF_FLAG_CLOCK_MASK	0x7000
>>> /* Possible Clocks */
>>> V4L2_BUF_FLAG_CLOCK_UNKNOWN	0x0000  /* system or monotonic, we don't know 
>> */
>>> V4L2_BUF_FLAG_CLOCK_MONOTONIC   0x1000
>>>
>>> you could tell the application which clock is used.
>>>
>>> This does allow for more clocks to be added in the future and clock
>>> selection would then be done by a control or possibly an ioctl.
>>
>> Clock selection could also be done by setting the buffer flag at QBUF time.
> 
> True. Not a bad idea, actually. You would have to specify that setting the
> clock to 0 (UNKNOWN) or any other unsupported clock, then that will be mapped
> to MONOTONIC for newer kernels, but that's no problem.
> 
> It has the advantage of not requiring any controls, ioctls, etc. The only
> disadvantage is that you can't check if a particular clock is actually
> supported. Although I guess you could do a QBUF followed by QUERYBUF to check
> the clock bits. But you can't change to a different clock for that buffer
> afterwards (at least, not until it is dequeued).

Buffer flags are not and will not be available on subdevs. If the
timestamp clock source is made selectable it should be selectable on
subdevs, too. I'm afraid otherwise it may end up being a useless
feature: the timestamps that the application gets from the devices must
be from the same clock, otherwise they cannot be compared in a
meaningful way. Or at least alternative mechanism should be provided to
subdevs, but I don't see then why that wouldn't be done on V4L2, too...

Kind regards,

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi all,

As a summar from the discussion, I think we have reached the following
conclusion. Please say if you agree or disagree with what's below. :-)

- The drivers will be moved to use monotonic timestamps for video buffers.
- The user space will learn about the type of the timestamp through buffer
flags.
- The timestamp source may be made selectable in the future, but buffer
flags won't be the means for this, primarily since they're not available on
subdevs. Possible way to do this include a new V4L2 control or a new IOCTL.

Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Sakari,

On Monday 15 October 2012 19:05:49 Sakari Ailus wrote:
> Hi all,
> 
> As a summar from the discussion, I think we have reached the following
> conclusion. Please say if you agree or disagree with what's below. :-)
> 
> - The drivers will be moved to use monotonic timestamps for video buffers.
> - The user space will learn about the type of the timestamp through buffer
> flags.
> - The timestamp source may be made selectable in the future, but buffer
> flags won't be the means for this, primarily since they're not available on
> subdevs. Possible way to do this include a new V4L2 control or a new IOCTL.

That's my understanding as well. For the concept,

Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Chris MacGregor]

Hi, all.

On 10/15/2012 11:45 AM, Laurent Pinchart wrote:
> Hi Sakari,
>
> On Monday 15 October 2012 19:05:49 Sakari Ailus wrote:
>> Hi all,
>>
>> As a summar from the discussion, I think we have reached the following
>> conclusion. Please say if you agree or disagree with what's below. :-)
>>
>> - The drivers will be moved to use monotonic timestamps for video buffers.
>> - The user space will learn about the type of the timestamp through buffer
>> flags.
>> - The timestamp source may be made selectable in the future, but buffer
>> flags won't be the means for this, primarily since they're not available on
>> subdevs. Possible way to do this include a new V4L2 control or a new IOCTL.
> That's my understanding as well. For the concept,
>
> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

I wasn't able to participate in the discussion that led to this, but I'd 
like to suggest and request now that an explicit requirement (of 
whatever scheme is selected) be that a userspace app have a reasonable 
and straightforward way to translate the timestamps to real wall-clock 
time, ideally with enough precision to allow synchronization of cameras 
across multiple computers.

In the systems I work on, for instance, we are recording real-world 
biological processes, some of which vary based on the time of day, and 
it is important to know when a given frame was captured so that 
information can be stored with the raw frame and the data derived from 
it. For many such purposes, an accuracy measured in multiple seconds (or 
even minutes) is fine.

However, when we are using multiple cameras on multiple computers (e.g., 
two or more BeagleBoard xM's, each with a camera connected), we would 
want to synchronize with an accuracy of less than 1 frame time - e.g. 10 
ms or less.

Thanks very much,
Chris MacGregor

Re: [RFC] Timestamps and V4L2

[Sakari Ailus]

Hi Chris,

On Mon, Oct 15, 2012 at 11:53:56AM -0700, Chris MacGregor wrote:
> On 10/15/2012 11:45 AM, Laurent Pinchart wrote:
> >Hi Sakari,
> >
> >On Monday 15 October 2012 19:05:49 Sakari Ailus wrote:
> >>Hi all,
> >>
> >>As a summar from the discussion, I think we have reached the following
> >>conclusion. Please say if you agree or disagree with what's below. :-)
> >>
> >>- The drivers will be moved to use monotonic timestamps for video buffers.
> >>- The user space will learn about the type of the timestamp through buffer
> >>flags.
> >>- The timestamp source may be made selectable in the future, but buffer
> >>flags won't be the means for this, primarily since they're not available on
> >>subdevs. Possible way to do this include a new V4L2 control or a new IOCTL.
> >That's my understanding as well. For the concept,
> >
> >Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> I wasn't able to participate in the discussion that led to this, but
> I'd like to suggest and request now that an explicit requirement (of
> whatever scheme is selected) be that a userspace app have a
> reasonable and straightforward way to translate the timestamps to
> real wall-clock time, ideally with enough precision to allow
> synchronization of cameras across multiple computers.
> 
> In the systems I work on, for instance, we are recording real-world
> biological processes, some of which vary based on the time of day,
> and it is important to know when a given frame was captured so that
> information can be stored with the raw frame and the data derived
> from it. For many such purposes, an accuracy measured in multiple
> seconds (or even minutes) is fine.
> 
> However, when we are using multiple cameras on multiple computers
> (e.g., two or more BeagleBoard xM's, each with a camera connected),
> we would want to synchronize with an accuracy of less than 1 frame
> time - e.g. 10 ms or less.

I think you have two use cases actually: knowing when an image has been
taken, and synchronisation of images from multiple sources. The first one is
easy: you just call clock_gettime() for the realtime clock. The precision
will certainly be enough.

For the latter the realtime clock fits poorly to begin with: it jumps around
e.g. when the daylight saving time changes.

I think what I'd do is this: figure out the difference between the monotonic
clocks of your systems and use that as basis for synchronisation. I wonder
if there are existing solutions for this based e.g. on the NTP.

The pace of the monotonic clocks on different systems is the same as the
real-time ones; the same NTP adjustments are done to the monotonic clock as
well. As an added bonus you also won't be affected by daylight saving time
or someone setting the clock manually.

The conversion of the two clocks requires the knowledge of the values of
kernel internal variables, so performing the conversion in user space later
on is not an option.

Alternatively you could just call clock_gettime() after every DQBUF call,
but that's indeed less precise than if the driver would get the timestamp
for you.

How would this work for you?

Best regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [RFC] Timestamps and V4L2

[Rémi Denis-Courmont]

Le lundi 15 octobre 2012 22:59:06, Sakari Ailus a écrit :
> For the latter the realtime clock fits poorly to begin with: it jumps
> around e.g. when the daylight saving time changes.

Wrong. The real time clock is always UTC. It is not subject to time zone 
offsets. It only jumps when the clock is manually adjusted.

(That is not to deny that clock warping is a problem. All serious multimedia 
frameworks and network protocol stacks use the monotonic clock nowadays.)

-- 
Rémi Denis-Courmont
http://www.remlab.net/

Re: [RFC] Timestamps and V4L2

[Chris MacGregor]

Hi, Sakari.

On 10/15/2012 12:59 PM, Sakari Ailus wrote:
> Hi Chris,
>
> On Mon, Oct 15, 2012 at 11:53:56AM -0700, Chris MacGregor wrote:
>> On 10/15/2012 11:45 AM, Laurent Pinchart wrote:
>>> Hi Sakari,
>>>
>>> On Monday 15 October 2012 19:05:49 Sakari Ailus wrote:
>>>> Hi all,
>>>>
>>>> As a summar from the discussion, I think we have reached the following
>>>> conclusion. Please say if you agree or disagree with what's below. :-)
>>>>
>>>> - The drivers will be moved to use monotonic timestamps for video buffers.
>>>> - The user space will learn about the type of the timestamp through buffer
>>>> flags.
>>>> - The timestamp source may be made selectable in the future, but buffer
>>>> flags won't be the means for this, primarily since they're not available on
>>>> subdevs. Possible way to do this include a new V4L2 control or a new IOCTL.
>>> That's my understanding as well. For the concept,
>>>
>>> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> I wasn't able to participate in the discussion that led to this, but
>> I'd like to suggest and request now that an explicit requirement (of
>> whatever scheme is selected) be that a userspace app have a
>> reasonable and straightforward way to translate the timestamps to
>> real wall-clock time, ideally with enough precision to allow
>> synchronization of cameras across multiple computers.
>>
>> In the systems I work on, for instance, we are recording real-world
>> biological processes, some of which vary based on the time of day,
>> and it is important to know when a given frame was captured so that
>> information can be stored with the raw frame and the data derived
>> from it. For many such purposes, an accuracy measured in multiple
>> seconds (or even minutes) is fine.
>>
>> However, when we are using multiple cameras on multiple computers
>> (e.g., two or more BeagleBoard xM's, each with a camera connected),
>> we would want to synchronize with an accuracy of less than 1 frame
>> time - e.g. 10 ms or less.
> I think you have two use cases actually: knowing when an image has been
> taken, and synchronisation of images from multiple sources. The first one is
> easy: you just call clock_gettime() for the realtime clock. The precision
> will certainly be enough.

I assume you mean that I would, for instance:

clock_gettime(CLOCK_REALTIME, &realtime);
clock_gettime(CLOCK_MONOTONIC, &monotime);
(compute realtime-monotime and save the result)
...
(later add that result to the timestamp on a frame to recover the 
approximate real-world capture time)

Agreed, for this purpose - getting a reasonable real-world timestamp on 
the frame - the precision is fine...worst case, my process gets 
rescheduled between the two clock_gettime() calls, but even that won't 
matter for this purpose.

> For the latter the realtime clock fits poorly to begin with: it jumps around
> e.g. when the daylight saving time changes.
>
> I think what I'd do is this: figure out the difference between the monotonic
> clocks of your systems and use that as basis for synchronisation. I wonder
> if there are existing solutions for this based e.g. on the NTP.
>
> The pace of the monotonic clocks on different systems is the same as the
> real-time ones; the same NTP adjustments are done to the monotonic clock as
> well. As an added bonus you also won't be affected by daylight saving time
> or someone setting the clock manually.

Yes, I believe that this could work. My concern is that there is some 
unpredictability in the timing of the two clock_gettime() calls, and 
thus some inaccuracy in the conversion, and while I could likely get the 
two systems sufficiently sync'd using NTP or the like at the real-world 
level, the conversion from that (on each system) to the system's 
monotonic time would contain an unpredictable (though bounded) 
inaccuracy. I suppose that as long as I sync the cameras at the hardware 
level (e.g. by tying the strobe line of one to the trigger lines of the 
rest, or tying all the trigger lines to a GPIO), the inaccuracy would be 
less than a frame time, and so I could know reliably enough which frames 
go together.

However, it seems much cleaner to have a more direct way to convert the 
monotonic time to real-world time, or to get real-world time on the 
frames in the first place (for applications that want that). I don't 
know of a way to do the former.

> The conversion of the two clocks requires the knowledge of the values of
> kernel internal variables, so performing the conversion in user space later
> on is not an option.

Sorry, you lost me on this one, unless you're talking about what I refer 
to in my paragraph just above - converting the monotonic timestamp on 
the frame to real-world time...?

> Alternatively you could just call clock_gettime() after every DQBUF call,
> but that's indeed less precise than if the driver would get the timestamp
> for you.

And also less efficient. The platforms I'm working on are already 
hard-pressed to keep up with all the pixels I'm trying to capture and 
process, so I don't really want to waste time trapping into kernel mode 
again if I can avoid it.

> How would this work for you?

Better than nothing, and probably I could live with it. But I think 
perhaps we can do better than that, and now seems like the right time to 
figure it out.

> Best regards,

Cheers,
Chris MacGregor (the Seattle one)

Re: [RFC] Timestamps and V4L2

[Laurent Pinchart]

Hi Chris,

On Monday 15 October 2012 18:25:32 Chris MacGregor wrote:
> On 10/15/2012 12:59 PM, Sakari Ailus wrote:
> > On Mon, Oct 15, 2012 at 11:53:56AM -0700, Chris MacGregor wrote:
> >> On 10/15/2012 11:45 AM, Laurent Pinchart wrote:
> >>> On Monday 15 October 2012 19:05:49 Sakari Ailus wrote:
> >>>> Hi all,
> >>>> 
> >>>> As a summar from the discussion, I think we have reached the following
> >>>> conclusion. Please say if you agree or disagree with what's below. :-)
> >>>> 
> >>>> - The drivers will be moved to use monotonic timestamps for video
> >>>> buffers.
> >>>> - The user space will learn about the type of the timestamp through
> >>>> buffer flags.
> >>>> - The timestamp source may be made selectable in the future, but buffer
> >>>> flags won't be the means for this, primarily since they're not
> >>>> available on subdevs. Possible way to do this include a new V4L2
> >>>> control or a new IOCTL.
> >>> 
> >>> That's my understanding as well. For the concept,
> >>> 
> >>> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >> 
> >> I wasn't able to participate in the discussion that led to this, but
> >> I'd like to suggest and request now that an explicit requirement (of
> >> whatever scheme is selected) be that a userspace app have a
> >> reasonable and straightforward way to translate the timestamps to
> >> real wall-clock time, ideally with enough precision to allow
> >> synchronization of cameras across multiple computers.
> >> 
> >> In the systems I work on, for instance, we are recording real-world
> >> biological processes, some of which vary based on the time of day,
> >> and it is important to know when a given frame was captured so that
> >> information can be stored with the raw frame and the data derived
> >> from it. For many such purposes, an accuracy measured in multiple
> >> seconds (or even minutes) is fine.
> >> 
> >> However, when we are using multiple cameras on multiple computers
> >> (e.g., two or more BeagleBoard xM's, each with a camera connected),
> >> we would want to synchronize with an accuracy of less than 1 frame
> >> time - e.g. 10 ms or less.
> > 
> > I think you have two use cases actually: knowing when an image has been
> > taken, and synchronisation of images from multiple sources. The first one
> > is easy: you just call clock_gettime() for the realtime clock. The
> > precision will certainly be enough.
> 
> I assume you mean that I would, for instance:
> 
> clock_gettime(CLOCK_REALTIME, &realtime);
> clock_gettime(CLOCK_MONOTONIC, &monotime);
> (compute realtime-monotime and save the result)
> ...
> (later add that result to the timestamp on a frame to recover the
> approximate real-world capture time)

That would be a very simple way to do it. You might or might not need 
something more complex in practice, depending on your use cases. The point is 
that you need to synchronize images on multiple machines, so you need a common 
clock between those machines. Whether you use wall clock time, possibly 
synchronized through NTP, or another clock source is irrelevant here. The 
kernel driver has no access to the cross-machines clock, so you will need to 
convert the timestamp provided by the kernel to a timestamp relative to your 
clock.

> Agreed, for this purpose - getting a reasonable real-world timestamp on
> the frame - the precision is fine...worst case, my process gets
> rescheduled between the two clock_gettime() calls, but even that won't
> matter for this purpose.
> 
> > For the latter the realtime clock fits poorly to begin with: it jumps
> > around e.g. when the daylight saving time changes.
> > 
> > I think what I'd do is this: figure out the difference between the
> > monotonic clocks of your systems and use that as basis for
> > synchronisation. I wonder if there are existing solutions for this based
> > e.g. on the NTP.
> > 
> > The pace of the monotonic clocks on different systems is the same as the
> > real-time ones; the same NTP adjustments are done to the monotonic clock
> > as
> > well. As an added bonus you also won't be affected by daylight saving time
> > or someone setting the clock manually.
> 
> Yes, I believe that this could work. My concern is that there is some
> unpredictability in the timing of the two clock_gettime() calls, and
> thus some inaccuracy in the conversion, and while I could likely get the
> two systems sufficiently sync'd using NTP or the like at the real-world
> level, the conversion from that (on each system) to the system's
> monotonic time would contain an unpredictable (though bounded)
> inaccuracy. I suppose that as long as I sync the cameras at the hardware
> level (e.g. by tying the strobe line of one to the trigger lines of the
> rest, or tying all the trigger lines to a GPIO), the inaccuracy would be
> less than a frame time, and so I could know reliably enough which frames
> go together.

I'm pretty sure there's lots of literature on clock synchronization algorithms 
:-) The topic can quickly become pretty complex, and I'm not expert there. I 
don't expect too much problems to reach the require precision if you 
synchronize your cameras.

> However, it seems much cleaner to have a more direct way to convert the
> monotonic time to real-world time, or to get real-world time on the
> frames in the first place (for applications that want that). I don't
> know of a way to do the former.
> 
> > The conversion of the two clocks requires the knowledge of the values of
> > kernel internal variables, so performing the conversion in user space
> > later on is not an option.

We might be missing a kernel API here then.

> Sorry, you lost me on this one, unless you're talking about what I refer
> to in my paragraph just above - converting the monotonic timestamp on
> the frame to real-world time...?
> 
> > Alternatively you could just call clock_gettime() after every DQBUF call,
> > but that's indeed less precise than if the driver would get the timestamp
> > for you.
> 
> And also less efficient. The platforms I'm working on are already
> hard-pressed to keep up with all the pixels I'm trying to capture and
> process, so I don't really want to waste time trapping into kernel mode
> again if I can avoid it.
> 
> > How would this work for you?
> 
> Better than nothing, and probably I could live with it. But I think
> perhaps we can do better than that, and now seems like the right time to
> figure it out.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Timestamps and V4L2

[Hans Verkuil]

On Mon October 15 2012 20:45:45 Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Monday 15 October 2012 19:05:49 Sakari Ailus wrote:
> > Hi all,
> > 
> > As a summar from the discussion, I think we have reached the following
> > conclusion. Please say if you agree or disagree with what's below. :-)
> > 
> > - The drivers will be moved to use monotonic timestamps for video buffers.
> > - The user space will learn about the type of the timestamp through buffer
> > flags.
> > - The timestamp source may be made selectable in the future, but buffer
> > flags won't be the means for this, primarily since they're not available on
> > subdevs. Possible way to do this include a new V4L2 control or a new IOCTL.
> 
> That's my understanding as well. For the concept,
> 
> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Also for the concept:

Acked-by: Hans Verkuil <hans.verkuil@cisco.com>