Re: [RFC] Timestamps and V4L2

[Christian Gmeiner <christian.gmeiner@gmail.com>]

2012/9/20 Sakari Ailus <sakari.ailus@iki.fi>:
> Hi all,
>
>
> This RFC intends to summarise and further the recent discussion on
> linux-media regarding the proposed changes of timestamping V4L2 buffers.
>
>
> The problem
> ===========
>
> The V4L2 has long used realtime timestamps (such as
> clock_gettime(CLOCK_REALTIME, ...)) to stamp the video buffers before
> handing them over to the user. This has been found problematic in
> associating the video buffers with data from other sources: realtime clock
> may jump around due to daylight saving time, for example, and ALSA
> (audio-video synchronisation is a common use case) user space API does not
> provide the user with realtime timestamps, but instead uses monotonic time
> (i.e. clock_gettime(CLOCK_MONOTONIC, ...)).
>
> This is especially an issue in embedded systems where video recording is a
> common use case. Drivers typically used in such systems have silently
> switched to use monotonic timestamps. While against the spec, this is
> necessary for those systems to operate properly.
>
> In general, realtime timestamps are seen of little use in other than
> debugging purposes, but monotonic timestamps are fine for that as well. It's
> still possible that an application I'm not aware of uses them in a peculiar
> way that would be adversely affected by changing to monotonic timestamps.
> Nevertheless, we're not supposed to break the API (or ABI). It'd be also
> very important for the application to know what kind of timestamps are
> provided by the device.
>
>
> Requirements, wishes and constraints
> ====================================
>
> Now that it seems to be about the time to fix these issues, it's worth
> looking a little bit to the future to anticipate the coming changes to be
> able to accommodate them better later on.
>
> - The new default should be monotonic. As the monotonic timestamps are seen
> to be the most useful, they should be made the default.
>
> - timeval vs. timespec. The two structs can be used to store timestamp
> information. They are not compatible with each other. It's a little bit
> uncertain what's the case with all the architectures but it looks like the
> timespec fits into the space of timeval in all cases. If timespec is
> considered to be used somewhere the compatibility must be ensured. Timespec
> is better than timeval since timespec has more precision and it's the same
> struct that's used everywhere else in the V4L2 API: timespec does not need
> conversion to timespec in the user space.
>
> struct timespec {
>         __kernel_time_t tv_sec;                 /* seconds */
>         long            tv_nsec;                /* nanoseconds */
> };
>
> struct timeval {
>         __kernel_time_t         tv_sec;         /* seconds */
>         __kernel_suseconds_t    tv_usec;        /* microseconds */
> };
>
> To be able to use timespec, the user would have to most likely explicitly
> choose to do that.
>
> - Users should know what kind of timestamps the device produces. This
> includes existing and future kernels. What should be considered are
> uninformed porting drivers back and forth across kernel versions and
> out-of-date kernel header files.
>
> - Device-dependent timestamps. Some devices such as the uvcvideo ones
> produce device-dependent timestamps for synchronising video and audio, both
> produced by the same physical hardware device. For uvcvideo these timestamps
> are unsigned 32-bit integers.


What about pure output devices like old-school mpeg2 cards? The timestamps for
audio and video are pure PTS values.

>
> - There's also another clock, Linux-specific raw monotonic clock (as in
> clock_gettime(CLOCK_RAW_MONOTONIC, ...)) that could be better in some use
> cases than the regular monotonic clock. The difference is that the raw
> monotonic clock is free from the NTP adjustments. It would be nice for the
> user to be able to choose the clock used for timestamps. This is especially
> important for device-dependent timestamps: not all applications can be
> expected to be able to use them.
>
> - The field adjacent to timestamp, timecode, is 128 bits wide, and not used
> by a single driver. This field could be re-used.
>
>
> Possible solutions
> ==================
>
> Not all of the solutions below that have been proposed are mutually
> exclusive. That's also what's making the choice difficult: the ultimate
> solution to the issue of timestamping may involve several of these --- or
> possibly something better that's not on the list.
>
>
> Use of timespec
> ---------------
>
> If we can conclude timespec will always fit into the size of timeval (or
> timecode) we could use timespec instead. The solution should still make
> the use of timespec explicit to the user space. This seems to conflict with
> the idea of making monotonic timestamps the default: the default can't be
> anything incompatible with timeval, and at the same time it's the most
> important that the monotonic timestamps are timespec.
>
>
> Kernel version as indicator of timestamp
> ----------------------------------------
>
> Conversion of drivers to use monotonic timestamp is trivial, so the
> conversion could be done once and for all drivers. The kernel version could
> be used to indicate the type of the timestamp.
>
> If this approach is taken care must be taken when new drivers are
> integrated: developers sometimes use old kernels for development and might
> also use an old driver for guidance on timestamps, thus using real-time
> timestamps when monotonic timestamps should be used. This approach has an
> advantage over the capability flag below: which is that we don't populate
> the interface with essentially dead definitions.
>
>
> Capability flag for monotonic timestamps
> ----------------------------------------
>
> A capability flag can be used to tell whether the timestamp is monotonic.
> However, it's not extensible cleanly to provide selectable timestamps. These
> are not features that are needed right now, though.
>
> The upside of this option is ease of implementation and use, but it's not
> extensible. Also we're left with a flag that's set for all drivers: in the
> end it provides no information to the user and is only noise in the spec.
>
>
> Control for timestamp type
> --------------------------
>
> Using a control to tell the type of the timestamp is extensible but not as
> easy to implement than the capability flag: each and every device would get
> an additional control. The value should likely be also file handle specific,
> and we do not have file handle specific controls yet.
>
> In the meantime the control could be read-only, and later made read-write
> when the timestamp type can be made selectable. Much of he work of
> timestamping can be done by the framework: drivers can use a single helper
> function and need to create one extra standard control.
>
> Should the control also have an effect on the types of the timestamps in
> V4L2 events? Likely yes.
>
>
> Device-dependent timestamp
> --------------------------
>
> Should we agree on selectable timestamps, the existing timestamp field (or a
> union with another field of different type) could be used for the
> device-dependent timestamps. Alternatively we can choose to re-use the
> existing timecode field.
>
> At the moment there's no known use case for passing device-dependent
> timestamps at the same time with monotonic timestamps.
>
>
> Now what?
> =========
>
> Almost as many options have been presented as there were opinions, but we
> need to agree to have a single one. My personal leaning is on using a
> control for the purpose as it is the most flexible alternative. I'd still
> need to see an implementation of that but it doesn't seem that difficult,
> especially when it's read-only. And even for read-write control the vast
> majority of the work can be done by the V4L2 framework.
>
> Questions, comments and opinions are very, very welcome.
>
>

---
Christian Gmeiner, MSc