Missing CLOCK_BOOTTIME_RAW?

Missing CLOCK_BOOTTIME_RAW?

[Pali Rohár]

Hello Thomas!

I'm looking at clocks which are provided by kernel for userspace
applications via clock_gettime() syscall and I think that there is
missing clock variant CLOCK_BOOTTIME_RAW.

If userspace application wants to measure time difference between two
events then I think all of available clocks CLOCK_REALTIME,
CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW and CLOCK_BOOTTIME have some issues
for such task.

CLOCK_REALTIME may be changed but userspace and therefore time
difference between two events may be also negative.

CLOCK_MONOTONIC does not have to precise as there may be incremental
jumps caused by NTP or measured time difference may be smaller as
elapsed when system is suspended. During system suspend is clock
CLOCK_MONOTONIC stopped.

CLOCK_MONOTONIC_RAW is not affected by NTP jumps but still is not
suitable for measuring time differences when system is suspended.

CLOCK_BOOTTIME is affected by NTP jumps but provides correct time
difference when system was suspended during measurement.

So for me it looks like that there is missing CLOCK_BOOTTIME_RAW clock
which would not be affected by NTP jumps (like CLOCK_MONOTONIC_RAW) and
also would not be affected by system suspend (like CLOCK_BOOTTIME).

Please correct me if I'm wrong in some of my above assumptions. It is
how I understood documentation for clock_gettime() function from Linux
manpage.

Is there any reason why kernel does not provide such CLOCK_BOOTTIME_RAW
clock for userspace applications which would be interested in measuring
time difference which occurred between two events?

Re: Missing CLOCK_BOOTTIME_RAW?

[Thomas Gleixner]

Pali,

Pali Rohár <pali@kernel.org> writes:

> I'm looking at clocks which are provided by kernel for userspace
> applications via clock_gettime() syscall and I think that there is
> missing clock variant CLOCK_BOOTTIME_RAW.
>
> If userspace application wants to measure time difference between two
> events then I think all of available clocks CLOCK_REALTIME,
> CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW and CLOCK_BOOTTIME have some issues
> for such task.
...
> CLOCK_BOOTTIME is affected by NTP jumps but provides correct time
> difference when system was suspended during measurement.

What do you mean with NTP jumps?

Neither CLOCK_BOOTTIME nor CLOCK_MONOTONIC jump. They are frequency
corrected when NTP, PTP or PPS are in use. The frequency correction is
incremental an smothed. They really don't jump and they give you proper
time in nanoseconds which is as close to the real nanoseconds as it
gets.

CLOCK_MONOTONIC_RAW converts some assumed clock frequency into something
which looks like time. But it's raw and subject to drift. The only
reason why it's exposed is because NTP/PTP need it to calculate the
frequency difference between the hardware counter and the master clock.

> So for me it looks like that there is missing CLOCK_BOOTTIME_RAW clock
> which would not be affected by NTP jumps (like CLOCK_MONOTONIC_RAW) and
> also would not be affected by system suspend (like CLOCK_BOOTTIME).
>
> Please correct me if I'm wrong in some of my above assumptions. It is
> how I understood documentation for clock_gettime() function from Linux
> manpage.

I don't know how you read jumps into this:

  CLOCK_MONOTONIC

   Clock that cannot be set and represents monotonic time since some
   unspecified starting point.  This clock is not affected by
   discontinuous jumps in the system time (e.g., if the system
   administrator manually changes the clock), but is affected by the
   incremental adjustments performed by adjtime(3) and NTP.

And CLOCK_BOOTTIME is the same except that it accounts time in suspend
while MONOTONIC stops in suspend.

> Is there any reason why kernel does not provide such CLOCK_BOOTTIME_RAW
> clock for userspace applications which would be interested in measuring
> time difference which occurred between two events?

Nobody needs it and there is even no guarantee that it can be
reconstructed on resume.

If you want accurate time deltas then use either CLOCK_MONOTONIC or
CLOCK_BOOTTIME depending on your interest in suspend time.

Thanks,

        tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Pali Rohár]

On Friday 08 May 2020 22:59:57 Thomas Gleixner wrote:
> Pali,
> 
> Pali Rohár <pali@kernel.org> writes:
> 
> > I'm looking at clocks which are provided by kernel for userspace
> > applications via clock_gettime() syscall and I think that there is
> > missing clock variant CLOCK_BOOTTIME_RAW.
> >
> > If userspace application wants to measure time difference between two
> > events then I think all of available clocks CLOCK_REALTIME,
> > CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW and CLOCK_BOOTTIME have some issues
> > for such task.
> ...
> > CLOCK_BOOTTIME is affected by NTP jumps but provides correct time
> > difference when system was suspended during measurement.
> 
> What do you mean with NTP jumps?
> 
> Neither CLOCK_BOOTTIME nor CLOCK_MONOTONIC jump. They are frequency
> corrected when NTP, PTP or PPS are in use. The frequency correction is
> incremental an smothed. They really don't jump and they give you proper
> time in nanoseconds which is as close to the real nanoseconds as it
> gets.

Hello! I should have been more precise about it. CLOCK_BOOTTIME and
CLOCK_MONOTONIC do not jump but I understood that they are affected by
adjtime(). So these clocks may tick faster or slower than real time. NTP
daemon when see that CLOCK_REALTIME is incorrect, it may speed up or
slow down its ticking. And this is affected also by CLOCK_BOOTTIME and
CLOCK_MONOTONIC, right?

> CLOCK_MONOTONIC_RAW converts some assumed clock frequency into something
> which looks like time. But it's raw and subject to drift. The only
> reason why it's exposed is because NTP/PTP need it to calculate the
> frequency difference between the hardware counter and the master clock.

  CLOCK_MONOTONIC_RAW
      Similar to CLOCK_MONOTONIC, but provides access to a raw
      hardware-based time that is not subject to NTP adjustments or the
      incremental adjustments performed by adjtime(3).

I understood it that it is like CLOCK_MONOTONIC but is not affected by
adjtime() which is used by NTP daemon to slow down or speed up system
clock to synchronize it (when system clock is incorrect).

So I thought that this clock is better for time differences as measured
time would not be affected when NTP daemon speeded up system clock via
adjtime().

You wrote that this clock is subject to drifts. What exactly may happen
with CLOCK_MONOTONIC_RAW?

> > So for me it looks like that there is missing CLOCK_BOOTTIME_RAW clock
> > which would not be affected by NTP jumps (like CLOCK_MONOTONIC_RAW) and
> > also would not be affected by system suspend (like CLOCK_BOOTTIME).
> >
> > Please correct me if I'm wrong in some of my above assumptions. It is
> > how I understood documentation for clock_gettime() function from Linux
> > manpage.
> 
> I don't know how you read jumps into this:
> 
>   CLOCK_MONOTONIC
> 
>    Clock that cannot be set and represents monotonic time since some
>    unspecified starting point.  This clock is not affected by
>    discontinuous jumps in the system time (e.g., if the system
>    administrator manually changes the clock), but is affected by the
>    incremental adjustments performed by adjtime(3) and NTP.

Sorry, by jump I mean that clock may be adjusted (even smoothed
adjustment).

> And CLOCK_BOOTTIME is the same except that it accounts time in suspend
> while MONOTONIC stops in suspend.
> 
> > Is there any reason why kernel does not provide such CLOCK_BOOTTIME_RAW
> > clock for userspace applications which would be interested in measuring
> > time difference which occurred between two events?
> 
> Nobody needs it and there is even no guarantee that it can be
> reconstructed on resume.
> 
> If you want accurate time deltas then use either CLOCK_MONOTONIC or
> CLOCK_BOOTTIME depending on your interest in suspend time.
> 
> Thanks,
> 
>         tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Thomas Gleixner]

Pali,

Pali Rohár <pali@kernel.org> writes:
> On Friday 08 May 2020 22:59:57 Thomas Gleixner wrote:
>> Pali Rohár <pali@kernel.org> writes:
>> Neither CLOCK_BOOTTIME nor CLOCK_MONOTONIC jump. They are frequency
>> corrected when NTP, PTP or PPS are in use. The frequency correction is
>> incremental an smothed. They really don't jump and they give you proper
>> time in nanoseconds which is as close to the real nanoseconds as it
>> gets.
>
> Hello! I should have been more precise about it. CLOCK_BOOTTIME and
> CLOCK_MONOTONIC do not jump but I understood that they are affected by
> adjtime(). So these clocks may tick faster or slower than real time. NTP
> daemon when see that CLOCK_REALTIME is incorrect, it may speed up or
> slow down its ticking. And this is affected also by CLOCK_BOOTTIME and
> CLOCK_MONOTONIC, right?

Sure, but what's the problem? The adjustemt is done to make the observed
time as correct as possible.

> You wrote that this clock is subject to drifts. What exactly may happen
> with CLOCK_MONOTONIC_RAW?

  1) As the frequency of the raw clock is an estimate, resulting time
     is drifting apart vs. the correct frequency.

  2) Depending on the crystal/oszillator there can be temperatur drift as
     well.

Just for clarification. Even with NTP/PTP adjustment the resulting time
stamps are never going to be precise vs. an atomic clock, but good
enough for 99.9999% of the problems.

TBH, I have no idea what real world problem you are trying to solve.

Thanks,

        tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Pali Rohár]

On Saturday 09 May 2020 11:49:27 Thomas Gleixner wrote:
> Pali,
> 
> Pali Rohár <pali@kernel.org> writes:
> > On Friday 08 May 2020 22:59:57 Thomas Gleixner wrote:
> >> Pali Rohár <pali@kernel.org> writes:
> >> Neither CLOCK_BOOTTIME nor CLOCK_MONOTONIC jump. They are frequency
> >> corrected when NTP, PTP or PPS are in use. The frequency correction is
> >> incremental an smothed. They really don't jump and they give you proper
> >> time in nanoseconds which is as close to the real nanoseconds as it
> >> gets.
> >
> > Hello! I should have been more precise about it. CLOCK_BOOTTIME and
> > CLOCK_MONOTONIC do not jump but I understood that they are affected by
> > adjtime(). So these clocks may tick faster or slower than real time. NTP
> > daemon when see that CLOCK_REALTIME is incorrect, it may speed up or
> > slow down its ticking. And this is affected also by CLOCK_BOOTTIME and
> > CLOCK_MONOTONIC, right?
> 
> Sure, but what's the problem? The adjustemt is done to make the observed
> time as correct as possible.

Yes. But correction may take lot of time, e.g. also more then one day.

So during this period when correction is in progress, measuring time
difference via CLOCK_MONITONIC will have incorrect results.

It already happened for me, system clock was shifted by one hour and
chronyd started adjustment, it slow down system clock. 6 real hours
passed and via system clock was measured time difference only about
5 hours and 20 minutes (correction was still in progress as system
clock at that time was still shifted by about 20 minutes).

So measuring time differences via clock affected by NTP adjustment
resulted in error which was more then 30 minutes.

CLOCK_MONOTONIC_RAW is not affected by this correction progress, so it
should gives better results. Or not?

> > You wrote that this clock is subject to drifts. What exactly may happen
> > with CLOCK_MONOTONIC_RAW?
> 
>   1) As the frequency of the raw clock is an estimate, resulting time
>      is drifting apart vs. the correct frequency.
> 
>   2) Depending on the crystal/oszillator there can be temperatur drift as
>      well.
> 
> Just for clarification. Even with NTP/PTP adjustment the resulting time
> stamps are never going to be precise vs. an atomic clock, but good
> enough for 99.9999% of the problems.

Ok, so it looks like that CLOCK_MONOTONIC_RAW is not the best choice.

> TBH, I have no idea what real world problem you are trying to solve.

Problem is simple: Measure time difference between two events and not to
be affected by the fact that system clock on machine is incorrect or
that time daemon is actually adjusting time.

I need to know if difference between those two events in more then some
period or not.

So if I want to know if time difference is more then 5 hours and 40
minutes then measurement via clock which is affected by NTP adjustment
would give me wrong result. As described above in reality 6 hours
passed but clock measured only 5 hours and 20 minutes.

> Thanks,
> 
>         tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Thomas Gleixner]

Pali Rohár <pali@kernel.org> writes:
> On Saturday 09 May 2020 11:49:27 Thomas Gleixner wrote:
>> Sure, but what's the problem? The adjustemt is done to make the observed
>> time as correct as possible.
>
> Yes. But correction may take lot of time, e.g. also more then one day.
>
> So during this period when correction is in progress, measuring time
> difference via CLOCK_MONITONIC will have incorrect results.
>
> It already happened for me, system clock was shifted by one hour and
> chronyd started adjustment, it slow down system clock. 6 real hours
> passed and via system clock was measured time difference only about
> 5 hours and 20 minutes (correction was still in progress as system
> clock at that time was still shifted by about 20 minutes).

System clock shifted by one hour? You mean DST change? If chronyd
adjusts that by smoothing the frequency, that's just broken and really
not the kernel's problem.

Thanks,

        tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Pali Rohár]

On Monday 18 May 2020 13:26:14 Thomas Gleixner wrote:
> Pali Rohár <pali@kernel.org> writes:
> > On Saturday 09 May 2020 11:49:27 Thomas Gleixner wrote:
> >> Sure, but what's the problem? The adjustemt is done to make the observed
> >> time as correct as possible.
> >
> > Yes. But correction may take lot of time, e.g. also more then one day.
> >
> > So during this period when correction is in progress, measuring time
> > difference via CLOCK_MONITONIC will have incorrect results.
> >
> > It already happened for me, system clock was shifted by one hour and
> > chronyd started adjustment, it slow down system clock. 6 real hours
> > passed and via system clock was measured time difference only about
> > 5 hours and 20 minutes (correction was still in progress as system
> > clock at that time was still shifted by about 20 minutes).
> 
> System clock shifted by one hour? You mean DST change?

Yes, clock was shifted by one hour.

> If chronyd
> adjusts that by smoothing the frequency, that's just broken and really
> not the kernel's problem.

And what other can time synchronization daemon does?

Well, I think this is not related to chronyd. Any userspace application
may call adjtime(). It is not privilege that only chronyd is allowed to
use that syscall.

I agree that this is not a kernel problem.

But I'm asking, how my userspace application can measure time difference?
As I wrote CLOCK_MONITONIC is not suitable as it is affected by those
NTP adjustments and that is why I thought that CLOCK_MONITONIC_RAW is
better as it is not affected by these NTP problems.

But CLOCK_MONITONIC_RAW has a problem that is stopped during system
sleep and that is why I thought that CLOCK_BOOTTIME_RAW should be there.

> Thanks,
> 
>         tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Thomas Gleixner]

Pali Rohár <pali@kernel.org> writes:
> On Monday 18 May 2020 13:26:14 Thomas Gleixner wrote:
>> System clock shifted by one hour? You mean DST change?
>
> Yes, clock was shifted by one hour.
>
>> If chronyd
>> adjusts that by smoothing the frequency, that's just broken and really
>> not the kernel's problem.
>
> And what other can time synchronization daemon does?

DST switching is a matter of switching time zones and has absolutely
nothing to do with frequencies. In fact smearing DST is a blantant
violation of the timekeeping guarantees.

I've never heard about that before. All I know is that chronyd can be
configured to smear leap seconds, but that doesn't take 6 hours and does
not screw with the time accuracy in the range of 20 minutes.

> Well, I think this is not related to chronyd. Any userspace application
> may call adjtime(). It is not privilege that only chronyd is allowed to
> use that syscall.

Of course not, but the kernel relies on that application behaving
sanely. If it does not then the time stamps you are complaining about
are the least of your worries.

> I agree that this is not a kernel problem.
>
> But I'm asking, how my userspace application can measure time difference?
> As I wrote CLOCK_MONITONIC is not suitable as it is affected by those
> NTP adjustments and that is why I thought that CLOCK_MONITONIC_RAW is
> better as it is not affected by these NTP problems.
>
> But CLOCK_MONITONIC_RAW has a problem that is stopped during system
> sleep and that is why I thought that CLOCK_BOOTTIME_RAW should be there.

And how do you make CLOCK_BOOTTIME_RAW accurate? The clock hardware
can stop accross suspend/resume and the kernel then uses RTC or some
other hardware to inject the sleep time. That injection is not and
cannot be correct vs. the raw clock.

So exposing CLOCK_BOOTTIME_RAW would just provide yet another illusion
of time.

Thanks,

        tglx

Re: Missing CLOCK_BOOTTIME_RAW?

[Pali Rohár]

On Monday 18 May 2020 14:13:48 Thomas Gleixner wrote:
> Pali Rohár <pali@kernel.org> writes:
> > On Monday 18 May 2020 13:26:14 Thomas Gleixner wrote:
> >> System clock shifted by one hour? You mean DST change?
> >
> > Yes, clock was shifted by one hour.
> >
> >> If chronyd
> >> adjusts that by smoothing the frequency, that's just broken and really
> >> not the kernel's problem.
> >
> > And what other can time synchronization daemon does?
> 
> DST switching is a matter of switching time zones and has absolutely
> nothing to do with frequencies. In fact smearing DST is a blantant
> violation of the timekeeping guarantees.

Problem was that "external source" changed RTC.

But I think this is irrelevant here.

Important was state of machine. Machine was in state that had incorrect
system clock, had running NTP daemon and there was need to somehow make
system clock again correct.

And such situation may happen at any time. Either user unintentionally
change "current time" in his desktop GUI application or call "date"
application with "set" parameter (instead of get), ...

> I've never heard about that before. All I know is that chronyd can be
> configured to smear leap seconds, but that doesn't take 6 hours and does
> not screw with the time accuracy in the range of 20 minutes.
> 
> > Well, I think this is not related to chronyd. Any userspace application
> > may call adjtime(). It is not privilege that only chronyd is allowed to
> > use that syscall.
> 
> Of course not, but the kernel relies on that application behaving
> sanely. If it does not then the time stamps you are complaining about
> are the least of your worries.

I do not thing it is too bad... When I needed to deal in userspace with
time/date/clock I just needed either "current time in UTC" to show it to
user (possible in different timezone and pretty formatted) or I needed
"timestamp since some epoch" suitable for measuring time differences.

For first case I used CLOCK_REALTIME and for second case I used
CLOCK_MONOTONIC_RAW (as it was not affected by adjtime()).

And I would like to know, it is correct to use these two clocks in those
situations?

> > I agree that this is not a kernel problem.
> >
> > But I'm asking, how my userspace application can measure time difference?
> > As I wrote CLOCK_MONITONIC is not suitable as it is affected by those
> > NTP adjustments and that is why I thought that CLOCK_MONITONIC_RAW is
> > better as it is not affected by these NTP problems.
> >
> > But CLOCK_MONITONIC_RAW has a problem that is stopped during system
> > sleep and that is why I thought that CLOCK_BOOTTIME_RAW should be there.
> 
> And how do you make CLOCK_BOOTTIME_RAW accurate? The clock hardware
> can stop accross suspend/resume and the kernel then uses RTC or some
> other hardware to inject the sleep time. That injection is not and
> cannot be correct vs. the raw clock.
> 
> So exposing CLOCK_BOOTTIME_RAW would just provide yet another illusion
> of time.

Now I see. Usage of external source during suspend would just lead to
another problems. So idea about CLOCK_BOOTTIME_RAW is nice but there is
no way how to provide it.

Anyway, what would happen with CLOCK_BOOTTIME when during suspend is
that external RTC source shifted back? Is kernel guarantee that
CLOCK_BOOTTIME is always monotonic even in this case?

Re: Missing CLOCK_BOOTTIME_RAW?

[Thomas Gleixner]

Pali Rohár <pali@kernel.org> writes:
> On Monday 18 May 2020 14:13:48 Thomas Gleixner wrote:
>> Of course not, but the kernel relies on that application behaving
>> sanely. If it does not then the time stamps you are complaining about
>> are the least of your worries.
>
> I do not thing it is too bad... When I needed to deal in userspace with
> time/date/clock I just needed either "current time in UTC" to show it to
> user (possible in different timezone and pretty formatted) or I needed
> "timestamp since some epoch" suitable for measuring time differences.
>
> For first case I used CLOCK_REALTIME and for second case I used
> CLOCK_MONOTONIC_RAW (as it was not affected by adjtime()).
>
> And I would like to know, it is correct to use these two clocks in those
> situations?

It's your choice to do so. I prefer CLOCK_MONOTONIC simply because it's
in human understandable units and not some assumed frequency.

> Anyway, what would happen with CLOCK_BOOTTIME when during suspend is
> that external RTC source shifted back? Is kernel guarantee that
> CLOCK_BOOTTIME is always monotonic even in this case?

If the RTC delta is negative, then it's ignored, i.e. 0 sleep time
injected.

Thanks,

        tglx