Ensuring wall_to_monotonic is not positive breaks use case

Ensuring wall_to_monotonic is not positive breaks use case

[Rick Ratzel]

Hello,

We have a use case that was broken by the commit e1d7ba873555 (time: Always make sure wall_to_monotonic isn't positive).  We've been reverting the commit in our builds, but we'd greatly prefer a solution consistent with the mainline.  We also think our use case isn't unique to us, and may become more common in the near future.

Our use case is as follows: we have devices that have no notion of traceable time and often boot up with a time value of 0 (the Epoch).  These devices are networked and share time using protocols such as IEEE 1588 (PTP) or IEEE 802.1AS.  These protocols involve automatically electing a device to act as the source of time for all other devices on the network (the "grandmaster" in PTP speak) to transmit its time to the other "slave" devices.  This common shared time is used as a means to synchronize I/O operations across all devices to create a distributed measurement or control system.  The devices often interoperate with other 3rd party devices that also share time using the same protocol, and may also boot up with a time very near the Epoch.  We have no control over the 3rd party devices and cannot change the time that they boot up with, or the standardized algorithm they use to elect a common grandmaster.

In this case, time is used only as a means to synchronize periodic operations, where stable monotonically-increasing counts (this also implies no leap seconds!) are all that's needed and traceability to a standardized timescale is not necessary.

The problem arises when a device that's been elected grandmaster is sending out time at or very near (maybe only a few seconds past) the Epoch, and a slave device has an uptime of, say, several minutes past the Epoch.  The slave device will never be able to synchronize to the master in this situation, since the master is sending out time values lower than the slave's Epoch+uptime lower bound.

The presence of an RTC helps mitigate this situation, but only if the RTC has been set accordingly and its batteries have not failed.  We cannot guarantee these conditions, and many of the networked devices participating will not even have RTCs.

We're looking for suggestions on how best to proceed with a new change that ideally both supports the use case described above, as well as addresses the symptoms brought up in the initial commit (negative boot time causes get_expiry() to overflow time_t, and show_stat() uses "unsigned long" to print negative btime).  Any thoughts on this would be greatly appreciated.

Link to initial commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e1d7ba8735551ed79c7a0463a042353574b96da3

Thanks,
Rick Ratzel - National Instruments

Re: Ensuring wall_to_monotonic is not positive breaks use case

[Thomas Gleixner]

Rick,

On Wed, 5 Sep 2018, Rick Ratzel wrote:

> We're looking for suggestions on how best to proceed with a new change
> that ideally both supports the use case described above, as well as
> addresses the symptoms brought up in the initial commit (negative boot
> time causes get_expiry() to overflow time_t, and show_stat() uses
> "unsigned long" to print negative btime).  Any thoughts on this would be
> greatly appreciated.

Those symptoms are just the tip of the iceberg. For sure it screws up
everything around boot time and a lot of things use boottime nowadays.

So reverting this is not really an option.

Chosing a PTP grandmaster which populates random time is a really great
idea. Why has this industry the tendency to turn everything into a
trainwreck?

Thanks,

	tglx

Re: Ensuring wall_to_monotonic is not positive breaks use case

[Thomas Gleixner]

On Thu, 6 Sep 2018, Thomas Gleixner wrote:
> On Wed, 5 Sep 2018, Rick Ratzel wrote:
> 
> > We're looking for suggestions on how best to proceed with a new change
> > that ideally both supports the use case described above, as well as
> > addresses the symptoms brought up in the initial commit (negative boot
> > time causes get_expiry() to overflow time_t, and show_stat() uses
> > "unsigned long" to print negative btime).  Any thoughts on this would be
> > greatly appreciated.
> 
> Those symptoms are just the tip of the iceberg. For sure it screws up
> everything around boot time and a lot of things use boottime nowadays.

I had a second look and actually it's not that bad. My brain snapped on
boot time and we actually have two variants of boot time. One is the
monotonic time since boot aka CLOCK_BOOTTIME and the other one is the wall
clock time when the system was booted.

> So reverting this is not really an option.

Maybe we can :)

We have 3 users in tree:

1) /proc/stat btime

   It's trivial enough to clamp that value to 0, though it might be
   surprising for some users as uptime will tell something different.

2) sunrpc

   I think that can be converted to actually use CLOCK_BOOTTIME. This needs
   a bit of trickery to preserve the user space interfaces which are
   CLOCK_REALTIME based, but it should be doable.

3) x86/kvm

   That's actually the trickiest part of all and I haven't yet fully
   analyzed it, whether there is an issue that time can go negative.

   Actually there shouldn't be one as this is about wall time and that
   cannot go before the epoch. Needs some close inspection and while it
   might be trivial to adapt the code in question, it could be tricky or
   even impossible to fix it without breaking existing guests.

Thanks,

	tglx

RE: Ensuring wall_to_monotonic is not positive breaks use case

[Rick Ratzel]

On Sat, 15 Sep 2018, Thomas Gleixner wrote:
> > 
> > So reverting this is not really an option.

> Maybe we can :)

Thanks for checking into this more.  We've been looking at the uses you described, and hopefully we can have something to submit for review (or at least discuss) relatively soon.

>
> We have 3 users in tree:
>
> 1) /proc/stat btime
>
>    It's trivial enough to clamp that value to 0, though it might be
>    surprising for some users as uptime will tell something different.
>

Agreed.

> 2) sunrpc
>
>    I think that can be converted to actually use CLOCK_BOOTTIME. This needs
>    a bit of trickery to preserve the user space interfaces which are
>    CLOCK_REALTIME based, but it should be doable.
>

It looks like this can use CLOCK_BOOTTIME as you describe, but "properly" converting to the REALTIME timescale might be hard, as you noticed. One proposal is to floor the conversion to the Epoch, just like your proposal to floor /proc/stat btime to 0, but we're not sure how that'll impact users yet.

> 3) x86/kvm
>
>    That's actually the trickiest part of all and I haven't yet fully
>    analyzed it, whether there is an issue that time can go negative.
>
>    Actually there shouldn't be one as this is about wall time and that
>    cannot go before the epoch. Needs some close inspection and while it
>    might be trivial to adapt the code in question, it could be tricky or
>    even impossible to fix it without breaking existing guests.

We're still looking at this one as well...

-Rick