Issue with time going backwards on OSX

Issue with time going backwards on OSX

[Joe Tanen]

[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]

Hello,

I posted this on qemu-discuss, but it doesn't seem like anybody else has seen this issue. I figured I'd try here to see if anybody had any thoughts.

I'm using qemu-system-aarch64 v5.2 on OSX 11.6. While reading cntvct_el0, I've seen the ticks count go backwards. I dug into the qemu source a bit, and this register is eventually backed by a call to clock_gettime(CLOCK_MONOTONIC), which should not go backwards. This code seems to be the same on master as the branch I'm on.

Before I started delving into debugging qemu, I wanted to verify that this wasn't an OS issue. I wrote a simple test that read CLOCK_MONOTONIC every 100ms, compared the values to make sure they always increased, and let it run overnight. Lo and behold, I saw time go backwards. I ran a concurrent test with CLOCK_MONOTONIC_RAW, and I did not see time go backwards. Successive reads of the CLOCK_MONOTONIC after it jumped backwards tracked the new time, so the jump wasn't a spurious error. When I ran this test on a Fedora 34 PC, I never saw time go backwards.

Someone suggested as a test disabling networking so the NTP daemon couldn't mess with the clock, but that's not something I can do IRL. In any case, my understanding is that CLOCK_MONOTONIC should never go backwards and that any adjustments are made by temporarily slowing down or speeding up the tick rate. I'm going to pursue what I believe to be the underlying issue separately, but my qemu questions are these:

- Has anybody else seen this or other timer registers go backwards under these or other conditions? I only see the behavior on OSX 11.6, but I'm a small sample size.

- Would it make sense to use CLOCK_MONOTONIC_RAW (if available) instead of CLOCK_MONOTONIC in qemu? My issue aside, CLOCK_MONOTONIC_RAW feels closer to giving the guest direct access to a hardware clock. e.g., if I wanted to write my own NTP time adjustment daemon, I'd prefer to have an unadjusted clock. The code already checks to see if CLOCK_MONOTONIC is available, and, if not, it reverts to a simple gettimeofday(). It was easy to add another check to init_get_clock() to add a case checking for the availability of CLOCK_MONOTONIC_RAW. I've tested this myself, but I wanted to see if there was upstream interest in this and/or if there were reasons to prefer the current implementation.

Thanks, and best regards,
Joe


[-- Attachment #2: Type: text/html, Size: 4719 bytes --]

Re: Issue with time going backwards on OSX

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2864 bytes --]

On Tue, Sep 28, 2021 at 8:48 AM Joe Tanen <jtanen@fb.com> wrote:

> Hello,
>
>
>
> I posted this on qemu-discuss, but it doesn't seem like anybody else has
> seen this issue. I figured I'd try here to see if anybody had any thoughts.
>
>
>
> I'm using qemu-system-aarch64 v5.2 on OSX 11.6. While reading cntvct_el0,
> I've seen the ticks count go backwards. I dug into the qemu source a bit,
> and this register is eventually backed by a call to
> clock_gettime(CLOCK_MONOTONIC), which should not go backwards. This code
> seems to be the same on master as the branch I'm on.
>
>
>
> Before I started delving into debugging qemu, I wanted to verify that this
> wasn't an OS issue. I wrote a simple test that read CLOCK_MONOTONIC every
> 100ms, compared the values to make sure they always increased, and let it
> run overnight. Lo and behold, I saw time go backwards. I ran a concurrent
> test with CLOCK_MONOTONIC_RAW, and I did not see time go backwards.
> Successive reads of the CLOCK_MONOTONIC after it jumped backwards tracked
> the new time, so the jump wasn't a spurious error. When I ran this test on
> a Fedora 34 PC, I never saw time go backwards.
>
>
>
> Someone suggested as a test disabling networking so the NTP daemon
> couldn't mess with the clock, but that's not something I can do IRL. In any
> case, my understanding is that CLOCK_MONOTONIC should never go backwards
> and that any adjustments are made by temporarily slowing down or speeding
> up the tick rate. I'm going to pursue what I believe to be the underlying
> issue separately, but my qemu questions are these:
>
>
>
> - Has anybody else seen this or other timer registers go backwards under
> these or other conditions? I only see the behavior on OSX 11.6, but I'm a
> small sample size.
>
>
>
> - Would it make sense to use CLOCK_MONOTONIC_RAW (if available) instead of
> CLOCK_MONOTONIC in qemu? My issue aside, CLOCK_MONOTONIC_RAW feels closer
> to giving the guest direct access to a hardware clock. e.g., if I wanted to
> write my own NTP time adjustment daemon, I'd prefer to have an unadjusted
> clock. The code already checks to see if CLOCK_MONOTONIC is available, and,
> if not, it reverts to a simple gettimeofday(). It was easy to add another
> check to init_get_clock() to add a case checking for the availability of
> CLOCK_MONOTONIC_RAW. I've tested this myself, but I wanted to see if there
> was upstream interest in this and/or if there were reasons to prefer the
> current implementation.
>

Yea, CLOCK_MONOTONIC going backwards violates POSIX's description of it not
going backwards... Some operating systems with the not going backwards
property provide CLOCK_MONOTONIC_RAW as an alias for CLOCK_MONOTONIC, so
whatever you do, please make sure it works in this situation, or you'll
break at least FreeBSD...

Warner



> Thanks, and best regards,
>
> Joe
>
>
>

[-- Attachment #2: Type: text/html, Size: 4747 bytes --]