Monotonic clock with KVM pv-clock

Monotonic clock with KVM pv-clock

[Nadav Har'El]

Hi,

I'm trying to figure out how a guest OS can get a monotonic clock using
KVM's paravirtual clock.

At first, I thought that the clock I get using KVM_SYSTEM_TIME is a
monotonic clock, based on the host's monotonic clock.
But I'm no longer sure this is actually done correctly:

1. What happens on live migration? On the new host, will the
   KVM_SYSTEM_TIME continue from the same point? Or will it jump because
   KVM_WALL_CLOCK (the wall time during boot) is different on the new host?

2. What happens when the wall-clock time is set on the host? I was
   hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't,
   but am no longer sure this is actually the case.

If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
from KVM, is there a correct way?

Thanks,
Nadav.

-- 
Nadav Har'El                        |      Monday, Jan 20 2014, 19 Shevat 5774
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Diplomat: A man who always remembers a
http://nadav.harel.org.il           |woman's birthday but never her age.

Re: Monotonic clock with KVM pv-clock

[Marcelo Tosatti]

On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
> Hi,
> 
> I'm trying to figure out how a guest OS can get a monotonic clock using
> KVM's paravirtual clock.
> 
> At first, I thought that the clock I get using KVM_SYSTEM_TIME is a
> monotonic clock, based on the host's monotonic clock.

It is. However, it is used in conjunction with TSC delta, part
of the structure which is written at KVM_SYSTEM_TIME GPA. See
pvclock_clocksource_read at arch/x86/kernel/pvclock.c.

> But I'm no longer sure this is actually done correctly:
> 
> 1. What happens on live migration? On the new host, will the
>    KVM_SYSTEM_TIME continue from the same point? 

Yes, from the same point where the VM was stopped on the source.

> Or will it jump because
>    KVM_WALL_CLOCK (the wall time during boot) is different on the new host?

No, it won't, see the kernel commit which introduces
KVM_SET_CLOCK/KVM_GET_CLOCK.

> 2. What happens when the wall-clock time is set on the host? I was
>    hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't,
>    but am no longer sure this is actually the case.

Yes, it is the case. The host clock which backs system_timestamp field
of pvclock structure is        

CLOCK_MONOTONIC
Clock  that  cannot be set and represents monotonic time
since some unspecified starting point.  This clock is not affected by
discontinuous jumps in the system time (e.g., if the system
administrator manually changes the clock), but is affected by the
incremental adjustments per‐formed by adjtime(3) and NTP.

> If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
> from KVM, is there a correct way?

Inside a Linux guest? Can use sched_clock().

If not a Linux guest, either implement kvmclock-like driver
(kvm-unit-test contains one).

> 
> Thanks,
> Nadav.
> 
> -- 
> Nadav Har'El                        |      Monday, Jan 20 2014, 19 Shevat 5774
> nyh@math.technion.ac.il             |-----------------------------------------
> Phone +972-523-790466, ICQ 13349191 |Diplomat: A man who always remembers a
> http://nadav.harel.org.il           |woman's birthday but never her age.

Re: Monotonic clock with KVM pv-clock

[Fernando Luis Vazquez Cao]

(2014/01/20 22:33), Marcelo Tosatti wrote:
> On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
>> If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
>> from KVM, is there a correct way?
> Inside a Linux guest? Can use sched_clock().

I would like to mention that Linux guests usually do not use sched_clock()
directly. The reason being that the kvm_clock based sched_clock() is not
marked stable (sched_clock_stable is 0), which means that the pair of
wrappers sched_clock_local()/sched_clock_remote() is used instead.

Re: Monotonic clock with KVM pv-clock

[Marcelo Tosatti]

On Mon, Jan 20, 2014 at 11:59:39PM +0900, Fernando Luis Vazquez Cao wrote:
> (2014/01/20 22:33), Marcelo Tosatti wrote:
> >On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
> >>If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
> >>from KVM, is there a correct way?
> >Inside a Linux guest? Can use sched_clock().
> 
> I would like to mention that Linux guests usually do not use sched_clock()
> directly. The reason being that the kvm_clock based sched_clock() is not
> marked stable (sched_clock_stable is 0), which means that the pair of
> wrappers sched_clock_local()/sched_clock_remote() is used instead.

Should verify the requirements of sched_clock_cpu() and enable
sched_clock_stable in case it fulfills requirements (kvmclock_read can
be nondecreasing due to TSC->nanosecond scaling, and not increase for a
longer duration with global accumulator, due to cmpxchg).

Re: Monotonic clock with KVM pv-clock

[Nadav Har'El]

On Mon, Jan 20, 2014, Marcelo Tosatti wrote about "Re: Monotonic clock with KVM pv-clock":
> On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
> > Hi,
> > 
> > I'm trying to figure out how a guest OS can get a monotonic clock using
> > KVM's paravirtual clock.
> > 
> > At first, I thought that the clock I get using KVM_SYSTEM_TIME is a
> > monotonic clock, based on the host's monotonic clock.
> 
> It is. However, it is used in conjunction with TSC delta, part
> of the structure which is written at KVM_SYSTEM_TIME GPA. See
> pvclock_clocksource_read at arch/x86/kernel/pvclock.c.

Sure, I'm using the whole protocol for KVM_SYSTEM_TIME. I just wondered
if it has a meaningful definition (namely, of being a monotonic clock) when
being used not in conjunction with the KVM_WALL_CLOCK protocol.

> > 2. What happens when the wall-clock time is set on the host? I was
> >    hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't,
> >    but am no longer sure this is actually the case.
> 
> Yes, it is the case. The host clock which backs system_timestamp field
> of pvclock structure is        
> 
> CLOCK_MONOTONIC
> Clock  that  cannot be set and represents monotonic time
> since some unspecified starting point.  This clock is not affected by
> discontinuous jumps in the system time (e.g., if the system
> administrator manually changes the clock), but is affected by the
> incremental adjustments per???formed by adjtime(3) and NTP.

Excellent.

> 
> > If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
> > from KVM, is there a correct way?
> 
> Inside a Linux guest? Can use sched_clock().
> 
> If not a Linux guest, either implement kvmclock-like driver
> (kvm-unit-test contains one).

This is actually an OSv guest (http://osv.io/).

Until now we were using kvmclock by adding up the wallclock and
systemtime protocol, and this resulted with a good wall-time clock
(CLOCK_REALTIME), but I really wanted to also have a monotonic clock
(CLOCK_MONOTONIC) for the guest.

I took a look and how a Linux guest uses the KVM pv clock, and thought
that the "system time" part of the kvm clock protocol will be a good
monotonic clock, but wasn't sure I was understanding this right.

Thanks for the clarifications!

Nadav.

-- 
Nadav Har'El                        |     Tuesday, Jan 21 2014, 20 Shevat 5774
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |"The average person thinks he isn't." -
http://nadav.harel.org.il           |Father Larry Lorenzoni

Re: Monotonic clock with KVM pv-clock

[Fernando Luis Vazquez Cao]

(2014/01/21 9:19), Marcelo Tosatti wrote:
> On Mon, Jan 20, 2014 at 11:59:39PM +0900, Fernando Luis Vazquez Cao wrote:
>> (2014/01/20 22:33), Marcelo Tosatti wrote:
>>> On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
>>>> If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
>>> >from KVM, is there a correct way?
>>> Inside a Linux guest? Can use sched_clock().
>> I would like to mention that Linux guests usually do not use sched_clock()
>> directly. The reason being that the kvm_clock based sched_clock() is not
>> marked stable (sched_clock_stable is 0), which means that the pair of
>> wrappers sched_clock_local()/sched_clock_remote() is used instead.
> Should verify the requirements of sched_clock_cpu() and enable
> sched_clock_stable in case it fulfills requirements

The requirements of cpu_clock() are (from code comments in 
kernel/sched/clock.c):

1. cpu_clock(i) provides a fast (execution time) high resolution
2. Clock with bounded drift between CPUs.
3. The value of cpu_clock(i) is monotonic for constant i (when comparing 
cpu_clock(i)
    to cpu_clock(j) for i != j, time can go backwards)
4. The timestamp returned is in nanoseconds.

Besides, for sched_clock() to be considered stable

5. it has to be synchronized across CPUs.


> (kvmclock_read can be nondecreasing due to TSC->nanosecond scaling,

s/nondecreasing/nonincreasing/? I guess you mean that consecutive calls to
kvm_clock_read() can return the same nanosecond value when the TSC
frequency is greater than 1GHz or there are frequency changes.


> and not increase for a longer duration with global accumulator, due
> to cmpxchg)

Yes. If I understand correctly, that can happen if 
PVCLOCK_TSC_STABLE_BIT is not set.

I think that when PVCLOCK_TSC_STABLE_BIT is set we should be fine as
long as backward motion is filtered out.

Re: Monotonic clock with KVM pv-clock

[Marcelo Tosatti]

On Tue, Jan 21, 2014 at 03:24:43PM +0200, Nadav Har'El wrote:
> On Mon, Jan 20, 2014, Marcelo Tosatti wrote about "Re: Monotonic clock with KVM pv-clock":
> > On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
> > > Hi,
> > > 
> > > I'm trying to figure out how a guest OS can get a monotonic clock using
> > > KVM's paravirtual clock.
> > > 
> > > At first, I thought that the clock I get using KVM_SYSTEM_TIME is a
> > > monotonic clock, based on the host's monotonic clock.
> > 
> > It is. However, it is used in conjunction with TSC delta, part
> > of the structure which is written at KVM_SYSTEM_TIME GPA. See
> > pvclock_clocksource_read at arch/x86/kernel/pvclock.c.
> 
> Sure, I'm using the whole protocol for KVM_SYSTEM_TIME. I just wondered
> if it has a meaningful definition (namely, of being a monotonic clock) when
> being used not in conjunction with the KVM_WALL_CLOCK protocol.

If flags field contains PVCLOCK_TSC_STABLE_BIT, yes, otherwise, no.

Should use a global accumulator with cmpxchg if PVCLOCK_TSC_STABLE_BIT
is clear.

> > > 2. What happens when the wall-clock time is set on the host? I was
> > >    hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't,
> > >    but am no longer sure this is actually the case.
> > 
> > Yes, it is the case. The host clock which backs system_timestamp field
> > of pvclock structure is        
> > 
> > CLOCK_MONOTONIC
> > Clock  that  cannot be set and represents monotonic time
> > since some unspecified starting point.  This clock is not affected by
> > discontinuous jumps in the system time (e.g., if the system
> > administrator manually changes the clock), but is affected by the
> > incremental adjustments per???formed by adjtime(3) and NTP.
> 
> Excellent.
> 
> > 
> > > If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
> > > from KVM, is there a correct way?
> > 
> > Inside a Linux guest? Can use sched_clock().
> > 
> > If not a Linux guest, either implement kvmclock-like driver
> > (kvm-unit-test contains one).
> 
> This is actually an OSv guest (http://osv.io/).
> 
> Until now we were using kvmclock by adding up the wallclock and
> systemtime protocol, and this resulted with a good wall-time clock
> (CLOCK_REALTIME), but I really wanted to also have a monotonic clock
> (CLOCK_MONOTONIC) for the guest.
> 
> I took a look and how a Linux guest uses the KVM pv clock, and thought
> that the "system time" part of the kvm clock protocol will be a good
> monotonic clock, but wasn't sure I was understanding this right.
> 
> Thanks for the clarifications!
> 
> Nadav.
> 
> -- 
> Nadav Har'El                        |     Tuesday, Jan 21 2014, 20 Shevat 5774
> nyh@math.technion.ac.il             |-----------------------------------------
> Phone +972-523-790466, ICQ 13349191 |"The average person thinks he isn't." -
> http://nadav.harel.org.il           |Father Larry Lorenzoni

Re: Monotonic clock with KVM pv-clock

[Marcelo Tosatti]

On Tue, Jan 21, 2014 at 11:23:37PM +0900, Fernando Luis Vazquez Cao wrote:
> (2014/01/21 9:19), Marcelo Tosatti wrote:
> >On Mon, Jan 20, 2014 at 11:59:39PM +0900, Fernando Luis Vazquez Cao wrote:
> >>(2014/01/20 22:33), Marcelo Tosatti wrote:
> >>>On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote:
> >>>>If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock
> >>>>from KVM, is there a correct way?
> >>>Inside a Linux guest? Can use sched_clock().
> >>I would like to mention that Linux guests usually do not use sched_clock()
> >>directly. The reason being that the kvm_clock based sched_clock() is not
> >>marked stable (sched_clock_stable is 0), which means that the pair of
> >>wrappers sched_clock_local()/sched_clock_remote() is used instead.
> >Should verify the requirements of sched_clock_cpu() and enable
> >sched_clock_stable in case it fulfills requirements
> 
> The requirements of cpu_clock() are (from code comments in
> kernel/sched/clock.c):
> 
> 1. cpu_clock(i) provides a fast (execution time) high resolution
> 2. Clock with bounded drift between CPUs.

How high of a resolution ? Or what is the effect of lowering resolution?

> 3. The value of cpu_clock(i) is monotonic for constant i (when
> comparing cpu_clock(i)
>    to cpu_clock(j) for i != j, time can go backwards)

OK.

> 4. The timestamp returned is in nanoseconds.

OK.

> Besides, for sched_clock() to be considered stable
> 
> 5. it has to be synchronized across CPUs.

This contradicts item 3 ?

> >(kvmclock_read can be nondecreasing due to TSC->nanosecond scaling,
> 
> s/nondecreasing/nonincreasing/? 

Yes.

> I guess you mean that consecutive calls to
> kvm_clock_read() can return the same nanosecond value when the TSC
> frequency is greater than 1GHz or there are frequency changes.

Yes, when CPU can execute more than one kvm_clock_read's in one
nanosecond.

> >and not increase for a longer duration with global accumulator, due
> >to cmpxchg)
> 
> Yes. If I understand correctly, that can happen if
> PVCLOCK_TSC_STABLE_BIT is not set.

Yes.

> I think that when PVCLOCK_TSC_STABLE_BIT is set we should be fine as
> long as backward motion is filtered out.

There can be no backward motion with PVCLOCK_TSC_STABLE_BIT (in theory,
and no reports have been seen yet).

Do you have any numbers for the change?