* Monotonic clock with KVM pv-clock @ 2014-01-20 9:56 Nadav Har'El 2014-01-20 13:33 ` Marcelo Tosatti 0 siblings, 1 reply; 8+ messages in thread From: Nadav Har'El @ 2014-01-20 9:56 UTC (permalink / raw) To: mtosatti, kvm Hi, I'm trying to figure out how a guest OS can get a monotonic clock using KVM's paravirtual clock. At first, I thought that the clock I get using KVM_SYSTEM_TIME is a monotonic clock, based on the host's monotonic clock. But I'm no longer sure this is actually done correctly: 1. What happens on live migration? On the new host, will the KVM_SYSTEM_TIME continue from the same point? Or will it jump because KVM_WALL_CLOCK (the wall time during boot) is different on the new host? 2. What happens when the wall-clock time is set on the host? I was hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't, but am no longer sure this is actually the case. If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock from KVM, is there a correct way? Thanks, Nadav. -- Nadav Har'El | Monday, Jan 20 2014, 19 Shevat 5774 nyh@math.technion.ac.il |----------------------------------------- Phone +972-523-790466, ICQ 13349191 |Diplomat: A man who always remembers a http://nadav.harel.org.il |woman's birthday but never her age. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-20 9:56 Monotonic clock with KVM pv-clock Nadav Har'El @ 2014-01-20 13:33 ` Marcelo Tosatti 2014-01-20 14:59 ` Fernando Luis Vazquez Cao 2014-01-21 13:24 ` Nadav Har'El 0 siblings, 2 replies; 8+ messages in thread From: Marcelo Tosatti @ 2014-01-20 13:33 UTC (permalink / raw) To: Nadav Har'El; +Cc: kvm On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: > Hi, > > I'm trying to figure out how a guest OS can get a monotonic clock using > KVM's paravirtual clock. > > At first, I thought that the clock I get using KVM_SYSTEM_TIME is a > monotonic clock, based on the host's monotonic clock. It is. However, it is used in conjunction with TSC delta, part of the structure which is written at KVM_SYSTEM_TIME GPA. See pvclock_clocksource_read at arch/x86/kernel/pvclock.c. > But I'm no longer sure this is actually done correctly: > > 1. What happens on live migration? On the new host, will the > KVM_SYSTEM_TIME continue from the same point? Yes, from the same point where the VM was stopped on the source. > Or will it jump because > KVM_WALL_CLOCK (the wall time during boot) is different on the new host? No, it won't, see the kernel commit which introduces KVM_SET_CLOCK/KVM_GET_CLOCK. > 2. What happens when the wall-clock time is set on the host? I was > hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't, > but am no longer sure this is actually the case. Yes, it is the case. The host clock which backs system_timestamp field of pvclock structure is CLOCK_MONOTONIC Clock that cannot be set and represents monotonic time since some unspecified starting point. This clock is not affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the clock), but is affected by the incremental adjustments per‐formed by adjtime(3) and NTP. > If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock > from KVM, is there a correct way? Inside a Linux guest? Can use sched_clock(). If not a Linux guest, either implement kvmclock-like driver (kvm-unit-test contains one). > > Thanks, > Nadav. > > -- > Nadav Har'El | Monday, Jan 20 2014, 19 Shevat 5774 > nyh@math.technion.ac.il |----------------------------------------- > Phone +972-523-790466, ICQ 13349191 |Diplomat: A man who always remembers a > http://nadav.harel.org.il |woman's birthday but never her age. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-20 13:33 ` Marcelo Tosatti @ 2014-01-20 14:59 ` Fernando Luis Vazquez Cao 2014-01-21 0:19 ` Marcelo Tosatti 2014-01-21 13:24 ` Nadav Har'El 1 sibling, 1 reply; 8+ messages in thread From: Fernando Luis Vazquez Cao @ 2014-01-20 14:59 UTC (permalink / raw) To: Marcelo Tosatti, Nadav Har'El; +Cc: kvm (2014/01/20 22:33), Marcelo Tosatti wrote: > On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: >> If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock >> from KVM, is there a correct way? > Inside a Linux guest? Can use sched_clock(). I would like to mention that Linux guests usually do not use sched_clock() directly. The reason being that the kvm_clock based sched_clock() is not marked stable (sched_clock_stable is 0), which means that the pair of wrappers sched_clock_local()/sched_clock_remote() is used instead. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-20 14:59 ` Fernando Luis Vazquez Cao @ 2014-01-21 0:19 ` Marcelo Tosatti 2014-01-21 14:23 ` Fernando Luis Vazquez Cao 0 siblings, 1 reply; 8+ messages in thread From: Marcelo Tosatti @ 2014-01-21 0:19 UTC (permalink / raw) To: Fernando Luis Vazquez Cao; +Cc: Nadav Har'El, kvm On Mon, Jan 20, 2014 at 11:59:39PM +0900, Fernando Luis Vazquez Cao wrote: > (2014/01/20 22:33), Marcelo Tosatti wrote: > >On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: > >>If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock > >>from KVM, is there a correct way? > >Inside a Linux guest? Can use sched_clock(). > > I would like to mention that Linux guests usually do not use sched_clock() > directly. The reason being that the kvm_clock based sched_clock() is not > marked stable (sched_clock_stable is 0), which means that the pair of > wrappers sched_clock_local()/sched_clock_remote() is used instead. Should verify the requirements of sched_clock_cpu() and enable sched_clock_stable in case it fulfills requirements (kvmclock_read can be nondecreasing due to TSC->nanosecond scaling, and not increase for a longer duration with global accumulator, due to cmpxchg). ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-21 0:19 ` Marcelo Tosatti @ 2014-01-21 14:23 ` Fernando Luis Vazquez Cao 2014-01-22 15:41 ` Marcelo Tosatti 0 siblings, 1 reply; 8+ messages in thread From: Fernando Luis Vazquez Cao @ 2014-01-21 14:23 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: Nadav Har'El, kvm (2014/01/21 9:19), Marcelo Tosatti wrote: > On Mon, Jan 20, 2014 at 11:59:39PM +0900, Fernando Luis Vazquez Cao wrote: >> (2014/01/20 22:33), Marcelo Tosatti wrote: >>> On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: >>>> If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock >>> >from KVM, is there a correct way? >>> Inside a Linux guest? Can use sched_clock(). >> I would like to mention that Linux guests usually do not use sched_clock() >> directly. The reason being that the kvm_clock based sched_clock() is not >> marked stable (sched_clock_stable is 0), which means that the pair of >> wrappers sched_clock_local()/sched_clock_remote() is used instead. > Should verify the requirements of sched_clock_cpu() and enable > sched_clock_stable in case it fulfills requirements The requirements of cpu_clock() are (from code comments in kernel/sched/clock.c): 1. cpu_clock(i) provides a fast (execution time) high resolution 2. Clock with bounded drift between CPUs. 3. The value of cpu_clock(i) is monotonic for constant i (when comparing cpu_clock(i) to cpu_clock(j) for i != j, time can go backwards) 4. The timestamp returned is in nanoseconds. Besides, for sched_clock() to be considered stable 5. it has to be synchronized across CPUs. > (kvmclock_read can be nondecreasing due to TSC->nanosecond scaling, s/nondecreasing/nonincreasing/? I guess you mean that consecutive calls to kvm_clock_read() can return the same nanosecond value when the TSC frequency is greater than 1GHz or there are frequency changes. > and not increase for a longer duration with global accumulator, due > to cmpxchg) Yes. If I understand correctly, that can happen if PVCLOCK_TSC_STABLE_BIT is not set. I think that when PVCLOCK_TSC_STABLE_BIT is set we should be fine as long as backward motion is filtered out. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-21 14:23 ` Fernando Luis Vazquez Cao @ 2014-01-22 15:41 ` Marcelo Tosatti 0 siblings, 0 replies; 8+ messages in thread From: Marcelo Tosatti @ 2014-01-22 15:41 UTC (permalink / raw) To: Fernando Luis Vazquez Cao; +Cc: Nadav Har'El, kvm On Tue, Jan 21, 2014 at 11:23:37PM +0900, Fernando Luis Vazquez Cao wrote: > (2014/01/21 9:19), Marcelo Tosatti wrote: > >On Mon, Jan 20, 2014 at 11:59:39PM +0900, Fernando Luis Vazquez Cao wrote: > >>(2014/01/20 22:33), Marcelo Tosatti wrote: > >>>On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: > >>>>If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock > >>>>from KVM, is there a correct way? > >>>Inside a Linux guest? Can use sched_clock(). > >>I would like to mention that Linux guests usually do not use sched_clock() > >>directly. The reason being that the kvm_clock based sched_clock() is not > >>marked stable (sched_clock_stable is 0), which means that the pair of > >>wrappers sched_clock_local()/sched_clock_remote() is used instead. > >Should verify the requirements of sched_clock_cpu() and enable > >sched_clock_stable in case it fulfills requirements > > The requirements of cpu_clock() are (from code comments in > kernel/sched/clock.c): > > 1. cpu_clock(i) provides a fast (execution time) high resolution > 2. Clock with bounded drift between CPUs. How high of a resolution ? Or what is the effect of lowering resolution? > 3. The value of cpu_clock(i) is monotonic for constant i (when > comparing cpu_clock(i) > to cpu_clock(j) for i != j, time can go backwards) OK. > 4. The timestamp returned is in nanoseconds. OK. > Besides, for sched_clock() to be considered stable > > 5. it has to be synchronized across CPUs. This contradicts item 3 ? > >(kvmclock_read can be nondecreasing due to TSC->nanosecond scaling, > > s/nondecreasing/nonincreasing/? Yes. > I guess you mean that consecutive calls to > kvm_clock_read() can return the same nanosecond value when the TSC > frequency is greater than 1GHz or there are frequency changes. Yes, when CPU can execute more than one kvm_clock_read's in one nanosecond. > >and not increase for a longer duration with global accumulator, due > >to cmpxchg) > > Yes. If I understand correctly, that can happen if > PVCLOCK_TSC_STABLE_BIT is not set. Yes. > I think that when PVCLOCK_TSC_STABLE_BIT is set we should be fine as > long as backward motion is filtered out. There can be no backward motion with PVCLOCK_TSC_STABLE_BIT (in theory, and no reports have been seen yet). Do you have any numbers for the change? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-20 13:33 ` Marcelo Tosatti 2014-01-20 14:59 ` Fernando Luis Vazquez Cao @ 2014-01-21 13:24 ` Nadav Har'El 2014-01-21 15:27 ` Marcelo Tosatti 1 sibling, 1 reply; 8+ messages in thread From: Nadav Har'El @ 2014-01-21 13:24 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: kvm On Mon, Jan 20, 2014, Marcelo Tosatti wrote about "Re: Monotonic clock with KVM pv-clock": > On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: > > Hi, > > > > I'm trying to figure out how a guest OS can get a monotonic clock using > > KVM's paravirtual clock. > > > > At first, I thought that the clock I get using KVM_SYSTEM_TIME is a > > monotonic clock, based on the host's monotonic clock. > > It is. However, it is used in conjunction with TSC delta, part > of the structure which is written at KVM_SYSTEM_TIME GPA. See > pvclock_clocksource_read at arch/x86/kernel/pvclock.c. Sure, I'm using the whole protocol for KVM_SYSTEM_TIME. I just wondered if it has a meaningful definition (namely, of being a monotonic clock) when being used not in conjunction with the KVM_WALL_CLOCK protocol. > > 2. What happens when the wall-clock time is set on the host? I was > > hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't, > > but am no longer sure this is actually the case. > > Yes, it is the case. The host clock which backs system_timestamp field > of pvclock structure is > > CLOCK_MONOTONIC > Clock that cannot be set and represents monotonic time > since some unspecified starting point. This clock is not affected by > discontinuous jumps in the system time (e.g., if the system > administrator manually changes the clock), but is affected by the > incremental adjustments per???formed by adjtime(3) and NTP. Excellent. > > > If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock > > from KVM, is there a correct way? > > Inside a Linux guest? Can use sched_clock(). > > If not a Linux guest, either implement kvmclock-like driver > (kvm-unit-test contains one). This is actually an OSv guest (http://osv.io/). Until now we were using kvmclock by adding up the wallclock and systemtime protocol, and this resulted with a good wall-time clock (CLOCK_REALTIME), but I really wanted to also have a monotonic clock (CLOCK_MONOTONIC) for the guest. I took a look and how a Linux guest uses the KVM pv clock, and thought that the "system time" part of the kvm clock protocol will be a good monotonic clock, but wasn't sure I was understanding this right. Thanks for the clarifications! Nadav. -- Nadav Har'El | Tuesday, Jan 21 2014, 20 Shevat 5774 nyh@math.technion.ac.il |----------------------------------------- Phone +972-523-790466, ICQ 13349191 |"The average person thinks he isn't." - http://nadav.harel.org.il |Father Larry Lorenzoni ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Monotonic clock with KVM pv-clock 2014-01-21 13:24 ` Nadav Har'El @ 2014-01-21 15:27 ` Marcelo Tosatti 0 siblings, 0 replies; 8+ messages in thread From: Marcelo Tosatti @ 2014-01-21 15:27 UTC (permalink / raw) To: Nadav Har'El; +Cc: kvm On Tue, Jan 21, 2014 at 03:24:43PM +0200, Nadav Har'El wrote: > On Mon, Jan 20, 2014, Marcelo Tosatti wrote about "Re: Monotonic clock with KVM pv-clock": > > On Mon, Jan 20, 2014 at 11:56:56AM +0200, Nadav Har'El wrote: > > > Hi, > > > > > > I'm trying to figure out how a guest OS can get a monotonic clock using > > > KVM's paravirtual clock. > > > > > > At first, I thought that the clock I get using KVM_SYSTEM_TIME is a > > > monotonic clock, based on the host's monotonic clock. > > > > It is. However, it is used in conjunction with TSC delta, part > > of the structure which is written at KVM_SYSTEM_TIME GPA. See > > pvclock_clocksource_read at arch/x86/kernel/pvclock.c. > > Sure, I'm using the whole protocol for KVM_SYSTEM_TIME. I just wondered > if it has a meaningful definition (namely, of being a monotonic clock) when > being used not in conjunction with the KVM_WALL_CLOCK protocol. If flags field contains PVCLOCK_TSC_STABLE_BIT, yes, otherwise, no. Should use a global accumulator with cmpxchg if PVCLOCK_TSC_STABLE_BIT is clear. > > > 2. What happens when the wall-clock time is set on the host? I was > > > hoping that only KVM_WALL_CLOCK changes and KVM_SYSTEM_TIME doesn't, > > > but am no longer sure this is actually the case. > > > > Yes, it is the case. The host clock which backs system_timestamp field > > of pvclock structure is > > > > CLOCK_MONOTONIC > > Clock that cannot be set and represents monotonic time > > since some unspecified starting point. This clock is not affected by > > discontinuous jumps in the system time (e.g., if the system > > administrator manually changes the clock), but is affected by the > > incremental adjustments per???formed by adjtime(3) and NTP. > > Excellent. > > > > > > If KVM_SYSTEM_TIME is not a correct way to get a monotonic paravirtual clock > > > from KVM, is there a correct way? > > > > Inside a Linux guest? Can use sched_clock(). > > > > If not a Linux guest, either implement kvmclock-like driver > > (kvm-unit-test contains one). > > This is actually an OSv guest (http://osv.io/). > > Until now we were using kvmclock by adding up the wallclock and > systemtime protocol, and this resulted with a good wall-time clock > (CLOCK_REALTIME), but I really wanted to also have a monotonic clock > (CLOCK_MONOTONIC) for the guest. > > I took a look and how a Linux guest uses the KVM pv clock, and thought > that the "system time" part of the kvm clock protocol will be a good > monotonic clock, but wasn't sure I was understanding this right. > > Thanks for the clarifications! > > Nadav. > > -- > Nadav Har'El | Tuesday, Jan 21 2014, 20 Shevat 5774 > nyh@math.technion.ac.il |----------------------------------------- > Phone +972-523-790466, ICQ 13349191 |"The average person thinks he isn't." - > http://nadav.harel.org.il |Father Larry Lorenzoni ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-01-22 16:21 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-01-20 9:56 Monotonic clock with KVM pv-clock Nadav Har'El 2014-01-20 13:33 ` Marcelo Tosatti 2014-01-20 14:59 ` Fernando Luis Vazquez Cao 2014-01-21 0:19 ` Marcelo Tosatti 2014-01-21 14:23 ` Fernando Luis Vazquez Cao 2014-01-22 15:41 ` Marcelo Tosatti 2014-01-21 13:24 ` Nadav Har'El 2014-01-21 15:27 ` Marcelo Tosatti
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.