Domain-Virtual time

Domain-Virtual time

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 555 bytes --]

Sorry for multiple emails. I sent the last one from the wrong address.

Can anyone please tell me if the value returned by a time querying
instruction like gettimeofday() on a Xen (Linux) HVM is the true (System)
time or the Domain-virtual time?

PS: Domain virtual time is defined as the time that progresses at the same
pace as cycle counter, but only while a domain is executing. It stops while
the domain is de-scheduled where as System time accurately reflects the
passage of real time.

I am facing issues because my HVMs show a time drift.

Thanks

[-- Attachment #1.2: Type: text/html, Size: 585 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

RE: [Xen-devel] Domain-Virtual time

[Dan Magenheimer]

Should be "true" system time, i.e. should be very close to what
you see on a "wallclock" (clock on the wall).

HVM's are sadly very widely varied in the parameters needed
to minimize time drift.  In general in the past, timer_mode=0
(or timer_mode unspecified) would be best for 32-bit Linux
domains, timer_mode=1 would be best for Windows domains,
and timer_mode=2 would be best for 64-bit Linux domains.
However, for best results on Linux, this must be combined with
kernel boot parameters that properly select a clock -- and
on some Linux kernel versions, the parameters needed are
different between 32-bit and 64-bit versions of the same
kernel version.  It is up to providers of HVM templates
(aka "appliances") to choose parameters wisely.

Also, you haven't specified your Xen version, but I believe
Xen 4.0 switches the timer_mode default from 0 to 1 so, sadly,
clock behavior may change when moving an unchanged HVM
domain from pre-4.0 to 4.0.

So for best results you should run ntpd in any Linux HVM
domain (and I don't know what you do in Windows).  But
even ntpd may be inadequate to avoid drift if poor parameters
are chosen.

==========

From: Priya [mailto:pbhat@acis.ufl.edu] 
Sent: Thursday, February 25, 2010 9:04 AM
To: xen-devel@lists.xensource.com; xen-users@lists.xensource.com
Subject: [Xen-devel] Domain-Virtual time

Sorry for multiple emails. I sent the last one from the wrong address.

Can anyone please tell me if the value returned by a time querying instruction like gettimeofday() on a Xen (Linux) HVM is the true (System) time or the Domain-virtual time?

PS: Domain virtual time is defined as the time that progresses at the same pace as cycle counter, but only while a domain is executing. It stops while the domain is de-scheduled where as System time accurately reflects the passage of real time.

I am facing issues because my HVMs show a time drift. 

Thanks

Re: [Xen-devel] Domain-Virtual time

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 3195 bytes --]

Thanks Dan! That's a lot of useful information.

Since yesterday, I started NTP on my machines to correct and measure the
amount of drift. (I am running 3 HVMs with a Ubuntu 8.04 Linux (tick-less)
kernel which were all installed in an identical manner. At the time of
installing the HVMs I did not change the default timer_mode).

The funny thing is that NTP is measuring a very different drift on my three
machines (-189.206, -108.373 and -71.321 parts per million). The drift
reported on Domain-0 is -11.393. So I don't think my machines are showing
the system time.

In addition, the negative sign on the drift means that my machines are
running *faster* that the real time, which is again puzzling. I found out
that VMWare has issues with overcompensation on its linux kernels that cause
the VM time to run faster. Could Xen be having a similar problem ?

The fact that all three on my machines are showing different drifts makes me
doubt that they are showing the system time. I checked the CPU frequency
that the three machines are reporting (from /proc/cpuinfo) and they are
similar but not identical.

Any thoughts?

On Thu, Feb 25, 2010 at 3:36 PM, Dan Magenheimer <dan.magenheimer@oracle.com
> wrote:

> Should be "true" system time, i.e. should be very close to what
> you see on a "wallclock" (clock on the wall).
>
> HVM's are sadly very widely varied in the parameters needed
> to minimize time drift.  In general in the past, timer_mode=0
> (or timer_mode unspecified) would be best for 32-bit Linux
> domains, timer_mode=1 would be best for Windows domains,
> and timer_mode=2 would be best for 64-bit Linux domains.
> However, for best results on Linux, this must be combined with
> kernel boot parameters that properly select a clock -- and
> on some Linux kernel versions, the parameters needed are
> different between 32-bit and 64-bit versions of the same
> kernel version.  It is up to providers of HVM templates
> (aka "appliances") to choose parameters wisely.
>
> Also, you haven't specified your Xen version, but I believe
> Xen 4.0 switches the timer_mode default from 0 to 1 so, sadly,
> clock behavior may change when moving an unchanged HVM
> domain from pre-4.0 to 4.0.
>
> So for best results you should run ntpd in any Linux HVM
> domain (and I don't know what you do in Windows).  But
> even ntpd may be inadequate to avoid drift if poor parameters
> are chosen.
>
> ==========
>
> From: Priya [mailto:pbhat@acis.ufl.edu]
> Sent: Thursday, February 25, 2010 9:04 AM
> To: xen-devel@lists.xensource.com; xen-users@lists.xensource.com
> Subject: [Xen-devel] Domain-Virtual time
>
> Sorry for multiple emails. I sent the last one from the wrong address.
>
> Can anyone please tell me if the value returned by a time querying
> instruction like gettimeofday() on a Xen (Linux) HVM is the true (System)
> time or the Domain-virtual time?
>
> PS: Domain virtual time is defined as the time that progresses at the same
> pace as cycle counter, but only while a domain is executing. It stops while
> the domain is de-scheduled where as System time accurately reflects the
> passage of real time.
>
> I am facing issues because my HVMs show a time drift.
>
> Thanks
>
>

[-- Attachment #1.2: Type: text/html, Size: 3890 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-devel] Domain-Virtual time

[Tim Deegan]

Hi, 

At 16:46 +0000 on 26 Feb (1267202798), Priya wrote:
> The funny thing is that NTP is measuring a very different drift on my
> three machines (-189.206, -108.373 and -71.321 parts per million). The
> drift reported on Domain-0 is -11.393. So I don't think my machines
> are showing the system time.

No; they are showing system time modified according to the timer_mode
setting, then extrapolated into a collection of virtual timers and
reconstituted by the linux kernel.

I don't know a lot about linux HVM because I always run Linux with PV
kernels, but on Windows I've found that the (virtual) ACPI PM timer is a
better time-source than the HPET or RTC.

> In addition, the negative sign on the drift means that my machines are
> running faster that the real time, which is again puzzling.

Xen system time generally does drift forward, IIRC that's because xen
always tends to catch up to the fastest CPU, so cross-CPU jitter turns
into forward drift.  Linux may be doing the same thing inside the HVM
VM, where the jitter is higher.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: [Xen-devel] Domain-Virtual time

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 1814 bytes --]

Thanks Tim!

I have some follow-up questions:

   1. How can I check/change the timer_mode on a *XenServer* HVM and on a
   Xen HVM?
   2. How can I check/change the time-source on my machines?
   3.
>
>    No; they are showing system time modified according to the timer_mode
>    setting, then extrapolated into a collection of virtual timers and
>    reconstituted by the linux kernel.

   Can you please elaborate a little bit. Or please point me to a reference.


Thanks again !
Pr



On Fri, Feb 26, 2010 at 12:01 PM, Tim Deegan <Tim.Deegan@citrix.com> wrote:

> Hi,
>
> At 16:46 +0000 on 26 Feb (1267202798), Priya wrote:
> > The funny thing is that NTP is measuring a very different drift on my
> > three machines (-189.206, -108.373 and -71.321 parts per million). The
> > drift reported on Domain-0 is -11.393. So I don't think my machines
> > are showing the system time.
>
> No; they are showing system time modified according to the timer_mode
> setting, then extrapolated into a collection of virtual timers and
> reconstituted by the linux kernel.
>
> I don't know a lot about linux HVM because I always run Linux with PV
> kernels, but on Windows I've found that the (virtual) ACPI PM timer is a
> better time-source than the HPET or RTC.
>
> > In addition, the negative sign on the drift means that my machines are
> > running faster that the real time, which is again puzzling.
>
> Xen system time generally does drift forward, IIRC that's because xen
> always tends to catch up to the fastest CPU, so cross-CPU jitter turns
> into forward drift.  Linux may be doing the same thing inside the HVM
> VM, where the jitter is higher.
>
> Tim.
>
> --
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, XenServer Engineering
> Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
>
>

[-- Attachment #1.2: Type: text/html, Size: 2510 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

RE: Domain-Virtual time

[Dan Magenheimer]

Hi Priya --

You need only run additional domains (thus "overcommitting" the CPU(s) and guaranteeing that NO domain is getting close to an entire CPU) to demonstrate that it is certainly not domain time.  It IS system time but, as the VMware paper describes, virtualizing time on an older (buggy) OS is more of an art than a science.  Unless one builds directly into the hypervisor a complete list of all OS's (including all versions and patch-levels) and all of their bugs and idiosyncrasies, the result is a s*it-load of options at different levels of the virtualization stack (including hardware configuration such as in a boot-time BIOS menu).

As you may also have read, even on a physical machine running a native OS, time drift is very possible as time in every system is based on one or more inexpensive crystals with frequencies that are only estimates of a (extremely expensive) cesium clock (which for lack of a better term we will call "true" time) and may themselves drift relative to true time and relative to each other on the same system or even an apparently identical system.  This can often be demonstrated as you saw by checking /proc/cpuinfo... but identical values don't mean that the clocks are true, just that the difference is lower than the kernel code measuring it can discern or choose to report.

When virtualizing time, some of the problems manifest in time drift in the virtual system which moves faster than true time and some slower than true time.

NTP does its best to find a source of true time and then adjusts the clock on the system it is controlling so that it asymptotically approaches true time withOUT allowing time to go backwards (as this can cause all sorts of challenging system problems... imagine a "make" where time randomly goes backwards in the middle!).  But NTP may not have a good source for true time (as it is an administrator who configures it), NTP may subtly conflict with some combinations of the options set by administrators in the virtualization stack (because not everyone has access to "true" time... for example suppose your virtual machine has not networking configured), and NTP may silently give up if the drift is too bad for it to safely compensate.  (On a physical machine, this would be considered a "hardware bug" and you would ship your box back to the vendor to get one without a "broken" clock.)

So if you have ever heard the old pop song by Chicago "Does anybody really know what time it is?", the title is truer than most people believe ;-)

Hope that helps!
Dan

From: Priya [mailto:pbhat@acis.ufl.edu] 
Sent: Friday, February 26, 2010 9:47 AM
To: Dan Magenheimer
Cc: xen-devel@lists.xensource.com; xen-users@lists.xensource.com
Subject: Re: [Xen-devel] Domain-Virtual time

Thanks Dan! That's a lot of useful information. 

Since yesterday, I started NTP on my machines to correct and measure the amount of drift. (I am running 3 HVMs with a Ubuntu 8.04 Linux (tick-less) kernel which were all installed in an identical manner. At the time of installing the HVMs I did not change the default timer_mode).

The funny thing is that NTP is measuring a very different drift on my three machines (-189.206, -108.373 and -71.321 parts per million). The drift reported on Domain-0 is -11.393. So I don't think my machines are showing the system time. 

In addition, the negative sign on the drift means that my machines are running faster that the real time, which is again puzzling. I found out that VMWare has issues with overcompensation on its linux kernels that cause the VM time to run faster. Could Xen be having a similar problem ?

The fact that all three on my machines are showing different drifts makes me doubt that they are showing the system time. I checked the CPU frequency that the three machines are reporting (from /proc/cpuinfo) and they are similar but not identical.

Any thoughts?
On Thu, Feb 25, 2010 at 3:36 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
Should be "true" system time, i.e. should be very close to what
you see on a "wallclock" (clock on the wall).

HVM's are sadly very widely varied in the parameters needed
to minimize time drift.  In general in the past, timer_mode=0
(or timer_mode unspecified) would be best for 32-bit Linux
domains, timer_mode=1 would be best for Windows domains,
and timer_mode=2 would be best for 64-bit Linux domains.
However, for best results on Linux, this must be combined with
kernel boot parameters that properly select a clock -- and
on some Linux kernel versions, the parameters needed are
different between 32-bit and 64-bit versions of the same
kernel version.  It is up to providers of HVM templates
(aka "appliances") to choose parameters wisely.

Also, you haven't specified your Xen version, but I believe
Xen 4.0 switches the timer_mode default from 0 to 1 so, sadly,
clock behavior may change when moving an unchanged HVM
domain from pre-4.0 to 4.0.

So for best results you should run ntpd in any Linux HVM
domain (and I don't know what you do in Windows).  But
even ntpd may be inadequate to avoid drift if poor parameters
are chosen.

==========

From: Priya [mailto:pbhat@acis.ufl.edu]
Sent: Thursday, February 25, 2010 9:04 AM
To: xen-devel@lists.xensource.com; xen-users@lists.xensource.com
Subject: [Xen-devel] Domain-Virtual time

Sorry for multiple emails. I sent the last one from the wrong address.

Can anyone please tell me if the value returned by a time querying instruction like gettimeofday() on a Xen (Linux) HVM is the true (System) time or the Domain-virtual time?

PS: Domain virtual time is defined as the time that progresses at the same pace as cycle counter, but only while a domain is executing. It stops while the domain is de-scheduled where as System time accurately reflects the passage of real time.

I am facing issues because my HVMs show a time drift.

Thanks

RE: Domain-Virtual time

[Dan Magenheimer]

The timer_mode=X can be set in the "VM configuration" file
in which you specify things like the disk, memory, etc.

Setting the Xen time source on your machine is done by
adding clocksource=xxxxx to your Xen boot line, but unless
your system has one broken clocksource and another is fine,
this is rarely the source of extreme drift in an HVM.
(And, by the way, doing something like enabling HPET in
your BIOS may cause Xen to choose a different clocksource.
Xen tries to choose what it deems to be the best, and
the clocksource= is just a manual override.)

There are similar Linux kernel parameters that must
be specified within the boot-time grub file for the HVM.
These parameters vary from release to release (and
between 32-bit and 64-bit).  Many more recent kernels
allow you to change the clocksource dynamically
using a sysfs file.

Be aware that in some cases, a misspelled option name
or one that works fine on one version but not on the
currently booting Xen/kernel, may be completely and silently
ignored.

And finally, time has been a moving target on Xen,
Linux, VMware, etc, so problems that appear in older
versions may have been resolved in later versions.
ANY time you post questions/problems regarding time
(or for that matter even non-time-related problems),
it is best to describe what version of Xen and kernels
you are using.

======

From: Priya [mailto:pbhat@acis.ufl.edu] 
Sent: Friday, February 26, 2010 10:20 AM
To: Tim Deegan
Cc: Dan Magenheimer; xen-devel@lists.xensource.com; xen-users@lists.xensource.com
Subject: Re: [Xen-devel] Domain-Virtual time

Thanks Tim!

I have some follow-up questions:
1. How can I check/change the timer_mode on a XenServer HVM and on a Xen HVM?
2. How can I check/change the time-source on my machines?
3. No; they are showing system time modified according to the timer_mode
setting, then extrapolated into a collection of virtual timers and
reconstituted by the linux kernel.
Can you please elaborate a little bit. Or please point me to a reference. 
Thanks again !
Pr


On Fri, Feb 26, 2010 at 12:01 PM, Tim Deegan <Tim.Deegan@citrix.com> wrote:
Hi,

At 16:46 +0000 on 26 Feb (1267202798), Priya wrote:
> The funny thing is that NTP is measuring a very different drift on my
> three machines (-189.206, -108.373 and -71.321 parts per million). The
> drift reported on Domain-0 is -11.393. So I don't think my machines
> are showing the system time.
No; they are showing system time modified according to the timer_mode
setting, then extrapolated into a collection of virtual timers and
reconstituted by the linux kernel.

I don't know a lot about linux HVM because I always run Linux with PV
kernels, but on Windows I've found that the (virtual) ACPI PM timer is a
better time-source than the HPET or RTC.

> In addition, the negative sign on the drift means that my machines are
> running faster that the real time, which is again puzzling.
Xen system time generally does drift forward, IIRC that's because xen
always tends to catch up to the fastest CPU, so cross-CPU jitter turns
into forward drift.  Linux may be doing the same thing inside the HVM
VM, where the jitter is higher.

Tim.

--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Domain-Virtual time

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 3583 bytes --]

Thanks guys ! I am really grateful !

On Fri, Feb 26, 2010 at 12:46 PM, Dan Magenheimer <
dan.magenheimer@oracle.com> wrote:

> The timer_mode=X can be set in the "VM configuration" file
> in which you specify things like the disk, memory, etc.
>
> Setting the Xen time source on your machine is done by
> adding clocksource=xxxxx to your Xen boot line, but unless
> your system has one broken clocksource and another is fine,
> this is rarely the source of extreme drift in an HVM.
> (And, by the way, doing something like enabling HPET in
> your BIOS may cause Xen to choose a different clocksource.
> Xen tries to choose what it deems to be the best, and
> the clocksource= is just a manual override.)
>
> There are similar Linux kernel parameters that must
> be specified within the boot-time grub file for the HVM.
> These parameters vary from release to release (and
> between 32-bit and 64-bit).  Many more recent kernels
> allow you to change the clocksource dynamically
> using a sysfs file.
>
> Be aware that in some cases, a misspelled option name
> or one that works fine on one version but not on the
> currently booting Xen/kernel, may be completely and silently
> ignored.
>
> And finally, time has been a moving target on Xen,
> Linux, VMware, etc, so problems that appear in older
> versions may have been resolved in later versions.
> ANY time you post questions/problems regarding time
> (or for that matter even non-time-related problems),
> it is best to describe what version of Xen and kernels
> you are using.
>
> ======
>
> From: Priya [mailto:pbhat@acis.ufl.edu]
> Sent: Friday, February 26, 2010 10:20 AM
> To: Tim Deegan
> Cc: Dan Magenheimer; xen-devel@lists.xensource.com;
> xen-users@lists.xensource.com
> Subject: Re: [Xen-devel] Domain-Virtual time
>
> Thanks Tim!
>
> I have some follow-up questions:
> 1. How can I check/change the timer_mode on a XenServer HVM and on a Xen
> HVM?
> 2. How can I check/change the time-source on my machines?
> 3. No; they are showing system time modified according to the timer_mode
> setting, then extrapolated into a collection of virtual timers and
> reconstituted by the linux kernel.
> Can you please elaborate a little bit. Or please point me to a reference.
> Thanks again !
> Pr
>
>
> On Fri, Feb 26, 2010 at 12:01 PM, Tim Deegan <Tim.Deegan@citrix.com>
> wrote:
> Hi,
>
> At 16:46 +0000 on 26 Feb (1267202798), Priya wrote:
> > The funny thing is that NTP is measuring a very different drift on my
> > three machines (-189.206, -108.373 and -71.321 parts per million). The
> > drift reported on Domain-0 is -11.393. So I don't think my machines
> > are showing the system time.
> No; they are showing system time modified according to the timer_mode
> setting, then extrapolated into a collection of virtual timers and
> reconstituted by the linux kernel.
>
> I don't know a lot about linux HVM because I always run Linux with PV
> kernels, but on Windows I've found that the (virtual) ACPI PM timer is a
> better time-source than the HPET or RTC.
>
> > In addition, the negative sign on the drift means that my machines are
> > running faster that the real time, which is again puzzling.
> Xen system time generally does drift forward, IIRC that's because xen
> always tends to catch up to the fastest CPU, so cross-CPU jitter turns
> into forward drift.  Linux may be doing the same thing inside the HVM
> VM, where the jitter is higher.
>
> Tim.
>
> --
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, XenServer Engineering
> Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
>
>

[-- Attachment #1.2: Type: text/html, Size: 4477 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Domain-virtual time

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 375 bytes --]

Can anyone tell me if the value returned by gettimeofday() on a Xen (Linux)
HVM is the true System time or the Domain-virtual time?

PS: Domain virtual time is defined as the time progresses at the same pace
as cycle counter time, but only while a domain is executing. It stops while
the domain is de-scheduled.

I am facing issues because my HVMs show a time drift.

Thanks

[-- Attachment #1.2: Type: text/html, Size: 402 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel