Re: [Xen-users] Xen timekeeping best practices

Re: [Xen-users] Xen timekeeping best practices

[Todd Deshane]

Adding Xen-devel.

What are the current timekeeping best practices now?

Thanks,
Todd

---------- Forwarded message ----------
From: Remi Gacogne <rgacogne-xen@valombre.net>
Date: Fri, Apr 13, 2012 at 12:31 PM
Subject: Re: [Xen-users] Xen timekeeping best practices
To: Todd Deshane <todd.deshane@xen.org>
Cc: Lloyd Dizon <thecarrionkind@gmail.com>, xen-users@lists.xen.org



> Does this help at all:
> http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F


Well, thank you but independent_wallclock doesn't exist anymore on
pvops kernel, and the link to blogspot seems broken so, no, not very
much :)

--

Regards,

Remi Gacogne



-- 
Todd Deshane
http://www.linkedin.com/in/deshantm
http://blog.xen.org/
http://wiki.xen.org/

Re: [Xen-users] Xen timekeeping best practices

[Ian Campbell]

On Fri, 2012-04-13 at 18:22 +0100, Todd Deshane wrote:
> Adding Xen-devel.
> 
> What are the current timekeeping best practices now?

pvops kernels behave as if independent_wallclock == 1, so the existing
advice:
        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
        domU. 
still applies.

Ian.

> 
> Thanks,
> Todd
> 
> ---------- Forwarded message ----------
> From: Remi Gacogne <rgacogne-xen@valombre.net>
> Date: Fri, Apr 13, 2012 at 12:31 PM
> Subject: Re: [Xen-users] Xen timekeeping best practices
> To: Todd Deshane <todd.deshane@xen.org>
> Cc: Lloyd Dizon <thecarrionkind@gmail.com>, xen-users@lists.xen.org
> 
> 
> 
> > Does this help at all:
> > http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F
> 
> 
> Well, thank you but independent_wallclock doesn't exist anymore on
> pvops kernel, and the link to blogspot seems broken so, no, not very
> much :)
> 
> --
> 
> Regards,
> 
> Remi Gacogne
> 
> 
> 

Re: [Xen-devel] Xen timekeeping best practices

[Remi Gacogne]

Hi Ian,

>> What are the current timekeeping best practices now?
>
> pvops kernels behave as if independent_wallclock == 1, so the existing
> advice:
>        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
>        domU.
> still applies.

O, thanks!

The thing is, we have several servers in Xen 4.0.1 using NTP in dom0 and 
domU. We didn't have any problem keeping a correct time with Xen 3 on the 
same servers, but with Xen 4 we have a recurring problem where the clock 
goes out of sync for an hour or so, in dom0 and domU at the same time.

We have done more tests since friday, and it seems that this issue does 
not occur on servers having a Intel Xeon L5630, but on older servers 
having E5405 or L5410 processors.

Looking at the output from xm dmesg, olders servers have :
(XEN) checking TSC synchronization across 8 CPUs: passed.
(XEN) Platform timer appears to have unexpectedly wrapped 10 or more 
times.
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) CPUIDLE: disabled due to no HPET. Force enable with 'cpuidle'.

where newer have :

(XEN) TSC is reliable, synchronization unnecessary
(XEN) Platform timer is 14.318MHz HPET

I'm wondering if my issue could simply be linked to the fact that HPET is 
not available ?

On newer servers, the ouput of
# xm debug-key s; xm dmesg | tail
is :
(XEN) TSC marked as reliable, warp = 0 (count=4)
(XEN) dom3: mode=0,ofs=0x42f57c62a36,khz=2133464,inc=1
(XEN) dom4: mode=0,ofs=0x430b996b94f,khz=2133464,inc=1
(XEN) dom5(hvm): mode=0,ofs=0x470ba656d26,khz=2133464,inc=1
(XEN) dom10(hvm): mode=0,ofs=0x6baa148c81f66,khz=2133464,inc=1
(XEN) No domains have emulated TSC

whereas it is on olders servers :
(XEN) Platform timer appears to have unexpectedly wrapped 10 or more times
(XEN) TSC has constant rate, deep Cstates possible, so not reliable, 
warp=2310 (count=8)
(XEN) dom1: mode=0,ofs=0x4379529ffa3,khz=2000068,inc=1,vtsc count: 
5579056116 kernel, 18906545 user
(XEN) dom2: mode=0,ofs=0x43a55ddb79e,khz=2000068,inc=1,vtsc count: 
12684066480 kernel, 25277162 user
(XEN) dom3: mode=0,ofs=0x43bd882b29e,khz=2000068,inc=1,vtsc count: 
35568040315 kernel, 27356321 user

Any idea would be appreciated !

Thanks,

Re: [Xen-devel] Xen timekeeping best practices

[Lloyd Dizon]

On Mon, Apr 16, 2012 at 11:57 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>
> On Fri, 2012-04-13 at 18:22 +0100, Todd Deshane wrote:
> > Adding Xen-devel.
> >
> > What are the current timekeeping best practices now?
>
> pvops kernels behave as if independent_wallclock == 1, so the existing
> advice:
>        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
>        domU.
> still applies.

This also applies to Xen 4.0 (4.0.0) if I understood correctly?

If so and based on previous posts can we resume that this is what is
recommended:

        |        Xen3                     |          Xen4               |
---------------------------------------------------------------------
PV    | independent_wallclock = 0 or 1  | independent_wallclock =  1  |
---------------------------------------------------------------------
HVM | independent_wallclock = 0       | independent_wallclock =  1  |
---------------------------------------------------------------------

With priority to independent_wallclock=1 and if activated, ntp required.

Re: [Xen-users] Xen timekeeping best practices

[Ian Campbell]

On Mon, 2012-04-16 at 15:45 +0100, Lloyd Dizon wrote:
> On Mon, Apr 16, 2012 at 11:57 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >
> > On Fri, 2012-04-13 at 18:22 +0100, Todd Deshane wrote:
> > > Adding Xen-devel.
> > >
> > > What are the current timekeeping best practices now?
> >
> > pvops kernels behave as if independent_wallclock == 1, so the existing
> > advice:
> >        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
> >        domU.
> > still applies.
> 
> This also applies to Xen 4.0 (4.0.0) if I understood correctly?

This is entirely a function of the guest kernel version and not the
hypervisor

> If so and based on previous posts can we resume that this is what is
> recommended:

If you s/Xen3/Classic XenoLinux Kernels/ and s/Xen4/Paravirt Ops Linux
Kernels/g then I think it is somewhat accurate, at least for the PV
line.

I must confess I don't really know for the HVM line. I suspect that it
is actually:

Classic XenoLinux -- no PV time source, independent_wallclock means
nothing, run NTP in the guest

Paravirt ops kernels _without_ PV time PVHVM extensions -- no PV time
source, independent_wallclock means nothing, run NTP in the guest

Paravirt ops kernels _with_ PV time PVHVM extensions -- PV time source,
independent_wallclock always == 1, run NTP.

So in short always run NTP in an HVM guest, regardless of the presence
or absence of PV time extensions.

>         |        Xen3                     |          Xen4               |
> ---------------------------------------------------------------------
> PV    | independent_wallclock = 0 or 1  | independent_wallclock =  1  |
> ---------------------------------------------------------------------
> HVM | independent_wallclock = 0       | independent_wallclock =  1  |
> ---------------------------------------------------------------------
> 
> With priority to independent_wallclock=1 and if activated, ntp required.

Re: [Xen-devel] Xen timekeeping best practices

[Lloyd Dizon]

On Mon, Apr 16, 2012 at 4:57 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Mon, 2012-04-16 at 15:45 +0100, Lloyd Dizon wrote:
>> On Mon, Apr 16, 2012 at 11:57 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >
>> > On Fri, 2012-04-13 at 18:22 +0100, Todd Deshane wrote:
>> > > Adding Xen-devel.
>> > >
>> > > What are the current timekeeping best practices now?
>> >
>> > pvops kernels behave as if independent_wallclock == 1, so the existing
>> > advice:
>> >        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
>> >        domU.
>> > still applies.
>>
>> This also applies to Xen 4.0 (4.0.0) if I understood correctly?
>
> This is entirely a function of the guest kernel version and not the
> hypervisor
>
>> If so and based on previous posts can we resume that this is what is
>> recommended:
>
> If you s/Xen3/Classic XenoLinux Kernels/ and s/Xen4/Paravirt Ops Linux
> Kernels/g then I think it is somewhat accurate, at least for the PV
> line.
>
> I must confess I don't really know for the HVM line. I suspect that it
> is actually:
>
> Classic XenoLinux -- no PV time source, independent_wallclock means
> nothing, run NTP in the guest
>
> Paravirt ops kernels _without_ PV time PVHVM extensions -- no PV time
> source, independent_wallclock means nothing, run NTP in the guest
>
> Paravirt ops kernels _with_ PV time PVHVM extensions -- PV time source,
> independent_wallclock always == 1, run NTP.
>
> So in short always run NTP in an HVM guest, regardless of the presence
> or absence of PV time extensions

I do not fully understand the difference yet between and Classic
XenoLinux and Paravirt Ops Kernel (will dig the info later) but here
is the resume:

       -------------------------------------------------------------------------------------------------------------------
       |  Xen3: Classic XenoLinux Kern                     |
    Xen4: Paravirt Ops Linux Kern                 |
--------------------------------------------------------------------------------------------------------------------------
 PV    | independent_wallclock = 0 or 1                    |
independent_wallclock =  1                                  |
--------------------------------------------------------------------------------------------------------------------------
 HVM   | -> no PV timesource, no ind. wallclock, run NTP   | -> w/o PV
time PVHVM extensions, no ind. wallclock, run NTP |
       |                                                   | -> PV
time source, ind. wallclock always 1, run NTP         |
--------------------------------------------------------------------------------------------------------------------------


Should we put this somewhere (wiki) until somebody else proves this is
inaccurate ?

Re: [Xen-users] Xen timekeeping best practices

[Ian Campbell]

On Tue, 2012-04-17 at 08:41 +0100, Lloyd Dizon wrote:
> On Mon, Apr 16, 2012 at 4:57 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Mon, 2012-04-16 at 15:45 +0100, Lloyd Dizon wrote:
> >> On Mon, Apr 16, 2012 at 11:57 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> >
> >> > On Fri, 2012-04-13 at 18:22 +0100, Todd Deshane wrote:
> >> > > Adding Xen-devel.
> >> > >
> >> > > What are the current timekeeping best practices now?
> >> >
> >> > pvops kernels behave as if independent_wallclock == 1, so the existing
> >> > advice:
> >> >        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
> >> >        domU.
> >> > still applies.
> >>
> >> This also applies to Xen 4.0 (4.0.0) if I understood correctly?
> >
> > This is entirely a function of the guest kernel version and not the
> > hypervisor
> >
> >> If so and based on previous posts can we resume that this is what is
> >> recommended:
> >
> > If you s/Xen3/Classic XenoLinux Kernels/ and s/Xen4/Paravirt Ops Linux
> > Kernels/g then I think it is somewhat accurate, at least for the PV
> > line.
> >
> > I must confess I don't really know for the HVM line. I suspect that it
> > is actually:
> >
> > Classic XenoLinux -- no PV time source, independent_wallclock means
> > nothing, run NTP in the guest
> >
> > Paravirt ops kernels _without_ PV time PVHVM extensions -- no PV time
> > source, independent_wallclock means nothing, run NTP in the guest
> >
> > Paravirt ops kernels _with_ PV time PVHVM extensions -- PV time source,
> > independent_wallclock always == 1, run NTP.
> >
> > So in short always run NTP in an HVM guest, regardless of the presence
> > or absence of PV time extensions
> 
> I do not fully understand the difference yet between and Classic
> XenoLinux and Paravirt Ops Kernel (will dig the info later)

Roughly:

classic == old out-of-tree Xen patches, static compile time support for
Xen. This is the old "2.6.18-xen.hg" tree and includes the "SuSE forward
port kernels" which are the classic patches kept up to date with recent
kernels.

pvops == patches in upstream Linux, dynamic support for Xen enabled at
kernel boot time. pvops supported domU from ~2.6.24 and dom0 from 3.0
onwards.

>  but here
> is the resume:
> 
>        -------------------------------------------------------------------------------------------------------------------
>        |  Xen3: Classic XenoLinux Kern                     |
>     Xen4: Paravirt Ops Linux Kern                 |
> --------------------------------------------------------------------------------------------------------------------------
>  PV    | independent_wallclock = 0 or 1                    |
> independent_wallclock =  1                                  |
> --------------------------------------------------------------------------------------------------------------------------
>  HVM   | -> no PV timesource, no ind. wallclock, run NTP   | -> w/o PV
> time PVHVM extensions, no ind. wallclock, run NTP |
>        |                                                   | -> PV
> time source, ind. wallclock always 1, run NTP         |
> --------------------------------------------------------------------------------------------------------------------------

It got a bit mangled in transit but it looks reasonable.

You should remove "Xen3" and "Xen4" from the titles since the hypervisor
version has no bearing on any of this. You can just as easily run a
pvops kernel on Xen3.x or a classic kernel on Xen4.x (and of course you
can mix and match those on a single host)

The 3->4 major number transition was mostly just a version number bump,
it didn't indicate any major change in the API or anything like that --
any PV kernel should run on any Xen from 3.0 onwards, right through to
4.2 (and beyond).

> Should we put this somewhere (wiki) until somebody else proves this is
> inaccurate ?

Please do put it on the wiki, I think there was a reference to a page (a
FAQ?) further up thread which seems like as good a place as any to add
it.

Ian.

Re: [Xen-devel] Xen timekeeping best practices

[Lloyd Dizon]

On Tue, Apr 17, 2012 at 10:16 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2012-04-17 at 08:41 +0100, Lloyd Dizon wrote:
>> On Mon, Apr 16, 2012 at 4:57 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Mon, 2012-04-16 at 15:45 +0100, Lloyd Dizon wrote:
>> >> On Mon, Apr 16, 2012 at 11:57 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> >
>> >> > On Fri, 2012-04-13 at 18:22 +0100, Todd Deshane wrote:
>> >> > > Adding Xen-devel.
>> >> > >
>> >> > > What are the current timekeeping best practices now?
>> >> >
>> >> > pvops kernels behave as if independent_wallclock == 1, so the existing
>> >> > advice:
>> >> >        set /proc/sys/xen/independent_wallclock to 1 and run ntp on
>> >> >        domU.
>> >> > still applies.
>> >>
>> >> This also applies to Xen 4.0 (4.0.0) if I understood correctly?
>> >
>> > This is entirely a function of the guest kernel version and not the
>> > hypervisor
>> >
>> >> If so and based on previous posts can we resume that this is what is
>> >> recommended:
>> >
>> > If you s/Xen3/Classic XenoLinux Kernels/ and s/Xen4/Paravirt Ops Linux
>> > Kernels/g then I think it is somewhat accurate, at least for the PV
>> > line.
>> >
>> > I must confess I don't really know for the HVM line. I suspect that it
>> > is actually:
>> >
>> > Classic XenoLinux -- no PV time source, independent_wallclock means
>> > nothing, run NTP in the guest
>> >
>> > Paravirt ops kernels _without_ PV time PVHVM extensions -- no PV time
>> > source, independent_wallclock means nothing, run NTP in the guest
>> >
>> > Paravirt ops kernels _with_ PV time PVHVM extensions -- PV time source,
>> > independent_wallclock always == 1, run NTP.
>> >
>> > So in short always run NTP in an HVM guest, regardless of the presence
>> > or absence of PV time extensions
>>
>> I do not fully understand the difference yet between and Classic
>> XenoLinux and Paravirt Ops Kernel (will dig the info later)
>
> Roughly:
>
> classic == old out-of-tree Xen patches, static compile time support for
> Xen. This is the old "2.6.18-xen.hg" tree and includes the "SuSE forward
> port kernels" which are the classic patches kept up to date with recent
> kernels.
>
> pvops == patches in upstream Linux, dynamic support for Xen enabled at
> kernel boot time. pvops supported domU from ~2.6.24 and dom0 from 3.0
> onwards.

Thanks also found info here:
http://wiki.xensource.com/xenwiki/XenDom0Kernels.html


>>  but here
>> is the resume:
>>
>>        -------------------------------------------------------------------------------------------------------------------
>>        |  Xen3: Classic XenoLinux Kern                     |
>>     Xen4: Paravirt Ops Linux Kern                 |
>> --------------------------------------------------------------------------------------------------------------------------
>>  PV    | independent_wallclock = 0 or 1                    |
>> independent_wallclock =  1                                  |
>> --------------------------------------------------------------------------------------------------------------------------
>>  HVM   | -> no PV timesource, no ind. wallclock, run NTP   | -> w/o PV
>> time PVHVM extensions, no ind. wallclock, run NTP |
>>        |                                                   | -> PV
>> time source, ind. wallclock always 1, run NTP         |
>> --------------------------------------------------------------------------------------------------------------------------
>
> It got a bit mangled in transit but it looks reasonable.
>
> You should remove "Xen3" and "Xen4" from the titles since the hypervisor
> version has no bearing on any of this. You can just as easily run a
> pvops kernel on Xen3.x or a classic kernel on Xen4.x (and of course you
> can mix and match those on a single host)
>
> The 3->4 major number transition was mostly just a version number bump,
> it didn't indicate any major change in the API or anything like that --
> any PV kernel should run on any Xen from 3.0 onwards, right through to
> 4.2 (and beyond).
>
>> Should we put this somewhere (wiki) until somebody else proves this is
>> inaccurate ?
>
> Please do put it on the wiki, I think there was a reference to a page (a
> FAQ?) further up thread which seems like as good a place as any to add
> it.

Done.
Anybody please correct the information if inaccurate.
http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F