timekeepeing in XEN - request fo comments

timekeepeing in XEN - request fo comments

[Bartosz Lis]

Dear developers,

I have some issues with time synchronization among dom0 and domUs. In my case 
helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for domUs and 
running ntpd also on domUs. I think, running ntpd on domU is not a good idea, 
but it works acceptably good as a workaround for my configuration: XEN 3.4.1 + 
kernel 3.6.31.5 with pv_ops (taken from git last week). When I don't run ntpd 
on domU clock slowly drifts. I have choosen xen clocksource in both dom0 and 
domU.

I'm not the only one to have time synchronization issues. Please write some 
document specifying how time synchronization between dom0 and domU should be 
achieved. I have a number of special questions, but probably there are more 
questions that should be aswered:

1. What is the mechanics behind the clocksource named xen? How does it depend 
on phisical time sources available on different platforms?

2. How gets domU's clock initialized when domU boots? (I observe that just 
after domU has started its clock differs from dom0. The difference is about 
10s.)

3. What are the consequences of not selecting xen clocksource (by selecting 
one of hpet, acpi_pm, etc.) for dom0 and for domU?

4. What are the best practices to achieve clock synchronization?

5. What has changed recently in timekeeping? What is going to change? Are 
there any improvements/bugfixes added to redhat or suse kernels? Are these 
bugfixes planned to be added upstream?

Thank you in advance,

-- 
Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland
   bartoszl @ ics.p.lodz.pl

RE: timekeepeing in XEN - request fo comments

[Dan Magenheimer]

What kernels are running in your domUs?  Are they
also (all) 2.6.31.5?  If your domU's are older kernels
(e.g. SLES10 or older, RHEL5 or older) there are
many issues that cannot be entirely solved by
Xen or dom0.

> -----Original Message-----
> From: Bartosz Lis [mailto:bartoszl@ics.p.lodz.pl]
> Sent: Monday, November 09, 2009 5:21 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] timekeepeing in XEN - request fo comments
> 
> 
> Dear developers,
> 
> I have some issues with time synchronization among dom0 and 
> domUs. In my case 
> helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 
> for domUs and 
> running ntpd also on domUs. I think, running ntpd on domU is 
> not a good idea, 
> but it works acceptably good as a workaround for my 
> configuration: XEN 3.4.1 + 
> kernel 3.6.31.5 with pv_ops (taken from git last week). When 
> I don't run ntpd 
> on domU clock slowly drifts. I have choosen xen clocksource 
> in both dom0 and 
> domU.
> 
> I'm not the only one to have time synchronization issues. 
> Please write some 
> document specifying how time synchronization between dom0 and 
> domU should be 
> achieved. I have a number of special questions, but probably 
> there are more 
> questions that should be aswered:
> 
> 1. What is the mechanics behind the clocksource named xen? 
> How does it depend 
> on phisical time sources available on different platforms?
> 
> 2. How gets domU's clock initialized when domU boots? (I 
> observe that just 
> after domU has started its clock differs from dom0. The 
> difference is about 
> 10s.)
> 
> 3. What are the consequences of not selecting xen clocksource 
> (by selecting 
> one of hpet, acpi_pm, etc.) for dom0 and for domU?
> 
> 4. What are the best practices to achieve clock synchronization?
> 
> 5. What has changed recently in timekeeping? What is going to 
> change? Are 
> there any improvements/bugfixes added to redhat or suse 
> kernels? Are these 
> bugfixes planned to be added upstream?
> 
> Thank you in advance,
> 
> -- 
> Bartosz Lis @ Inst. of Information Technology, Technical 
> Univ. of Lodz, Poland
>    bartoszl @ ics.p.lodz.pl
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
>

Re: timekeepeing in XEN - request fo comments

[Pasi Kärkkäinen]

On Mon, Nov 09, 2009 at 08:50:52AM -0800, Dan Magenheimer wrote:
> What kernels are running in your domUs?  Are they
> also (all) 2.6.31.5?  If your domU's are older kernels
> (e.g. SLES10 or older, RHEL5 or older) there are
> many issues that cannot be entirely solved by
> Xen or dom0.
> 

Could you tell more about these timekeeping problems with 
SLES10/RHEL5 PV kernels?

-- Pasi

> > -----Original Message-----
> > From: Bartosz Lis [mailto:bartoszl@ics.p.lodz.pl]
> > Sent: Monday, November 09, 2009 5:21 AM
> > To: xen-devel@lists.xensource.com
> > Subject: [Xen-devel] timekeepeing in XEN - request fo comments
> > 
> > 
> > Dear developers,
> > 
> > I have some issues with time synchronization among dom0 and 
> > domUs. In my case 
> > helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 
> > for domUs and 
> > running ntpd also on domUs. I think, running ntpd on domU is 
> > not a good idea, 
> > but it works acceptably good as a workaround for my 
> > configuration: XEN 3.4.1 + 
> > kernel 3.6.31.5 with pv_ops (taken from git last week). When 
> > I don't run ntpd 
> > on domU clock slowly drifts. I have choosen xen clocksource 
> > in both dom0 and 
> > domU.
> > 
> > I'm not the only one to have time synchronization issues. 
> > Please write some 
> > document specifying how time synchronization between dom0 and 
> > domU should be 
> > achieved. I have a number of special questions, but probably 
> > there are more 
> > questions that should be aswered:
> > 
> > 1. What is the mechanics behind the clocksource named xen? 
> > How does it depend 
> > on phisical time sources available on different platforms?
> > 
> > 2. How gets domU's clock initialized when domU boots? (I 
> > observe that just 
> > after domU has started its clock differs from dom0. The 
> > difference is about 
> > 10s.)
> > 
> > 3. What are the consequences of not selecting xen clocksource 
> > (by selecting 
> > one of hpet, acpi_pm, etc.) for dom0 and for domU?
> > 
> > 4. What are the best practices to achieve clock synchronization?
> > 
> > 5. What has changed recently in timekeeping? What is going to 
> > change? Are 
> > there any improvements/bugfixes added to redhat or suse 
> > kernels? Are these 
> > bugfixes planned to be added upstream?
> > 
> > Thank you in advance,
> > 
> > -- 
> > Bartosz Lis @ Inst. of Information Technology, Technical 
> > Univ. of Lodz, Poland
> >    bartoszl @ ics.p.lodz.pl
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: timekeepeing in XEN - request fo comments

[Jeremy Fitzhardinge]

On 11/09/09 04:21, Bartosz Lis wrote:
> I have some issues with time synchronization among dom0 and domUs. In my case 
> helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for domUs and 
> running ntpd also on domUs. I think, running ntpd on domU is not a good idea, 
>   

Why not?

> but it works acceptably good as a workaround for my configuration: XEN 3.4.1 + 
> kernel 3.6.31.5 with pv_ops (taken from git last week). When I don't run ntpd 
> on domU clock slowly drifts. I have choosen xen clocksource in both dom0 and 
> domU.
>   

Drifts with respect to what?

If you run all domains without ntp then they should remain in lockstep,
because they are all being run off the same timebase.  But the xen
clocksource does have a drift correction factor which can be updated via
ntpd/adjtimex, so if each kernel updates its factor differently then
they will appear to drift with respect to each other.

> I'm not the only one to have time synchronization issues. Please write some 
> document specifying how time synchronization between dom0 and domU should be 
> achieved. I have a number of special questions, but probably there are more 
> questions that should be aswered:
>
> 1. What is the mechanics behind the clocksource named xen? How does it depend 
> on phisical time sources available on different platforms?
>   

You can think of it as an augmented tsc; the tsc is used to convey
fine-grained time information, and a set of additional parameters allow
the guest to correct for things like tsc skew between cpus.  (And,
ideally, differing tsc rates, but it is hard to get things perfectly
monotonic in that case.)

> 2. How gets domU's clock initialized when domU boots? (I observe that just 
> after domU has started its clock differs from dom0. The difference is about 
> 10s.)
>   

It reads the Xen wallclock at boot to initialize the time of day.  Xen's
wallclock time may not be correct/match dom0 if dom0 has failed to
update it (which is currently the case in pvops dom0 kernels).

> 3. What are the consequences of not selecting xen clocksource (by selecting 
> one of hpet, acpi_pm, etc.) for dom0 and for domU?
>   
Xen itself uses and claims ownership of hardware resources like hpet and
acpi_pm, so they aren't available to guest kernels.  DomU kernels simply
don't have any hardware access; dom0 is specifically not allowed to
touch hpet (and I think acpi_pm).

You could force a kernel to use tsc a clocksource, but that will only
work correctly if the system has inherently synchronized tscs, and there
would be discontinuities over suspend/resume/migrate.

> 4. What are the best practices to achieve clock synchronization?
>   

ntp is it at present.

> 5. What has changed recently in timekeeping? What is going to change? Are 
> there any improvements/bugfixes added to redhat or suse kernels? Are these 
> bugfixes planned to be added upstream?
>   

As far as I know the pvops kernel is state of the art.  It currently
fails to set the Xen wallclock time when asked (not implemented), but
aside from that it should be OK.

The basic principle is that Xen doesn't know all that much about time of
day, and doesn't really care.  It offers a token get/set time of day
interface to access the corresponding hardware, but very little beyond
that.  Time is really considered to be a local matter for domains to
work out.  If they want to synchronize, then ntp is a perfectly
reasonable mechanism for doing that.

Would you like to see any particular changes?  What are your requirements?

    J

Re: timekeepeing in XEN - request fo comments

[Bartosz Lis]

Dnia poniedziałek, 9 listopada 2009 o 17:50:52 Dan Magenheimer napisał(a):
> What kernels are running in your domUs?  Are they
> also (all) 2.6.31.5?  If your domU's are older kernels
> (e.g. SLES10 or older, RHEL5 or older) there are
> many issues that cannot be entirely solved by
> Xen or dom0.

Dan,

Both mine are 2.6.31.5 from Jeremy's git tree, but Jordi Espasa Clofent 
reported on xen-users list problems with 2.6.18-6-xen-amd64. See his post:
http://lists.xensource.com/archives/html/xen-users/2009-11/msg00115.html .

-- 
Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland
   bartoszl @ ics.p.lodz.pl

Re: timekeepeing in XEN - request fo comments

[Bartosz Lis]

Jeremy,

Thanks very much for your answers. I think, now, I understand what causes my 
problems.

Dnia wtorek, 10 listopada 2009 o 01:59:13 Jeremy Fitzhardinge napisał(a):
> On 11/09/09 04:21, Bartosz Lis wrote:
> > I have some issues with time synchronization among dom0 and domUs. In my
> > case helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for
> > domUs and running ntpd also on domUs. I think, running ntpd on domU is
> > not a good idea,
> 
> Why not?
> 

I run ntpd in dom0, so it is synchronized to internet time. This 
synchronization (done by a daemon) costs in terms of processing power, memory 
usage and network traffic. This cost generated by dom0's ntpd is small and 
acceptable. I think it would be much cheaper to synchronize domUs' clocks to 
dom0 clock using hypercalls rather than running ntpd daemon on each of them.

> > but it works acceptably good as a workaround for my configuration: XEN
> > 3.4.1 + kernel 3.6.31.5 with pv_ops (taken from git last week). When I
> > don't run ntpd on domU clock slowly drifts. I have choosen xen
> > clocksource in both dom0 and domU.
> 
> Drifts with respect to what?

Drifts with respect to dom0's clock, which is synchronized to internet time by 
ntpd.

> 
> If you run all domains without ntp then they should remain in lockstep,
> because they are all being run off the same timebase.  But the xen
> clocksource does have a drift correction factor which can be updated via
> ntpd/adjtimex, so if each kernel updates its factor differently then
> they will appear to drift with respect to each other.

I had run ntpd in domUs for several hours, then I switched it off, and 
observed what happens then. Clocks started drifting from dom0 (which was still 
synchronized to global time by ntpd) and from each other. Yes, it's possible 
that during the period that each domU had ntpd running their drift correction 
factors were set in a different way. Domains had different loads, so ntpd on 
more loaded domain could have had worse accuracy.

> 
> > I'm not the only one to have time synchronization issues. Please write
> > some document specifying how time synchronization between dom0 and domU
> > should be achieved. I have a number of special questions, but probably
> > there are more questions that should be aswered:
> >
> > 1. What is the mechanics behind the clocksource named xen? How does it
> > depend on phisical time sources available on different platforms?
> 
> You can think of it as an augmented tsc; the tsc is used to convey
> fine-grained time information, and a set of additional parameters allow
> the guest to correct for things like tsc skew between cpus.  (And,
> ideally, differing tsc rates, but it is hard to get things perfectly
> monotonic in that case.)
> 
> > 2. How gets domU's clock initialized when domU boots? (I observe that
> > just after domU has started its clock differs from dom0. The difference
> > is about 10s.)
> 
> It reads the Xen wallclock at boot to initialize the time of day.  Xen's
> wallclock time may not be correct/match dom0 if dom0 has failed to
> update it (which is currently the case in pvops dom0 kernels).

Please, tell me if I'm right (I'm 99% sure, but, please, assure me):
* date - displays and sets system clock (dom0 or domU).
* hwclock - displays and sets cmos clock if run in dom0 (in domU doesn't work 
from obvious reasons).

Is there any shell command (or xm subcommand) to display xen wallclock?

> 
> > 3. What are the consequences of not selecting xen clocksource (by
> > selecting one of hpet, acpi_pm, etc.) for dom0 and for domU?
> 
> Xen itself uses and claims ownership of hardware resources like hpet and
> acpi_pm, so they aren't available to guest kernels.  DomU kernels simply
> don't have any hardware access; dom0 is specifically not allowed to
> touch hpet (and I think acpi_pm).
> 
> You could force a kernel to use tsc a clocksource, but that will only
> work correctly if the system has inherently synchronized tscs, and there
> would be discontinuities over suspend/resume/migrate.
> 
> > 4. What are the best practices to achieve clock synchronization?
> 
> ntp is it at present.
> 
> > 5. What has changed recently in timekeeping? What is going to change? Are
> > there any improvements/bugfixes added to redhat or suse kernels? Are
> > these bugfixes planned to be added upstream?
> 
> As far as I know the pvops kernel is state of the art.  It currently
> fails to set the Xen wallclock time when asked (not implemented), but
> aside from that it should be OK.
> 
> The basic principle is that Xen doesn't know all that much about time of
> day, and doesn't really care.  It offers a token get/set time of day
> interface to access the corresponding hardware, but very little beyond
> that.  Time is really considered to be a local matter for domains to
> work out.  If they want to synchronize, then ntp is a perfectly
> reasonable mechanism for doing that.
> 
> Would you like to see any particular changes?  What are your requirements?

No, I think, that's enough from the user's point of view.

Kind regards,

-- 
Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland
   bartoszl @ ics.p.lodz.pl

RE: timekeepeing in XEN - request fo comments

[Dan Magenheimer]

Hi Pasi --

The timekeeping problems I am referring to are
only(?) for hvm guests.  HVM guests with different
OS's and different versions of a Linux kernel (32-bit
vs 64-bit, even kernel updates between EL5uX and
EL5uX+1) behave differently when ticks are delivered
at imperfect intervals (which happens frequently
for Xen and other hypervisors).  Xen distros
and "virtual appliance" distributors may specify
certain settings of timer_mode and certain kernel
parameters (e.g. "clock=pit") to ensure time
is kept as accurately as possible.  There is no
"one size fits all" solution unfortunately.

For PV guests, note that there have been patches
to dom0 (2.6.18-xen) that improve timekeeping
in dom0 and this may affect pv guests with
independent_wallclock==0.  I don't know if the
patches have been applied to pv_ops dom0, or
if they are even necessary.  In fact, I don't
know if there is a concept of independent_wallclock
in pv_ops.

Dan

> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Sent: Monday, November 09, 2009 11:50 AM
> To: Dan Magenheimer
> Cc: Bartosz Lis; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] timekeepeing in XEN - request fo comments
> 
> 
> On Mon, Nov 09, 2009 at 08:50:52AM -0800, Dan Magenheimer wrote:
> > What kernels are running in your domUs?  Are they
> > also (all) 2.6.31.5?  If your domU's are older kernels
> > (e.g. SLES10 or older, RHEL5 or older) there are
> > many issues that cannot be entirely solved by
> > Xen or dom0.
> > 
> 
> Could you tell more about these timekeeping problems with 
> SLES10/RHEL5 PV kernels?
> 
> -- Pasi
> 
> > > -----Original Message-----
> > > From: Bartosz Lis [mailto:bartoszl@ics.p.lodz.pl]
> > > Sent: Monday, November 09, 2009 5:21 AM
> > > To: xen-devel@lists.xensource.com
> > > Subject: [Xen-devel] timekeepeing in XEN - request fo comments
> > > 
> > > 
> > > Dear developers,
> > > 
> > > I have some issues with time synchronization among dom0 and 
> > > domUs. In my case 
> > > helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 
> > > for domUs and 
> > > running ntpd also on domUs. I think, running ntpd on domU is 
> > > not a good idea, 
> > > but it works acceptably good as a workaround for my 
> > > configuration: XEN 3.4.1 + 
> > > kernel 3.6.31.5 with pv_ops (taken from git last week). When 
> > > I don't run ntpd 
> > > on domU clock slowly drifts. I have choosen xen clocksource 
> > > in both dom0 and 
> > > domU.
> > > 
> > > I'm not the only one to have time synchronization issues. 
> > > Please write some 
> > > document specifying how time synchronization between dom0 and 
> > > domU should be 
> > > achieved. I have a number of special questions, but probably 
> > > there are more 
> > > questions that should be aswered:
> > > 
> > > 1. What is the mechanics behind the clocksource named xen? 
> > > How does it depend 
> > > on phisical time sources available on different platforms?
> > > 
> > > 2. How gets domU's clock initialized when domU boots? (I 
> > > observe that just 
> > > after domU has started its clock differs from dom0. The 
> > > difference is about 
> > > 10s.)
> > > 
> > > 3. What are the consequences of not selecting xen clocksource 
> > > (by selecting 
> > > one of hpet, acpi_pm, etc.) for dom0 and for domU?
> > > 
> > > 4. What are the best practices to achieve clock synchronization?
> > > 
> > > 5. What has changed recently in timekeeping? What is going to 
> > > change? Are 
> > > there any improvements/bugfixes added to redhat or suse 
> > > kernels? Are these 
> > > bugfixes planned to be added upstream?
> > > 
> > > Thank you in advance,
> > > 
> > > -- 
> > > Bartosz Lis @ Inst. of Information Technology, Technical 
> > > Univ. of Lodz, Poland
> > >    bartoszl @ ics.p.lodz.pl
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > > 
> > >
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>