[PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Du, Hang]

[-- Attachment #1.1: Type: text/plain, Size: 755 bytes --]

Signed-off-by: Shunli Yi <shyi@websense.com>
Signed-off-by: Hang Du <hdu@websense.com>

When sync up time from dom0 to domU, the monotonic time should be reset or domU cannot sync up backward time from dom0.

diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-xen.c

--- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000 +0800
+++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000 +0800
@@ -715,6 +715,8 @@
    }

    if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
+        if (!independent_wallclock)
+            monotonic_reset();
        update_wallclock();
        schedule_clock_was_set_work = 1;
    }



 Protected by Websense Hosted Email Security -- www.websense.com 

[-- Attachment #1.2: Type: text/html, Size: 6581 bytes --]

[-- Attachment #2: time-xen.patch --]
[-- Type: application/octet-stream, Size: 517 bytes --]

Subject:Reset monotonic time when sync up time from dom0 to domU

Signed-off-by: Shunli Yi <shyi@websense.com>
Signed-off-by: Hang Du <hdu@websense.com>

--- a/arch/i386/kernel/time-xen.c	2010-10-11 10:41:06.000000000 +0800
+++ b/arch/i386/kernel/time-xen.c	2010-10-11 10:43:32.000000000 +0800
@@ -715,6 +715,8 @@
 	}
 
 	if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
+        if (!independent_wallclock)
+            monotonic_reset();
 		update_wallclock();
 		schedule_clock_was_set_work = 1;
 	}

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Jan Beulich]

>>> On 12.10.10 at 12:19, "Du, Hang" <hdu@websense.com> wrote:
> Signed-off-by: Shunli Yi <shyi@websense.com>
> Signed-off-by: Hang Du <hdu@websense.com>
> 
> When sync up time from dom0 to domU, the monotonic time should be reset or 
> domU cannot sync up backward time from dom0.

Without a better description of what you try to do and why there
is no is_initial_xendomain() check in this code despite talk being
about sync-ing into a single direction (admittedly the word "backward"
above is confusing me quite a bit), I would ask that this patch not be
applied.

Jan

> diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-xen.c
> 
> --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000 +0800
> +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000 +0800
> @@ -715,6 +715,8 @@
>     }
> 
>     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> +        if (!independent_wallclock)
> +            monotonic_reset();
>         update_wallclock();
>         schedule_clock_was_set_work = 1;
>     }
> 
> 
> 
>  Protected by Websense Hosted Email Security -- www.websense.com 

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Dan Magenheimer]

[-- Attachment #1.1: Type: text/plain, Size: 1572 bytes --]

I'm not sure what kernel tree you are proposing this against, but as I said in the previous thread, I think this is a bad idea for general use.  It is too easy for a dom0 system administrator to unwittingly wreak subtle and mysterious havoc on possibly dozens of guests each with dozens of users.  The only advantage is that it saves time for that dom0 administrator who would otherwise have to go into each guest individually and change the time (backwards, which is never recommended in any running system).

 

From: Du, Hang [mailto:hdu@websense.com] 
Sent: Tuesday, October 12, 2010 4:19 AM
To: xen-devel@lists.xensource.com
Cc: Liu, Saipu; Yi, Shunli
Subject: [Xen-devel] [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

 

Signed-off-by: Shunli Yi <shyi@websense.com>

Signed-off-by: Hang Du <hdu@websense.com>

 

When sync up time from dom0 to domU, the monotonic time should be reset or domU cannot sync up backward time from dom0.

 

diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-xen.c   

 

--- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000 +0800

+++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000 +0800

@@ -715,6 +715,8 @@

    }   

 

    if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {

+        if (!independent_wallclock)

+            monotonic_reset();

        update_wallclock();

        schedule_clock_was_set_work = 1;

    }

 

 

Protected by Websense Hosted Email Security - www.websense.com

[-- Attachment #1.2: Type: text/html, Size: 8420 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Du, Hang]

[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]

Jan,

Sorry for my brief description in previous mail and missing is_initial_xendomain check. The kernel I submit this patch is linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-xendomain check.

In this patch, we support the backward time changing sync to all domUs which configured to use "dependent wall clock".
Currently, without the backward time syncing, when we change the time backward in Dom0, the time in DomU would be froze. 
And this cause some commands hang and don't executed until the time catch up with the domU time.
For example:
"rpm -q kernel-xen"
"sleep 1"
Monotonic time should be reset when sync up time from dom0 to domU to support domU backward time syncing.

diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-xen.c   

--- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000 +0800
+++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000 +0800
@@ -715,6 +715,8 @@
    }   
 
    if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
+        if (!is_initial_xendomain() && !independent_wallclock)
+            monotonic_reset();
        update_wallclock();
        schedule_clock_was_set_work = 1;
    }


 Protected by Websense Hosted Email Security -- www.websense.com 

[-- Attachment #2: time-xen.patch --]
[-- Type: application/octet-stream, Size: 544 bytes --]

Subject:Reset monotonic time when sync up time from dom0 to domU

Signed-off-by: Shunli Yi <shyi@websense.com>
Signed-off-by: Hang Du <hdu@websense.com>

--- a/arch/i386/kernel/time-xen.c	2010-10-11 10:41:06.000000000 +0800
+++ b/arch/i386/kernel/time-xen.c	2010-10-11 10:43:32.000000000 +0800
@@ -715,6 +715,8 @@
 	}
 
 	if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
+        if (!is_initial_xendomain() && !independent_wallclock)
+            monotonic_reset();
 		update_wallclock();
 		schedule_clock_was_set_work = 1;
 	}

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Jan Beulich]

>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
> Sorry for my brief description in previous mail and missing 
> is_initial_xendomain check. The kernel I submit this patch is 
> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-xendomain 
> check.

Actually, I didn't expect you to blindly insert that check, but rather
either explain why only DomU-s need the changed behavior (as your
original description suggested), or refine the description of the
problem you're trying to solve.

> In this patch, we support the backward time changing sync to all domUs which 
> configured to use "dependent wall clock".
> Currently, without the backward time syncing, when we change the time 
> backward in Dom0, the time in DomU would be froze. 
> And this cause some commands hang and don't executed until the time catch up 
> with the domU time.
> For example:
> "rpm -q kernel-xen"
> "sleep 1"
> Monotonic time should be reset when sync up time from dom0 to domU to 
> support domU backward time syncing.

I can see your point, however ...

> diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-xen.c   
> 
> --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000 +0800
> +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000 +0800
> @@ -715,6 +715,8 @@
>     }   
>  
>     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> +        if (!is_initial_xendomain() && !independent_wallclock)
> +            monotonic_reset();
>         update_wallclock();

... you definitely need to call monotonic_reset() *after* the
update_wallclock() (or else another vCPU, preempted in
do_gettimeofday() between the end of the xtime read loop
and the acquiring of the monotonic lock, would be able to
overwrite monotonic.tv with values obtained before the wall
clock update - similar to the secondary problem addressed by
c/s 1021).

Further, blindly running a reset here doesn't seem nice in
the (general) case of the time getting updated forwards.

>         schedule_clock_was_set_work = 1;
>     }

You'll also want to address Dan's concerns, and you will want to
get your patch formatted so that it would actually apply (read:
no tab -> space conversion) if you expect it to be committed
by Keir at some point.

Jan

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Dan Magenheimer]

(bringing the topic up to a more theoretical level and including
Keir and Jeremy)

I wonder if the "bug" here is that "dependent wall clock" is
an incredibly poor replacement for NTP (or a Windows Time Server)
and a hack and really shouldn't even exist?  I suppose one could
argue that it makes some sense in a XCP environment, and maybe
in a server environment where a single physical server is extremely
isolated, but does it ever make sense in a real world server
environment?

In other words, I'm arguing that the correct "fix" here is:
Don't use dependent wallclock.

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@novell.com]
> Sent: Wednesday, October 13, 2010 6:38 AM
> To: Hang Du
> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> sync up time from dom0 to domU
> 
> >>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
> > Sorry for my brief description in previous mail and missing
> > is_initial_xendomain check. The kernel I submit this patch is
> > linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
> xendomain
> > check.
> 
> Actually, I didn't expect you to blindly insert that check, but rather
> either explain why only DomU-s need the changed behavior (as your
> original description suggested), or refine the description of the
> problem you're trying to solve.
> 
> > In this patch, we support the backward time changing sync to all
> domUs which
> > configured to use "dependent wall clock".
> > Currently, without the backward time syncing, when we change the time
> > backward in Dom0, the time in DomU would be froze.
> > And this cause some commands hang and don't executed until the time
> catch up
> > with the domU time.
> > For example:
> > "rpm -q kernel-xen"
> > "sleep 1"
> > Monotonic time should be reset when sync up time from dom0 to domU to
> > support domU backward time syncing.
> 
> I can see your point, however ...
> 
> > diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-
> xen.c
> >
> > --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000
> +0800
> > +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000
> +0800
> > @@ -715,6 +715,8 @@
> >     }
> >
> >     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> > +        if (!is_initial_xendomain() && !independent_wallclock)
> > +            monotonic_reset();
> >         update_wallclock();
> 
> ... you definitely need to call monotonic_reset() *after* the
> update_wallclock() (or else another vCPU, preempted in
> do_gettimeofday() between the end of the xtime read loop
> and the acquiring of the monotonic lock, would be able to
> overwrite monotonic.tv with values obtained before the wall
> clock update - similar to the secondary problem addressed by
> c/s 1021).
> 
> Further, blindly running a reset here doesn't seem nice in
> the (general) case of the time getting updated forwards.
> 
> >         schedule_clock_was_set_work = 1;
> >     }
> 
> You'll also want to address Dan's concerns, and you will want to
> get your patch formatted so that it would actually apply (read:
> no tab -> space conversion) if you expect it to be committed
> by Keir at some point.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Jeremy Fitzhardinge]

 On 10/13/2010 08:56 AM, Dan Magenheimer wrote:
> (bringing the topic up to a more theoretical level and including
> Keir and Jeremy)
>
> I wonder if the "bug" here is that "dependent wall clock" is
> an incredibly poor replacement for NTP (or a Windows Time Server)
> and a hack and really shouldn't even exist?  I suppose one could
> argue that it makes some sense in a XCP environment, and maybe
> in a server environment where a single physical server is extremely
> isolated, but does it ever make sense in a real world server
> environment?
>
> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.

Yes, that's always been my view.  pvops kernels don't implement it.

    J

>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@novell.com]
>> Sent: Wednesday, October 13, 2010 6:38 AM
>> To: Hang Du
>> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
>> sync up time from dom0 to domU
>>
>>>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
>>> Sorry for my brief description in previous mail and missing
>>> is_initial_xendomain check. The kernel I submit this patch is
>>> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
>> xendomain
>>> check.
>> Actually, I didn't expect you to blindly insert that check, but rather
>> either explain why only DomU-s need the changed behavior (as your
>> original description suggested), or refine the description of the
>> problem you're trying to solve.
>>
>>> In this patch, we support the backward time changing sync to all
>> domUs which
>>> configured to use "dependent wall clock".
>>> Currently, without the backward time syncing, when we change the time
>>> backward in Dom0, the time in DomU would be froze.
>>> And this cause some commands hang and don't executed until the time
>> catch up
>>> with the domU time.
>>> For example:
>>> "rpm -q kernel-xen"
>>> "sleep 1"
>>> Monotonic time should be reset when sync up time from dom0 to domU to
>>> support domU backward time syncing.
>> I can see your point, however ...
>>
>>> diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-
>> xen.c
>>> --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000
>> +0800
>>> +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000
>> +0800
>>> @@ -715,6 +715,8 @@
>>>     }
>>>
>>>     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
>>> +        if (!is_initial_xendomain() && !independent_wallclock)
>>> +            monotonic_reset();
>>>         update_wallclock();
>> ... you definitely need to call monotonic_reset() *after* the
>> update_wallclock() (or else another vCPU, preempted in
>> do_gettimeofday() between the end of the xtime read loop
>> and the acquiring of the monotonic lock, would be able to
>> overwrite monotonic.tv with values obtained before the wall
>> clock update - similar to the secondary problem addressed by
>> c/s 1021).
>>
>> Further, blindly running a reset here doesn't seem nice in
>> the (general) case of the time getting updated forwards.
>>
>>>         schedule_clock_was_set_work = 1;
>>>     }
>> You'll also want to address Dan's concerns, and you will want to
>> get your patch formatted so that it would actually apply (read:
>> no tab -> space conversion) if you expect it to be committed
>> by Keir at some point.
>>
>> Jan
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Keir Fraser]

On 13/10/2010 16:56, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> (bringing the topic up to a more theoretical level and including
> Keir and Jeremy)
> 
> I wonder if the "bug" here is that "dependent wall clock" is
> an incredibly poor replacement for NTP (or a Windows Time Server)
> and a hack and really shouldn't even exist?  I suppose one could
> argue that it makes some sense in a XCP environment, and maybe
> in a server environment where a single physical server is extremely
> isolated, but does it ever make sense in a real world server
> environment?

Yes it's a bad replacement for ntp. Pv_ops doesn't bother to implement it,
syncs with Xen wallclock once during boot and thereafter expects ntp to be
used within the guest.

 -- Keir

> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.
> 
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@novell.com]
>> Sent: Wednesday, October 13, 2010 6:38 AM
>> To: Hang Du
>> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
>> sync up time from dom0 to domU
>> 
>>>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
>>> Sorry for my brief description in previous mail and missing
>>> is_initial_xendomain check. The kernel I submit this patch is
>>> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
>> xendomain
>>> check.
>> 
>> Actually, I didn't expect you to blindly insert that check, but rather
>> either explain why only DomU-s need the changed behavior (as your
>> original description suggested), or refine the description of the
>> problem you're trying to solve.
>> 
>>> In this patch, we support the backward time changing sync to all
>> domUs which
>>> configured to use "dependent wall clock".
>>> Currently, without the backward time syncing, when we change the time
>>> backward in Dom0, the time in DomU would be froze.
>>> And this cause some commands hang and don't executed until the time
>> catch up
>>> with the domU time.
>>> For example:
>>> "rpm -q kernel-xen"
>>> "sleep 1"
>>> Monotonic time should be reset when sync up time from dom0 to domU to
>>> support domU backward time syncing.
>> 
>> I can see your point, however ...
>> 
>>> diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-
>> xen.c
>>> 
>>> --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000
>> +0800
>>> +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000
>> +0800
>>> @@ -715,6 +715,8 @@
>>>     }
>>> 
>>>     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
>>> +        if (!is_initial_xendomain() && !independent_wallclock)
>>> +            monotonic_reset();
>>>         update_wallclock();
>> 
>> ... you definitely need to call monotonic_reset() *after* the
>> update_wallclock() (or else another vCPU, preempted in
>> do_gettimeofday() between the end of the xtime read loop
>> and the acquiring of the monotonic lock, would be able to
>> overwrite monotonic.tv with values obtained before the wall
>> clock update - similar to the secondary problem addressed by
>> c/s 1021).
>> 
>> Further, blindly running a reset here doesn't seem nice in
>> the (general) case of the time getting updated forwards.
>> 
>>>         schedule_clock_was_set_work = 1;
>>>     }
>> 
>> You'll also want to address Dan's concerns, and you will want to
>> get your patch formatted so that it would actually apply (read:
>> no tab -> space conversion) if you expect it to be committed
>> by Keir at some point.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Tim Deegan]

At 16:56 +0100 on 13 Oct (1286988987), Dan Magenheimer wrote:
> I wonder if the "bug" here is that "dependent wall clock" is
> an incredibly poor replacement for NTP (or a Windows Time Server)
> and a hack and really shouldn't even exist?

I've seen Windows Time server do some pretty odd things, but leaving
that aside - the problem with dependent wallclock is that it distributes
only the wallclock time and not the drift parameters, so you get 
sawtoothing.

Running NTP in guests has its own problems, particularly around live
migration, where NTP can lose stability quite badly.

There was a paper about this at OSDI last week:
http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf

Tim.

>  I suppose one could
> argue that it makes some sense in a XCP environment, and maybe
> in a server environment where a single physical server is extremely
> isolated, but does it ever make sense in a real world server
> environment?
> 
> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.
> 
> > -----Original Message-----
> > From: Jan Beulich [mailto:JBeulich@novell.com]
> > Sent: Wednesday, October 13, 2010 6:38 AM
> > To: Hang Du
> > Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
> > Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> > sync up time from dom0 to domU
> > 
> > >>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
> > > Sorry for my brief description in previous mail and missing
> > > is_initial_xendomain check. The kernel I submit this patch is
> > > linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
> > xendomain
> > > check.
> > 
> > Actually, I didn't expect you to blindly insert that check, but rather
> > either explain why only DomU-s need the changed behavior (as your
> > original description suggested), or refine the description of the
> > problem you're trying to solve.
> > 
> > > In this patch, we support the backward time changing sync to all
> > domUs which
> > > configured to use "dependent wall clock".
> > > Currently, without the backward time syncing, when we change the time
> > > backward in Dom0, the time in DomU would be froze.
> > > And this cause some commands hang and don't executed until the time
> > catch up
> > > with the domU time.
> > > For example:
> > > "rpm -q kernel-xen"
> > > "sleep 1"
> > > Monotonic time should be reset when sync up time from dom0 to domU to
> > > support domU backward time syncing.
> > 
> > I can see your point, however ...
> > 
> > > diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-
> > xen.c
> > >
> > > --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000
> > +0800
> > > +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000
> > +0800
> > > @@ -715,6 +715,8 @@
> > >     }
> > >
> > >     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> > > +        if (!is_initial_xendomain() && !independent_wallclock)
> > > +            monotonic_reset();
> > >         update_wallclock();
> > 
> > ... you definitely need to call monotonic_reset() *after* the
> > update_wallclock() (or else another vCPU, preempted in
> > do_gettimeofday() between the end of the xtime read loop
> > and the acquiring of the monotonic lock, would be able to
> > overwrite monotonic.tv with values obtained before the wall
> > clock update - similar to the secondary problem addressed by
> > c/s 1021).
> > 
> > Further, blindly running a reset here doesn't seem nice in
> > the (general) case of the time getting updated forwards.
> > 
> > >         schedule_clock_was_set_work = 1;
> > >     }
> > 
> > You'll also want to address Dan's concerns, and you will want to
> > get your patch formatted so that it would actually apply (read:
> > no tab -> space conversion) if you expect it to be committed
> > by Keir at some point.
> > 
> > Jan
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Jeremy Fitzhardinge]

 On 10/13/2010 09:16 AM, Tim Deegan wrote:
> At 16:56 +0100 on 13 Oct (1286988987), Dan Magenheimer wrote:
>> I wonder if the "bug" here is that "dependent wall clock" is
>> an incredibly poor replacement for NTP (or a Windows Time Server)
>> and a hack and really shouldn't even exist?
> I've seen Windows Time server do some pretty odd things, but leaving
> that aside - the problem with dependent wallclock is that it distributes
> only the wallclock time and not the drift parameters, so you get 
> sawtoothing.
>
> Running NTP in guests has its own problems, particularly around live
> migration, where NTP can lose stability quite badly.

There seems to be some work on replacements for NTP which converge
quicker and are shinier and better.

I've thought that having some kind of lightweight dom0<->domU sync
protocol would be a better replacement for dependent time than NTP
(unless full NTP is required).

> There was a paper about this at OSDI last week:
> http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf

Ooh, look, RADclock, just what I was thinking about.

    J

> Tim.
>
>>  I suppose one could
>> argue that it makes some sense in a XCP environment, and maybe
>> in a server environment where a single physical server is extremely
>> isolated, but does it ever make sense in a real world server
>> environment?
>>
>> In other words, I'm arguing that the correct "fix" here is:
>> Don't use dependent wallclock.
>>
>>> -----Original Message-----
>>> From: Jan Beulich [mailto:JBeulich@novell.com]
>>> Sent: Wednesday, October 13, 2010 6:38 AM
>>> To: Hang Du
>>> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
>>> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
>>> sync up time from dom0 to domU
>>>
>>>>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
>>>> Sorry for my brief description in previous mail and missing
>>>> is_initial_xendomain check. The kernel I submit this patch is
>>>> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
>>> xendomain
>>>> check.
>>> Actually, I didn't expect you to blindly insert that check, but rather
>>> either explain why only DomU-s need the changed behavior (as your
>>> original description suggested), or refine the description of the
>>> problem you're trying to solve.
>>>
>>>> In this patch, we support the backward time changing sync to all
>>> domUs which
>>>> configured to use "dependent wall clock".
>>>> Currently, without the backward time syncing, when we change the time
>>>> backward in Dom0, the time in DomU would be froze.
>>>> And this cause some commands hang and don't executed until the time
>>> catch up
>>>> with the domU time.
>>>> For example:
>>>> "rpm -q kernel-xen"
>>>> "sleep 1"
>>>> Monotonic time should be reset when sync up time from dom0 to domU to
>>>> support domU backward time syncing.
>>> I can see your point, however ...
>>>
>>>> diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-
>>> xen.c
>>>> --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000
>>> +0800
>>>> +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000
>>> +0800
>>>> @@ -715,6 +715,8 @@
>>>>     }
>>>>
>>>>     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
>>>> +        if (!is_initial_xendomain() && !independent_wallclock)
>>>> +            monotonic_reset();
>>>>         update_wallclock();
>>> ... you definitely need to call monotonic_reset() *after* the
>>> update_wallclock() (or else another vCPU, preempted in
>>> do_gettimeofday() between the end of the xtime read loop
>>> and the acquiring of the monotonic lock, would be able to
>>> overwrite monotonic.tv with values obtained before the wall
>>> clock update - similar to the secondary problem addressed by
>>> c/s 1021).
>>>
>>> Further, blindly running a reset here doesn't seem nice in
>>> the (general) case of the time getting updated forwards.
>>>
>>>>         schedule_clock_was_set_work = 1;
>>>>     }
>>> You'll also want to address Dan's concerns, and you will want to
>>> get your patch formatted so that it would actually apply (read:
>>> no tab -> space conversion) if you expect it to be committed
>>> by Keir at some point.
>>>
>>> Jan
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Yi, Shunli]

[-- Attachment #1: Type: text/plain, Size: 5246 bytes --]


> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: 2010年10月13日 23:56
> To: Jan Beulich; Du, Hang; Keir Fraser; Jeremy Fitzhardinge
> Cc: Liu, Saipu; Yi, Shunli; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when sync up
> time from dom0 to domU
> 
> (bringing the topic up to a more theoretical level and including Keir and Jeremy)
> 
> I wonder if the "bug" here is that "dependent wall clock" is an incredibly poor
> replacement for NTP (or a Windows Time Server) and a hack and really shouldn't
> even exist?  I suppose one could argue that it makes some sense in a XCP
> environment, and maybe in a server environment where a single physical server
> is extremely isolated, but does it ever make sense in a real world server
> environment?
> 
[Shunli] We encounter the time-syncing issue when deploy enterprise applications. 
We suppose the "dependent wall clock" feature is for some enterprise environment (or products) which have uniform administrator and manager tool.
So, they can maintain the time on Dom0 only, and Xen would maintain the time in DomainUs.

Why we think this defect should be fixed?
1. We implemented a forward time syncing from Dom0 to DomU, also the backward time syncing should be supported.
2. Currently, backward time setting in Dom0 would cause some DomU applications hang (when set "independent_wallclock=0").
     For example : rpm
3. We don't think the backward time setting is risk for DomUs, this only impacts the DomUs who set "independent_wallclock=0", and the administrator should always know what they are doing.

> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.

[Shunli] Of cause, we can, don't use the dependent wallclock. 
But, we still want to do something for those peoples who want the "dependent wallclock".
It's easy to fix, and we didn't hear known risks about changing time backward in DomU.


> 
> > -----Original Message-----
> > From: Jan Beulich [mailto:JBeulich@novell.com]
> > Sent: Wednesday, October 13, 2010 6:38 AM
> > To: Hang Du
> > Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
> > Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> > sync up time from dom0 to domU
> >
> > >>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
> > > Sorry for my brief description in previous mail and missing
> > > is_initial_xendomain check. The kernel I submit this patch is
> > > linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
> > xendomain
> > > check.
> >
> > Actually, I didn't expect you to blindly insert that check, but rather
> > either explain why only DomU-s need the changed behavior (as your
> > original description suggested), or refine the description of the
> > problem you're trying to solve.
> >
> > > In this patch, we support the backward time changing sync to all
> > domUs which
> > > configured to use "dependent wall clock".
> > > Currently, without the backward time syncing, when we change the
> > > time backward in Dom0, the time in DomU would be froze.
> > > And this cause some commands hang and don't executed until the time
> > catch up
> > > with the domU time.
> > > For example:
> > > "rpm -q kernel-xen"
> > > "sleep 1"
> > > Monotonic time should be reset when sync up time from dom0 to domU
> > > to support domU backward time syncing.
> >
> > I can see your point, however ...
> >
> > > diff -urN a/arch/i386/kernel/time-xen.c   b/arch/i386/kernel/time-
> > xen.c
> > >
> > > --- a/arch/i386/kernel/time-xen.c   2010-10-11 10:41:06.000000000
> > +0800
> > > +++ b/arch/i386/kernel/time-xen.c   2010-10-11 10:43:32.000000000
> > +0800
> > > @@ -715,6 +715,8 @@
> > >     }
> > >
> > >     if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> > > +        if (!is_initial_xendomain() && !independent_wallclock)
> > > +            monotonic_reset();
> > >         update_wallclock();
> >
> > ... you definitely need to call monotonic_reset() *after* the
> > update_wallclock() (or else another vCPU, preempted in
> > do_gettimeofday() between the end of the xtime read loop and the
> > acquiring of the monotonic lock, would be able to overwrite
> > monotonic.tv with values obtained before the wall clock update -
> > similar to the secondary problem addressed by c/s 1021).
> >
> > Further, blindly running a reset here doesn't seem nice in the
> > (general) case of the time getting updated forwards.
> >
> > >         schedule_clock_was_set_work = 1;
> > >     }
> >
> > You'll also want to address Dan's concerns, and you will want to get
> > your patch formatted so that it would actually apply (read:
> > no tab -> space conversion) if you expect it to be committed by Keir
> > at some point.
> >
> > Jan
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
>  To report this as spam, please forward to spam@websense.com.  Thank you.


 Protected by Websense Hosted Email Security -- www.websense.com 

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Tim Deegan]

At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> > There was a paper about this at OSDI last week:
> > http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> 
> Ooh, look, RADclock, just what I was thinking about.

Yes, it looks pretty good.  Also they can use Xen stime as the local
oscillator and distribute drift numbers from xenstore, so no hypervisor
patches (and no hypervisor-interface changes) required. :)

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Jeremy Fitzhardinge]

 On 10/14/2010 02:07 AM, Tim Deegan wrote:
> At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
>>> There was a paper about this at OSDI last week:
>>> http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
>> Ooh, look, RADclock, just what I was thinking about.
> Yes, it looks pretty good.  Also they can use Xen stime as the local
> oscillator and distribute drift numbers from xenstore, so no hypervisor
> patches (and no hypervisor-interface changes) required. :)

Neat!  Is it something we should encourage them to submit for xen-unstable?

    J

RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU)

[Tim Deegan]

At 00:35 +0100 on 15 Oct (1287102942), Jeremy Fitzhardinge wrote:
>  On 10/14/2010 02:07 AM, Tim Deegan wrote:
> > At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> >>> There was a paper about this at OSDI last week:
> >>> http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> >> Ooh, look, RADclock, just what I was thinking about.
> > Yes, it looks pretty good.  Also they can use Xen stime as the local
> > oscillator and distribute drift numbers from xenstore, so no hypervisor
> > patches (and no hypervisor-interface changes) required. :)
> 
> Neat!  Is it something we should encourage them to submit for xen-unstable?

Well I did, but there are no hypervisor patches. :) 

CC'ing the speaker: are there any dom0 tools or kernel patches we could
be looking at for xen-unstable?

Regards,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Dan Magenheimer]

> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Thursday, October 14, 2010 3:08 AM
> To: Jeremy Fitzhardinge
> Cc: Dan Magenheimer; Jan Beulich; Hang Du; Keir Fraser; Saipu Liu;
> Shunli Yi; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> sync up time from dom0 to domU
> 
> At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> > > There was a paper about this at OSDI last week:
> > > http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> >
> > Ooh, look, RADclock, just what I was thinking about.
> 
> Yes, it looks pretty good.  Also they can use Xen stime as the local
> oscillator and distribute drift numbers from xenstore, so no hypervisor
> patches (and no hypervisor-interface changes) required. :)

Maybe I'm misunderstanding the paper, but isn't it required
that Xen stime be directly readable for every attempt to
sample time (e.g. requiring at least some small interface
change such as adding a hypercall to obtain Xen stime)?
The current pvclock mechanism still depends on TSC to
interpolate between "Xen stime samples periodically written
to memory" to get adequate precision.

Also, for those certain enterprise applications that want
to sample time 10000+ samples/second/processor, and need
to know "immediately" when a sample might be bad (due
to, for example, live migration), I think each sample would
need to check xenstore.  Is xenstore up to that kind of
pounding (and is it fast enough)?

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Tim Deegan]

At 15:09 +0100 on 15 Oct (1287155382), Dan Magenheimer wrote:
> > At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> > > > There was a paper about this at OSDI last week:
> > > > http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> > >
> > > Ooh, look, RADclock, just what I was thinking about.
> > 
> > Yes, it looks pretty good.  Also they can use Xen stime as the local
> > oscillator and distribute drift numbers from xenstore, so no hypervisor
> > patches (and no hypervisor-interface changes) required. :)
> 
> Maybe I'm misunderstanding the paper, but isn't it required
> that Xen stime be directly readable for every attempt to
> sample time (e.g. requiring at least some small interface
> change such as adding a hypercall to obtain Xen stime)?

You can obtain xen stime directly from the shared-into page and RDTSC.
There might need to be _kernel_ changes to make that available to
userspace, though my impression is that they push packet-timestamping
into the kernel.

> Also, for those certain enterprise applications that want
> to sample time 10000+ samples/second/processor, and need
> to know "immediately" when a sample might be bad (due
> to, for example, live migration), I think each sample would
> need to check xenstore.  Is xenstore up to that kind of
> pounding (and is it fast enough)?

No it's certainly not (either of those things), but:
 - the problem of turning a distributed wallclock into something suitable
   for timestamping that aggressively is orthogonal to the problem of
   distributing that wallclock in the first place; and
 - all the user really needs is a generation counter to know that the 
   clock correction values are stale, and the kernel can provide that 
   alongside the stime.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Dan Magenheimer]

> > Also, for those certain enterprise applications that want
> > to sample time 10000+ samples/second/processor, and need
> > to know "immediately" when a sample might be bad (due
> > to, for example, live migration), I think each sample would
> > need to check xenstore.  Is xenstore up to that kind of
> > pounding (and is it fast enough)?
> 
> No it's certainly not (either of those things), but:
>  - the problem of turning a distributed wallclock into something
> suitable
>    for timestamping that aggressively is orthogonal to the problem of
>    distributing that wallclock in the first place; and

Agreed that this is a better solution to a very important problem.
I'm just trying to determine if it helps solve another time-related
real world problem.

>  - all the user really needs is a generation counter to know that the
>    clock correction values are stale, and the kernel can provide that
>    alongside the stime.

Agreed that a generation counter solves this.  BUT...

(1) afaik there is no generation counter in any time-related
kernel API; and

(2) afaict this generation counter would need to be "pushed"
from the dom0 kernel so would either (a) require domU kernels
to read from xenstore, or (b) require some kind of privileged
hypercall from dom0 to the hypervisor telling the hypervisor
to change the generation counter (and scaling values?) for all
domUs.

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Dan Magenheimer]

(RADclock authors cc'ed in attempt to re-merge the two threads...
Tim, if it's not too late, please reply to this email rather
than the previous one without the full cc list)

> From: Dan Magenheimer
> Sent: Friday, October 15, 2010 8:47 AM
> To: Tim Deegan
> Cc: Jeremy Fitzhardinge; Jan Beulich; Hang Du; Keir Fraser; Saipu Liu;
> Shunli Yi; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> sync up time from dom0 to domU
> 
> > > Also, for those certain enterprise applications that want
> > > to sample time 10000+ samples/second/processor, and need
> > > to know "immediately" when a sample might be bad (due
> > > to, for example, live migration), I think each sample would
> > > need to check xenstore.  Is xenstore up to that kind of
> > > pounding (and is it fast enough)?
> >
> > No it's certainly not (either of those things), but:
> >  - the problem of turning a distributed wallclock into something
> > suitable
> >    for timestamping that aggressively is orthogonal to the problem of
> >    distributing that wallclock in the first place; and
> 
> Agreed that this is a better solution to a very important problem.
> I'm just trying to determine if it helps solve another time-related
> real world problem.
> 
> >  - all the user really needs is a generation counter to know that the
> >    clock correction values are stale, and the kernel can provide that
> >    alongside the stime.
> 
> Agreed that a generation counter solves this.  BUT...
> 
> (1) afaik there is no generation counter in any time-related
> kernel API; and
> 
> (2) afaict this generation counter would need to be "pushed"
> from the dom0 kernel so would either (a) require domU kernels
> to read from xenstore, or (b) require some kind of privileged
> hypercall from dom0 to the hypervisor telling the hypervisor
> to change the generation counter (and scaling values?) for all
> domUs.

Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Tim Deegan]

At 15:46 +0100 on 15 Oct (1287157606), Dan Magenheimer wrote:
> Agreed that this is a better solution to a very important problem.
> I'm just trying to determine if it helps solve another time-related
> real world problem.

Yep.  AFAICS it doesn't really affect that at all. :|

> >  - all the user really needs is a generation counter to know that the
> >    clock correction values are stale, and the kernel can provide that
> >    alongside the stime.
> 
> Agreed that a generation counter solves this.  BUT...
> 
> (1) afaik there is no generation counter in any time-related
> kernel API; and

OK, but one could be added wherever the stime is exposed.  This line of
reasoning runs into the earlier discussions about PV time protocols fopr
userspace (inluding process migration &c), and I've nothing useful to
add to that now. :)

> (2) afaict this generation counter would need to be "pushed"
> from the dom0 kernel so would either (a) require domU kernels
> to read from xenstore, or (b) require some kind of privileged
> hypercall from dom0 to the hypervisor telling the hypervisor
> to change the generation counter (and scaling values?) for all
> domUs.

I was under the impression that the only time you'd really need to
invalidate the xenstore values was on live migration, which the domU
kernel can certainly know about.  One reason they use xenstore is that
the correction numbers are quite stable and don't need to be polled
every time.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU)

[Jeremy Fitzhardinge]

 On 10/15/2010 05:10 AM, Darryl Veitch wrote:
> Hi Tim,
>
> good to hear from you.  Dom0 will need the usual RADclock patches
> which now integrate the Xen support. These supply two main things:  
> i)  kernel level raw timestamping (ie reading of the chosen counter)
> of packet arrivals and departures in `good but safe' locations and an
> ability to pass those timestamps up (works transparently with pcap,
> and doesn't affect its normal operation)

Are you trying to get these changes into mainline Linux?

> ii)  the ability to access the raw counter of choice from both kernel
> and user context

The kernel already has this info, and I have patches for mapping it into
userspace.  Or would a syscall suffice?

> In the Xen context, the choice of counter will end up being Xen
> Clocksource which is already accessible to guests as is already a
> suitable counter of the right size (64-bit).
>
> Take a look at the RADclock downloads
> <http://www.cubinlab.ee.unimelb.edu.au/radclock/download.php> page.
> There have been a number of changes recently in particular in the
> integration of the Xen support so I will direct you to Julien Ridoux
> for finer-grained answers as he and Tim Broomhead actually wrote the
> code. 
>
> We'll be quizzing you shortly about our needs to make this work under
> HV using that way to access the Xenstore you mentioned, and we'd like
> to talk to you about how to improve packet timestamping under Dom0. 

OK.

    J

RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

[Dan Magenheimer]

[-- Attachment #1.1: Type: text/plain, Size: 5067 bytes --]

Hi Darryl -

 

So how would a userland enterprise app go about doing this?


For example, the Oracle DB (and other Oracle enterprise apps) collects timestamps at a high frequency, sometimes more than 10K timestamps/second.  (These are used primarily for collecting transaction statistics, but these statistics are collected by default and must be explicitly disabled by a DB administrator.)   As one might expect, the difference between any two of these timestamps is the key result and the two timestamps may differ by seconds or minutes so, as  a result, the two timestamps of interest may cross a live migration boundary.

 

I am guessing that to use the approach you suggest, both the counter and some form of "generation indicator" must be stored for each timestamp, then when the difference between the two is to be calculated, the RADclock history is consulted, essentially to lookup, if they are different, the two generation indicators and use them to adjust the two counters.

 

Ideally, this would be entirely hidden from userland in the kernel, but the cost of a "real" system call is too high to be incurred 10K/sec, and I'm not sure how RADclock history could be consulted in a vsyscall.

 

Thanks,

Dan

 

P.S. Since you probably don't closely follow the xen-devel list, some discussion has occurred on other threads without cc'ing you.  If you are interested, search for RADclock in xen.markmail.org.

 

From: Darryl Veitch [mailto:dveitch@unimelb.edu.au] 
Sent: Friday, October 15, 2010 4:57 PM
To: Dan Magenheimer
Cc: Tim Deegan; Jeremy Fitzhardinge; Jan Beulich; Hang Du; Keir Fraser; Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com; Julien Ridoux; Timothy Broomhead
Subject: Re: [Xen-devel] [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU

 

Hi All,

 

in terms of pounding on xenstore, one way to get around this is to note that raw timestamps (that is what we call counter readings) can be taken at the required events but only converted to walltime later. This could be done in batches asynchronously, and perhaps even much later off-line, depending on the application.  This is one of the advantages of the feedforward approach. The RADclock keeps historical data which enables raw timestamps to be converted to walltime at any time with identically zero loss of precision- they would be exactly the same wallclock values as if they were calculated immediately.

 

Darryl

 

 

On 15/10/2010, at 4:53 PM, Dan Magenheimer wrote:





(RADclock authors cc'ed in attempt to re-merge the two threads...
Tim, if it's not too late, please reply to this email rather
than the previous one without the full cc list)




From: Dan Magenheimer

Sent: Friday, October 15, 2010 8:47 AM

To: Tim Deegan

Cc: Jeremy Fitzhardinge; Jan Beulich; Hang Du; Keir Fraser; Saipu Liu;

Shunli Yi; HYPERLINK "mailto:xen-devel@lists.xensource.com"xen-devel@lists.xensource.com

Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when

sync up time from dom0 to domU

 

Also, for those certain enterprise applications that want

to sample time 10000+ samples/second/processor, and need

to know "immediately" when a sample might be bad (due

to, for example, live migration), I think each sample would

need to check xenstore.  Is xenstore up to that kind of

pounding (and is it fast enough)?

 

No it's certainly not (either of those things), but:

- the problem of turning a distributed wallclock into something

suitable

  for timestamping that aggressively is orthogonal to the problem of

  distributing that wallclock in the first place; and

 

Agreed that this is a better solution to a very important problem.

I'm just trying to determine if it helps solve another time-related

real world problem.

 

- all the user really needs is a generation counter to know that the

  clock correction values are stale, and the kernel can provide that

  alongside the stime.

 

Agreed that a generation counter solves this.  BUT...

 

(1) afaik there is no generation counter in any time-related

kernel API; and

 

(2) afaict this generation counter would need to be "pushed"

from the dom0 kernel so would either (a) require domU kernels

to read from xenstore, or (b) require some kind of privileged

hypercall from dom0 to the hypervisor telling the hypervisor

to change the generation counter (and scaling values?) for all

domUs.

 

-----------------------------------------------

Darryl Veitch

Sabbatical Address:

TREC INRIA

23 Avenue D'Italie

75214 Paris Cedex 13, France 

Tel:  +33 (0)1 39 63 55 40

&

Technicolor Paris Research Lab

1, rue Jeanne d'Arc

92443 Issy-les-Moulineaux Cedex, France

Tel:  +33 (0)1 41 86 52 19

 

Global:

dveitch AT unimelb edu au

http://www.cubinlab.ee.unimelb.edu.au/~darryl/

AIM id:  darrylveitch

Skype:  darrylveitch

 

 





 

 

 

 





 

[-- Attachment #1.2: Type: text/html, Size: 22412 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU)

[Darryl Veitch]

[-- Attachment #1.1: Type: text/plain, Size: 1650 bytes --]


On 16/10/2010, at 10:21 AM, Jeremy Fitzhardinge wrote:

> On 10/15/2010 05:10 AM, Darryl Veitch wrote:
>> Hi Tim,
>> 
>> good to hear from you.  Dom0 will need the usual RADclock patches
>> which now integrate the Xen support. These supply two main things:  
>> i)  kernel level raw timestamping (ie reading of the chosen counter)
>> of packet arrivals and departures in `good but safe' locations and an
>> ability to pass those timestamps up (works transparently with pcap,
>> and doesn't affect its normal operation)
> 
> Are you trying to get these changes into mainline Linux?

We are intending to do this, but have not pushed comprehensively yet, though we hope to start soon after we have the adoption into FreeBSD finalised (theoretically around end-2010).  We have had some discussions with John Stultz in preparation.   Would you be interested in being involved in encouraging adoption here ? :-)  


> 
>> ii)  the ability to access the raw counter of choice from both kernel
>> and user context
> 
> The kernel already has this info, and I have patches for mapping it into
> userspace.  Or would a syscall suffice?

We use syscall's currently.

Darryl



-----------------------------------------------
Darryl Veitch
Sabbatical Address:
TREC INRIA
23 Avenue D'Italie
75214 Paris Cedex 13, France 
Tel:  +33 (0)1 39 63 55 40
&
Technicolor Paris Research Lab
1, rue Jeanne d'Arc
92443 Issy-les-Moulineaux Cedex, France
Tel:  +33 (0)1 41 86 52 19

Global:
dveitch AT unimelb edu au
http://www.cubinlab.ee.unimelb.edu.au/~darryl/
AIM id:  darrylveitch
Skype:  darrylveitch











[-- Attachment #1.2: Type: text/html, Size: 34812 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU)

[Jeremy Fitzhardinge]

 On 10/20/2010 08:36 PM, Darryl Veitch wrote:
>
> On 16/10/2010, at 10:21 AM, Jeremy Fitzhardinge wrote:
>
>> On 10/15/2010 05:10 AM, Darryl Veitch wrote:
>>> Hi Tim,
>>>
>>> good to hear from you.  Dom0 will need the usual RADclock patches
>>> which now integrate the Xen support. These supply two main things:  
>>> i)  kernel level raw timestamping (ie reading of the chosen counter)
>>> of packet arrivals and departures in `good but safe' locations and an
>>> ability to pass those timestamps up (works transparently with pcap,
>>> and doesn't affect its normal operation)
>>
>> Are you trying to get these changes into mainline Linux?
>
> We are intending to do this, but have not pushed comprehensively yet,
> though we hope to start soon after we have the adoption into FreeBSD
> finalised (theoretically around end-2010).  We have had some
> discussions with John Stultz in preparation.   Would you be interested
> in being involved in encouraging adoption here ? :-) 

I'm happy to give you help out where I can.  John is definitely a person
who should be involved.  Where's the current set of patches?

>
>>
>>> ii)  the ability to access the raw counter of choice from both kernel
>>> and user context
>>
>> The kernel already has this info, and I have patches for mapping it into
>> userspace.  Or would a syscall suffice?
>
> We use syscall's currently.

OK, that's simple.   What do you add?  A new lock time for clock_gettime?

    J

Re: RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU)

[Darryl Veitch]

[-- Attachment #1.1: Type: text/plain, Size: 997 bytes --]







On 22/10/2010, at 7:49 AM, Jeremy Fitzhardinge wrote:
>>>  
>>> Are you trying to get these changes into mainline Linux?
>> 
>> We are intending to do this, but have not pushed comprehensively yet,
>> though we hope to start soon after we have the adoption into FreeBSD
>> finalised (theoretically around end-2010).  We have had some
>> discussions with John Stultz in preparation.   Would you be interested
>> in being involved in encouraging adoption here ? :-) 
> 
> I'm happy to give you help out where I can.  John is definitely a person
> who should be involved.  Where's the current set of patches?


http://www.cubinlab.ee.unimelb.edu.au/radclock/download.php



--------------------------------------------------------
Darryl Veitch
Principal Research Fellow
Dept. of E&E Engineering
University of Melbourne
Rm 5.13, EEE building
Tel:      +61(0)3 8344 3817
Fax:     +61(0)3 8344 3823
Skype:       darrylveitch

dveitch AT unimelb.edu.au
www.cubinlab.ee.unimelb.edu.au/~darryl/





[-- Attachment #1.2: Type: text/html, Size: 22011 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel