Time goes backwards in dom0 in xen-unstable

Time goes backwards in dom0 in xen-unstable

[Dan Magenheimer]

I'm seeing some "Time went backwards" errors reported
in dom0 in near-tip (c/s 19515) xen-unstable build.
It's rare and random and not reproducible, but here's
the report that just showed up.  There was no load
running at the time.

I can move to 3.4-rc1 if that would be helpful but I
don't remember seeing any time-related changes recently.

This was on my dual-core test machine which reports
lots of power management info during Xen boot.

Dan

Timer ISR/0: Time went backwards: delta=-14971734 delta_cpu=25028266 shadow=3655
618577922 off=580668482 processed=3656214217729 cpu_processed=3656174217729
 0: 3656174217729
 1: 3656214217729
Timer ISR/0: Time went backwards: delta=-14971447 delta_cpu=25028553 shadow=3655
618577922 off=590668621 processed=3656224217729 cpu_processed=3656184217729
 0: 3656184217729
 1: 3656224217729
Timer ISR/0: Time went backwards: delta=-14970691 delta_cpu=25029309 shadow=3655
618577922 off=620669383 processed=3656254217729 cpu_processed=3656214217729
 0: 3656214217729
 1: 3656254217729
Timer ISR/0: Time went backwards: delta=-14970623 delta_cpu=25029377 shadow=3655
618577922 off=630669373 processed=3656264217729 cpu_processed=3656224217729
 0: 3656224217729
 1: 3656254217729
Timer ISR/0: Time went backwards: delta=-14969392 delta_cpu=25030608 shadow=3655
618577922 off=680670676 processed=3656314217729 cpu_processed=3656274217729
 0: 3656274217729
 1: 3656304217729
Timer ISR/0: Time went backwards: delta=-14967619 delta_cpu=25032381 shadow=3655
618577922 off=740672580 processed=3656374217729 cpu_processed=3656334217729
 0: 3656334217729
 1: 3656374217729
Timer ISR/0: Time went backwards: delta=-14967315 delta_cpu=25032685 shadow=3655
618577922 off=750672771 processed=3656384217729 cpu_processed=3656344217729
 0: 3656344217729
 1: 3656374217729

RE: Time goes backwards in dom0 in xen-unstable

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 2927 bytes --]

>From: Dan Magenheimer
>Sent: 2009年4月15日 3:37
>
>I'm seeing some "Time went backwards" errors reported
>in dom0 in near-tip (c/s 19515) xen-unstable build.
>It's rare and random and not reproducible, but here's
>the report that just showed up.  There was no load
>running at the time.
>
>I can move to 3.4-rc1 if that would be helpful but I
>don't remember seeing any time-related changes recently.
>
>This was on my dual-core test machine which reports
>lots of power management info during Xen boot.

could you add "hpetbroadcast" in grub.conf? This is possibly
caused by PIT broadcast again as in your Mcycles thread.
http://markmail.org/thread/uzqcmjuu6uc5xxqd is one
write-up about those side-effects when cpuidle gets enabled
by default in Xen 3.4. Preferred way to do broadcast is always
to use HPET, however by far one missing feature makes PIT
as default broadcast on most platforms. That missing feature
is to emulate RTC interrupt with IRQ8 replaced by HPET if
using legacy replacement mode. We're adding that emulation
now and hopefully get it ready in several days and in final
Xen 3.4 release. At that time, PIT will be only turned to
on some old platform (then no deep C-state support) or weird 
BIOS not exporting HPET (then simply disable cpuidle).

Thanks
Kevin

>
>Dan
>
>Timer ISR/0: Time went backwards: delta=-14971734 
>delta_cpu=25028266 shadow=3655
>618577922 off=580668482 processed=3656214217729 
>cpu_processed=3656174217729
> 0: 3656174217729
> 1: 3656214217729
>Timer ISR/0: Time went backwards: delta=-14971447 
>delta_cpu=25028553 shadow=3655
>618577922 off=590668621 processed=3656224217729 
>cpu_processed=3656184217729
> 0: 3656184217729
> 1: 3656224217729
>Timer ISR/0: Time went backwards: delta=-14970691 
>delta_cpu=25029309 shadow=3655
>618577922 off=620669383 processed=3656254217729 
>cpu_processed=3656214217729
> 0: 3656214217729
> 1: 3656254217729
>Timer ISR/0: Time went backwards: delta=-14970623 
>delta_cpu=25029377 shadow=3655
>618577922 off=630669373 processed=3656264217729 
>cpu_processed=3656224217729
> 0: 3656224217729
> 1: 3656254217729
>Timer ISR/0: Time went backwards: delta=-14969392 
>delta_cpu=25030608 shadow=3655
>618577922 off=680670676 processed=3656314217729 
>cpu_processed=3656274217729
> 0: 3656274217729
> 1: 3656304217729
>Timer ISR/0: Time went backwards: delta=-14967619 
>delta_cpu=25032381 shadow=3655
>618577922 off=740672580 processed=3656374217729 
>cpu_processed=3656334217729
> 0: 3656334217729
> 1: 3656374217729
>Timer ISR/0: Time went backwards: delta=-14967315 
>delta_cpu=25032685 shadow=3655
>618577922 off=750672771 processed=3656384217729 
>cpu_processed=3656344217729
> 0: 3656344217729
> 1: 3656374217729
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Time goes backwards in dom0 in xen-unstable

[Keir Fraser]

On 14/04/2009 20:36, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> I'm seeing some "Time went backwards" errors reported
> in dom0 in near-tip (c/s 19515) xen-unstable build.
> It's rare and random and not reproducible, but here's
> the report that just showed up.  There was no load
> running at the time.
> 
> I can move to 3.4-rc1 if that would be helpful but I
> don't remember seeing any time-related changes recently.
> 
> This was on my dual-core test machine which reports
> lots of power management info during Xen boot.

If you specify 'cpuidle=off' as a Xen boot parameter, does that make the
timer_interrupt scalability problem go away, and this time backwards
problem? I was going to enable by default in 3.4 but could go the other way.

 -- Keir

Re: Time goes backwards in dom0 in xen-unstable

[Keir Fraser]

On 15/04/2009 00:31, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> At that time, PIT will be only turned to
> on some old platform (then no deep C-state support) or weird
> BIOS not exporting HPET (then simply disable cpuidle).

Changeset 19545 now disables cpuidle by default if HPET broadcast is
unavailable. Obviously the PIT alternative is not up to being enabled by
default.

 -- Keir

Re: Time goes backwards in dom0 in xen-unstable

[Keir Fraser]

On 15/04/2009 00:31, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> That missing feature
> is to emulate RTC interrupt with IRQ8 replaced by HPET if
> using legacy replacement mode. We're adding that emulation
> now and hopefully get it ready in several days and in final
> Xen 3.4 release.

It's a bit late for 3.4 really. We'll be past -rc2 by the time you get this
patch out so it's missed the boat.

 -- Keir

RE: Time goes backwards in dom0 in xen-unstable

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
>Sent: 2009年4月15日 15:41
>
>On 15/04/2009 00:31, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> At that time, PIT will be only turned to
>> on some old platform (then no deep C-state support) or weird
>> BIOS not exporting HPET (then simply disable cpuidle).
>
>Changeset 19545 now disables cpuidle by default if HPET broadcast is
>unavailable. Obviously the PIT alternative is not up to being 
>enabled by
>default.
>

Thanks, that's a wise decision. 
Kevin

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Time goes backwards in dom0 in xen-unstable

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
>Sent: 2009年4月15日 15:44
>
>On 15/04/2009 00:31, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> That missing feature
>> is to emulate RTC interrupt with IRQ8 replaced by HPET if
>> using legacy replacement mode. We're adding that emulation
>> now and hopefully get it ready in several days and in final
>> Xen 3.4 release.
>
>It's a bit late for 3.4 really. We'll be past -rc2 by the time 
>you get this
>patch out so it's missed the boat.
>

We revised our plan here, after realizing not-negligible effort to
enable RTC emulation. Instead we want to turn to an alternative
that hpet broadcast is always enabled once available (current
situation is to always give up if no MSI delivery support), and
then disable it on the fly (disable HPET interrupt and reduce
max_cstate to a level not requiring broadcast) once detecting
user trying to enable UIE/PIE/AIE on RTC. That way is far 
simpler and makes cpuidle really available on most platforms
with HPET (true for most with VT support), in the meantime
not breaking occasional usage on RTC interrupt.

Will you accept such patch which handles mostly about policy? :-)
We'll try to send it out today.

Thanks,
Kevin

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Time goes backwards in dom0 in xen-unstable

[Keir Fraser]

On 15/04/2009 08:51, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> We revised our plan here, after realizing not-negligible effort to
> enable RTC emulation. Instead we want to turn to an alternative
> that hpet broadcast is always enabled once available (current
> situation is to always give up if no MSI delivery support), and
> then disable it on the fly (disable HPET interrupt and reduce
> max_cstate to a level not requiring broadcast) once detecting
> user trying to enable UIE/PIE/AIE on RTC. That way is far
> simpler and makes cpuidle really available on most platforms
> with HPET (true for most with VT support), in the meantime
> not breaking occasional usage on RTC interrupt.
> 
> Will you accept such patch which handles mostly about policy? :-)
> We'll try to send it out today.

It's got a better chance than RTC emulation. I'll certainly consider it.

 -- Keir

[PATCH] cpuidle: Enable hpet broadcast by default (RE: Time goes backwards in dom0 in xen-unstable)

[Wei, Gang]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

cpuidle: Enable hpet broadcast by default

And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.

Signed-off-by: Wei Gang <gang.wei@intel.com>
Signed-off-by: Tian Kevin <kevin.tian@intel.com>

On Wednesday, April 15, 2009 5:19 PM, Keir Fraser wrote:
> On 15/04/2009 08:51, "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
>> We revised our plan here, after realizing not-negligible effort to
>> enable RTC emulation. Instead we want to turn to an alternative
>> that hpet broadcast is always enabled once available (current
>> situation is to always give up if no MSI delivery support), and
>> then disable it on the fly (disable HPET interrupt and reduce
>> max_cstate to a level not requiring broadcast) once detecting
>> user trying to enable UIE/PIE/AIE on RTC. That way is far
>> simpler and makes cpuidle really available on most platforms
>> with HPET (true for most with VT support), in the meantime
>> not breaking occasional usage on RTC interrupt.
>> 
>> Will you accept such patch which handles mostly about policy? :-)
>> We'll try to send it out today.
> 
> It's got a better chance than RTC emulation. I'll certainly consider it.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[-- Attachment #2: cpuidle_rtc_mitigation.patch --]
[-- Type: application/octet-stream, Size: 5880 bytes --]

cpuidle: Enable hpet broadcast by default

And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.

Signed-off-by: Wei Gang <gang.wei@intel.com>
Signed-off-by: Tian Kevin <kevin.tian@intel.com>

diff -r 34dca01addc9 xen/arch/x86/acpi/cpu_idle.c
--- a/xen/arch/x86/acpi/cpu_idle.c	Wed Apr 15 08:40:12 2009 +0100
+++ b/xen/arch/x86/acpi/cpu_idle.c	Wed Apr 15 20:41:21 2009 +0800
@@ -221,8 +221,8 @@ static void acpi_processor_idle(void)
         return;
     }
 
-    next_state = power ? cpuidle_current_governor->select(power) : -1;
-    if ( next_state > 0 )
+    if ( max_cstate > 0 && power && 
+         (next_state = cpuidle_current_governor->select(power)) > 0 )
     {
         cx = &power->states[next_state];
         if ( power->flags.bm_check && acpi_idle_bm_check()
@@ -853,3 +853,18 @@ int pmstat_reset_cx_stat(uint32_t cpuid)
     return 0;
 }
 
+void cpuidle_disable_deep_cstate(void)
+{
+    if ( max_cstate > 1 )
+    {
+        if ( local_apic_timer_c2_ok )
+            max_cstate = 2;
+        else
+            max_cstate = 1;
+    }
+
+    mb();
+
+    hpet_disable_legacy_broadcast();
+}
+
diff -r 34dca01addc9 xen/arch/x86/hpet.c
--- a/xen/arch/x86/hpet.c	Wed Apr 15 08:40:12 2009 +0100
+++ b/xen/arch/x86/hpet.c	Wed Apr 15 20:41:21 2009 +0800
@@ -22,8 +22,10 @@
 
 #define MAX_HPET_NUM 32
 
-#define HPET_EVT_USED_BIT   2
+#define HPET_EVT_USED_BIT    0
 #define HPET_EVT_USED       (1 << HPET_EVT_USED_BIT)
+#define HPET_EVT_DISABLE_BIT 1
+#define HPET_EVT_DISALBE    (1 << HPET_EVT_DISABLE_BIT)
 
 struct hpet_event_channel
 {
@@ -53,8 +55,11 @@ unsigned long hpet_address;
 
 void msi_compose_msg(struct pci_dev *pdev, int vector, struct msi_msg *msg);
 
-/* force_hpet_broadcast: if true, force using hpet_broadcast to fix lapic stop
-   issue for deep C state with pit disabled */
+/*
+ * force_hpet_broadcast: by default legacy hpet broadcast will be stopped
+ * if RTC interrupts are enabled. Enable this option if want to always enable
+ * legacy hpet broadcast for deep C state
+ */
 int force_hpet_broadcast;
 boolean_param("hpetbroadcast", force_hpet_broadcast);
 
@@ -113,6 +118,9 @@ static int reprogram_hpet_evt_channel(
 {
     int64_t delta;
     int ret;
+
+    if ( ch->flags & HPET_EVT_DISALBE )
+        return 0;
 
     if ( unlikely(expire < 0) )
     {
@@ -483,6 +491,32 @@ static void (*hpet_attach_channel)(int c
 static void (*hpet_attach_channel)(int cpu, struct hpet_event_channel *ch);
 static void (*hpet_detach_channel)(int cpu);
 
+#include <asm/mc146818rtc.h>
+void cpuidle_disable_deep_cstate(void);
+
+void (*pv_rtc_handler)(unsigned int port, uint8_t value);
+
+static void handle_rtc_once(unsigned int port, uint8_t value)
+{
+    static int index;
+
+    if ( port == 0x70 )
+    {
+        index = value;
+        return;
+    }
+
+    if ( index != RTC_REG_B )
+        return;
+    
+    /* RTC Reg B, contain PIE/AIE/UIE */
+    if ( value & (RTC_PIE | RTC_AIE | RTC_UIE ) )
+    {
+        cpuidle_disable_deep_cstate();
+        pv_rtc_handler = NULL;
+    }
+}
+
 void hpet_broadcast_init(void)
 {
     u64 hpet_rate;
@@ -526,8 +560,11 @@ void hpet_broadcast_init(void)
         return;
     }
 
+    if ( legacy_hpet_event.flags & HPET_EVT_DISALBE )
+        return;
+
     hpet_id = hpet_read32(HPET_ID);
-    if ( !(hpet_id & HPET_ID_LEGSUP) || !force_hpet_broadcast )
+    if ( !(hpet_id & HPET_ID_LEGSUP) )
         return;
 
     /* Start HPET legacy interrupts */
@@ -555,6 +592,32 @@ void hpet_broadcast_init(void)
 
     for_each_cpu(i)
         per_cpu(cpu_bc_channel, i) = &legacy_hpet_event;
+
+    if ( !force_hpet_broadcast )
+        pv_rtc_handler = handle_rtc_once;
+}
+
+void hpet_disable_legacy_broadcast(void)
+{
+    u32 cfg;
+
+    spin_lock_irq(&legacy_hpet_event.lock);
+
+    legacy_hpet_event.flags |= HPET_EVT_DISALBE;
+
+    /* disable HPET T0 */
+    cfg = hpet_read32(HPET_T0_CFG);
+    cfg &= ~HPET_TN_ENABLE;
+    hpet_write32(cfg, HPET_T0_CFG);
+
+    /* Stop HPET legacy interrupts */
+    cfg = hpet_read32(HPET_CFG);
+    cfg &= ~HPET_CFG_LEGACY;
+    hpet_write32(cfg, HPET_CFG);
+
+    spin_unlock_irq(&legacy_hpet_event.lock);
+
+    smp_send_event_check_mask(cpu_online_map);
 }
 
 void hpet_broadcast_enter(void)
diff -r 34dca01addc9 xen/arch/x86/traps.c
--- a/xen/arch/x86/traps.c	Wed Apr 15 08:40:12 2009 +0100
+++ b/xen/arch/x86/traps.c	Wed Apr 15 20:41:21 2009 +0800
@@ -1551,6 +1551,8 @@ static uint32_t guest_io_read(
     return data;
 }
 
+extern void (*pv_rtc_handler)(unsigned int port, uint8_t value);
+
 static void guest_io_write(
     unsigned int port, unsigned int bytes, uint32_t data,
     struct vcpu *v, struct cpu_user_regs *regs)
@@ -1565,6 +1567,8 @@ static void guest_io_write(
             outb((uint8_t)data, port);
             if ( pv_post_outb_hook )
                 pv_post_outb_hook(port, (uint8_t)data);
+            if ( ((port == 0x71) || (port == 0x70)) && pv_rtc_handler )
+                pv_rtc_handler(port, (uint8_t)data);
             break;
         case 2:
             outw((uint16_t)data, port);
@@ -1936,6 +1940,8 @@ static int emulate_privileged_op(struct 
             io_emul(regs);            
             if ( (op_bytes == 1) && pv_post_outb_hook )
                 pv_post_outb_hook(port, regs->eax);
+            if ( ((port == 0x71) || (port == 0x70)) && pv_rtc_handler )
+                pv_rtc_handler(port, regs->eax);
         }
         else
         {
diff -r 34dca01addc9 xen/include/asm-x86/hpet.h
--- a/xen/include/asm-x86/hpet.h	Wed Apr 15 08:40:12 2009 +0100
+++ b/xen/include/asm-x86/hpet.h	Wed Apr 15 20:41:22 2009 +0800
@@ -78,5 +78,6 @@ void hpet_broadcast_enter(void);
 void hpet_broadcast_enter(void);
 void hpet_broadcast_exit(void);
 int hpet_broadcast_is_available(void);
+void hpet_disable_legacy_broadcast(void);
 
 #endif /* __X86_HPET_H__ */

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Time goes backwards in dom0 in xen-unstable

[Dan Magenheimer]

I can confirm that cpuidle=off makes the timer_interrupt
scaleability problem go away.

It also appears that the max cycles for the MSI
interrupts becomes reasonably small again.  Was
this expected?

I'll leave it running for awhile but may not be
able to confirm the "Time went backwards" error
goes away as it seemed to appear only after a
random long period of time.

(BTW, Kevin, hpetbroadcast did not make the problem
go away.)

Dan

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Wednesday, April 15, 2009 1:31 AM
> To: Dan Magenheimer; Xen-Devel (E-mail)
> Subject: Re: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> 
> 
> On 14/04/2009 20:36, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > I'm seeing some "Time went backwards" errors reported
> > in dom0 in near-tip (c/s 19515) xen-unstable build.
> > It's rare and random and not reproducible, but here's
> > the report that just showed up.  There was no load
> > running at the time.
> > 
> > I can move to 3.4-rc1 if that would be helpful but I
> > don't remember seeing any time-related changes recently.
> > 
> > This was on my dual-core test machine which reports
> > lots of power management info during Xen boot.
> 
> If you specify 'cpuidle=off' as a Xen boot parameter, does 
> that make the
> timer_interrupt scalability problem go away, and this time backwards
> problem? I was going to enable by default in 3.4 but could go 
> the other way.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[PATCH] cpuidle: Enable hpet broadcast by default (RE: Time goes backwards in dom0 in xen-unstable)

[Jan Beulich]

>>> "Wei, Gang" <gang.wei@intel.com> 15.04.09 16:09 >>>
>cpuidle: Enable hpet broadcast by default
>
>And stop legacy hpet broadcast and limit max C-state to shallower state if RTC interrupts are enabled.

Shouldn't the pv_rtc_handler() hook be called *before* the output actually
happens?

Jan

RE: Time goes backwards in dom0 in xen-unstable

[Dan Magenheimer]

Hmmm... after only a few minutes with cpuidle=off,
my test domPV froze up after printing a number of
call traces starting with:

INFO: task xxx:nnn blocked for more than 480 seconds.

At the top of all of the traces is either
getnstimeofday+51 or io_schedule+44.

(Note that this PV domain is a 2.6.29 kernel... don't
know if the messages are the same on an older kernel.)

> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Wednesday, April 15, 2009 8:22 AM
> To: Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
> Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> 
> 
> I can confirm that cpuidle=off makes the timer_interrupt
> scaleability problem go away.
> 
> It also appears that the max cycles for the MSI
> interrupts becomes reasonably small again.  Was
> this expected?
> 
> I'll leave it running for awhile but may not be
> able to confirm the "Time went backwards" error
> goes away as it seemed to appear only after a
> random long period of time.
> 
> (BTW, Kevin, hpetbroadcast did not make the problem
> go away.)
> 
> Dan
> 
> > -----Original Message-----
> > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> > Sent: Wednesday, April 15, 2009 1:31 AM
> > To: Dan Magenheimer; Xen-Devel (E-mail)
> > Subject: Re: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> > 
> > 
> > On 14/04/2009 20:36, "Dan Magenheimer" 
> > <dan.magenheimer@oracle.com> wrote:
> > 
> > > I'm seeing some "Time went backwards" errors reported
> > > in dom0 in near-tip (c/s 19515) xen-unstable build.
> > > It's rare and random and not reproducible, but here's
> > > the report that just showed up.  There was no load
> > > running at the time.
> > > 
> > > I can move to 3.4-rc1 if that would be helpful but I
> > > don't remember seeing any time-related changes recently.
> > > 
> > > This was on my dual-core test machine which reports
> > > lots of power management info during Xen boot.
> > 
> > If you specify 'cpuidle=off' as a Xen boot parameter, does 
> > that make the
> > timer_interrupt scalability problem go away, and this time backwards
> > problem? I was going to enable by default in 3.4 but could go 
> > the other way.
> > 
> >  -- Keir
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >

RE: Time goes backwards in dom0 in xen-unstable

[Dan Magenheimer]

Double hmmm... after destroying the domain and restarting
it has been running fine for over two hours.  So
apparently it was a coincidence.

> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Wednesday, April 15, 2009 8:59 AM
> To: Dan Magenheimer; Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
> Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> 
> 
> Hmmm... after only a few minutes with cpuidle=off,
> my test domPV froze up after printing a number of
> call traces starting with:
> 
> INFO: task xxx:nnn blocked for more than 480 seconds.
> 
> At the top of all of the traces is either
> getnstimeofday+51 or io_schedule+44.
> 
> (Note that this PV domain is a 2.6.29 kernel... don't
> know if the messages are the same on an older kernel.)
> 
> > -----Original Message-----
> > From: Dan Magenheimer 
> > Sent: Wednesday, April 15, 2009 8:22 AM
> > To: Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
> > Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> > 
> > 
> > I can confirm that cpuidle=off makes the timer_interrupt
> > scaleability problem go away.
> > 
> > It also appears that the max cycles for the MSI
> > interrupts becomes reasonably small again.  Was
> > this expected?
> > 
> > I'll leave it running for awhile but may not be
> > able to confirm the "Time went backwards" error
> > goes away as it seemed to appear only after a
> > random long period of time.
> > 
> > (BTW, Kevin, hpetbroadcast did not make the problem
> > go away.)
> > 
> > Dan
> > 
> > > -----Original Message-----
> > > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> > > Sent: Wednesday, April 15, 2009 1:31 AM
> > > To: Dan Magenheimer; Xen-Devel (E-mail)
> > > Subject: Re: [Xen-devel] Time goes backwards in dom0 in 
> xen-unstable
> > > 
> > > 
> > > On 14/04/2009 20:36, "Dan Magenheimer" 
> > > <dan.magenheimer@oracle.com> wrote:
> > > 
> > > > I'm seeing some "Time went backwards" errors reported
> > > > in dom0 in near-tip (c/s 19515) xen-unstable build.
> > > > It's rare and random and not reproducible, but here's
> > > > the report that just showed up.  There was no load
> > > > running at the time.
> > > > 
> > > > I can move to 3.4-rc1 if that would be helpful but I
> > > > don't remember seeing any time-related changes recently.
> > > > 
> > > > This was on my dual-core test machine which reports
> > > > lots of power management info during Xen boot.
> > > 
> > > If you specify 'cpuidle=off' as a Xen boot parameter, does 
> > > that make the
> > > timer_interrupt scalability problem go away, and this 
> time backwards
> > > problem? I was going to enable by default in 3.4 but could go 
> > > the other way.
> > > 
> > >  -- Keir
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > >

RE: Time goes backwards in dom0 in xen-unstable

[Dan Magenheimer]

Back to the first hmmm...  The same problem
arose after about three hours.  Exactly the
same symptoms.  Note that my test VM has
4 vcpus but is otherwise unremarkable.

I'll try again with tip (post-rc2) tomorrow.

> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Wednesday, April 15, 2009 11:12 AM
> To: Dan Magenheimer; Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
> Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> 
> 
> Double hmmm... after destroying the domain and restarting
> it has been running fine for over two hours.  So
> apparently it was a coincidence.
> 
> > -----Original Message-----
> > From: Dan Magenheimer 
> > Sent: Wednesday, April 15, 2009 8:59 AM
> > To: Dan Magenheimer; Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
> > Subject: RE: [Xen-devel] Time goes backwards in dom0 in xen-unstable
> > 
> > 
> > Hmmm... after only a few minutes with cpuidle=off,
> > my test domPV froze up after printing a number of
> > call traces starting with:
> > 
> > INFO: task xxx:nnn blocked for more than 480 seconds.
> > 
> > At the top of all of the traces is either
> > getnstimeofday+51 or io_schedule+44.
> > 
> > (Note that this PV domain is a 2.6.29 kernel... don't
> > know if the messages are the same on an older kernel.)
> > 
> > > -----Original Message-----
> > > From: Dan Magenheimer 
> > > Sent: Wednesday, April 15, 2009 8:22 AM
> > > To: Keir Fraser; Xen-Devel (E-mail); Tian, Kevin
> > > Subject: RE: [Xen-devel] Time goes backwards in dom0 in 
> xen-unstable
> > > 
> > > 
> > > I can confirm that cpuidle=off makes the timer_interrupt
> > > scaleability problem go away.
> > > 
> > > It also appears that the max cycles for the MSI
> > > interrupts becomes reasonably small again.  Was
> > > this expected?
> > > 
> > > I'll leave it running for awhile but may not be
> > > able to confirm the "Time went backwards" error
> > > goes away as it seemed to appear only after a
> > > random long period of time.
> > > 
> > > (BTW, Kevin, hpetbroadcast did not make the problem
> > > go away.)
> > > 
> > > Dan
> > > 
> > > > -----Original Message-----
> > > > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> > > > Sent: Wednesday, April 15, 2009 1:31 AM
> > > > To: Dan Magenheimer; Xen-Devel (E-mail)
> > > > Subject: Re: [Xen-devel] Time goes backwards in dom0 in 
> > xen-unstable
> > > > 
> > > > 
> > > > On 14/04/2009 20:36, "Dan Magenheimer" 
> > > > <dan.magenheimer@oracle.com> wrote:
> > > > 
> > > > > I'm seeing some "Time went backwards" errors reported
> > > > > in dom0 in near-tip (c/s 19515) xen-unstable build.
> > > > > It's rare and random and not reproducible, but here's
> > > > > the report that just showed up.  There was no load
> > > > > running at the time.
> > > > > 
> > > > > I can move to 3.4-rc1 if that would be helpful but I
> > > > > don't remember seeing any time-related changes recently.
> > > > > 
> > > > > This was on my dual-core test machine which reports
> > > > > lots of power management info during Xen boot.
> > > > 
> > > > If you specify 'cpuidle=off' as a Xen boot parameter, does 
> > > > that make the
> > > > timer_interrupt scalability problem go away, and this 
> > time backwards
> > > > problem? I was going to enable by default in 3.4 but could go 
> > > > the other way.
> > > > 
> > > >  -- Keir
> > > > 
> > > > 
> > > > 
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: [PATCH] cpuidle: Enable hpet broadcast by default (RE: Time goes backwards in dom0 in xen-unstable)

[Wei, Gang]

On Wednesday, April 15, 2009 10:38 PM, Jan Beulich wrote:
>>>> "Wei, Gang" <gang.wei@intel.com> 15.04.09 16:09 >>>
>> cpuidle: Enable hpet broadcast by default
>> 
>> And stop legacy hpet broadcast and limit max C-state to shallower state if
>> RTC interrupts are enabled. 
> 
> Shouldn't the pv_rtc_handler() hook be called *before* the output actually
> happens?

Your are right, that would be safer. As this patch has already been checked in, could you send out the fix? It is your good point. :)

Jimmy