2.6.37 dom0 under Xen 4.1 clocksource not working

2.6.37 dom0 under Xen 4.1 clocksource not working

[Daniel De Graaf]

When booting 2.6.37 (from kernel.org) under Xen 4.1, there are
apparently issues with timer interrupts being delivered to the dom0.
Repeatedly causing an interrupt (for example, the ACPI power button) will
allow the system to boot, as does adding "clocksource=jiffies" to the
kernel command line. This is only present under Xen, not when booting the
same kernel on real hardware.

When boot with clocksource=jiffies, the system will sometimes freeze
for about 30 seconds and then continue; however, this could be related
to kernel debugging that I have enabled.

The only clock-related message in dmesg is:
[  153.483813] Clocksource tsc unstable (delta = -77312555973 ns)

/sys/devices/system/clocksource/clocksource0/available_clocksource:
  xen jiffies tsc

I can attach kernel config or other information as required. I have
observed this on at least two platforms; the 2.6.32.x xen kernels do not
exhibit this behavior.

-- 
Daniel De Graaf
National Security Agency

Re: 2.6.37 dom0 under Xen 4.1 clocksource not working

[Konrad Rzeszutek Wilk]

On Mon, Jan 10, 2011 at 10:06:19AM -0500, Daniel De Graaf wrote:
> When booting 2.6.37 (from kernel.org) under Xen 4.1, there are
> apparently issues with timer interrupts being delivered to the dom0.
> Repeatedly causing an interrupt (for example, the ACPI power button) will
> allow the system to boot, as does adding "clocksource=jiffies" to the
> kernel command line. This is only present under Xen, not when booting the
> same kernel on real hardware.
> 
> When boot with clocksource=jiffies, the system will sometimes freeze
> for about 30 seconds and then continue; however, this could be related
> to kernel debugging that I have enabled.
> 
> The only clock-related message in dmesg is:
> [  153.483813] Clocksource tsc unstable (delta = -77312555973 ns)

I see it too.
> 
> /sys/devices/system/clocksource/clocksource0/available_clocksource:
>   xen jiffies tsc
> 
> I can attach kernel config or other information as required. I have
> observed this on at least two platforms; the 2.6.32.x xen kernels do not
> exhibit this behavior.

I think we are just missing some of the Xen time code that is in 2.6.32
but has not been ported over to 2.6.37. I don't remember the name of the
patches thought..

Re: 2.6.37 dom0 under Xen 4.1 clocksource not working

[Jeremy Fitzhardinge]

On 01/10/2011 07:06 AM, Daniel De Graaf wrote:
> When booting 2.6.37 (from kernel.org) under Xen 4.1, there are
> apparently issues with timer interrupts being delivered to the dom0.
> Repeatedly causing an interrupt (for example, the ACPI power button) will
> allow the system to boot, as does adding "clocksource=jiffies" to the
> kernel command line. This is only present under Xen, not when booting the
> same kernel on real hardware.

Ah, yes.  This is from the cpuidle stuff, I think.  Does booting with
"idle=halt" work?

> When boot with clocksource=jiffies, the system will sometimes freeze
> for about 30 seconds and then continue; however, this could be related
> to kernel debugging that I have enabled.
>
> The only clock-related message in dmesg is:
> [  153.483813] Clocksource tsc unstable (delta = -77312555973 ns)
>
> /sys/devices/system/clocksource/clocksource0/available_clocksource:
>   xen jiffies tsc
>
> I can attach kernel config or other information as required. I have
> observed this on at least two platforms; the 2.6.32.x xen kernels do not
> exhibit this behavior.
>

    J

Re: 2.6.37 dom0 under Xen 4.1 clocksource not working

[Daniel De Graaf]

On 01/26/2011 10:40 AM, Jeremy Fitzhardinge wrote:
> On 01/10/2011 07:06 AM, Daniel De Graaf wrote:
>> When booting 2.6.37 (from kernel.org) under Xen 4.1, there are
>> apparently issues with timer interrupts being delivered to the dom0.
>> Repeatedly causing an interrupt (for example, the ACPI power button) will
>> allow the system to boot, as does adding "clocksource=jiffies" to the
>> kernel command line. This is only present under Xen, not when booting the
>> same kernel on real hardware.
> 
> Ah, yes.  This is from the cpuidle stuff, I think.  Does booting with
> "idle=halt" work?
> 

Yes, clocksource=xen idle=halt works.

>> When boot with clocksource=jiffies, the system will sometimes freeze
>> for about 30 seconds and then continue; however, this could be related
>> to kernel debugging that I have enabled.
>>
>> The only clock-related message in dmesg is:
>> [  153.483813] Clocksource tsc unstable (delta = -77312555973 ns)
>>
>> /sys/devices/system/clocksource/clocksource0/available_clocksource:
>>   xen jiffies tsc
>>
>> I can attach kernel config or other information as required. I have
>> observed this on at least two platforms; the 2.6.32.x xen kernels do not
>> exhibit this behavior.
>>

Re: 2.6.37 dom0 under Xen 4.1 clocksource not working

[Stefano Stabellini]

On Wed, 26 Jan 2011, Daniel De Graaf wrote:
> On 01/26/2011 10:40 AM, Jeremy Fitzhardinge wrote:
> > On 01/10/2011 07:06 AM, Daniel De Graaf wrote:
> >> When booting 2.6.37 (from kernel.org) under Xen 4.1, there are
> >> apparently issues with timer interrupts being delivered to the dom0.
> >> Repeatedly causing an interrupt (for example, the ACPI power button) will
> >> allow the system to boot, as does adding "clocksource=jiffies" to the
> >> kernel command line. This is only present under Xen, not when booting the
> >> same kernel on real hardware.
> > 
> > Ah, yes.  This is from the cpuidle stuff, I think.  Does booting with
> > "idle=halt" work?
> > 
> 
> Yes, clocksource=xen idle=halt works.

I have a testbox that can reproduce this problem.
It seems to be caused by the cpuidle driver that overrides pm_idle
unless boot_option_idle_override is set to something other than the
default.
This patch fixes the problem for me:

---

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index a3d28a1..7e85979 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -353,6 +353,7 @@ void __init xen_arch_setup(void)
 	boot_cpu_data.hlt_works_ok = 1;
 #endif
 	pm_idle = default_idle;
+	boot_option_idle_override = IDLE_HALT;
 
 	fiddle_vdso();
 }

Re: 2.6.37 dom0 under Xen 4.1 clocksource not working

[Konrad Rzeszutek Wilk]

On Wed, Jan 26, 2011 at 05:07:27PM +0000, Stefano Stabellini wrote:
> On Wed, 26 Jan 2011, Daniel De Graaf wrote:
> > On 01/26/2011 10:40 AM, Jeremy Fitzhardinge wrote:
> > > On 01/10/2011 07:06 AM, Daniel De Graaf wrote:
> > >> When booting 2.6.37 (from kernel.org) under Xen 4.1, there are
> > >> apparently issues with timer interrupts being delivered to the dom0.
> > >> Repeatedly causing an interrupt (for example, the ACPI power button) will
> > >> allow the system to boot, as does adding "clocksource=jiffies" to the
> > >> kernel command line. This is only present under Xen, not when booting the
> > >> same kernel on real hardware.
> > > 
> > > Ah, yes.  This is from the cpuidle stuff, I think.  Does booting with
> > > "idle=halt" work?
> > > 
> > 
> > Yes, clocksource=xen idle=halt works.
> 
> I have a testbox that can reproduce this problem.
> It seems to be caused by the cpuidle driver that overrides pm_idle
> unless boot_option_idle_override is set to something other than the
> default.
> This patch fixes the problem for me:
> 
> ---
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index a3d28a1..7e85979 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -353,6 +353,7 @@ void __init xen_arch_setup(void)
>  	boot_cpu_data.hlt_works_ok = 1;
>  #endif
>  	pm_idle = default_idle;
> +	boot_option_idle_override = IDLE_HALT;

Is this how 2.6.32 did it? Or was there no cpuidle driver in 2.6.32 so
we never had tried this? With this being overwridden to HALT does that
mean the hypervisor cycles through the Cx states? What about using
MWAIT instead? How does this impact AMD platforms?

>  
>  	fiddle_vdso();
>  }
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: 2.6.37 dom0 under Xen 4.1 clocksource not working

[Jeremy Fitzhardinge]

On 01/26/2011 12:12 PM, Konrad Rzeszutek Wilk wrote:
> Is this how 2.6.32 did it? Or was there no cpuidle driver in 2.6.32 so
> we never had tried this?

CPUIDLE is new.

>  With this being overwridden to HALT does that
> mean the hypervisor cycles through the Cx states? What about using
> MWAIT instead? How does this impact AMD platforms?

"halt" in this case is the safe_halt pvop, so it ends up going into the
appropriate hypercall.

    J