Kernel crash with acpi_processor, cpu_idle and intel_idle =y

Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Hi everyone,

When I set CONFIG_ACPI_PROCESSOR, CONFIG_CPU_IDLE and CONFIG_INTEL_IDLE
to y then I cannot boot Xen; there is a crash. If I turn
CONFIG_INTEL_IDLE off then the boot goes well and, after dom0 has
booted, xenpm works and gives some sane output, see below. I have
tested this with kernels 3.2 to 3.4.6. 

Is it impossible to use INTEL_IDLE with Xen? If this is a known issue
then maybe someone can add info to the INTEL_IDLE help text in the
kernel configuration...

I am using this on a dual CPU motherboard, it has two Xeon W3530 CPUs
(i.e. family 6, model 26, stepping 5). I have also tested this with a
single-CPU Core2 Quad Q6600 and the same situation occurs here, but the
below output is of the W3530 Xeon system.

While I'm not that familiar with CPUidle, one thing that seems to be
not right is that the maximum idle state here is C3 while the processor
should be able to reach as far as C7.

Here is the snipped output of two commands:

# xenpm get-cpufreq-states
cpu id               : 0
total P-states       : 11
usable P-states      : 11
current frequency    : 1600 MHz
P0         [2801 MHz]: transition [                  11]
                       residency  [                 866 ms]
P1         [2800 MHz]: transition [                   1]
                       residency  [                   0 ms]
P2         [2667 MHz]: transition [                   0]
                       residency  [                   0 ms]
P3         [2533 MHz]: transition [                   1]
                       residency  [                   1 ms]
P4         [2400 MHz]: transition [                   0]
                       residency  [                   0 ms]
P5         [2267 MHz]: transition [                   2]
                       residency  [                  36 ms]
P6         [2133 MHz]: transition [                   1]
                       residency  [                   0 ms]
P7         [2000 MHz]: transition [                   0]
                       residency  [                   0 ms]
P8         [1867 MHz]: transition [                   0]
                       residency  [                   0 ms]
P9         [1733 MHz]: transition [                   2]
                       residency  [                   0 ms]
*P10       [1600 MHz]: transition [                   9]
                       residency  [                 960 ms]


# xenpm get-cpuidle-states           
Max possible C-state: C7

cpu id               : 0
total C-states       : 4
idle time(ms)        : 240266
C0                   : transition [               39457]
                       residency  [               10534 ms]
C1                   : transition [                3965]
                       residency  [                2274 ms]
C2                   : transition [                 371]
                       residency  [                 495 ms]
C3                   : transition [               35121]
                       residency  [              230822 ms]
pc2                  : [                   0 ms]
pc3                  : [                   0 ms]
pc6                  : [                   0 ms]
pc7                  : [                   0 ms]
cc3                  : [                   0 ms]
cc6                  : [                   0 ms]
cc7                  : [                   0 ms]

# xenpm get-cpufreq-para             
cpu id               : 0
affected_cpus        : 0
cpuinfo frequency    : max [2801000] min [1600000] cur [1600000]
scaling_driver       : acpi-cpufreq
scaling_avail_gov    : userspace performance powersave ondemand
current_governor     : ondemand
  ondemand specific  :
    sampling_rate    : max [10000000] min [10000] cur [20000]
    up_threshold     : 80
scaling_avail_freq   : 2801000 2800000 2667000 2533000 2400000 2267000
2133000 2000000 1867000 1733000 *1600000 scaling frequency    : max
[2801000] min [1600000] cur [1600000] turbo mode           : enabled

Kind regards,
Mark van Dijk

PS I am not receiving xen-devel messages but this message does
probably belong there so I'm posting it there too.



-- 

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Sun, Jul 22, 2012 at 06:16:11PM +0200, Mark van Dijk wrote:
> Hi everyone,
> 
> When I set CONFIG_ACPI_PROCESSOR, CONFIG_CPU_IDLE and CONFIG_INTEL_IDLE
> to y then I cannot boot Xen; there is a crash. If I turn
> CONFIG_INTEL_IDLE off then the boot goes well and, after dom0 has
> booted, xenpm works and gives some sane output, see below. I have
> tested this with kernels 3.2 to 3.4.6. 
> 
> Is it impossible to use INTEL_IDLE with Xen? If this is a known issue
> then maybe someone can add info to the INTEL_IDLE help text in the
> kernel configuration...

Could be - without a serial crash it is hard to figure out. 

Why do you want to use the intel idle driver? Can't you use
the xen-acpi-processor driver which does the job of uploading
the power management data to the hypervisor.

(CONFIG_XEN_ACPI_PROCESSOR is the option you need to have enabled).

> 
> I am using this on a dual CPU motherboard, it has two Xeon W3530 CPUs
> (i.e. family 6, model 26, stepping 5). I have also tested this with a
> single-CPU Core2 Quad Q6600 and the same situation occurs here, but the
> below output is of the W3530 Xeon system.
> 
> While I'm not that familiar with CPUidle, one thing that seems to be
> not right is that the maximum idle state here is C3 while the processor
> should be able to reach as far as C7.

Right, it won't unless you don't compile acpi_pad (CONFIG_ACPI_PROCESSOR).
Is # CONFIG_ACPI_PROCESSOR  is not set"  in your .config?


> 
> Here is the snipped output of two commands:
> 
> # xenpm get-cpufreq-states
> cpu id               : 0
> total P-states       : 11
> usable P-states      : 11
> current frequency    : 1600 MHz
> P0         [2801 MHz]: transition [                  11]
>                        residency  [                 866 ms]
> P1         [2800 MHz]: transition [                   1]
>                        residency  [                   0 ms]
> P2         [2667 MHz]: transition [                   0]
>                        residency  [                   0 ms]
> P3         [2533 MHz]: transition [                   1]
>                        residency  [                   1 ms]
> P4         [2400 MHz]: transition [                   0]
>                        residency  [                   0 ms]
> P5         [2267 MHz]: transition [                   2]
>                        residency  [                  36 ms]
> P6         [2133 MHz]: transition [                   1]
>                        residency  [                   0 ms]
> P7         [2000 MHz]: transition [                   0]
>                        residency  [                   0 ms]
> P8         [1867 MHz]: transition [                   0]
>                        residency  [                   0 ms]
> P9         [1733 MHz]: transition [                   2]
>                        residency  [                   0 ms]
> *P10       [1600 MHz]: transition [                   9]
>                        residency  [                 960 ms]
> 
> 
> # xenpm get-cpuidle-states           
> Max possible C-state: C7
> 
> cpu id               : 0
> total C-states       : 4
> idle time(ms)        : 240266
> C0                   : transition [               39457]
>                        residency  [               10534 ms]
> C1                   : transition [                3965]
>                        residency  [                2274 ms]
> C2                   : transition [                 371]
>                        residency  [                 495 ms]
> C3                   : transition [               35121]
>                        residency  [              230822 ms]
> pc2                  : [                   0 ms]
> pc3                  : [                   0 ms]
> pc6                  : [                   0 ms]
> pc7                  : [                   0 ms]
> cc3                  : [                   0 ms]
> cc6                  : [                   0 ms]
> cc7                  : [                   0 ms]
> 
> # xenpm get-cpufreq-para             
> cpu id               : 0
> affected_cpus        : 0
> cpuinfo frequency    : max [2801000] min [1600000] cur [1600000]
> scaling_driver       : acpi-cpufreq
> scaling_avail_gov    : userspace performance powersave ondemand
> current_governor     : ondemand
>   ondemand specific  :
>     sampling_rate    : max [10000000] min [10000] cur [20000]
>     up_threshold     : 80
> scaling_avail_freq   : 2801000 2800000 2667000 2533000 2400000 2267000
> 2133000 2000000 1867000 1733000 *1600000 scaling frequency    : max
> [2801000] min [1600000] cur [1600000] turbo mode           : enabled
> 
> Kind regards,
> Mark van Dijk
> 
> PS I am not receiving xen-devel messages but this message does
> probably belong there so I'm posting it there too.
> 
> 
> 
> -- 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Jan Beulich]

>>> On 22.07.12 at 18:16, Mark van Dijk <lists+xen@internecto.net> wrote:
> When I set CONFIG_ACPI_PROCESSOR, CONFIG_CPU_IDLE and CONFIG_INTEL_IDLE
> to y then I cannot boot Xen; there is a crash.

Without providing us with the details of the crash, I don't think
anyone will be able to help here.

> Is it impossible to use INTEL_IDLE with Xen? If this is a known issue
> then maybe someone can add info to the INTEL_IDLE help text in the
> kernel configuration...

All CPU idle management happens in Xen, and the corresponding
code in the Dom0 (and DomU) kernel gets turned off. There's no
need to clutter the kernel config option with Xen specific information
(the kernel ought to work irrespective of its setting).

> PS I am not receiving xen-devel messages but this message does
> probably belong there so I'm posting it there too.

But nevertheless please don't cross-post.

Jan

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Hi Konrad,

> > When I set CONFIG_ACPI_PROCESSOR, CONFIG_CPU_IDLE and
> > CONFIG_INTEL_IDLE to y then I cannot boot Xen; there is a crash. If
> > I turn CONFIG_INTEL_IDLE off then the boot goes well and, after
> > dom0 has booted, xenpm works and gives some sane output, see below.
> > I have tested this with kernels 3.2 to 3.4.6. 
> > 
> > Is it impossible to use INTEL_IDLE with Xen? If this is a known
> > issue then maybe someone can add info to the INTEL_IDLE help text
> > in the kernel configuration...
> 
> Could be - without a serial crash it is hard to figure out.

Alright, I will enable INTEL_IDLE again so that I can report the crash
message. Won't be now but it will be within the next few days.

> Why do you want to use the intel idle driver? Can't you use
> the xen-acpi-processor driver which does the job of uploading
> the power management data to the hypervisor.
> 
> (CONFIG_XEN_ACPI_PROCESSOR is the option you need to have enabled).

I just checked - I have enabled this option. It is in Device Drivers ->
Xen driver support which did confuse me somewhat, partially because the
rest of the ACPI options are under 'Power management and ACPI
options', also because the options look alike so much, I suppose
fatigue kicked in. :)

Anyway, It seems that CONFIG_XEN_ACPI_PROCESSOR depends on
ACPI_PROCESSOR, see below.

To answer your question about why I want to use the intel idle driver.
Actually I don't want/need to use it per se.. I simply thought it was
the logical choice to enable it in order to be able to use Intel
specific idle instructions.

> > I am using this on a dual CPU motherboard, it has two Xeon W3530
> > CPUs (i.e. family 6, model 26, stepping 5). I have also tested this
> > with a single-CPU Core2 Quad Q6600 and the same situation occurs
> > here, but the below output is of the W3530 Xeon system.
> > 
> > While I'm not that familiar with CPUidle, one thing that seems to be
> > not right is that the maximum idle state here is C3 while the
> > processor should be able to reach as far as C7.
> 
> Right, it won't unless you don't compile acpi_pad
> (CONFIG_ACPI_PROCESSOR). Is # CONFIG_ACPI_PROCESSOR  is not set"  in
> your .config?

No, and this is where I start to wonder. I have built in the CPUFREQ
drivers and the kernel help text says that when this is the case,
CONFIG_XEN_ACPI_PROCESSOR must be Y as well. For this to work I have to
set CONFIG_ACPI_PROCESSOR to Y because:

XEN_ACPI_PROCESSOR [=y]
Depends on: XEN [=y] && X86 [=y] && ACPI_PROCESSOR [=y] && CPU_FREQ [=y]

The full .config is here:
http://pastebin.com/36a2tVYj

What I am after: cpu scheduling (xenpm set-cpufreq-governor) and
maximum idle states. But you can see that I am not as knowledgable as
yourself, so further hints or requests for debugging are welcome (the
INTEL_IDLE boot crash message will be posted when I am able to fiddle
with the kernel again) and thanks so far for your answers.

Mark

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Hi Jan,

> > When I set CONFIG_ACPI_PROCESSOR, CONFIG_CPU_IDLE and
> > CONFIG_INTEL_IDLE to y then I cannot boot Xen; there is a crash.
> 
> Without providing us with the details of the crash, I don't think
> anyone will be able to help here.

Yes, I'll post those as soon as I can.

> > Is it impossible to use INTEL_IDLE with Xen? If this is a known
> > issue then maybe someone can add info to the INTEL_IDLE help text
> > in the kernel configuration...
> 
> All CPU idle management happens in Xen, and the corresponding
> code in the Dom0 (and DomU) kernel gets turned off. There's no
> need to clutter the kernel config option with Xen specific information
> (the kernel ought to work irrespective of its setting).

Here is a message I posted a month ago:
http://lists.xen.org/archives/html/xen-users/2012-06/msg00403.html

Since then I found that the behaviour of xenpm changed to a working
state when CONFIG_ACPI_PROCESSOR=y. See my reply, posted a couple of
minutes ago, to Konrad and xen-devel for more info.

> > PS I am not receiving xen-devel messages but this message does
> > probably belong there so I'm posting it there too.
> 
> But nevertheless please don't cross-post.

Alright sorry, xen-users has been removed.

Mark

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Jan Beulich]

>>> On 26.07.12 at 02:25, Mark van Dijk <lists+xen@internecto.net> wrote:
> To answer your question about why I want to use the intel idle driver.
> Actually I don't want/need to use it per se.. I simply thought it was
> the logical choice to enable it in order to be able to use Intel
> specific idle instructions.

Just to reiterate - this is only for the case where you want to
run the same kernel natively (i.e. without Xen underneath).
When Xen is involved, all enhanced CPU idle and frequency
management are done in the hypervisor, merely requiring
Dom0 to forward relevant ACPI data to it.

However - given that the specific Intel idle driver appear to
be preferred over the ACPI based one in native Linux, I have
already on my todo list an item to check whether that driver
can reasonably be ported over into the hypervisor. The major
factor preventing me from doing this is that I don't have
immediate access to a system where that driver would
actually get involved, i.e. I wouldn't be able to test the
ported code.

Jan

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Thu, Jul 26, 2012 at 02:25:27AM +0200, Mark van Dijk wrote:
> Hi Konrad,
> 
> > > When I set CONFIG_ACPI_PROCESSOR, CONFIG_CPU_IDLE and
> > > CONFIG_INTEL_IDLE to y then I cannot boot Xen; there is a crash. If
> > > I turn CONFIG_INTEL_IDLE off then the boot goes well and, after
> > > dom0 has booted, xenpm works and gives some sane output, see below.
> > > I have tested this with kernels 3.2 to 3.4.6. 
> > > 
> > > Is it impossible to use INTEL_IDLE with Xen? If this is a known
> > > issue then maybe someone can add info to the INTEL_IDLE help text
> > > in the kernel configuration...
> > 
> > Could be - without a serial crash it is hard to figure out.
> 
> Alright, I will enable INTEL_IDLE again so that I can report the crash
> message. Won't be now but it will be within the next few days.
> 
> > Why do you want to use the intel idle driver? Can't you use
> > the xen-acpi-processor driver which does the job of uploading
> > the power management data to the hypervisor.
> > 
> > (CONFIG_XEN_ACPI_PROCESSOR is the option you need to have enabled).
> 
> I just checked - I have enabled this option. It is in Device Drivers ->
> Xen driver support which did confuse me somewhat, partially because the
> rest of the ACPI options are under 'Power management and ACPI
> options', also because the options look alike so much, I suppose
> fatigue kicked in. :)
> 
> Anyway, It seems that CONFIG_XEN_ACPI_PROCESSOR depends on
> ACPI_PROCESSOR, see below.
> 
> To answer your question about why I want to use the intel idle driver.
> Actually I don't want/need to use it per se.. I simply thought it was
> the logical choice to enable it in order to be able to use Intel
> specific idle instructions.
> 
> > > I am using this on a dual CPU motherboard, it has two Xeon W3530
> > > CPUs (i.e. family 6, model 26, stepping 5). I have also tested this
> > > with a single-CPU Core2 Quad Q6600 and the same situation occurs
> > > here, but the below output is of the W3530 Xeon system.
> > > 
> > > While I'm not that familiar with CPUidle, one thing that seems to be
> > > not right is that the maximum idle state here is C3 while the
> > > processor should be able to reach as far as C7.
> > 
> > Right, it won't unless you don't compile acpi_pad
> > (CONFIG_ACPI_PROCESSOR). Is # CONFIG_ACPI_PROCESSOR  is not set"  in
> > your .config?
> 
> No, and this is where I start to wonder. I have built in the CPUFREQ
> drivers and the kernel help text says that when this is the case,
> CONFIG_XEN_ACPI_PROCESSOR must be Y as well. For this to work I have to
> set CONFIG_ACPI_PROCESSOR to Y because:
> 
> XEN_ACPI_PROCESSOR [=y]
> Depends on: XEN [=y] && X86 [=y] && ACPI_PROCESSOR [=y] && CPU_FREQ [=y]
> 
> The full .config is here:
> http://pastebin.com/36a2tVYj
> 
> What I am after: cpu scheduling (xenpm set-cpufreq-governor) and
> maximum idle states. But you can see that I am not as knowledgable as
> yourself, so further hints or requests for debugging are welcome (the
> INTEL_IDLE boot crash message will be posted when I am able to fiddle
> with the kernel again) and thanks so far for your answers.

So.. in that case make sure you have XEN_ACPI_PROCESSOR=y and
"CONFIG_INTEL_IDLE is not set" and that should do it.

I got the ACPI_PAD and ACPI_PROCESSOR confused. The option I wanted
you to unset is: CONFIG_ACPI_PROCESSOR_AGGREGATOR, sorry about that.
If you unset that ("# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set") that
should allow MWAIT and lower states to be reached by the hypervisor.

You can verify that by booting Xen with 'cpufreq=verbose' and during
the bootup should see in the 'xl dmesg' something like this:

(XEN) Set CPU acpi_id(0) cpuid(0) Px State info:^M
(XEN)   _PCT: descriptor=130, length=12, space_id=127, bit_width=64, bit_offset=0, reserved=0, address=3221291106^M
(XEN)   _PCT: descriptor=130, length=12, space_id=127, bit_width=64, bit_offset=0, reserved=0, address=0^M
(XEN)   _PSS: state_count=4^M
(XEN)   State0: 2900MHz 32062mW 8us 8us 0x0 0x0^M
(XEN)   State1: 2200MHz 27030mW 8us 8us 0x1 0x1^M
(XEN)   State2: 1700MHz 23152mW 8us 8us 0x2 0x2^M
(XEN)   State3: 800MHz 8325mW 8us 8us 0x3 0x3^M
(XEN)   _PSD: num_entries=5 rev=0 domain=0 coord_type=253 num_processors=1^M
(XEN)   _PPC: 0^M
(XEN) max_freq: 2900000    second_max_freq: 2200000^M
(XEN) CPU 0 initialization completed^M
(XEN) Set CPU acpi_id(1) cpuid(1) Px State info:^M
(XEN)   _PCT: descriptor=130, length=12, space_id=127, bit_width=64, bit_offset=0, reserved=0, address=3221291106^M
(XEN)   _PCT: descriptor=130, length=12, space_id=127, bit_width=64, bit_offset=0, reserved=0, address=0^M
(XEN)   _PSS: state_count=4^M
(XEN)   State0: 2900MHz 32062mW 8us 8us 0x0 0x0^M
(XEN)   State1: 2200MHz 27030mW 8us 8us 0x1 0x1^M
(XEN)   State2: 1700MHz 23152mW 8us 8us 0x2 0x2^M
(XEN)   State3: 800MHz 8325mW 8us 8us 0x3 0x3^M

or running xenpm get-cpufreq to check out the details.
> 
> Mark
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Thu, Jul 26, 2012 at 03:57:01PM +0100, Jan Beulich wrote:
> >>> On 26.07.12 at 16:05, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > So.. in that case make sure you have XEN_ACPI_PROCESSOR=y and
> > "CONFIG_INTEL_IDLE is not set" and that should do it.
> 
> Didn't he say that with INTEL_IDLE disabled things work (perhaps
> in a limited way, but at least don't crash)? My understanding is
> that things should work even with it enabled, and I was under the
> impression that you had already taken care of disabling cpuidle
> and cpufreq in the kernel when running on Xen...

So did I - both of those (cpufreq and cpuidle) are disabled.

Mark, any chance you can collect the serial output when it crashes please?
Actually, can you get the whole bootup log with 'loglevel=8 debug' on the Linux
command line?

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Jan Beulich]

>>> On 26.07.12 at 16:05, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> So.. in that case make sure you have XEN_ACPI_PROCESSOR=y and
> "CONFIG_INTEL_IDLE is not set" and that should do it.

Didn't he say that with INTEL_IDLE disabled things work (perhaps
in a limited way, but at least don't crash)? My understanding is
that things should work even with it enabled, and I was under the
impression that you had already taken care of disabling cpuidle
and cpufreq in the kernel when running on Xen...

Jan

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

> > > So.. in that case make sure you have XEN_ACPI_PROCESSOR=y and
> > > "CONFIG_INTEL_IDLE is not set" and that should do it.
> > 
> > Didn't he say that with INTEL_IDLE disabled things work (perhaps
> > in a limited way, but at least don't crash)? My understanding is
> > that things should work even with it enabled, and I was under the
> > impression that you had already taken care of disabling cpuidle
> > and cpufreq in the kernel when running on Xen...
> 
> So did I - both of those (cpufreq and cpuidle) are disabled.
> 
> Mark, any chance you can collect the serial output when it crashes
> please? Actually, can you get the whole bootup log with 'loglevel=8
> debug' on the Linux command line?
> 

The sad part is that I don't have physical access to the box, the good
part is that I do have KVM and IPMI (SOL) access.

This is what I could capture:

http://i.imgur.com/t66FM.png

Note: this crash only occurs with Xen, not when I boot it natively.

Does it tell you anything, or should I find a way to grab more output?
I can login via SSH to the IPMI console but unfortunately it emulates a
small screen, if you know how I can get it to not do that and just use
my current width/height I'm happy to hear it.

The relevant kernel config file is here:
http://pastebin.com/G4hR9rB9

Then, in response to your previous message Konrad about
CONFIG_ACPI_PROCESSOR_AGGREGATOR, note that this is not set.

> If you unset that ("# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set")
> that should allow MWAIT and lower states to be reached by the
> hypervisor.

I am not sure where MWAIT is, lower than C3 I suppose. In any case, the
output of the xenpm commands in my first email of this topic are on a
kernel without CONFIG_ACPI_PROCESSOR_AGGREGATOR.

So on a working kernel (i.e. without intel_idle), here is the output of
xl dmesg when booted with 'cpufreq=verbose':

http://pastebin.com/WuaLVFLG

Here is the output of 'dmesg' on Linux:

http://pastebin.com/kq1qVWv5

Stay in touch,
Mark

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Jan Beulich]

>>> On 28.07.12 at 14:55, Mark van Dijk <lists+xen@internecto.net> wrote:
>> > > So.. in that case make sure you have XEN_ACPI_PROCESSOR=y and
>> > > "CONFIG_INTEL_IDLE is not set" and that should do it.
>> > 
>> > Didn't he say that with INTEL_IDLE disabled things work (perhaps
>> > in a limited way, but at least don't crash)? My understanding is
>> > that things should work even with it enabled, and I was under the
>> > impression that you had already taken care of disabling cpuidle
>> > and cpufreq in the kernel when running on Xen...
>> 
>> So did I - both of those (cpufreq and cpuidle) are disabled.
>> 
>> Mark, any chance you can collect the serial output when it crashes
>> please? Actually, can you get the whole bootup log with 'loglevel=8
>> debug' on the Linux command line?
>> 
> 
> The sad part is that I don't have physical access to the box, the good
> part is that I do have KVM and IPMI (SOL) access.
> 
> This is what I could capture:
> 
> http://i.imgur.com/t66FM.png 

The first thing intel_idle_init() does is check
boot_option_idle_override, and I thought this got forced to
something other than IDLE_NO_OVERRIDE by the Xen code.
But at least in the current kernel that doesn't seem to be
the case anymore, and thus I suppose that it's dying on the
subsequent

	retval = cpuidle_register_driver(&intel_idle_driver);
	if (retval) {
		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
			cpuidle_get_driver()->name);
		return retval;
	}

since cpuidle_get_driver() is presumably returning NULL (after
all xen_arch_setup() does call disable_cpuidle()).

Jan

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Message 501657C70200007800091357@nat28.tlf.novell.com contained:
>The first thing intel_idle_init() does is check
>boot_option_idle_override, and I thought this got forced to
>something other than IDLE_NO_OVERRIDE by the Xen code.
>But at least in the current kernel that doesn't seem to be
>the case anymore, and thus I suppose that it's dying on the
>subsequent
>
>	retval = cpuidle_register_driver(&intel_idle_driver);
>	if (retval) {
>		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
>			cpuidle_get_driver()->name);
>		return retval;
>	}
>
>since cpuidle_get_driver() is presumably returning NULL (after
>all xen_arch_setup() does call disable_cpuidle()).

This is way beyond my knowledge. I'm no programmer so unfortunately I
can't say anything about it.

Not using INTEL_IDLE is no problem, in fact, I now understand that
INTEL_IDLE is not (should not) even used with Xen.

But I have to ask whether we can look at the other issue a bit more,
regarding CPU states and how I can get states C4-C7 to work.

Perhaps you can tell me whether the BIOS options "ACPI T-state",
"C3 Auto Demotion" and "C1 Auto Demotion" should be disabled. I think
they should be, as I read it this sets respectively C4-C7 to C3, and
C3-C2 to C1 right? I have (and had) those disabled, when I enabled them
the output from xenpm did not differ.

Much obliged,

Mark.

-- 
Stay in touch,
Mark van Dijk.              ,--------------------------------
---------------------------'        Tue Jul 31 11:28 UTC 2012
Today is Boomtime, the 66th day of Confusion in the YOLD 3178

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Mon, Jul 30, 2012 at 08:45:43AM +0100, Jan Beulich wrote:
> >>> On 28.07.12 at 14:55, Mark van Dijk <lists+xen@internecto.net> wrote:
> >> > > So.. in that case make sure you have XEN_ACPI_PROCESSOR=y and
> >> > > "CONFIG_INTEL_IDLE is not set" and that should do it.
> >> > 
> >> > Didn't he say that with INTEL_IDLE disabled things work (perhaps
> >> > in a limited way, but at least don't crash)? My understanding is
> >> > that things should work even with it enabled, and I was under the
> >> > impression that you had already taken care of disabling cpuidle
> >> > and cpufreq in the kernel when running on Xen...
> >> 
> >> So did I - both of those (cpufreq and cpuidle) are disabled.
> >> 
> >> Mark, any chance you can collect the serial output when it crashes
> >> please? Actually, can you get the whole bootup log with 'loglevel=8
> >> debug' on the Linux command line?
> >> 
> > 
> > The sad part is that I don't have physical access to the box, the good
> > part is that I do have KVM and IPMI (SOL) access.
> > 
> > This is what I could capture:
> > 
> > http://i.imgur.com/t66FM.png 
> 
> The first thing intel_idle_init() does is check
> boot_option_idle_override, and I thought this got forced to
> something other than IDLE_NO_OVERRIDE by the Xen code.

Yeah, by me thinking everything is peachy and that override is
no longer needed :-(

> But at least in the current kernel that doesn't seem to be
> the case anymore, and thus I suppose that it's dying on the
> subsequent
> 
> 	retval = cpuidle_register_driver(&intel_idle_driver);
> 	if (retval) {
> 		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
> 			cpuidle_get_driver()->name);
> 		return retval;
> 	}
> 
> since cpuidle_get_driver() is presumably returning NULL (after
> all xen_arch_setup() does call disable_cpuidle()).

Duh! That is easy enough to fix. Mark, can you please try testing with
this patch (and obviously enable the CONFIG_INTEL_IDLE)

diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c
index d0f59c3..46a9884 100644
--- a/drivers/idle/intel_idle.c
+++ b/drivers/idle/intel_idle.c
@@ -557,7 +557,7 @@ static int __init intel_idle_init(void)
 	retval = cpuidle_register_driver(&intel_idle_driver);
 	if (retval) {
 		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
-			cpuidle_get_driver()->name);
+			cpuidle_get_driver() ? cpuidle_get_driver()->name : "none");
 		return retval;
 	}

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Tue, Jul 31, 2012 at 01:58:48PM +0200, Mark van Dijk wrote:
> Message 501657C70200007800091357@nat28.tlf.novell.com contained:
> >The first thing intel_idle_init() does is check
> >boot_option_idle_override, and I thought this got forced to
> >something other than IDLE_NO_OVERRIDE by the Xen code.
> >But at least in the current kernel that doesn't seem to be
> >the case anymore, and thus I suppose that it's dying on the
> >subsequent
> >
> >	retval = cpuidle_register_driver(&intel_idle_driver);
> >	if (retval) {
> >		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
> >			cpuidle_get_driver()->name);
> >		return retval;
> >	}
> >
> >since cpuidle_get_driver() is presumably returning NULL (after
> >all xen_arch_setup() does call disable_cpuidle()).
> 
> This is way beyond my knowledge. I'm no programmer so unfortunately I
> can't say anything about it.
> 
> Not using INTEL_IDLE is no problem, in fact, I now understand that
> INTEL_IDLE is not (should not) even used with Xen.

Sure, but it should be smart enought not to crash. Can you please try the
patch I sent in the other email.
> 
> But I have to ask whether we can look at the other issue a bit more,
> regarding CPU states and how I can get states C4-C7 to work.

You seem to be using MWAIT states, which is what the C4-C7 are.
> 
> Perhaps you can tell me whether the BIOS options "ACPI T-state",
> "C3 Auto Demotion" and "C1 Auto Demotion" should be disabled. I think
> they should be, as I read it this sets respectively C4-C7 to C3, and
> C3-C2 to C1 right? I have (and had) those disabled, when I enabled them
> the output from xenpm did not differ.

I am not sure what those do. What did the manual say that they are doing?

BTW, C4-C7 are basically C3 states with MWAIT - and from your pastebins
you seem to have those enabled.

> 
> Much obliged,
> 
> Mark.
> 
> -- 
> Stay in touch,
> Mark van Dijk.              ,--------------------------------
> ---------------------------'        Tue Jul 31 11:28 UTC 2012
> Today is Boomtime, the 66th day of Confusion in the YOLD 3178

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Message 20120731134054.GD4789@phenom.dumpdata.com contained:

>Duh! That is easy enough to fix. Mark, can you please try testing with
>this patch (and obviously enable the CONFIG_INTEL_IDLE)
>(patch omitted)

Sure, I'll try it right now.

Incidentally I came across a thread suggesting that there is quite a
lot happening in the world of cpuidle. So I want to make sure that
any implementation from your end isn't / won't be in conflict with code
from other sources.

https://lkml.org/lkml/2012/6/28/186
https://lkml.org/lkml/2012/6/29/48

I trust you'll be able to track this down far better than I can,
meanwhile I'll test this patch you sent me.

-- 
Kind regards,
Mark van Dijk.              ,--------------------------------
---------------------------'        Tue Jul 31 21:40 UTC 2012
Today is Boomtime, the 66th day of Confusion in the YOLD 3178

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Tue, Jul 31, 2012 at 11:50:39PM +0200, Mark van Dijk wrote:
> Message 20120731134054.GD4789@phenom.dumpdata.com contained:
> 
> >Duh! That is easy enough to fix. Mark, can you please try testing with
> >this patch (and obviously enable the CONFIG_INTEL_IDLE)
> >(patch omitted)
> 
> Sure, I'll try it right now.
> 
> Incidentally I came across a thread suggesting that there is quite a
> lot happening in the world of cpuidle. So I want to make sure that
> any implementation from your end isn't / won't be in conflict with code
> from other sources.
> 
> https://lkml.org/lkml/2012/6/28/186
> https://lkml.org/lkml/2012/6/29/48

ah yes. I recall seeing those. I ought to be OK.
> 
> I trust you'll be able to track this down far better than I can,
> meanwhile I'll test this patch you sent me.

Thank you.

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Konrad Rzeszutek Wilk]

On Wed, Aug 01, 2012 at 01:59:41AM +0200, Mark van Dijk wrote:
> Message 20120731134054.GD4789@phenom.dumpdata.com contained:
> 
> >Duh! That is easy enough to fix. Mark, can you please try testing with
> >this patch (and obviously enable the CONFIG_INTEL_IDLE)
> >
> >diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c
> >index d0f59c3..46a9884 100644
> >--- a/drivers/idle/intel_idle.c
> >+++ b/drivers/idle/intel_idle.c
> >@@ -557,7 +557,7 @@ static int __init intel_idle_init(void)
> > 	retval = cpuidle_register_driver(&intel_idle_driver);
> > 	if (retval) {
> > 		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
> >-			cpuidle_get_driver()->name);
> >+			cpuidle_get_driver() ?
> >cpuidle_get_driver()->name : "none");
> > 		return retval;
> > 	}
> > 
> 
> Good going Jan and Konrad. The above patch works: no more crashes. For
> fun I have also tested the patch from my previous email (the link to
> the lkml, it has been acked so was worth a try). I tested it in
> correlation with your patch and I still see no crashes.

Ok, is it Ok if I include your name as Reported-by and Tested-by?
Thx
> 
> 
> 
> -- 
> Stay in touch,
> Mark van Dijk.               ,--------------------------------
> ----------------------------'        Tue Jul 31 22:19 UTC 2012
> Today is Pungenday, the 67th day of Confusion in the YOLD 3178

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Message 20120731134054.GD4789@phenom.dumpdata.com contained:

>Duh! That is easy enough to fix. Mark, can you please try testing with
>this patch (and obviously enable the CONFIG_INTEL_IDLE)
>
>diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c
>index d0f59c3..46a9884 100644
>--- a/drivers/idle/intel_idle.c
>+++ b/drivers/idle/intel_idle.c
>@@ -557,7 +557,7 @@ static int __init intel_idle_init(void)
> 	retval = cpuidle_register_driver(&intel_idle_driver);
> 	if (retval) {
> 		printk(KERN_DEBUG PREFIX "intel_idle yielding to %s",
>-			cpuidle_get_driver()->name);
>+			cpuidle_get_driver() ?
>cpuidle_get_driver()->name : "none");
> 		return retval;
> 	}
> 

Good going Jan and Konrad. The above patch works: no more crashes. For
fun I have also tested the patch from my previous email (the link to
the lkml, it has been acked so was worth a try). I tested it in
correlation with your patch and I still see no crashes.



-- 
Stay in touch,
Mark van Dijk.               ,--------------------------------
----------------------------'        Tue Jul 31 22:19 UTC 2012
Today is Pungenday, the 67th day of Confusion in the YOLD 3178

Re: Kernel crash with acpi_processor, cpu_idle and intel_idle =y

[Mark van Dijk]

Message 20120731235527.GM32698@phenom.dumpdata.com contained:

>Ok, is it Ok if I include your name as Reported-by and Tested-by?

Yes, that's fine. Thanks for asking. If you want to include my email
address please substitute 'lists+xen' with 'mark'.

-- 
Stay in touch,
Mark van Dijk.               ,--------------------------------
----------------------------'        Wed Aug 01 00:22 UTC 2012
Today is Pungenday, the 67th day of Confusion in the YOLD 3178