Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi,

We are observing boot failures [1] & [2] on some of our servers (Dell r410) with 2.6.27-stable kernels. I found following commit in upstream tree via a quick search (but I haven't tried it yet) so I don't know whether I'm on the right track or not. 

Please let me know if anything else is needed...

commit d0af9eed5aa91b6b7b5049cae69e5ea956fd85c3
Author: Suresh Siddha <suresh.b.siddha@intel.com>
Date:   Wed Aug 19 18:05:36 2009 -0700

    x86, pat/mtrr: Rendezvous all the cpus for MTRR/PAT init
[…]

[1] http://www.cs.princeton.edu/~caglar/2.6.27/idle=mwait.png
[2] http://www.cs.princeton.edu/~caglar/2.6.27/idle=poll.png

Best regards,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Greg KH]

On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
> Hi,
> 
> We are observing boot failures [1] & [2] on some of our servers (Dell
> r410) with 2.6.27-stable kernels. I found following commit in upstream
> tree via a quick search (but I haven't tried it yet) so I don't know
> whether I'm on the right track or not. 
> 
> Please let me know if anything else is needed...

Has any .27 kernel worked on these machines?  If so, can you narrow it
down to the patch that caused the problem?

thanks,

greg k-h

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi Greg,

On May 25, 2010, at 11:53 AM, Greg KH wrote:

> On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
>> Hi,
>> 
>> We are observing boot failures [1] & [2] on some of our servers (Dell
>> r410) with 2.6.27-stable kernels. I found following commit in upstream
>> tree via a quick search (but I haven't tried it yet) so I don't know
>> whether I'm on the right track or not. 
>> 
>> Please let me know if anything else is needed...
> 
> Has any .27 kernel worked on these machines?  If so, can you narrow it
> down to the patch that caused the problem?

I don't think any .27 kernel worked on those machines before. But in any case I'll try (I don't have a physical access to those machines as they are located in China) an older .27 release to see.

> thanks,
> 
> greg k-h


Best regards,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Greg KH]

On Tue, May 25, 2010 at 04:32:16PM -0400, "S.Çağlar Onur" wrote:
> Hi Greg,
> 
> On May 25, 2010, at 11:53 AM, Greg KH wrote:
> 
> > On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
> >> Hi,
> >> 
> >> We are observing boot failures [1] & [2] on some of our servers (Dell
> >> r410) with 2.6.27-stable kernels. I found following commit in upstream
> >> tree via a quick search (but I haven't tried it yet) so I don't know
> >> whether I'm on the right track or not. 
> >> 
> >> Please let me know if anything else is needed...
> > 
> > Has any .27 kernel worked on these machines?  If so, can you narrow it
> > down to the patch that caused the problem?
> 
> I don't think any .27 kernel worked on those machines before. But in
> any case I'll try (I don't have a physical access to those machines as
> they are located in China) an older .27 release to see.

Ok, so it isn't a regression, which is what I want to know.

New hardware like this, probably will only work on newer kernel
versions, so please use those instead :)

thanks,

greg k-h

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Suresh Siddha]

On Tue, 2010-05-25 at 13:51 -0700, Greg KH wrote:
> On Tue, May 25, 2010 at 04:32:16PM -0400, "S.Çağlar Onur" wrote:
> > Hi Greg,
> > 
> > On May 25, 2010, at 11:53 AM, Greg KH wrote:
> > 
> > > On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
> > >> Hi,
> > >> 
> > >> We are observing boot failures [1] & [2] on some of our servers (Dell
> > >> r410) with 2.6.27-stable kernels. I found following commit in upstream
> > >> tree via a quick search (but I haven't tried it yet) so I don't know
> > >> whether I'm on the right track or not. 
> > >> 
> > >> Please let me know if anything else is needed...
> > > 
> > > Has any .27 kernel worked on these machines?  If so, can you narrow it
> > > down to the patch that caused the problem?
> > 
> > I don't think any .27 kernel worked on those machines before. But in
> > any case I'll try (I don't have a physical access to those machines as
> > they are located in China) an older .27 release to see.
> 
> Ok, so it isn't a regression, which is what I want to know.
> 
> New hardware like this, probably will only work on newer kernel
> versions, so please use those instead :)

I can't think of any known issues in 2.6.27 that can lead to this kind
of boot failure on WSM-EP. Caglar, It will be nice if you can do git
bisect between working and non-working kernels.

thanks,
suresh

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi Suresh,

On May 25, 2010, at 8:24 PM, Suresh Siddha wrote:

> On Tue, 2010-05-25 at 13:51 -0700, Greg KH wrote:
>> On Tue, May 25, 2010 at 04:32:16PM -0400, "S.Çağlar Onur" wrote:
>>> Hi Greg,
>>> 
>>> On May 25, 2010, at 11:53 AM, Greg KH wrote:
>>> 
>>>> On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
>>>>> Hi,
>>>>> 
>>>>> We are observing boot failures [1] & [2] on some of our servers (Dell
>>>>> r410) with 2.6.27-stable kernels. I found following commit in upstream
>>>>> tree via a quick search (but I haven't tried it yet) so I don't know
>>>>> whether I'm on the right track or not. 
>>>>> 
>>>>> Please let me know if anything else is needed...
>>>> 
>>>> Has any .27 kernel worked on these machines?  If so, can you narrow it
>>>> down to the patch that caused the problem?
>>> 
>>> I don't think any .27 kernel worked on those machines before. But in
>>> any case I'll try (I don't have a physical access to those machines as
>>> they are located in China) an older .27 release to see.
>> 
>> Ok, so it isn't a regression, which is what I want to know.
>> 
>> New hardware like this, probably will only work on newer kernel
>> versions, so please use those instead :)
> 
> I can't think of any known issues in 2.6.27 that can lead to this kind
> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
> bisect between working and non-working kernels.

Sure, I'll try to boot those with plain 2.6.27 to see but please note that it's going to be little time consuming. For each one I have to create a custom boot image for them to download and test (a.k.a. don't expect to hear from me soon :))

> thanks,
> suresh


Best regards,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Youquan Song]

> > I can't think of any known issues in 2.6.27 that can lead to this kind
> > of boot failure on WSM-EP. Caglar, It will be nice if you can do git
> > bisect between working and non-working kernels.
> 
> Sure, I'll try to boot those with plain 2.6.27 to see but please note that it's going to be little time consuming. For each one I have to create a custom boot image for them to download and test (a.k.a. don't expect to hear from me soon :))

Hi Caglar,

What's the microcode version? Get by "dmesg | grep microcode" if you
success boot other version of kernel.
Can you try to disable C-state(C6) in BIOS before your boot 2.6.27?

Thanks
-Youquan

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi Suresh,

On May 25, 2010, at 8:24 PM, Suresh Siddha wrote:

> On Tue, 2010-05-25 at 13:51 -0700, Greg KH wrote:
>> On Tue, May 25, 2010 at 04:32:16PM -0400, "S.Çağlar Onur" wrote:
>>> Hi Greg,
>>> 
>>> On May 25, 2010, at 11:53 AM, Greg KH wrote:
>>> 
>>>> On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
>>>>> Hi,
>>>>> 
>>>>> We are observing boot failures [1] & [2] on some of our servers (Dell
>>>>> r410) with 2.6.27-stable kernels. I found following commit in upstream
>>>>> tree via a quick search (but I haven't tried it yet) so I don't know
>>>>> whether I'm on the right track or not. 
>>>>> 
>>>>> Please let me know if anything else is needed...
>>>> 
>>>> Has any .27 kernel worked on these machines?  If so, can you narrow it
>>>> down to the patch that caused the problem?
>>> 
>>> I don't think any .27 kernel worked on those machines before. But in
>>> any case I'll try (I don't have a physical access to those machines as
>>> they are located in China) an older .27 release to see.
>> 
>> Ok, so it isn't a regression, which is what I want to know.
>> 
>> New hardware like this, probably will only work on newer kernel
>> versions, so please use those instead :)
> 
> I can't think of any known issues in 2.6.27 that can lead to this kind
> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
> bisect between working and non-working kernels.

Neither 2.6.27 nor 2.6.27.45 not able to boot those servers, according to the on-site admins the boot stuck at following stage[1].

> thanks,
> suresh


[1] http://www.cs.princeton.edu/~caglar/2.6.27/45.png

Best regards,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Greg KH]

On Wed, Jun 02, 2010 at 07:54:29PM -0400, "S.Çağlar Onur" wrote:
> Hi Suresh,
> 
> On May 25, 2010, at 8:24 PM, Suresh Siddha wrote:
> 
> > On Tue, 2010-05-25 at 13:51 -0700, Greg KH wrote:
> >> On Tue, May 25, 2010 at 04:32:16PM -0400, "S.Çağlar Onur" wrote:
> >>> Hi Greg,
> >>> 
> >>> On May 25, 2010, at 11:53 AM, Greg KH wrote:
> >>> 
> >>>> On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
> >>>>> Hi,
> >>>>> 
> >>>>> We are observing boot failures [1] & [2] on some of our servers (Dell
> >>>>> r410) with 2.6.27-stable kernels. I found following commit in upstream
> >>>>> tree via a quick search (but I haven't tried it yet) so I don't know
> >>>>> whether I'm on the right track or not. 
> >>>>> 
> >>>>> Please let me know if anything else is needed...
> >>>> 
> >>>> Has any .27 kernel worked on these machines?  If so, can you narrow it
> >>>> down to the patch that caused the problem?
> >>> 
> >>> I don't think any .27 kernel worked on those machines before. But in
> >>> any case I'll try (I don't have a physical access to those machines as
> >>> they are located in China) an older .27 release to see.
> >> 
> >> Ok, so it isn't a regression, which is what I want to know.
> >> 
> >> New hardware like this, probably will only work on newer kernel
> >> versions, so please use those instead :)
> > 
> > I can't think of any known issues in 2.6.27 that can lead to this kind
> > of boot failure on WSM-EP. Caglar, It will be nice if you can do git
> > bisect between working and non-working kernels.
> 
> Neither 2.6.27 nor 2.6.27.45 not able to boot those servers, according
> to the on-site admins the boot stuck at following stage[1].

Good, it's not a regression :)

Care to upgrade to a newer kernel version and see if that works?

thanks,

greg k-h

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi Greg,

On Jun 3, 2010, at 12:40 PM, Greg KH wrote:

> On Wed, Jun 02, 2010 at 07:54:29PM -0400, "S.Çağlar Onur" wrote:
>> Hi Suresh,
>> 
>> On May 25, 2010, at 8:24 PM, Suresh Siddha wrote:
>> 
>>> On Tue, 2010-05-25 at 13:51 -0700, Greg KH wrote:
>>>> On Tue, May 25, 2010 at 04:32:16PM -0400, "S.Çağlar Onur" wrote:
>>>>> Hi Greg,
>>>>> 
>>>>> On May 25, 2010, at 11:53 AM, Greg KH wrote:
>>>>> 
>>>>>> On Tue, May 25, 2010 at 11:10:28AM -0400, "S.Çağlar Onur" wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> We are observing boot failures [1] & [2] on some of our servers (Dell
>>>>>>> r410) with 2.6.27-stable kernels. I found following commit in upstream
>>>>>>> tree via a quick search (but I haven't tried it yet) so I don't know
>>>>>>> whether I'm on the right track or not. 
>>>>>>> 
>>>>>>> Please let me know if anything else is needed...
>>>>>> 
>>>>>> Has any .27 kernel worked on these machines?  If so, can you narrow it
>>>>>> down to the patch that caused the problem?
>>>>> 
>>>>> I don't think any .27 kernel worked on those machines before. But in
>>>>> any case I'll try (I don't have a physical access to those machines as
>>>>> they are located in China) an older .27 release to see.
>>>> 
>>>> Ok, so it isn't a regression, which is what I want to know.
>>>> 
>>>> New hardware like this, probably will only work on newer kernel
>>>> versions, so please use those instead :)
>>> 
>>> I can't think of any known issues in 2.6.27 that can lead to this kind
>>> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
>>> bisect between working and non-working kernels.
>> 
>> Neither 2.6.27 nor 2.6.27.45 not able to boot those servers, according
>> to the on-site admins the boot stuck at following stage[1].
> 
> Good, it's not a regression :)

Seems so :)

> Care to upgrade to a newer kernel version and see if that works?

I know 2.6.3x kernels boots those boxes without a problem, that was the first thing that I tried. 

In any case, I'll try to find a way to solve that problem as 2.6.27.46 is the kernel that we are using on some nodes of Planet-Lab. If you (any of you) have any ideas/patches/solutions please feel free to share it with me (and I'll let you know If I can find anything). 

> thanks,
> 
> greg k-h


Best regards,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi Youquan & Greg,


On May 26, 2010, at 9:23 AM, Youquan Song wrote:

>>> I can't think of any known issues in 2.6.27 that can lead to this kind
>>> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
>>> bisect between working and non-working kernels.
>> 
>> Sure, I'll try to boot those with plain 2.6.27 to see but please note that it's going to be little time consuming. For each one I have to create a custom boot image for them to download and test (a.k.a. don't expect to hear from me soon :))
> 
> Hi Caglar,
> 
> What's the microcode version? Get by "dmesg | grep microcode" if you
> success boot other version of kernel.
> Can you try to disable C-state(C6) in BIOS before your boot 2.6.27?

First of all I'm really sorry that it took really long time to reply this mail as I was waiting Dell to send a loaner machine to us. This morning I've got the hardware and following patch solved the boot problem that I was observing, Greg could you consider adding that to 2.6.27-stable tree?

commit a0bf284bfedd6dc95bbee7ebf5ccf3b5f753a008
Author: Len Brown <len.brown@intel.com>
Date:   Fri May 15 01:29:31 2009 -0400

    ACPI: Idle C-states disabled by max_cstate should not disable the TSC
    
    Processor idle power states C2 and C3 stop the TSC on many machines.
    Linux recognizes this situation and marks the TSC as unstable:
    
    Marking TSC unstable due to TSC halts in idle
    
    But if those same machines are booted with "processor.max_cstate=1",
    then there is no need to validate C2 and C3, and no need to
    disable the TSC, which can be reliably used as a clocksource.
    
    Signed-off-by: Len Brown <len.brown@intel.com>
    Acked-by: Thomas Gleixner <tglx@linutronix.de>

diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c
index e39a40a..e65476f 100644
--- a/drivers/acpi/processor_idle.c
+++ b/drivers/acpi/processor_idle.c
@@ -582,7 +582,7 @@ static int acpi_processor_power_verify(struct acpi_processor *pr)
 
        pr->power.timer_broadcast_on_state = INT_MAX;
 
-       for (i = 1; i < ACPI_PROCESSOR_MAX_POWER; i++) {
+       for (i = 1; i < ACPI_PROCESSOR_MAX_POWER && i <= max_cstate; i++) {
                struct acpi_processor_cx *cx = &pr->power.states[i];
 
                switch (cx->type) {


> Thanks
> -Youquan


Best,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Greg KH]

On Thu, Nov 18, 2010 at 04:24:40PM -0500, "S.Çağlar Onur" wrote:
> Hi Youquan & Greg,
> 
> 
> On May 26, 2010, at 9:23 AM, Youquan Song wrote:
> 
> >>> I can't think of any known issues in 2.6.27 that can lead to this kind
> >>> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
> >>> bisect between working and non-working kernels.
> >> 
> >> Sure, I'll try to boot those with plain 2.6.27 to see but please note that it's going to be little time consuming. For each one I have to create a custom boot image for them to download and test (a.k.a. don't expect to hear from me soon :))
> > 
> > Hi Caglar,
> > 
> > What's the microcode version? Get by "dmesg | grep microcode" if you
> > success boot other version of kernel.
> > Can you try to disable C-state(C6) in BIOS before your boot 2.6.27?
> 
> First of all I'm really sorry that it took really long time to reply this mail as I was waiting Dell to send a loaner machine to us. This morning I've got the hardware and following patch solved the boot problem that I was observing, Greg could you consider adding that to 2.6.27-stable tree?
> 
> commit a0bf284bfedd6dc95bbee7ebf5ccf3b5f753a008

No problem, now queued up.

thanks,

greg k-h

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

["S.Çağlar Onur"]

Hi Greg,

On Nov 18, 2010, at 4:54 PM, Greg KH wrote:

> On Thu, Nov 18, 2010 at 04:24:40PM -0500, "S.Çağlar Onur" wrote:
>> Hi Youquan & Greg,
>> 
>> 
>> On May 26, 2010, at 9:23 AM, Youquan Song wrote:
>> 
>>>>> I can't think of any known issues in 2.6.27 that can lead to this kind
>>>>> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
>>>>> bisect between working and non-working kernels.
>>>> 
>>>> Sure, I'll try to boot those with plain 2.6.27 to see but please note that it's going to be little time consuming. For each one I have to create a custom boot image for them to download and test (a.k.a. don't expect to hear from me soon :))
>>> 
>>> Hi Caglar,
>>> 
>>> What's the microcode version? Get by "dmesg | grep microcode" if you
>>> success boot other version of kernel.
>>> Can you try to disable C-state(C6) in BIOS before your boot 2.6.27?
>> 
>> First of all I'm really sorry that it took really long time to reply this mail as I was waiting Dell to send a loaner machine to us. This morning I've got the hardware and following patch solved the boot problem that I was observing, Greg could you consider adding that to 2.6.27-stable tree?
>> 
>> commit a0bf284bfedd6dc95bbee7ebf5ccf3b5f753a008
> 
> No problem, now queued up.

Umh please drop that patch for now, seems like I mixed my kernel configs and compiled/tested a 64 bit one (by the way 64bit kernel boots fine) but 32bit one still fails with the same error. I'll try to bisect...

> thanks,
> 
> greg k-h


Best,
--
S.Çağlar Onur <caglar@cs.princeton.edu>

Re: [stable] Boot failure with 2.6.27.46 on Xeon E5620 (Westmere EP)

[Greg KH]

On Thu, Nov 18, 2010 at 04:59:12PM -0500, "S.Çağlar Onur" wrote:
> Hi Greg,
> 
> On Nov 18, 2010, at 4:54 PM, Greg KH wrote:
> 
> > On Thu, Nov 18, 2010 at 04:24:40PM -0500, "S.Çağlar Onur" wrote:
> >> Hi Youquan & Greg,
> >> 
> >> 
> >> On May 26, 2010, at 9:23 AM, Youquan Song wrote:
> >> 
> >>>>> I can't think of any known issues in 2.6.27 that can lead to this kind
> >>>>> of boot failure on WSM-EP. Caglar, It will be nice if you can do git
> >>>>> bisect between working and non-working kernels.
> >>>> 
> >>>> Sure, I'll try to boot those with plain 2.6.27 to see but please note that it's going to be little time consuming. For each one I have to create a custom boot image for them to download and test (a.k.a. don't expect to hear from me soon :))
> >>> 
> >>> Hi Caglar,
> >>> 
> >>> What's the microcode version? Get by "dmesg | grep microcode" if you
> >>> success boot other version of kernel.
> >>> Can you try to disable C-state(C6) in BIOS before your boot 2.6.27?
> >> 
> >> First of all I'm really sorry that it took really long time to reply this mail as I was waiting Dell to send a loaner machine to us. This morning I've got the hardware and following patch solved the boot problem that I was observing, Greg could you consider adding that to 2.6.27-stable tree?
> >> 
> >> commit a0bf284bfedd6dc95bbee7ebf5ccf3b5f753a008
> > 
> > No problem, now queued up.
> 
> Umh please drop that patch for now, seems like I mixed my kernel
> configs and compiled/tested a 64 bit one (by the way 64bit kernel
> boots fine) but 32bit one still fails with the same error. I'll try to
> bisect...

Heh, ok, I'll go drop it now :)

that's what I get for trying to be quick...

thanks,

greg k-h