Kernel 2.6.32.60 64bit Crash/Hung

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

Hi,

I am facing crash/hang  with 2.6.32.60 Kernel (64bit). My maxcpus (CONFIG_NR_CPUS)  count is 64 and below is the call trace. I see log saying "BUG: spinlock lockup on CPU#74". Is it a  known bug? If its fixed let me know the patch.  Any pointers really appreciated.
Please note, if I change maxcpus=8 kernel boots successfully.

Thanks
Giri

[Jan 26 06:41:35.439] [ 67.606000] Non-volatile memory driver v1.3^M
[ 67.616000] Linux agpgart interface v0.103^M
[ 67.637000] [drm] Initialized drm 1.1.0 20060810^M
[ 67.652000] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled^M
[ 68.092000] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A^M
[Jan 26 06:41:36.679] [ 68.846000] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A^M
[ 68.910000] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A^M
[Jan 26 06:41:37.149] [ 69.316000] brd: module loaded^M
[Jan 26 06:41:41.015] [ 73.183000] loop: module loaded^M
[ 73.191000] lkdtm : Invalid command^M
[ 73.200000] kgdb: Registered I/O driver kgdbts.^M
[Jan 26 06:42:03.435] [ 73.552000] BUG: spinlock lockup on CPU#74, swapper/1, ffff880c9c1d2ac0^M <=======
[ 73.552000] Pid: 1, comm: swapper Not tainted 2.6.32.60 #2^M
[ 73.552000] Call Trace:^M
[ 73.552000] <IRQ> [<ffffffff81196df0>] _raw_spin_lock+0x103/0x12b^M
[ 73.552000] [<ffffffff81459c97>] _spin_lock+0x4f/0x62^M
[ 73.552000] [<ffffffff8103c1a3>] ? scheduler_tick+0x49/0x166^M
[ 73.552000] [<ffffffff8103c1a3>] scheduler_tick+0x49/0x166^M
[ 73.552000] [<ffffffff8104e6de>] update_process_times+0x42/0x52^M
[ 73.552000] [<ffffffff81061ef1>] tick_periodic+0x63/0x65^M
[ 73.552000] [<ffffffff81061f11>] tick_handle_periodic+0x1e/0x6b^M
[ 73.552000] [<ffffffff8101eb91>] smp_apic_timer_interrupt+0x84/0x97^M
[ 73.552000] [<ffffffff8100c773>] apic_timer_interrupt+0x13/0x20^M
[ 73.552000] <EOI> [<ffffffff8126a8a5>] ? kgdbts_break_test+0x1/0x22^M
[ 73.552000] [<ffffffff8126a8a4>] ? kgdbts_break_test+0x0/0x22^M
[ 73.552000] [<ffffffff8126b7d4>] ? configure_kgdbts+0x1ef/0x4bb^M
[ 73.552000] [<ffffffff81177ade>] ? blk_register_region+0x28/0x2a^M
[ 73.552000] [<ffffffff81b50f4c>] ? init_kgdbts+0x0/0x16^M
[ 73.552000] [<ffffffff81b50f60>] ? init_kgdbts+0x14/0x16^M
[ 73.552000] [<ffffffff8100905d>] ? do_one_initcall+0x57/0x157^M
[ 73.552000] [<ffffffff81b24d88>] ? kernel_init+0x1f0/0x241^M
[ 73.552000] [<ffffffff8100cc7a>] ? child_rip+0xa/0x20^M
[ 73.552000] [<ffffffff8100c63c>] ? restore_args+0x0/0x30^M
[ 73.552000] [<ffffffff81b24b98>] ? kernel_init+0x0/0x241^M
[ 73.552000] [<ffffffff8100cc70>] ? child_rip+0x0/0x20^M
[ 73.552000] sending NMI to all CPUs:^M
[ 73.552000] NMI backtrace for cpu 74^M
[ 73.552000] CPU 74:^M
[ 73.552000] Modules linked in:^M
[ 73.552000] Pid: 1, comm: swapper Not tainted 2.6.32.60 #2 UCSB-B420-M3^M
[ 73.552000] RIP: 0010:[<ffffffff81187fa0>] [<ffffffff81187fa0>] delay_loop+0x20/0x2a^M
[ 73.552000] RSP: 0018:ffff880c9c003da8 EFLAGS: 00000002^M
[ 73.552000] RAX: 000000000001d185 RBX: 0000000000000000 RCX: 0000000000000030^M
[ 73.552000] RDX: 000000000001d19a RSI: 0000000000000010 RDI: 000000000001d19b^M
[ 73.552000] RBP: ffff880c9c003da8 R08: 000000000000000b R09: ffffffff81947ad0^M
[ 73.552000] R10: 000000000000b4a0 R11: ffffffff81878da0 R12: 0000000000001000^M
[ 73.552000] R13: 0000000000000092 R14: 0000000000000002 R15: ffffffff81947ac0^M
[ 73.552000] FS: 0000000000000000(0000) GS:ffff880c9c000000(0000) knlGS:0000000000000000^M
[ 73.552000] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b^M
[ 73.552000] CR2: 0000000000000000 CR3: 0000000001001000 CR4: 00000000000406e0^M
[ 73.552000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 000000000000
[Jan 26 06:42:04.041] 0000^M
[ 73.552000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400^M
[ 73.552000] Call Trace:^M
[ 73.552000] <#DB[1]> <<EOE>> Pid: 1, comm: swapper Not tainted 2.6.32.60 #2^M
[ 73.552000] Call Trace:^M
[ 73.552000] <NMI> [<ffffffff8145b99d>] nmi_watchdog_tick+0x111/0x24d^M
[ 73.552000] [<ffffffff8145aee1>] do_nmi+0xbd/0x269^M
[ 73.552000] [<ffffffff8145ab1a>] nmi+0x1a/0x2c^M
[ 73.552000] [<ffffffff81187fa0>] ? delay_loop+0x20/0x2a^M
[ 73.552000] <<EOE>> <IRQ> [<ffffffff81188001>] __delay+0xa/0xc^M
[ 73.552000] [<ffffffff8118803f>] __const_udelay+0x3c/0x3e^M
[ 73.552000] [<ffffffff8101e898>] native_safe_apic_wait_icr_idle+0x31/0x44^M
[ 73.552000] [<ffffffff8101f479>] default_send_IPI_mask_sequence_phys+0x49/0x110^M
[ 73.552000] [<ffffffff810225f4>] physflat_send_IPI_mask+0x9/0xb^M
[ 73.552000] [<ffffffff8102262b>] physflat_send_IPI_all+0x14/0x16^M
[ 73.552000] [<ffffffff8101f8e6>] arch_trigger_all_cpu_backtrace+0x4e/0x7c^M
[ 73.552000] [<ffffffff81196df5>] _raw_spin_lock+0x108/0x12b^M
[ 73.552000] [<ffffffff81459c97>] _spin_lock+0x4f/0x62^M
[ 73.552000] [<ffffffff8103c1a3>] ? scheduler_tick+0x49/0x166^M
[ 73.552000] [<ffffffff8103c1a3>] scheduler_tick+0x49/0x166^M
[ 73.552000] [<ffffffff8104e6de>] update_process_times+0x42/0x52^M
[ 73.552000] [<ffffffff81061ef1>] tick_periodic+0x63/0x65^M
[ 73.552000] [<ffffffff81061f11>] tick_handle_periodic+0x1e/0x6b^M
[ 73.552000] [<ffffffff8101eb91>] smp_apic_timer_interrupt+0x84/0x97^M
[ 73.552000] [<ffffffff8100c773>] apic_timer_interrupt+0x13/0x20^M
[ 73.552000] <EOI> [<ffffffff8126a8a5>] ? kgdbts_break_test+0x1/0x22^M
[ 73.552000] [<ffffffff8126a8a4>] ? kgdbts_break_test+0x0/0x22^M
[ 73.552000] [<ffffffff8126b7d4>] ? configure_kgdbts+0x1ef/0x4bb^M
[ 73.552000] [<ffffffff81177ade>] ? blk_register_region+0x28/0x2a^M
[ 73.552000] [<ffffffff81b50f4c>] ? init_kgdbts+0x0/0x16^M
[ 73.552000] [<ffffffff81b50f60>] ? init_kgdbts+0x14/0x16^M
[ 73.552000] [<ffffffff8100905d>] ? do_one_initcall+0x57/0x157^M
[ 73.552000] [<ffffffff81b24d88>] ? kernel_init+0x1f0/0x241^M
[ 73.552000] [<ffffffff8100cc7a>] ? child_rip+0xa/0x20^M
[ 73.552000] [<ffffffff8100c63c>] ? restore_args+0x0/0x30^M
[ 73.552000] [<ffffffff81b24b98>] ? kernel_init+0x0/0x241^M
[ 73.552000] [<ffffffff8100cc70>] ? child_rip+0x0/0x20^M

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150127/4b50a0ef/attachment.html 

Kernel 2.6.32.60 64bit Crash/Hung

[Greg KH]

On Tue, Jan 27, 2015 at 12:06:56AM +0000, Giridhara RP (grp) wrote:
> Hi,
> 
>  
> 
> I am facing crash/hang  with 2.6.32.60 Kernel (64bit).

Note this is a _very_ old and obsolete kernel, please use a modern
kernel release if you wish to get help from the community.

If you are being forced to use this old version, then get support from
the company you are paying to provide this kernel, as that is what their
job is.

Best of luck,

greg k-h

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

Hi Greg,

Thanks for the quick turnaround. Which is the stable version? Kernel 2.6.32.60 was used in RHEL 6.2 and we thought this kernel version is stable.

Regards
Giri 

-----Original Message-----
From: Greg KH [mailto:greg at kroah.com] 
Sent: Tuesday, January 27, 2015 8:19 AM
To: Giridhara RP (grp)
Cc: kernelnewbies at kernelnewbies.org
Subject: Re: Kernel 2.6.32.60 64bit Crash/Hung

On Tue, Jan 27, 2015 at 12:06:56AM +0000, Giridhara RP (grp) wrote:
> Hi,
> 
>  
> 
> I am facing crash/hang  with 2.6.32.60 Kernel (64bit).

Note this is a _very_ old and obsolete kernel, please use a modern kernel release if you wish to get help from the community.

If you are being forced to use this old version, then get support from the company you are paying to provide this kernel, as that is what their job is.

Best of luck,

greg k-h

Kernel 2.6.32.60 64bit Crash/Hung

[Valdis.Kletnieks at vt.edu]

On Tue, 27 Jan 2015 03:52:05 +0000, "Giridhara RP (grp)" said:

> Thanks for the quick turnaround. Which is the stable version? Kernel
> 2.6.32.60 was used in RHEL 6.2 and we thought this kernel version is stable.

It's stable enough that RedHat can afford to keep supporting its paying
customers.  You probably will be quite disillusioned if you download the
.src.rpm for the kernel and start counting %patch lines in the .SPEC file.:)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150126/729ac729/attachment.bin 

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

-----Original Message-----
From: Valdis.Kletnieks@vt.edu [mailto:Valdis.Kletnieks at vt.edu] 
Sent: Tuesday, January 27, 2015 9:32 AM
To: Giridhara RP (grp)
Cc: Greg KH; kernelnewbies at kernelnewbies.org
Subject: Re: Kernel 2.6.32.60 64bit Crash/Hung

On Tue, 27 Jan 2015 03:52:05 +0000, "Giridhara RP (grp)" said:

> Thanks for the quick turnaround. Which is the stable version? Kernel
> 2.6.32.60 was used in RHEL 6.2 and we thought this kernel version is stable.

It's stable enough that RedHat can afford to keep supporting its paying customers.  You probably will be quite disillusioned if you download the .src.rpm for the kernel and start counting %patch lines in the .SPEC file.:)


---------------
Hi Valdis,
Which kernel version should I use to solve this crash/panic? 

Thanks
Giri

Kernel 2.6.32.60 64bit Crash/Hung

[Valdis.Kletnieks at vt.edu]

On Tue, 27 Jan 2015 04:06:00 +0000, "Giridhara RP (grp)" said:

> Which kernel version should I use to solve this crash/panic?

That will depend on exactly why you're still running a 2.6.32 kernel
from 6 years ago. Remember - the fact it's all the way up to 2.6.32.65
doesn't mean you have all the bugfixes.  It only means you have all the
bugfixes that qualified for the -stable side of the fence (which is a
small fraction of all the fixes that have happened in the past 6 years).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150126/91e9c9ad/attachment.bin 

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

-----Original Message-----
From: Valdis.Kletnieks@vt.edu [mailto:Valdis.Kletnieks at vt.edu] 
Sent: Tuesday, January 27, 2015 10:15 AM
To: Giridhara RP (grp)
Cc: Greg KH; kernelnewbies at kernelnewbies.org
Subject: Re: Kernel 2.6.32.60 64bit Crash/Hung

On Tue, 27 Jan 2015 04:06:00 +0000, "Giridhara RP (grp)" said:

> Which kernel version should I use to solve this crash/panic?

That will depend on exactly why you're still running a 2.6.32 kernel from 6 years ago. Remember - the fact it's all the way up to 2.6.32.65 doesn't mean you have all the bugfixes.  It only means you have all the bugfixes that qualified for the -stable side of the fence (which is a small fraction of all the fixes that have happened in the past 6 years).

-----------------------
Hi Valdis,

We were running 2.6.32.60 assuming it's a stable build and my assumption was wrong :(. Should I go ahead with stable 3.18.3 [2015-01-16 ].

Thanks
Giri

Kernel 2.6.32.60 64bit Crash/Hung

[Anand Moon]

Hi Giridhara,

http://ark.intel.com/products/64622/Intel-Xeon-Processor-E5-4650-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI

Show it has only 8 core of CPU. Why are you setting the CONFIG_NR_CPUS=64.
certainly it will have a issue in the kernel.

-Anand Moon




On Tuesday, January 27, 2015 10:27 AM, Giridhara RP (grp) <grp@cisco.com> wrote:
-----Original Message-----
From: Valdis.Kletnieks@vt.edu [mailto:Valdis.Kletnieks at vt.edu] 
Sent: Tuesday, January 27, 2015 10:15 AM
To: Giridhara RP (grp)
Cc: Greg KH; kernelnewbies at kernelnewbies.org
Subject: Re: Kernel 2.6.32.60 64bit Crash/Hung

On Tue, 27 Jan 2015 04:06:00 +0000, "Giridhara RP (grp)" said:

> Which kernel version should I use to solve this crash/panic?

That will depend on exactly why you're still running a 2.6.32 kernel from 6 years ago. Remember - the fact it's all the way up to 2.6.32.65 doesn't mean you have all the bugfixes.  It only means you have all the bugfixes that qualified for the -stable side of the fence (which is a small fraction of all the fixes that have happened in the past 6 years).

-----------------------
Hi Valdis,

We were running 2.6.32.60 assuming it's a stable build and my assumption was wrong :(. Should I go ahead with stable 3.18.3 [2015-01-16 ].

Thanks
Giri

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

-----Original Message-----
From: Anand Moon [mailto:moon.linux at yahoo.com] 
Sent: Tuesday, January 27, 2015 12:30 PM
To: Giridhara RP (grp); Valdis.Kletnieks at vt.edu
Cc: Greg KH; kernelnewbies at kernelnewbies.org
Subject: Re: Kernel 2.6.32.60 64bit Crash/Hung

Hi Giridhara,

http://ark.intel.com/products/64622/Intel-Xeon-Processor-E5-4650-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI

Show it has only 8 core of CPU. Why are you setting the CONFIG_NR_CPUS=64.
certainly it will have a issue in the kernel.

-Anand Moon
-------------------------------------------------------------------------------
Hi Anand,

Agree it?s a 8 core processor, but Hyper threading is supported in this CPU. 

Thanks
Giri



On Tuesday, January 27, 2015 10:27 AM, Giridhara RP (grp) <grp@cisco.com> wrote:
-----Original Message-----
From: Valdis.Kletnieks@vt.edu [mailto:Valdis.Kletnieks at vt.edu] 
Sent: Tuesday, January 27, 2015 10:15 AM
To: Giridhara RP (grp)
Cc: Greg KH; kernelnewbies at kernelnewbies.org
Subject: Re: Kernel 2.6.32.60 64bit Crash/Hung

On Tue, 27 Jan 2015 04:06:00 +0000, "Giridhara RP (grp)" said:

> Which kernel version should I use to solve this crash/panic?

That will depend on exactly why you're still running a 2.6.32 kernel from 6 years ago. Remember - the fact it's all the way up to 2.6.32.65 doesn't mean you have all the bugfixes.  It only means you have all the bugfixes that qualified for the -stable side of the fence (which is a small fraction of all the fixes that have happened in the past 6 years).

-----------------------
Hi Valdis,

We were running 2.6.32.60 assuming it's a stable build and my assumption was wrong :(. Should I go ahead with stable 3.18.3 [2015-01-16 ].

Thanks
Giri

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel 2.6.32.60 64bit Crash/Hung

[Jan Niggemann]

Zitat von Anand Moon <moon.linux@yahoo.com>:
> http://ark.intel.com/products/64622/Intel-Xeon-Processor-E5-4650-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI
>
> Show it has only 8 core of CPU. Why are you setting the CONFIG_NR_CPUS=64.
> certainly it will have a issue in the kernel.
Why would that lead to issues? AFAIK all it does is set the size of  
the array which contains CPU data, setting it higher would only result  
in some kilobytes of memory being used...

Jan

Kernel 2.6.32.60 64bit Crash/Hung

[Hinnerk van Bruinehsen]

On Tue, Jan 27, 2015 at 09:25:04AM +0100, Jan Niggemann wrote:
> Zitat von Anand Moon <moon.linux@yahoo.com>:
> > http://ark.intel.com/products/64622/Intel-Xeon-Processor-E5-4650-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI
> >
> > Show it has only 8 core of CPU. Why are you setting the CONFIG_NR_CPUS=64.
> > certainly it will have a issue in the kernel.
> Why would that lead to issues? AFAIK all it does is set the size of  
> the array which contains CPU data, setting it higher would only result  
> in some kilobytes of memory being used...

It is also the defined default in arch/x86/configs/x86_64_defconfig (on 3.18.3, but IIRC it's this way for quite a while).

WKR
Hinnerk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150127/c9c5128f/attachment.bin 

Kernel 2.6.32.60 64bit Crash/Hung

[Greg KH]

On Tue, Jan 27, 2015 at 03:52:05AM +0000, Giridhara RP (grp) wrote:
> Hi Greg,
> 
> Thanks for the quick turnaround. Which is the stable version? Kernel
> 2.6.32.60 was used in RHEL 6.2 and we thought this kernel version is
> stable.

What do you mean by "stable"?

Yes, it is "stable" if you have hardware that you bought before that
kernel was released, and you know how to support a kernel on your own,
but if you are doing this with newer hardware (as it seems you are), you
need to use a much newer kernel.

Again, the community can not help you out when you run old kernels like
this, only a company that you pay support to can (like Red Hat, if you
want to use their kernels, great, they will support you and you will pay
for it.)

So I strongly suggest, if you are doing this on your own, to use the
3.18 kernel release.

Best of luck,

greg k-h

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

Thanks Greg,
Giri

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

All,

I Changed CONFIG_NR_CPUS=64 to CONFIG_NR_CPUS=128 and kernel did not panic.
Please note my setup has 4CPUs , each has 10 cores and supports Hyper threading. So, linux creates 4 x 10 x 2 logical CPUs.

Thanks
Giri
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150130/29cf5899/attachment-0001.html 

Kernel 2.6.32.60 64bit Crash/Hung

[Giridhara RP (grp)]

Hello,

Please note my CPU is this.
Intel(R) Xeon(R) CPU E5-4650 v2 @ 2.40GHz stepping 04

From: Giridhara RP (grp)
Sent: Tuesday, January 27, 2015 5:37 AM
To: kernelnewbies at kernelnewbies.org
Subject: Kernel 2.6.32.60 64bit Crash/Hung

Hi,

I am facing crash/hang  with 2.6.32.60 Kernel (64bit). My maxcpus (CONFIG_NR_CPUS)  count is 64 and below is the call trace. I see log saying "BUG: spinlock lockup on CPU#74". Is it a  known bug? If its fixed let me know the patch.  Any pointers really appreciated.
Please note, if I change maxcpus=8 kernel boots successfully.

Thanks
Giri
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150127/26957844/attachment-0001.html