* Re: Linux 3.1-rc9
[not found] ` <20111024190203.GA24410@hostway.ca>
@ 2011-10-25 7:13 ` Linus Torvalds
2011-10-25 9:01 ` David Miller
2011-10-25 20:20 ` Simon Kirby
1 sibling, 1 reply; 31+ messages in thread
From: Linus Torvalds @ 2011-10-25 7:13 UTC (permalink / raw)
To: Simon Kirby, Network Development
Cc: Thomas Gleixner, David Miller, Peter Zijlstra,
Linux Kernel Mailing List, Dave Jones, Martin Schwidefsky,
Ingo Molnar
Added netdev, because this seems to be a generic networking bug (ABBA
between sk_lock and icsk_retransmit_timer if my quick scan looks
correct).
Davem?
Linus
On Mon, Oct 24, 2011 at 9:02 PM, Simon Kirby <sim@hostway.ca> wrote:
>
> Ok, hit the hang about 4 more times, but only this morning on a box with
> a serial cable attached. Yay!
>
> Simon-
>
> [216695.579770] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216695.589435]
> [216695.589437] =======================================================
> [216695.593380] [ INFO: possible circular locking dependency detected ]
> [216695.593380] 3.1.0-rc10-hw-lockdep+ #51
> [216695.593380] -------------------------------------------------------
> [216695.593380] kworker/0:1/0 is trying to acquire lock:
> [216695.593380] (&icsk->icsk_retransmit_timer){+.-.-.}, at: [<ffffffff8106cc88>] run_timer_softirq+0x198/0x410
> [216695.593380]
> [216695.593380] but task is already holding lock:
> [216695.593380] (slock-AF_INET){+.-.-.}, at: [<ffffffff81604540>] sk_clone+0x120/0x420
> [216695.593380]
> [216695.593380] which lock already depends on the new lock.
> [216695.593380]
> [216695.593380]
> [216695.593380] the existing dependency chain (in reverse order) is:
> [216695.593380]
> [216695.593380] -> #1 (slock-AF_INET){+.-.-.}:
> [216695.593380] [<ffffffff8109a7b9>] lock_acquire+0x109/0x140
> [216695.593380] [<ffffffff816f55fc>] _raw_spin_lock+0x3c/0x50
> [216695.593380] [<ffffffff81661cc3>] tcp_write_timer+0x23/0x230
> [216695.682901] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [216695.682901] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [216695.682901] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [216695.682901] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [216695.682901] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [216695.682901] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [216695.682901] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [216695.682901] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [216695.682901] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [216695.682901]
> [216695.682901] -> #0 (&icsk->icsk_retransmit_timer){+.-.-.}:
> [216695.682901] [<ffffffff8109a000>] __lock_acquire+0x2040/0x2180
> [216695.682901] [<ffffffff8109a7b9>] lock_acquire+0x109/0x140
> [216695.682901] [<ffffffff8106cd09>] run_timer_softirq+0x219/0x410
> [216695.682901] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [216695.682901] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [216695.682901] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [216695.682901] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [216695.682901] [<ffffffff81014b74>] do_IRQ+0x64/0xe0
> [216695.682901] [<ffffffff816f6273>] ret_from_intr+0x0/0x1a
> [216695.682901] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [216695.682901] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [216695.682901]
> [216695.682901] other info that might help us debug this:
> [216695.682901]
> [216695.682901] Possible unsafe locking scenario:
> [216695.682901]
> [216695.682901] CPU0 CPU1
> [216695.682901] ---- ----
> [216695.682901] lock(slock-AF_INET);
> [216695.682901] lock(&icsk->icsk_retransmit_timer);
> [216695.682901] lock(slock-AF_INET);
> [216695.682901] lock(&icsk->icsk_retransmit_timer);
> [216695.682901]
> [216695.682901] *** DEADLOCK ***
> [216695.682901]
> [216695.682901] 1 lock held by kworker/0:1/0:
> [216695.682901] #0: (slock-AF_INET){+.-.-.}, at: [<ffffffff81604540>] sk_clone+0x120/0x420
> [216695.682901]
> [216695.682901] stack backtrace:
> [216695.682901] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51
> [216695.682901] Call Trace:
> [216695.682901] <IRQ> [<ffffffff81097eab>] print_circular_bug+0x21b/0x330
> [216695.682901] [<ffffffff8109a000>] __lock_acquire+0x2040/0x2180
> [216695.682901] [<ffffffffa001d6e2>] ? nf_conntrack_free+0x42/0x50 [nf_conntrack]
> [216695.682901] [<ffffffffa001d6e2>] ? nf_conntrack_free+0x42/0x50 [nf_conntrack]
> [216695.682901] [<ffffffff8109a7b9>] lock_acquire+0x109/0x140
> [216695.682901] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [216695.682901] [<ffffffff81096b4c>] ? trace_hardirqs_on_caller+0x7c/0x1c0
> [216695.682901] [<ffffffff8106cd09>] run_timer_softirq+0x219/0x410
> [216695.682901] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [216695.682901] [<ffffffff816f16c1>] ? printk+0x67/0x69
> [216695.682901] [<ffffffff81661ca0>] ? tcp_delack_timer+0x230/0x230
> [216695.682901] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [216695.682901] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [216695.682901] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [216695.682901] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [216695.682901] [<ffffffff81014b74>] do_IRQ+0x64/0xe0
> [216695.682901] [<ffffffff816f6273>] common_interrupt+0x73/0x73
> [216695.682901] <EOI> [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [216695.682901] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [216695.682901] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [216695.682901] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [216696.019296] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000105?
> [216697.762956] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216698.597297] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216701.489681] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216701.667999] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216704.580592] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [216709.468971] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [216712.845904] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216716.588502] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [216725.072958] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [216725.603879] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216725.828374] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216727.588978] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216735.513864] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216740.581530] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [216756.278571] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [218855.312903] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [218855.323133] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [218858.293355] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [218864.301938] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [218876.333821] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [218885.332651] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [218900.313590] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [220821.012017] TCP: Peer 32.176.160.153:49226/80 unexpectedly shrunk window 665256753:665268993 (repaired)
> [221075.224300] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221075.234579] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221075.277593] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221075.780515] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221075.780713] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221077.349279] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221077.905587] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [221077.915567] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [221081.498430] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221081.703277] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [221082.088513] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221082.167985] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221089.772578] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221090.487927] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [221090.686394] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221094.587131] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221105.255699] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
> [221105.280699] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221105.291634] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221106.325794] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221107.286029] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221107.622736] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221107.734471] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [221120.381643] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
> [223936.264020] BUG: soft lockup - CPU#0 stuck for 22s! [swapper:0]
> [223936.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.268002] irq event stamp: 2595159887
> [223936.268002] hardirqs last enabled at (2595159887): [<ffffffff8101b805>] mwait_idle+0x145/0x170
> [223936.268002] hardirqs last disabled at (2595159886): [<ffffffff81013139>] cpu_idle+0x79/0xf0
> [223936.268002] softirqs last enabled at (2595159878): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
> [223936.268002] softirqs last disabled at (2595159873): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.268002] CPU 0
> [223936.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.268002]
> [223936.268002] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223936.268002] RIP: 0010:[<ffffffff813a4ee3>] [<ffffffff813a4ee3>] delay_tsc+0x73/0xd0
> [223936.268002] RSP: 0018:ffff88022fc03cf0 EFLAGS: 00000202
> [223936.268002] RAX: 00017b5d5932dd02 RBX: ffffffff816f6334 RCX: 000000005932dd02
> [223936.372028] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/0:0:0]
> [223936.372031] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.372042] irq event stamp: 2598787699
> [223936.372044] hardirqs last enabled at (2598787699): [<ffffffff8101b805>] mwait_idle+0x145/0x170
> [223936.372054] hardirqs last disabled at (2598787698): [<ffffffff81013139>] cpu_idle+0x79/0xf0
> [223936.372058] softirqs last enabled at (2598787696): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
> [223936.372063] softirqs last disabled at (2598787681): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.372069] CPU 1
> [223936.372070] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.372079]
> [223936.372081] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223936.372086] RIP: 0010:[<ffffffff8101afab>] [<ffffffff8101afab>] native_read_tsc+0xb/0x20
> [223936.372091] RSP: 0018:ffff88022fc43ce0 EFLAGS: 00000202
> [223936.372093] RAX: 0000000000017b5d RBX: ffffffff816f6334 RCX: 00000000652f810e
> [223936.372096] RDX: 0000000000017b5d RSI: 0000000000000000 RDI: 0000000000000001
> [223936.372098] RBP: ffff88022fc43ce0 R08: 00000000652f80c8 R09: 0000000000000000
> [223936.372101] R10: ffff88022690dd60 R11: 0000000000000000 R12: ffff88022fc43c58
> [223936.372103] R13: ffffffff816feb33 R14: ffff88022fc43ce0 R15: 00000000180bbeb8
> [223936.372106] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000
> [223936.372108] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223936.372111] CR2: 00007f472ba6a6fc CR3: 0000000126bb7000 CR4: 00000000000006e0
> [223936.372113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223936.372116] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223936.372119] Process kworker/0:0 (pid: 0, threadinfo ffff880226928000, task ffff88022690dd60)
> [223936.372121] Stack:
> [223936.372123] ffff88022fc43d30 ffffffff813a4eaf ffff880226928000 00000000652f8090
> [223936.372128] 000000012fc43d18 ffff88002e90e348 00000000180bbeb8 000000006efcdc62
> [223936.372132] 0000000000000001 ffff88022690dd60 ffff88022fc43d40 ffffffff813a4f6a
> [223936.372136] Call Trace:
> [223936.372139] <IRQ>
> [223936.372144] [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
> [223936.372148] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.372153] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.372159] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.372164] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.372168] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.372174] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.372178] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.372182] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.372186] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.372190] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.372194] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.372198] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.372203] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.372208] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.372210] <EOI>
> [223936.372214] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.372218] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.372222] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.372226] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.372230] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223936.372233] Code: a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 48 89 d0
> [223936.372253] c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 00 00 00 00 00
> [223936.372262] Call Trace:
> [223936.372264] <IRQ> [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
> [223936.372269] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.372272] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.372276] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.372280] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.372283] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.372286] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.372289] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.372293] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.372297] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.372300] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.372303] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.372307] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.372310] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.372313] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.372315] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.372321] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.372324] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.372327] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.372331] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223936.476032] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/0:1:0]
> [223936.476034] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.476043] irq event stamp: 2613824057
> [223936.476045] hardirqs last enabled at (2613824057): [<ffffffff8101b805>] mwait_idle+0x145/0x170
> [223936.476050] hardirqs last disabled at (2613824056): [<ffffffff81013139>] cpu_idle+0x79/0xf0
> [223936.476054] softirqs last enabled at (2613824048): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
> [223936.476058] softirqs last disabled at (2613824031): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.476062] CPU 2
> [223936.476063] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.476071]
> [223936.476073] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223936.476077] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
> [223936.476082] RSP: 0018:ffff88022fc83ce0 EFLAGS: 00000202
> [223936.476084] RAX: 0000000070ba7dfc RBX: ffffffff813a60ae RCX: 0000000070ba7dc4
> [223936.476086] RDX: 0000000000017b5d RSI: 0000000000000000 RDI: 0000000000000001
> [223936.476089] RBP: ffff88022fc83ce0 R08: 0000000070ba7d7e R09: 0000000000000000
> [223936.476091] R10: ffff880226948000 R11: 0000000000000000 R12: ffff88022fc83c58
> [223936.476093] R13: ffffffff816feb33 R14: ffff88022fc83ce0 R15: 00000000182285f9
> [223936.476096] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
> [223936.476099] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223936.476101] CR2: 00007f25e7874d7f CR3: 0000000124c0d000 CR4: 00000000000006e0
> [223936.476104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223936.476106] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223936.476109] Process kworker/0:1 (pid: 0, threadinfo ffff88022693e000, task ffff880226948000)
> [223936.476111] Stack:
> [223936.476113] ffff88022fc83d30 ffffffff813a4ee8 ffff88022693e000 0000000070ba7dc4
> [223936.476117] 00000002ffffff10 ffff88006afd8948 00000000182285f9 000000006efcdc62
> [223936.476121] 0000000000000001 ffff880226948000 ffff88022fc83d40 ffffffff813a4f6a
> [223936.476126] Call Trace:
> [223936.476128] <IRQ>
> [223936.476132] [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223936.476136] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.476141] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.476147] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.476153] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.476157] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.476163] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.476167] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.476171] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.476176] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.476180] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.476184] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.476187] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.476193] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.476197] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.476199] <EOI>
> [223936.476203] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.476207] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.476211] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.476215] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.476219] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223936.476222] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
> [223936.476241] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
> [223936.476251] Call Trace:
> [223936.476252] <IRQ> [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223936.476257] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.476261] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.476265] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.476268] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.476272] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.476275] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.476278] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.476282] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.476286] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.476289] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.476292] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.476295] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.476299] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.476302] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.476304] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.476310] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.476313] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.476316] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.476320] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223936.580039] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/0:1:0]
> [223936.580041] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.580050] irq event stamp: 2615464042
> [223936.580052] hardirqs last enabled at (2615464042): [<ffffffff816f5edb>] _raw_spin_unlock_irq+0x2b/0x50
> [223936.580057] hardirqs last disabled at (2615464041): [<ffffffff816f56a8>] _raw_spin_lock_irq+0x18/0x60
> [223936.580061] softirqs last enabled at (2615463964): [<ffffffff81063cce>] _local_bh_enable+0xe/0x10
> [223936.580065] softirqs last disabled at (2615463965): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.580069] CPU 3
> [223936.580070] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223936.580078]
> [223936.580080] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223936.580085] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
> [223936.580090] RSP: 0018:ffff88022fcc3ce0 EFLAGS: 00000202
> [223936.580092] RAX: 000000007c457b06 RBX: ffffffff816f6334 RCX: 000000007c457ad5
> [223936.580094] RDX: 0000000000017b5d RSI: ffffffff818f9896 RDI: 0000000000000001
> [223936.580097] RBP: ffff88022fcc3ce0 R08: 000000007c457a88 R09: 0000000000000000
> [223936.580099] R10: ffff880226981f20 R11: 0000000000000000 R12: ffff88022fcc3c58
> [223936.580101] R13: ffffffff816feb33 R14: ffff88022fcc3ce0 R15: 00000000183a1380
> [223936.580104] FS: 0000000000000000(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
> [223936.580107] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223936.580109] CR2: 0000000000f38820 CR3: 0000000104b52000 CR4: 00000000000006e0
> [223936.580112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223936.580114] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223936.580117] Process kworker/0:1 (pid: 0, threadinfo ffff880226988000, task ffff880226981f20)
> [223936.580119] Stack:
> [223936.580120] ffff88022fcc3d30 ffffffff813a4ee8 ffff880226988000 000000007c457ad5
> [223936.580125] 00000003ffffff10 ffff880031438948 00000000183a1380 000000006efcdc62
> [223936.580129] 0000000000000001 ffff880226981f20 ffff88022fcc3d40 ffffffff813a4f6a
> [223936.580133] Call Trace:
> [223936.580135] <IRQ>
> [223936.580138] [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223936.580142] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.580147] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.580151] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.580156] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.580160] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.580164] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.580168] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.580172] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.580176] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.580181] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.580185] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.580188] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.580192] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.580196] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.580199] <EOI>
> [223936.580202] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.580206] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.580211] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.580214] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.580218] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223936.580221] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
> [223936.580240] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
> [223936.580250] Call Trace:
> [223936.580251] <IRQ> [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223936.580256] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.580260] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.580264] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.580267] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.580270] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.580274] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.580277] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.580280] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.580284] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.580288] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.580291] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.580294] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.580297] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.580300] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.580302] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.580308] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.580312] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.580315] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.580318] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223936.268002] RDX: 000000005932dd02 RSI: ffffffff818f9896 RDI: 0000000000000001
> [223936.268002] RBP: ffff88022fc03d30 R08: 000000005932dcb5 R09: 0000000000000000
> [223936.268002] R10: ffffffff81a2b020 R11: 0000000000000000 R12: ffff88022fc03c68
> [223936.268002] R13: ffffffff816feb33 R14: ffff88022fc03d30 R15: 0000000017f328cd
> [223936.268002] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
> [223936.268002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223936.268002] CR2: 00007f25e7bc13a0 CR3: 00000001426fc000 CR4: 00000000000006f0
> [223936.268002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223936.268002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223936.268002] Process swapper (pid: 0, threadinfo ffffffff819a6000, task ffffffff81a2b020)
> [223936.268002] Stack:
> [223936.268002] ffffffff819a6000 000000005932dd02 000000002fc03d18 ffff8801f6c22448
> [223936.268002] 0000000017f328cd 000000006efcdc62 0000000000000001 ffffffff81a2b020
> [223936.268002] ffff88022fc03d40 ffffffff813a4f6a ffff88022fc03d80 ffffffff813ac2ab
> [223936.268002] Call Trace:
> [223936.268002] <IRQ>
> [223936.268002] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.268002] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.268002] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.268002] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.268002] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.268002] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.268002] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.268002] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.268002] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.268002] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.268002] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.268002] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.268002] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.268002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.268002] <EOI>
> [223936.268002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.268002] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.268002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.268002] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.268002] [<ffffffff816ca491>] rest_init+0xd1/0xe0
> [223936.268002] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
> [223936.268002] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
> [223936.268002] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
> [223936.268002] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
> [223936.268002] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
> [223936.268002] Code: 4c 89 7d c8 eb 1f 66 90 48 8b 45 c0 83 68 1c 01 f3 90 83 40 1c 01 65 44 8b 3c 25 50 d3 00 00 44 3b 7d d4 75 3b 66 66 90 0f ae e8 <e8> b8 60 c7 ff 66 90 4c 63 c0 4c 89 c0 48 2b 45 c8 48 39 d8 72
> [223936.268002] Call Trace:
> [223936.268002] <IRQ> [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223936.268002] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223936.268002] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223936.268002] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223936.268002] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223936.268002] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223936.268002] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223936.268002] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223936.268002] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223936.268002] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223936.268002] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223936.268002] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223936.268002] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223936.268002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223936.268002] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.268002] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223936.268002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223936.268002] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223936.268002] [<ffffffff816ca491>] rest_init+0xd1/0xe0
> [223936.268002] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
> [223936.268002] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
> [223936.268002] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
> [223936.268002] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
> [223936.268002] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
> [223964.264018] BUG: soft lockup - CPU#0 stuck for 22s! [swapper:0]
> [223964.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.268002] irq event stamp: 2595159887
> [223964.268002] hardirqs last enabled at (2595159887): [<ffffffff8101b805>] mwait_idle+0x145/0x170
> [223964.268002] hardirqs last disabled at (2595159886): [<ffffffff81013139>] cpu_idle+0x79/0xf0
> [223964.268002] softirqs last enabled at (2595159878): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
> [223964.268002] softirqs last disabled at (2595159873): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.268002] CPU 0
> [223964.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.268002]
> [223964.268002] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223964.268002] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
> [223964.268002] RSP: 0018:ffff88022fc03ce0 EFLAGS: 00000202
> [223964.268002] RAX: 000000007cb6c61b RBX: ffffffff816f6334 RCX: 000000007cb6c5e3
> [223964.372025] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/0:0:0]
> [223964.372027] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.372036] irq event stamp: 2598787699
> [223964.372037] hardirqs last enabled at (2598787699): [<ffffffff8101b805>] mwait_idle+0x145/0x170
> [223964.372042] hardirqs last disabled at (2598787698): [<ffffffff81013139>] cpu_idle+0x79/0xf0
> [223964.372045] softirqs last enabled at (2598787696): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
> [223964.372049] softirqs last disabled at (2598787681): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.372052] CPU 1
> [223964.372053] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.372061]
> [223964.372063] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223964.372067] RIP: 0010:[<ffffffff8101afa0>] [<ffffffff8101afa0>] read_persistent_clock+0x30/0x30
> [223964.372072] RSP: 0018:ffff88022fc43ce8 EFLAGS: 00000202
> [223964.372074] RAX: 0000000000000001 RBX: ffff88022fc43c68 RCX: 0000000088b369fd
> [223964.372076] RDX: 0000000000000200 RSI: 0000000000000000 RDI: 0000000000000001
> [223964.372078] RBP: ffff88022fc43d30 R08: ffffffff88b369fd R09: 0000000000000000
> [223964.372081] R10: ffff88022690dd60 R11: 0000000000000000 R12: ffff88022fc43c58
> [223964.372083] R13: ffffffff816feb33 R14: ffff88022fc43d30 R15: 00000000307e58b4
> [223964.372086] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000
> [223964.372089] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223964.372091] CR2: 00007f472ba6a6fc CR3: 0000000126bb7000 CR4: 00000000000006e0
> [223964.372093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223964.372096] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223964.372098] Process kworker/0:0 (pid: 0, threadinfo ffff880226928000, task ffff88022690dd60)
> [223964.372100] Stack:
> [223964.372102] ffffffff813a4eaf ffff880226928000 ffffffff88b369c5 000000012fc43d18
> [223964.372106] ffff88002e90e348 00000000307e58b4 000000006efcdc62 0000000000000001
> [223964.372111] ffff88022690dd60 ffff88022fc43d40 ffffffff813a4f6a ffff88022fc43d80
> [223964.372115] Call Trace:
> [223964.372116] <IRQ>
> [223964.372119] [<ffffffff813a4eaf>] ? delay_tsc+0x3f/0xd0
> [223964.372123] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.372127] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.372132] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.372136] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.372140] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.372144] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.372148] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.372153] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.372158] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.372162] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.372166] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.372170] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.372174] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.372178] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.372180] <EOI>
> [223964.372184] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.372188] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.372192] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.372196] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.372200] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223964.372203] Code: 48 89 fb 48 83 ec 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00
> [223964.372221] 48 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9
> [223964.372231] Call Trace:
> [223964.372232] <IRQ> [<ffffffff813a4eaf>] ? delay_tsc+0x3f/0xd0
> [223964.372237] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.372241] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.372245] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.372248] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.372251] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.372255] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.372258] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.372261] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.372265] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.372268] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.372271] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.372275] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.372278] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.372281] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.372282] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.372288] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.372292] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.372295] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.372298] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223964.476031] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/0:1:0]
> [223964.476033] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.476042] irq event stamp: 2613824057
> [223964.476043] hardirqs last enabled at (2613824057): [<ffffffff8101b805>] mwait_idle+0x145/0x170
> [223964.476048] hardirqs last disabled at (2613824056): [<ffffffff81013139>] cpu_idle+0x79/0xf0
> [223964.476051] softirqs last enabled at (2613824048): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
> [223964.476055] softirqs last disabled at (2613824031): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.476059] CPU 2
> [223964.476060] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.476067]
> [223964.476070] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223964.476074] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
> [223964.476078] RSP: 0018:ffff88022fc83ce0 EFLAGS: 00000206
> [223964.476080] RAX: 00000000943e6715 RBX: ffffffff816f6334 RCX: 00000000943e66dd
> [223964.476083] RDX: 0000000000017b69 RSI: 0000000000000000 RDI: 0000000000000001
> [223964.476085] RBP: ffff88022fc83ce0 R08: ffffffff943e6697 R09: 0000000000000000
> [223964.476087] R10: ffff880226948000 R11: 0000000000000000 R12: ffff88022fc83c58
> [223964.476090] R13: ffffffff816feb33 R14: ffff88022fc83ce0 R15: 000000003094ad30
> [223964.476092] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
> [223964.476095] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223964.476097] CR2: 00007f25e7874d7f CR3: 0000000124c0d000 CR4: 00000000000006e0
> [223964.476100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223964.476102] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223964.476105] Process kworker/0:1 (pid: 0, threadinfo ffff88022693e000, task ffff880226948000)
> [223964.476107] Stack:
> [223964.476108] ffff88022fc83d30 ffffffff813a4ee8 ffff88022693e000 ffffffff943e66dd
> [223964.476113] 00000002ffffff10 ffff88006afd8948 000000003094ad30 000000006efcdc62
> [223964.476117] 0000000000000001 ffff880226948000 ffff88022fc83d40 ffffffff813a4f6a
> [223964.476121] Call Trace:
> [223964.476123] <IRQ>
> [223964.476126] [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223964.476130] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.476134] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.476139] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.476143] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.476147] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.476151] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.476155] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.476159] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.476164] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.476168] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.476172] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.476176] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.476180] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.476184] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.476186] <EOI>
> [223964.476190] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.476194] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.476198] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.476202] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.476206] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223964.476208] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
> [223964.476227] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
> [223964.476236] Call Trace:
> [223964.476238] <IRQ> [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223964.476243] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.476246] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.476250] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.476254] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.476257] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.476260] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.476264] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.476267] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.476271] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.476274] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.476277] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.476281] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.476284] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.476287] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.476289] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.476295] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.476298] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.476301] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.476304] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223964.580038] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/0:1:0]
> [223964.580040] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.580049] irq event stamp: 2615464042
> [223964.580050] hardirqs last enabled at (2615464042): [<ffffffff816f5edb>] _raw_spin_unlock_irq+0x2b/0x50
> [223964.580054] hardirqs last disabled at (2615464041): [<ffffffff816f56a8>] _raw_spin_lock_irq+0x18/0x60
> [223964.580058] softirqs last enabled at (2615463964): [<ffffffff81063cce>] _local_bh_enable+0xe/0x10
> [223964.580062] softirqs last disabled at (2615463965): [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.580066] CPU 3
> [223964.580067] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223964.580075]
> [223964.580077] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223964.580081] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
> [223964.580086] RSP: 0018:ffff88022fcc3ce0 EFLAGS: 00000206
> [223964.580088] RAX: 000000009fc963af RBX: ffffffff816f6334 RCX: 000000009fc96377
> [223964.580090] RDX: 0000000000017b69 RSI: ffffffff818f9896 RDI: 0000000000000001
> [223964.580093] RBP: ffff88022fcc3ce0 R08: ffffffff9fc96331 R09: 0000000000000000
> [223964.580095] R10: ffff880226981f20 R11: 0000000000000000 R12: ffff88022fcc3c58
> [223964.580097] R13: ffffffff816feb33 R14: ffff88022fcc3ce0 R15: 0000000030ac88b0
> [223964.580100] FS: 0000000000000000(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
> [223964.580103] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223964.580105] CR2: 0000000000f38820 CR3: 0000000104b52000 CR4: 00000000000006e0
> [223964.580107] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223964.580110] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223964.580112] Process kworker/0:1 (pid: 0, threadinfo ffff880226988000, task ffff880226981f20)
> [223964.580114] Stack:
> [223964.580116] ffff88022fcc3d30 ffffffff813a4ee8 ffff880226988000 ffffffff9fc96377
> [223964.580120] 000000039c3b34d8 ffff880031438948 0000000030ac88b0 000000006efcdc62
> [223964.580124] 0000000000000001 ffff880226981f20 ffff88022fcc3d40 ffffffff813a4f6a
> [223964.580128] Call Trace:
> [223964.580130] <IRQ>
> [223964.580133] [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223964.580137] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.580141] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.580146] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.580150] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.580154] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.580158] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.580162] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.580167] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.580171] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.580176] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.580180] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.580184] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.580188] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.580192] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.580194] <EOI>
> [223964.580198] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.580202] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.580206] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.580210] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.580214] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223964.580217] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
> [223964.580235] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
> [223964.580245] Call Trace:
> [223964.580246] <IRQ> [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223964.580252] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.580255] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.580259] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.580262] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.580265] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.580269] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.580272] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.580276] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.580279] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.580283] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.580286] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.580289] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.580292] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.580295] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.580297] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.580303] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.580307] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.580310] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.580313] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223964.268002] RDX: 0000000000017b69 RSI: ffffffff818f9896 RDI: 0000000000000001
> [223964.268002] RBP: ffff88022fc03ce0 R08: 000000007cb6c596 R09: 0000000000000000
> [223964.268002] R10: ffffffff81a2b020 R11: 0000000000000000 R12: ffff88022fc03c58
> [223964.268002] R13: ffffffff816feb33 R14: ffff88022fc03ce0 R15: 000000002eb85d38
> [223964.268002] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
> [223964.268002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223964.268002] CR2: 00007f25e7bc13a0 CR3: 00000001426fc000 CR4: 00000000000006f0
> [223964.268002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223964.268002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223964.268002] Process swapper (pid: 0, threadinfo ffffffff819a6000, task ffffffff81a2b020)
> [223964.268002] Stack:
> [223964.268002] ffff88022fc03d30 ffffffff813a4ee8 ffffffff819a6000 000000007cb6c5e3
> [223964.268002] 000000007c44ac9c ffff8801f6c22448 000000002eb85d38 000000006efcdc62
> [223964.268002] 0000000000000001 ffffffff81a2b020 ffff88022fc03d40 ffffffff813a4f6a
> [223964.268002] Call Trace:
> [223964.268002] <IRQ>
> [223964.268002] [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223964.268002] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.268002] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.268002] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.268002] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.268002] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.268002] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.268002] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.268002] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.268002] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.268002] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.268002] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.268002] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.268002] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.268002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.268002] <EOI>
> [223964.268002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.268002] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.268002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.268002] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.268002] [<ffffffff816ca491>] rest_init+0xd1/0xe0
> [223964.268002] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
> [223964.268002] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
> [223964.268002] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
> [223964.268002] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
> [223964.268002] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
> [223964.268002] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
> [223964.268002] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
> [223964.268002] Call Trace:
> [223964.268002] <IRQ> [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223964.268002] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223964.268002] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223964.268002] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223964.268002] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223964.268002] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223964.268002] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223964.268002] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223964.268002] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223964.268002] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223964.268002] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223964.268002] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223964.268002] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223964.268002] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223964.268002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223964.268002] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.268002] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223964.268002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223964.268002] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223964.268002] [<ffffffff816ca491>] rest_init+0xd1/0xe0
> [223964.268002] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
> [223964.268002] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
> [223964.268002] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
> [223964.268002] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
> [223964.268002] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
> [223968.815995] INFO: rcu_sched_state detected stall on CPU 1 (t=15000 jiffies)
> [223968.819995] INFO: rcu_sched_state detected stalls on CPUs/tasks: { 1} (detected by 3, t=15002 jiffies)
> [223968.820000] sending NMI to all CPUs:
> [223968.820002] NMI backtrace for cpu 3
> [223968.820002] CPU 3
> [223968.820002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223968.820002]
> [223968.820002] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223968.820002] RIP: 0010:[<ffffffff813a4f86>] [<ffffffff813a4f86>] __const_udelay+0x16/0x40
> [223968.820002] RSP: 0018:ffff88022fcc3a90 EFLAGS: 00000002
> [223968.820002] RAX: 0000000000e34d8a RBX: 0000000000000001 RCX: 0000000001062560
> [223968.820002] RDX: 000000000071a6c5 RSI: 0000000000000002 RDI: 0000000000418958
> [223968.820002] RBP: ffff88022fcc3ab0 R08: 0000000000000002 R09: 0000000000000000
> [223968.820002] R10: 0000000000000006 R11: 000000000000000a R12: ffffffff81a40d80
> [223968.820002] R13: 0000000000000010 R14: ffffffff81a40e40 R15: ffffffff81a40fc0
> [223968.820002] FS: 0000000000000000(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
> [223968.820002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223968.820002] CR2: 0000000000f38820 CR3: 0000000104b52000 CR4: 00000000000006e0
> [223968.820002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223968.820002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223968.820002] Process kworker/0:1 (pid: 0, threadinfo ffff880226988000, task ffff880226981f20)
> [223968.820002] Stack:
> [223968.820002] ffff88022fcc3ab0 ffffffff81031695 ffff88022fccdfa0 ffff88022fccdfa0
> [223968.820002] ffff88022fcc3af0 ffffffff810bb9d2 ffffffff81a40fc0 0000000000000003
> [223968.820002] 0000000000000003 ffff880226981f20 ffffffff810921f0 ffff88022fcc3be0
> [223968.820002] Call Trace:
> [223968.820002] <IRQ>
> [223968.820002] [<ffffffff81031695>] ? arch_trigger_all_cpu_backtrace+0x65/0x90
> [223968.820002] [<ffffffff810bb9d2>] __rcu_pending+0x382/0x3b0
> [223968.820002] [<ffffffff810921f0>] ? tick_nohz_handler+0x100/0x100
> [223968.820002] [<ffffffff810bba67>] rcu_check_callbacks+0x67/0x130
> [223968.820002] [<ffffffff8106d861>] update_process_times+0x41/0x80
> [223968.820002] [<ffffffff81092256>] tick_sched_timer+0x66/0xc0
> [223968.820002] [<ffffffff810845ee>] __run_hrtimer+0xfe/0x1e0
> [223968.820002] [<ffffffff8108491d>] hrtimer_interrupt+0xcd/0x1f0
> [223968.820002] [<ffffffff810310c4>] smp_apic_timer_interrupt+0x64/0xa0
> [223968.820002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820002] [<ffffffff816f6334>] ? retint_restore_args+0x13/0x13
> [223968.820002] [<ffffffff8101afa6>] ? native_read_tsc+0x6/0x20
> [223968.820002] [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
> [223968.820002] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.820002] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.820002] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.820002] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.820002] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.820002] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.820002] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.820002] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.820002] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.820002] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.820002] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.820002] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.820002] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.820002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820002] <EOI>
> [223968.820002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820002] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.820002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820002] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.820002] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223968.820002] Code: 00 00 00 00 00 55 48 89 e5 ff 15 8e a5 6c 00 c9 c3 0f 1f 40 00 55 48 8d 0c bd 00 00 00 00 65 48 8b 14 25 58 2d 01 00 48 8d 04 12
> [223968.820002] c1 e2 06 48 89 e5 48 29 c2 48 89 c8 f7 e2 48 8d 7a 01 ff 15
> [223968.820002] Call Trace:
> [223968.820002] <IRQ> [<ffffffff81031695>] ? arch_trigger_all_cpu_backtrace+0x65/0x90
> [223968.820002] [<ffffffff810bb9d2>] __rcu_pending+0x382/0x3b0
> [223968.820002] [<ffffffff810921f0>] ? tick_nohz_handler+0x100/0x100
> [223968.820002] [<ffffffff810bba67>] rcu_check_callbacks+0x67/0x130
> [223968.820002] [<ffffffff8106d861>] update_process_times+0x41/0x80
> [223968.820002] [<ffffffff81092256>] tick_sched_timer+0x66/0xc0
> [223968.820002] [<ffffffff810845ee>] __run_hrtimer+0xfe/0x1e0
> [223968.820002] [<ffffffff8108491d>] hrtimer_interrupt+0xcd/0x1f0
> [223968.820002] [<ffffffff810310c4>] smp_apic_timer_interrupt+0x64/0xa0
> [223968.820002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820002] [<ffffffff816f6334>] ? retint_restore_args+0x13/0x13
> [223968.820002] [<ffffffff8101afa6>] ? native_read_tsc+0x6/0x20
> [223968.820002] [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
> [223968.820002] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.820002] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.820002] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.820002] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.820002] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.820002] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.820002] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.820002] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.820002] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.820002] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.820002] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.820002] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.820002] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.820002] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820002] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820002] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.820002] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820002] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.820002] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223968.820335] NMI backtrace for cpu 0
> [223968.820337] CPU 0
> [223968.820338] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223968.820347]
> [223968.820349] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223968.820353] RIP: 0010:[<ffffffff813a4ef0>] [<ffffffff813a4ef0>] delay_tsc+0x80/0xd0
> [223968.820358] RSP: 0018:ffff88022fc03cf0 EFLAGS: 00000206
> [223968.820360] RAX: 000000007659b10f RBX: 0000000000000001 RCX: 000000007659b10f
> [223968.820363] RDX: 000000007659b10f RSI: ffffffff818f9896 RDI: 0000000000000001
> [223968.820365] RBP: ffff88022fc03d30 R08: 000000007659b10f R09: 0000000000000000
> [223968.820367] R10: ffffffff81a2b020 R11: 0000000000000000 R12: 0000000031026962
> [223968.820370] R13: 000000006efcdc62 R14: ffffffff819a6000 R15: 000000007659b0de
> [223968.820373] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
> [223968.820375] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223968.820377] CR2: 00007f25e7bc13a0 CR3: 00000001426fc000 CR4: 00000000000006f0
> [223968.820380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223968.820382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223968.820385] Process swapper (pid: 0, threadinfo ffffffff819a6000, task ffffffff81a2b020)
> [223968.820387] Stack:
> [223968.820388] ffffffff819a6000 000000007659b0de 00000000818f9896 ffff8801f6c22448
> [223968.820393] 0000000031026962 000000006efcdc62 0000000000000001 ffffffff81a2b020
> [223968.820397] ffff88022fc03d40 ffffffff813a4f6a ffff88022fc03d80 ffffffff813ac2ab
> [223968.820401] Call Trace:
> [223968.820402] <IRQ>
> [223968.820406] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.820410] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.820414] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.820417] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.820420] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.820424] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.820427] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.820430] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.820434] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.820437] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.820441] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.820444] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.820447] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.820450] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820452] <EOI>
> [223968.820455] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820459] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.820462] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820465] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.820468] [<ffffffff816ca491>] rest_init+0xd1/0xe0
> [223968.820471] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
> [223968.820475] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
> [223968.820478] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
> [223968.820481] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
> [223968.820484] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
> [223968.820486] Code: 68 1c 01 f3 90 83 40 1c 01 65 44 8b 3c 25 50 d3 00 00 44 3b 7d d4 75 3b 66 66 90 0f ae e8 e8 b8 60 c7 ff 66 90 4c 63 c0 4c 89 c0
> [223968.820504] 2b 45 c8 48 39 d8 72 c7 65 48 8b 04 25 08 c4 00 00 83 a8 44
> [223968.820514] Call Trace:
> [223968.820515] <IRQ> [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.820521] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.820525] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.820528] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.820532] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.820535] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.820538] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.820542] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.820546] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.820549] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.820552] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.820556] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.820559] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.820562] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820564] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820570] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.820573] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820576] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.820579] [<ffffffff816ca491>] rest_init+0xd1/0xe0
> [223968.820583] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
> [223968.820586] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
> [223968.820589] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
> [223968.820593] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
> [223968.820596] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
> [223968.820599] NMI backtrace for cpu 2
> [223968.820600] CPU 2
> [223968.820602] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223968.820610]
> [223968.820612] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223968.820616] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
> [223968.820621] RSP: 0018:ffff88022fc83ce0 EFLAGS: 00000202
> [223968.820623] RAX: 000000007659b116 RBX: 0000000000000001 RCX: 000000007659b0e5
> [223968.820625] RDX: 0000000000017b6b RSI: 0000000000000000 RDI: 0000000000000001
> [223968.820628] RBP: ffff88022fc83ce0 R08: 000000007659b098 R09: 0000000000000000
> [223968.820630] R10: ffff880226948000 R11: 0000000000000000 R12: 00000000345f87d7
> [223968.820632] R13: 000000006efcdc62 R14: ffff88022693e000 R15: 000000007659b0e5
> [223968.820635] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
> [223968.820638] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223968.820640] CR2: 00007f25e7874d7f CR3: 0000000124c0d000 CR4: 00000000000006e0
> [223968.820642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223968.820645] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223968.820647] Process kworker/0:1 (pid: 0, threadinfo ffff88022693e000, task ffff880226948000)
> [223968.820649] Stack:
> [223968.820651] ffff88022fc83d30 ffffffff813a4ee8 ffff88022693e000 000000007659b0e5
> [223968.820655] 000000026b4044c5 ffff88006afd8948 00000000345f87d7 000000006efcdc62
> [223968.820659] 0000000000000001 ffff880226948000 ffff88022fc83d40 ffffffff813a4f6a
> [223968.820663] Call Trace:
> [223968.820665] <IRQ>
> [223968.820668] [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223968.820671] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.820674] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.820678] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.820682] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.820685] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.820688] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.820691] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.820695] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.820699] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.820702] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.820705] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.820708] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.820712] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.820715] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820717] <EOI>
> [223968.820720] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820723] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.820727] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820730] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.820733] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223968.820735] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
> [223968.820753] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
> [223968.820763] Call Trace:
> [223968.820764] <IRQ> [<ffffffff813a4ee8>] delay_tsc+0x78/0xd0
> [223968.820769] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.820773] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.820777] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.820780] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.820783] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.820787] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.820790] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.820793] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.820797] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.820801] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.820804] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.820807] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.820810] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.820813] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.820815] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820821] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.820824] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.820827] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.820831] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223968.816001] NMI backtrace for cpu 1
> [223968.816001] CPU 1
> [223968.816001] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
> [223968.816001]
> [223968.816001] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G
> [223968.816001] RIP: 0010:[<ffffffff81440955>] [<ffffffff81440955>] io_serial_out+0x15/0x20
> [223968.816001] RSP: 0018:ffff88022fc437f0 EFLAGS: 00000002
> [223968.816001] RAX: 0000000000000073 RBX: ffffffff8243eec0 RCX: 0000000000000000
> [223968.816001] RDX: 00000000000003f8 RSI: 00000000000003f8 RDI: ffffffff8243eec0
> [223968.816001] RBP: ffff88022fc437f0 R08: 000000007659a435 R09: 0000000000000000
> [223968.816001] R10: 0000000000000001 R11: 000000000000000a R12: 0000000000000073
> [223968.816001] R13: ffffffff81bc648d R14: 0000000000000050 R15: ffffffff8243eec0
> [223968.816001] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000
> [223968.816001] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [223968.816001] CR2: 00007f472ba6a6fc CR3: 0000000126bb7000 CR4: 00000000000006e0
> [223968.816001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [223968.816001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [223968.816001] Process kworker/0:0 (pid: 0, threadinfo ffff880226928000, task ffff88022690dd60)
> [223968.816001] Stack:
> [223968.816001] ffff88022fc43810 ffffffff814410dc 0000000000000030 ffffffff814410b0
> [223968.816001] ffff88022fc43850 ffffffff8143cdb5 0000000000000087 0000000000000000
> [223968.816001] ffffffff8243eec0 0000000000000001 0000000000000087 000000000000000d
> [223968.816001] Call Trace:
> [223968.816001] <IRQ>
> [223968.816001] [<ffffffff814410dc>] serial8250_console_putchar+0x2c/0x40
> [223968.816001] [<ffffffff814410b0>] ? wait_for_xmitr+0xa0/0xa0
> [223968.816001] [<ffffffff8143cdb5>] uart_console_write+0x35/0x70
> [223968.816001] [<ffffffff814417be>] serial8250_console_write+0xbe/0x1a0
> [223968.816001] [<ffffffff8105c78e>] __call_console_drivers+0x8e/0xb0
> [223968.816001] [<ffffffff8105c7f5>] _call_console_drivers+0x45/0x70
> [223968.816001] [<ffffffff8105d02f>] console_unlock+0x17f/0x2b0
> [223968.816001] [<ffffffff8105d64d>] vprintk+0x1fd/0x520
> [223968.816001] [<ffffffff816f16c1>] printk+0x67/0x69
> [223968.816001] [<ffffffff816f5fa6>] ? _raw_spin_unlock+0x26/0x40
> [223968.816001] [<ffffffff8105388b>] ? account_system_time+0xab/0x190
> [223968.816001] [<ffffffff810bb7e4>] __rcu_pending+0x194/0x3b0
> [223968.816001] [<ffffffff810921f0>] ? tick_nohz_handler+0x100/0x100
> [223968.816001] [<ffffffff810bba67>] rcu_check_callbacks+0x67/0x130
> [223968.816001] [<ffffffff8106d861>] update_process_times+0x41/0x80
> [223968.816001] [<ffffffff81092256>] tick_sched_timer+0x66/0xc0
> [223968.816001] [<ffffffff810845ee>] __run_hrtimer+0xfe/0x1e0
> [223968.816001] [<ffffffff8108491d>] hrtimer_interrupt+0xcd/0x1f0
> [223968.816001] [<ffffffff810310c4>] smp_apic_timer_interrupt+0x64/0xa0
> [223968.816001] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.816001] [<ffffffff816f6334>] ? retint_restore_args+0x13/0x13
> [223968.816001] [<ffffffff8101afa6>] ? native_read_tsc+0x6/0x20
> [223968.816001] [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
> [223968.816001] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.816001] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.816001] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.816001] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.816001] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.816001] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.816001] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.816001] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.816001] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.816001] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.816001] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.816001] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.816001] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.816001] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.816001] <EOI>
> [223968.816001] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.816001] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.816001] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.816001] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.816001] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
> [223968.816001] Code: 48 89 e5 d3 e2 03 57 38 ec 0f b6 c0 c9 c3 0f 1f 84 00 00 00 00 00 0f b6 8f 81 00 00 00 55 89 d0 48 89 e5 d3 e6 03 77 38 89 f2 ee <c9> c3 66 0f 1f 84 00 00 00 00 00 55 80 bf 82 00 00 00 08 48 89
> [223968.816001] Call Trace:
> [223968.816001] <IRQ> [<ffffffff814410dc>] serial8250_console_putchar+0x2c/0x40
> [223968.816001] [<ffffffff814410b0>] ? wait_for_xmitr+0xa0/0xa0
> [223968.816001] [<ffffffff8143cdb5>] uart_console_write+0x35/0x70
> [223968.816001] [<ffffffff814417be>] serial8250_console_write+0xbe/0x1a0
> [223968.816001] [<ffffffff8105c78e>] __call_console_drivers+0x8e/0xb0
> [223968.816001] [<ffffffff8105c7f5>] _call_console_drivers+0x45/0x70
> [223968.816001] [<ffffffff8105d02f>] console_unlock+0x17f/0x2b0
> [223968.816001] [<ffffffff8105d64d>] vprintk+0x1fd/0x520
> [223968.816001] [<ffffffff816f16c1>] printk+0x67/0x69
> [223968.816001] [<ffffffff816f5fa6>] ? _raw_spin_unlock+0x26/0x40
> [223968.816001] [<ffffffff8105388b>] ? account_system_time+0xab/0x190
> [223968.816001] [<ffffffff810bb7e4>] __rcu_pending+0x194/0x3b0
> [223968.816001] [<ffffffff810921f0>] ? tick_nohz_handler+0x100/0x100
> [223968.816001] [<ffffffff810bba67>] rcu_check_callbacks+0x67/0x130
> [223968.816001] [<ffffffff8106d861>] update_process_times+0x41/0x80
> [223968.816001] [<ffffffff81092256>] tick_sched_timer+0x66/0xc0
> [223968.816001] [<ffffffff810845ee>] __run_hrtimer+0xfe/0x1e0
> [223968.816001] [<ffffffff8108491d>] hrtimer_interrupt+0xcd/0x1f0
> [223968.816001] [<ffffffff810310c4>] smp_apic_timer_interrupt+0x64/0xa0
> [223968.816001] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.816001] [<ffffffff816f6334>] ? retint_restore_args+0x13/0x13
> [223968.816001] [<ffffffff8101afa6>] ? native_read_tsc+0x6/0x20
> [223968.816001] [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
> [223968.816001] [<ffffffff813a4f6a>] __delay+0xa/0x10
> [223968.816001] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
> [223968.816001] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
> [223968.816001] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
> [223968.816001] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
> [223968.816001] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
> [223968.816001] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
> [223968.816001] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
> [223968.816001] [<ffffffff810640b8>] __do_softirq+0x138/0x250
> [223968.816001] [<ffffffff817002bc>] call_softirq+0x1c/0x30
> [223968.816001] [<ffffffff810153c5>] do_softirq+0x95/0xd0
> [223968.816001] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
> [223968.816001] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
> [223968.816001] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
> [223968.816001] <EOI> [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.816001] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
> [223968.816001] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
> [223968.816001] [<ffffffff81013156>] cpu_idle+0x96/0xf0
> [223968.816001] [<ffffffff816ec4bb>] start_secondary+0x1ca/0x1ff
>
> [ goes on for another ~300kB, trimmed ]
>
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: Linux 3.1-rc9
[not found] ` <20111024190203.GA24410@hostway.ca>
2011-10-25 7:13 ` Linux 3.1-rc9 Linus Torvalds
@ 2011-10-25 20:20 ` Simon Kirby
2011-10-31 17:32 ` Simon Kirby
1 sibling, 1 reply; 31+ messages in thread
From: Simon Kirby @ 2011-10-25 20:20 UTC (permalink / raw)
To: Thomas Gleixner, David Miller
Cc: Peter Zijlstra, Linus Torvalds, Linux Kernel Mailing List,
Dave Jones, Martin Schwidefsky, Ingo Molnar, Network Development
On Mon, Oct 24, 2011 at 12:02:03PM -0700, Simon Kirby wrote:
> Ok, hit the hang about 4 more times, but only this morning on a box with
> a serial cable attached. Yay!
Here's lockdep output from another box. This one looks a bit different.
Simon-
[583223.799383] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[583223.805083] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[583223.805093]
[583223.805094] =================================
[583223.805096] [ INFO: inconsistent lock state ]
[583223.805098] 3.1.0-rc10-hw-lockdep+ #51
[583223.805100] ---------------------------------
[583223.805102] inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
[583223.805105] swapper/0 [HC0[0]:SC0[0]:HE1:SE1] takes:
[583223.805107] (slock-AF_INET){+.?.-.}, at: [<ffffffff81604540>] sk_clone+0x120/0x420
[583223.805116] {IN-SOFTIRQ-W} state was registered at:
[583223.805117] [<ffffffff81098c7c>] __lock_acquire+0xcbc/0x2180
[583223.805123] [<ffffffff8109a7b9>] lock_acquire+0x109/0x140
[583223.805126] [<ffffffff816f55fc>] _raw_spin_lock+0x3c/0x50
[583223.805131] [<ffffffff8166bd3d>] udp_queue_rcv_skb+0x26d/0x4b0
[583223.805135] [<ffffffff8166c6a3>] __udp4_lib_rcv+0x2f3/0x910
[583223.805139] [<ffffffff8166ccd5>] udp_rcv+0x15/0x20
[583223.805142] [<ffffffff81641960>] ip_local_deliver_finish+0x100/0x2f0
[583223.805146] [<ffffffff81641bdd>] ip_local_deliver+0x8d/0xa0
[583223.805149] [<ffffffff81641203>] ip_rcv_finish+0x1a3/0x510
[583223.805152] [<ffffffff816417e2>] ip_rcv+0x272/0x2f0
[583223.805154] [<ffffffff81610d67>] __netif_receive_skb+0x4d7/0x560
[583223.805158] [<ffffffff81610ec0>] process_backlog+0xd0/0x1e0
[583223.805161] [<ffffffff81613880>] net_rx_action+0x140/0x2c0
[583223.805164] [<ffffffff810640b8>] __do_softirq+0x138/0x250
[583223.805168] [<ffffffff817002bc>] call_softirq+0x1c/0x30
[583223.805172] [<ffffffff810153c5>] do_softirq+0x95/0xd0
[583223.805176] [<ffffffff81063ecd>] local_bh_enable+0xed/0x110
[583223.805179] [<ffffffff81614c48>] dev_queue_xmit+0x1a8/0x8a0
[583223.805181] [<ffffffff8161f1aa>] neigh_resolve_output+0x17a/0x220
[583223.805185] [<ffffffff81647d4c>] ip_finish_output+0x2ec/0x590
[583223.805188] [<ffffffff81648078>] ip_output+0x88/0xe0
[583223.805191] [<ffffffff81646cd8>] ip_local_out+0x28/0x80
[583223.805194] [<ffffffff81646d39>] ip_send_skb+0x9/0x40
[583223.805197] [<ffffffff8166aeb2>] udp_send_skb+0x122/0x390
[583223.805200] [<ffffffff8166db0c>] udp_sendmsg+0x7dc/0x920
[583223.805203] [<ffffffff81675e1f>] inet_sendmsg+0xbf/0x120
[583223.805207] [<ffffffff815ff333>] sock_sendmsg+0xe3/0x110
[583223.805209] [<ffffffff815ffc55>] sys_sendto+0x105/0x140
[583223.805212] [<ffffffff816fe052>] system_call_fastpath+0x16/0x1b
[583223.805217] irq event stamp: 4284605374
[583223.805219] hardirqs last enabled at (4284605372): [<ffffffff816101ad>] net_rps_action_and_irq_enable+0x8d/0xa0
[583223.805222] hardirqs last disabled at (4284605373): [<ffffffff8106412d>] __do_softirq+0x1ad/0x250
[583223.805226] softirqs last enabled at (4284605374): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
[583223.805230] softirqs last disabled at (4284605313): [<ffffffff817002bc>] call_softirq+0x1c/0x30
[583223.805233]
[583223.805233] other info that might help us debug this:
[583223.805235] Possible unsafe locking scenario:
[583223.805236]
[583223.805237] CPU0
[583223.805238] ----
[583223.805239] lock(slock-AF_INET);
[583223.805241] <Interrupt>
[583223.805242] lock(slock-AF_INET);
[583223.805244]
[583223.805245] *** DEADLOCK ***
[583223.805246]
[583223.805248] 1 lock held by swapper/0:
[583223.805249] #0: (slock-AF_INET){+.?.-.}, at: [<ffffffff81604540>] sk_clone+0x120/0x420
[583223.805254]
[583223.805254] stack backtrace:
[583223.805257] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51
[583223.805259] Call Trace:
[583223.805264] [<ffffffff81096033>] print_usage_bug+0x243/0x310
[583223.805267] [<ffffffff810965b4>] mark_lock+0x4b4/0x6c0
[583223.805271] [<ffffffff81097400>] ? check_usage_forwards+0x110/0x110
[583223.805275] [<ffffffff81096862>] mark_held_locks+0xa2/0x130
[583223.805278] [<ffffffff816f6334>] ? retint_restore_args+0x13/0x13
[583223.805281] [<ffffffff81096c0d>] trace_hardirqs_on_caller+0x13d/0x1c0
[583223.805286] [<ffffffff813a60ae>] trace_hardirqs_on_thunk+0x3a/0x3f
[583223.805290] [<ffffffff81092b8e>] ? tick_nohz_stop_sched_tick+0x2fe/0x430
[583223.805293] [<ffffffff816f6334>] ? retint_restore_args+0x13/0x13
[583223.805297] [<ffffffff8101b80e>] ? mwait_idle+0x14e/0x170
[583223.805301] [<ffffffff8101b805>] ? mwait_idle+0x145/0x170
[583223.805304] [<ffffffff81013156>] cpu_idle+0x96/0xf0
[583223.805307] [<ffffffff816ca491>] rest_init+0xd1/0xe0
[583223.805310] [<ffffffff816ca3c0>] ? csum_partial_copy_generic+0x170/0x170
[583223.805315] [<ffffffff81adcc55>] start_kernel+0x360/0x3ac
[583223.805318] [<ffffffff81adc2a2>] x86_64_start_reservations+0x82/0x89
[583223.805321] [<ffffffff81adc3b8>] x86_64_start_kernel+0x10f/0x12a
[583223.805325] [<ffffffff81adc140>] ? early_idt_handlers+0x140/0x140
[583226.813848] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[583232.802948] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[583244.833571] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
[583253.849631] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
[583268.837126] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
[587843.931805] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[587846.165584] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
[587850.602316] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
[587859.482841] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[587873.940136] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103?
[587877.240624] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102?
[590476.272022] BUG: soft lockup - CPU#0 stuck for 22s! [swapper:0]
[590476.276002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[590476.276002] irq event stamp: 4284605374
[590476.276002] hardirqs last enabled at (4284605372): [<ffffffff816101ad>] net_rps_action_and_irq_enable+0x8d/0xa0
[590476.276002] hardirqs last disabled at (4284605373): [<ffffffff8106412d>] __do_softirq+0x1ad/0x250
[590476.276002] softirqs last enabled at (4284605374): [<ffffffff81064176>] __do_softirq+0x1f6/0x250
[590476.276002] softirqs last disabled at (4284605313): [<ffffffff817002bc>] call_softirq+0x1c/0x30
[590476.276002] CPU 0
[590476.276002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[590476.276002]
[590476.276002] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0UR033
[590476.276002] RIP: 0010:[<ffffffff813a4ee3>] [<ffffffff813a4ee3>] delay_tsc+0x73/0xd0
[590476.276002] RSP: 0018:ffff88022fc03cf0 EFLAGS: 00000206
[590476.276002] RAX: 00042f884dcdaa24 RBX: ffff88022fc0d3c0 RCX: 000000004dcdaa24
[590476.380029] BUG: soft lockup - CPU#1 stuck for 22s! [php:10828]
[590476.380033] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[590476.380044] irq event stamp: 0
[590476.380045] hardirqs last enabled at (0): [< (null)>] (null)
[590476.380048] hardirqs last disabled at (0): [<ffffffff8105aa8b>] copy_process+0x65b/0x1450
[590476.380056] softirqs last enabled at (0): [<ffffffff8105aa8b>] copy_process+0x65b/0x1450
[590476.380060] softirqs last disabled at (0): [< (null)>] (null)
[590476.380063] CPU 1
[590476.380064] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[590476.380072]
[590476.380075] Pid: 10828, comm: php Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0UR033
[590476.380079] RIP: 0010:[<ffffffff8101afa6>] [<ffffffff8101afa6>] native_read_tsc+0x6/0x20
[590476.380086] RSP: 0000:ffff88022fc43ce0 EFLAGS: 00000206
[590476.380088] RAX: 000000005aa56d04 RBX: ffffffff816f6334 RCX: 000000005aa56c92
[590476.380091] RDX: 0000000000042f88 RSI: ffffffff818f9896 RDI: 0000000000000001
[590476.380093] RBP: ffff88022fc43ce0 R08: 000000005aa56c92 R09: 0000000000000000
[590476.380096] R10: ffff88014b9a9f20 R11: 0000000000000000 R12: ffff88022fc43c58
[590476.380098] R13: ffffffff816feb33 R14: ffff88022fc43ce0 R15: 000000000e27878c
[590476.380101] FS: 00007fb61c8fa720(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000
[590476.380103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[590476.380106] CR2: 00000000027914a0 CR3: 000000013a070000 CR4: 00000000000006e0
[590476.380108] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[590476.380110] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[590476.380113] Process php (pid: 10828, threadinfo ffff88014a1f2000, task ffff88014b9a9f20)
[590476.380115] Stack:
[590476.380117] ffff88022fc43d30 ffffffff813a4eaf ffff88014a1f2000 000000005aa56c38
[590476.380121] 00000001818f9896 ffff88001db58048 000000000e27878c 0000000076e96800
[590476.380125] 0000000000000001 ffff88014b9a9f20 ffff88022fc43d40 ffffffff813a4f6a
[590476.380129] Call Trace:
[590476.380132] <IRQ>
[590476.380137] [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
[590476.380141] [<ffffffff813a4f6a>] __delay+0xa/0x10
[590476.380145] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
[590476.380151] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
[590476.380157] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
[590476.380161] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
[590476.380166] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
[590476.380169] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
[590476.380174] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
[590476.380179] [<ffffffff810640b8>] __do_softirq+0x138/0x250
[590476.380184] [<ffffffff817002bc>] call_softirq+0x1c/0x30
[590476.380188] [<ffffffff810153c5>] do_softirq+0x95/0xd0
[590476.380191] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
[590476.380196] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
[590476.380200] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
[590476.380203] <EOI>
[590476.380206] [<ffffffff816f6319>] ? retint_swapgs+0x13/0x1b
[590476.380208] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31
[590476.380227] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84
[590476.380236] Call Trace:
[590476.380237] <IRQ> [<ffffffff813a4eaf>] delay_tsc+0x3f/0xd0
[590476.380242] [<ffffffff813a4f6a>] __delay+0xa/0x10
[590476.380246] [<ffffffff813ac2ab>] do_raw_spin_lock+0x13b/0x180
[590476.380249] [<ffffffff816f5604>] _raw_spin_lock+0x44/0x50
[590476.380252] [<ffffffff81661823>] ? tcp_keepalive_timer+0x23/0x270
[590476.380256] [<ffffffff81661823>] tcp_keepalive_timer+0x23/0x270
[590476.380259] [<ffffffff8106cd5d>] run_timer_softirq+0x26d/0x410
[590476.380262] [<ffffffff8106cc88>] ? run_timer_softirq+0x198/0x410
[590476.380265] [<ffffffff81661800>] ? tcp_init_xmit_timers+0x20/0x20
[590476.380268] [<ffffffff810640b8>] __do_softirq+0x138/0x250
[590476.380271] [<ffffffff817002bc>] call_softirq+0x1c/0x30
[590476.380274] [<ffffffff810153c5>] do_softirq+0x95/0xd0
[590476.380277] [<ffffffff81063c8d>] irq_exit+0xdd/0x110
[590476.380280] [<ffffffff810310c9>] smp_apic_timer_interrupt+0x69/0xa0
[590476.380283] [<ffffffff816feb33>] apic_timer_interrupt+0x73/0x80
[590476.380285] <EOI> [<ffffffff816f6319>] ? retint_swapgs+0x13/0x1b
[590476.484032] BUG: soft lockup - CPU#2 stuck for 23s! [suexec:10831]
...
^ permalink raw reply [flat|nested] 31+ messages in thread