From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756523Ab2AJQOD (ORCPT ); Tue, 10 Jan 2012 11:14:03 -0500 Received: from mail-wi0-f174.google.com ([209.85.212.174]:64059 "EHLO mail-wi0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756407Ab2AJQN7 (ORCPT ); Tue, 10 Jan 2012 11:13:59 -0500 Message-ID: <1326212033.19095.3.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> Subject: Re: [BUG] kernel freezes with latest tree From: Eric Dumazet To: Linus Torvalds Cc: linux-kernel Date: Tue, 10 Jan 2012 17:13:53 +0100 In-Reply-To: <1326183371.6638.6.camel@edumazet-laptop> References: <1326171444.6638.3.camel@edumazet-laptop> <1326171798.6638.4.camel@edumazet-laptop> <1326183371.6638.6.camel@edumazet-laptop> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.1- Content-Transfer-Encoding: 8bit Mime-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Le mardi 10 janvier 2012 à 09:16 +0100, Eric Dumazet a écrit : > Le mardi 10 janvier 2012 à 06:03 +0100, Eric Dumazet a écrit : > > Le mardi 10 janvier 2012 à 05:57 +0100, Eric Dumazet a écrit : > > > Hi Linus > > > > > > I got some freezes on two different machines, using latest kernel. > > > > > > while :; do hackbench 10 thread 4000; done > > > > > > Not sure I'll have time today to find the problem. > > > > > > It might be related to "perf top" also being run at least once. > > > > > > > Hmm, I can trigger the bug without ever using "perf". > > > > > > OK I managed to bisect it, but I have to run now. > > $ git bisect log > git bisect start > # bad: [31ae26ae7ba5ff6209b9ec5d1e1ba9442d6b87e6] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6 > git bisect bad 31ae26ae7ba5ff6209b9ec5d1e1ba9442d6b87e6 > # good: [805a6af8dba5dfdd35ec35dc52ec0122400b2610] Linux 3.2 > git bisect good 805a6af8dba5dfdd35ec35dc52ec0122400b2610 > # bad: [e4e88f31bcb5f05f24b9ae518d4ecb44e1a7774d] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc > git bisect bad e4e88f31bcb5f05f24b9ae518d4ecb44e1a7774d > # good: [ab9c17a009ee8eb8c667f22dc0be0709effceab9] mlx4_core: Modify driver initialization flow to accommodate SRIOV for Ethernet > git bisect good ab9c17a009ee8eb8c667f22dc0be0709effceab9 > # good: [117ff42fd43e92d24c6aa6f3e4f0f1e1edada140] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net > git bisect good 117ff42fd43e92d24c6aa6f3e4f0f1e1edada140 > # bad: [67b0243131150391125d8d0beb5359d7aec78b55] Merge branch 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect bad 67b0243131150391125d8d0beb5359d7aec78b55 > # good: [423d091dfe58d3109d84c408810a7cfa82f6f184] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect good 423d091dfe58d3109d84c408810a7cfa82f6f184 > # good: [f1ac18af219835fd5b8e19c14d2dd75c55f78737] perf: Add support for PERF_HW_COUNT_REF_CPU_CYCLES > git bisect good f1ac18af219835fd5b8e19c14d2dd75c55f78737 > # good: [1ac9bc6943edf7d181b4b1cc734981350d4f6bae] sched/tracing: Add a new tracepoint for sleeptime > git bisect good 1ac9bc6943edf7d181b4b1cc734981350d4f6bae > # bad: [0db49b72bce26341274b74fd968501489a361ae3] Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect bad 0db49b72bce26341274b74fd968501489a361ae3 > # good: [1b5495043d5bc058def21f9b66fd8feaa794eb44] perf tools: Moving code in some files > git bisect good 1b5495043d5bc058def21f9b66fd8feaa794eb44 > # good: [29c9862f1b818bf4caa4c48a30dbe5f25c84ee08] perf session: Remove impossible condition check > git bisect good 29c9862f1b818bf4caa4c48a30dbe5f25c84ee08 > # good: [466e2876bcb9ddc9b92502c46689679bee7d72a0] perf script: Kill script_spec__delete > git bisect good 466e2876bcb9ddc9b92502c46689679bee7d72a0 > # good: [35b740e4662ef386f0c60e1b60aaf5b44db9914c] Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect good 35b740e4662ef386f0c60e1b60aaf5b44db9914c > > > Could it be a merge error ? > > commit 0db49b72bce26341274b74fd968501489a361ae3 > Merge: 35b740e 1ac9bc6 > Author: Linus Torvalds > Date: Fri Jan 6 08:33:28 2012 -0800 Here are two traces of the lockups, since I had some free time to reproduce the bug on current tree (plus a minor change in bnx2 driver to avoid a netconsole related stack trace) 1st one : PREEMPT is enabled (CONFIG_PREEMPT=y) ----------------------------------------------- [ 3753.138509] ------------[ cut here ]------------ [ 3753.138509] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 3753.138509] Hardware name: ProLiant BL460c G1 [ 3753.138509] Watchdog detected hard LOCKUP on cpu 5 [ 3753.138509] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3753.138509] Pid: 5449, comm: IVR Not tainted 3.2.0-05066-g31ae26a-dirty #761 [ 3753.138509] Call Trace: [ 3753.138509] [] warn_slowpath_common+0x72/0xa0 [ 3753.138509] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.138509] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.138509] [] ? watchdog_timer_fn+0x170/0x170 [ 3753.138509] [] warn_slowpath_fmt+0x33/0x40 [ 3753.138509] [] watchdog_overflow_callback+0x9c/0xc0 [ 3753.138509] [] __perf_event_overflow+0x9a/0x240 [ 3753.138509] [] perf_event_overflow+0x15/0x20 [ 3753.138509] [] intel_pmu_handle_irq+0x129/0x250 [ 3753.138509] [] ? cpumask_next_and+0x28/0x40 [ 3753.138509] [] perf_event_nmi_handler+0x20/0x30 [ 3753.138509] [] nmi_handle.clone.0+0x38/0x50 [ 3753.138509] [] do_nmi+0x1b4/0x350 [ 3753.138509] [] nmi_stack_correct+0x28/0x2d [ 3753.138509] [] ? _raw_spin_lock+0x34/0x40 [ 3753.138509] [] load_balance+0x361/0x560 [ 3753.138509] [] idle_balance+0xd2/0x150 [ 3753.138509] [] __schedule+0x5e9/0x6b0 [ 3753.138509] [] ? timerqueue_add+0x60/0xc0 [ 3753.138509] [] ? get_parent_ip+0xb/0x40 [ 3753.138509] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.138509] [] ? get_parent_ip+0xb/0x40 [ 3753.138509] [] ? add_preempt_count+0x7b/0xc0 [ 3753.138509] [] ? _raw_spin_unlock_irqrestore+0x17/0x40 [ 3753.138509] [] schedule+0x35/0x50 [ 3753.138509] [] schedule_hrtimeout_range_clock+0xcc/0x180 [ 3753.138509] [] ? update_rmtp+0x80/0x80 [ 3753.138509] [] ? hrtimer_start_range_ns+0x26/0x30 [ 3753.138509] [] schedule_hrtimeout_range+0x17/0x20 [ 3753.138509] [] poll_schedule_timeout+0x3e/0x60 [ 3753.138509] [] do_select+0x552/0x690 [ 3753.138509] [] ? get_parent_ip+0xb/0x40 [ 3753.138509] [] ? poll_freewait+0xb0/0xb0 [ 3753.138509] [] ? __change_page_attr_set_clr+0xbb/0x8f0 [ 3753.138509] [] ? __alloc_pages_nodemask+0xde/0x600 [ 3753.138509] [] ? __change_page_attr_set_clr+0xbb/0x8f0 [ 3753.138509] [] ? native_sched_clock+0x25/0xc0 [ 3753.138509] [] ? native_sched_clock+0x25/0xc0 [ 3753.138509] [] ? sched_clock_local+0xe2/0x1e0 [ 3753.138509] [] ? native_sched_clock+0x25/0xc0 [ 3753.138509] [] ? sched_clock_local+0xe2/0x1e0 [ 3753.138509] [] ? sched_clock_cpu+0x119/0x1b0 [ 3753.138509] [] ? check_preempt_curr+0x72/0x90 [ 3753.138509] [] ? get_parent_ip+0xb/0x40 [ 3753.138509] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.138509] [] ? irq_exit+0x74/0xb0 [ 3753.138509] [] ? scheduler_ipi+0x73/0xc0 [ 3753.138509] [] ? smp_reschedule_interrupt+0x27/0x30 [ 3753.138509] [] ? reschedule_interrupt+0x2a/0x30 [ 3753.138509] [] core_sys_select+0x152/0x280 [ 3753.138509] [] ? native_sched_clock+0x25/0xc0 [ 3753.138509] [] ? sched_clock_local+0xe2/0x1e0 [ 3753.138509] [] ? native_sched_clock+0x25/0xc0 [ 3753.138509] [] ? sched_clock_local+0xe2/0x1e0 [ 3753.138509] [] ? ktime_get_ts+0xe7/0x110 [ 3753.138509] [] ? ktime_get_ts+0xe7/0x110 [ 3753.138509] [] sys_select+0x31/0xb0 [ 3753.138509] [] sysenter_do_call+0x12/0x22 [ 3753.138509] ---[ end trace 994a369c06c2c208 ]--- [ 3753.061448] ------------[ cut here ]------------ [ 3753.061448] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 3753.061448] Hardware name: ProLiant BL460c G1 [ 3753.061448] Watchdog detected hard LOCKUP on cpu 7 [ 3753.061448] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3753.061448] Pid: 8922, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #761 [ 3753.061448] Call Trace: [ 3753.061448] [] warn_slowpath_common+0x72/0xa0 [ 3753.061448] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.061448] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.061448] [] ? watchdog_timer_fn+0x170/0x170 [ 3753.061448] [] warn_slowpath_fmt+0x33/0x40 [ 3753.061448] [] watchdog_overflow_callback+0x9c/0xc0 [ 3753.061448] [] __perf_event_overflow+0x9a/0x240 [ 3753.061448] [] perf_event_overflow+0x15/0x20 [ 3753.061448] [] intel_pmu_handle_irq+0x129/0x250 [ 3753.061448] [] ? intel_pmu_handle_irq+0x16b/0x250 [ 3753.061448] [] ? cpumask_next_and+0x28/0x40 [ 3753.061448] [] perf_event_nmi_handler+0x20/0x30 [ 3753.061448] [] nmi_handle.clone.0+0x38/0x50 [ 3753.061448] [] do_nmi+0xae/0x350 [ 3753.061448] [] nmi_stack_correct+0x28/0x2d [ 3753.061448] [] ? _raw_spin_lock+0x32/0x40 [ 3753.061448] [] load_balance+0x361/0x560 [ 3753.061448] [] idle_balance+0xd2/0x150 [ 3753.061448] [] __schedule+0x5e9/0x6b0 [ 3753.061448] [] ? get_parent_ip+0xb/0x40 [ 3753.061448] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.061448] [] ? __probe_kernel_read+0x4e/0x90 [ 3753.061448] [] schedule+0x35/0x50 [ 3753.061448] [] schedule_timeout+0x21d/0x2a0 [ 3753.061448] [] ? __alloc_skb+0x31/0x110 [ 3753.061448] [] ? get_parent_ip+0xb/0x40 [ 3753.061448] [] ? get_parent_ip+0xb/0x40 [ 3753.061448] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.061448] [] ? _raw_spin_unlock_irqrestore+0x17/0x40 [ 3753.061448] [] ? prepare_to_wait+0x48/0x70 [ 3753.061448] [] sock_alloc_send_pskb+0x12c/0x290 [ 3753.061448] [] ? wake_up_bit+0x30/0x30 [ 3753.061448] [] sock_alloc_send_skb+0x1d/0x30 [ 3753.061448] [] unix_stream_sendmsg+0x26f/0x3b0 [ 3753.061448] [] sock_aio_write+0xf8/0x110 [ 3753.061448] [] do_sync_write+0xac/0xf0 [ 3753.061448] [] vfs_write+0x126/0x140 [ 3753.061448] [] ? fget_light+0x77/0xe0 [ 3753.061448] [] sys_write+0x42/0x70 [ 3753.061448] [] sysenter_do_call+0x12/0x22 [ 3753.061448] ---[ end trace 994a369c06c2c209 ]--- [ 3776.910004] BUG: soft lockup - CPU#3 stuck for 22s! [hackbench:9191] [ 3776.910040] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3776.910114] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3776.910187] [ 3776.910207] Pid: 9191, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #761 HP ProLiant BL460c G1 [ 3776.910262] EIP: 0060:[] EFLAGS: 00000202 CPU: 3 [ 3776.910290] EIP is at load_balance+0x30a/0x560 [ 3776.910316] EAX: 00000000 EBX: 00000000 ECX: 00000001 EDX: 00000100 [ 3776.910345] ESI: f4d9f060 EDI: 00000021 EBP: f4d5df6c ESP: f4d5df0c [ 3776.910373] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 3776.910401] Process hackbench (pid: 9191, ti=f4d5c000 task=f1a9a7e0 task.ti=ec502000) [ 3776.910446] Stack: [ 3776.910467] 00000001 f50c2c74 f4d5df90 00000202 f4d32348 f4d3231c f4d32300 f50c6ec0 [ 3776.910533] c0883ec0 00001133 c0883ec0 f50c2c74 f5086ec0 00001133 00000000 00000003 [ 3776.910598] 00000002 f5086f34 00001133 00001133 00001133 f4d32300 000544dc 00000007 [ 3776.910663] Call Trace: [ 3776.910687] [] rebalance_domains+0x89/0x160 [ 3776.910716] [] run_rebalance_domains+0x3d/0x120 [ 3776.910745] [] __do_softirq+0x8a/0x1e0 [ 3776.910772] [] ? local_bh_enable+0xa0/0xa0 [ 3776.910798] [ 3776.910822] [] ? irq_exit+0x8e/0xb0 [ 3776.910849] [] ? smp_apic_timer_interrupt+0x58/0x87 [ 3776.910880] [] ? kmem_cache_free+0x110/0x120 [ 3776.910907] [] ? __kfree_skb+0x3d/0x90 [ 3776.910934] [] ? apic_timer_interrupt+0x2a/0x30 [ 3776.910962] [] ? unix_stream_recvmsg+0x18f/0x590 [ 3776.910990] [] ? wake_up_bit+0x30/0x30 [ 3776.911017] [] ? sock_aio_read+0x107/0x110 [ 3776.911045] [] ? do_sync_read+0xac/0xf0 [ 3776.911073] [] ? vfs_read+0x11f/0x130 [ 3776.911099] [] ? fget_light+0x77/0xe0 [ 3776.911131] [] ? sys_read+0x42/0x70 [ 3776.911158] [] ? sysenter_do_call+0x12/0x22 [ 3776.911185] Code: d8 39 4d c4 76 08 85 c0 0f 85 eb fe ff ff 8b 45 bc 31 db 83 7d d8 00 0f 95 c3 e8 02 73 3b 00 8b 45 d0 e8 fa 72 3b 00 ff 75 ac 9d <85> db 0f 85 a2 00 00 00 f6 45 e0 04 0f 85 f7 00 00 00 f6 45 e0 [ 3776.911525] Call Trace: [ 3776.911548] [] rebalance_domains+0x89/0x160 [ 3776.911575] [] run_rebalance_domains+0x3d/0x120 [ 3776.911603] [] __do_softirq+0x8a/0x1e0 [ 3776.911630] [] ? local_bh_enable+0xa0/0xa0 [ 3776.911656] [] ? irq_exit+0x8e/0xb0 [ 3776.911686] [] ? smp_apic_timer_interrupt+0x58/0x87 [ 3776.911715] [] ? kmem_cache_free+0x110/0x120 [ 3776.911742] [] ? __kfree_skb+0x3d/0x90 [ 3776.911768] [] ? apic_timer_interrupt+0x2a/0x30 [ 3776.911796] [] ? unix_stream_recvmsg+0x18f/0x590 [ 3776.911825] [] ? wake_up_bit+0x30/0x30 [ 3776.911851] [] ? sock_aio_read+0x107/0x110 [ 3776.911879] [] ? do_sync_read+0xac/0xf0 [ 3776.911906] [] ? vfs_read+0x11f/0x130 [ 3776.911933] [] ? fget_light+0x77/0xe0 [ 3776.911959] [] ? sys_read+0x42/0x70 [ 3776.911985] [] ? sysenter_do_call+0x12/0x22 [ 3776.912011] Kernel panic - not syncing: softlockup: hung tasks [ 3753.128530] ------------[ cut here ]------------ [ 3753.128530] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 3753.128530] Hardware name: ProLiant BL460c G1 [ 3753.128530] Watchdog detected hard LOCKUP on cpu 1 [ 3753.128530] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3753.128530] Pid: 9073, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #761 [ 3753.128530] Call Trace: [ 3753.128530] [] warn_slowpath_common+0x72/0xa0 [ 3753.128530] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.128530] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.128530] [] ? watchdog_timer_fn+0x170/0x170 [ 3753.128530] [] warn_slowpath_fmt+0x33/0x40 [ 3753.128530] [] watchdog_overflow_callback+0x9c/0xc0 [ 3753.128530] [] __perf_event_overflow+0x9a/0x240 [ 3753.128530] [] perf_event_overflow+0x15/0x20 [ 3753.128530] [] intel_pmu_handle_irq+0x129/0x250 [ 3753.128530] [] ? intel_pmu_handle_irq+0x16b/0x250 [ 3753.128530] [] ? cpumask_next_and+0x28/0x40 [ 3753.128530] [] perf_event_nmi_handler+0x20/0x30 [ 3753.128530] [] nmi_handle.clone.0+0x38/0x50 [ 3753.128530] [] do_nmi+0xae/0x350 [ 3753.128530] [] nmi_stack_correct+0x28/0x2d [ 3753.128530] [] ? _raw_spin_lock+0x30/0x40 [ 3753.128530] [] load_balance+0x1a8/0x560 [ 3753.128530] [] idle_balance+0xd2/0x150 [ 3753.128530] [] __schedule+0x5e9/0x6b0 [ 3753.128530] [] ? __slab_free+0x1b9/0x350 [ 3753.128530] [] ? smp_apic_timer_interrupt+0x58/0x87 [ 3753.128530] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.128530] [] schedule+0x35/0x50 [ 3753.128530] [] schedule_timeout+0x21d/0x2a0 [ 3753.128530] [] ? kmem_cache_free+0x110/0x120 [ 3753.128530] [] ? __kfree_skb+0x3d/0x90 [ 3753.128530] [] ? __kfree_skb+0x3d/0x90 [ 3753.128530] [] ? get_parent_ip+0xb/0x40 [ 3753.128530] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.128530] [] unix_stream_recvmsg+0x38e/0x590 [ 3753.128530] [] ? wake_up_bit+0x30/0x30 [ 3753.128530] [] sock_aio_read+0x107/0x110 [ 3753.128530] [] do_sync_read+0xac/0xf0 [ 3753.128530] [] ? sched_clock_local+0xe2/0x1e0 [ 3753.128530] [] ? sched_clock_cpu+0x119/0x1b0 [ 3753.128530] [] vfs_read+0x11f/0x130 [ 3753.128530] [] ? fget_light+0x77/0xe0 [ 3753.128530] [] sys_read+0x42/0x70 [ 3753.128530] [] sysenter_do_call+0x12/0x22 [ 3753.128530] ---[ end trace 994a369c06c2c20a ]--- [ 3753.060025] ------------[ cut here ]------------ [ 3753.060025] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 3753.060025] Hardware name: ProLiant BL460c G1 [ 3753.060025] Watchdog detected hard LOCKUP on cpu 0 [ 3753.060025] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3753.060025] Pid: 9083, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #761 [ 3753.060025] Call Trace: [ 3753.060025] [] warn_slowpath_common+0x72/0xa0 [ 3753.060025] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.060025] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.060025] [] ? watchdog_timer_fn+0x170/0x170 [ 3753.060025] [] warn_slowpath_fmt+0x33/0x40 [ 3753.060025] [] watchdog_overflow_callback+0x9c/0xc0 [ 3753.060025] [] __perf_event_overflow+0x9a/0x240 [ 3753.060025] [] perf_event_overflow+0x15/0x20 [ 3753.060025] [] intel_pmu_handle_irq+0x129/0x250 [ 3753.060025] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.060025] [] perf_event_nmi_handler+0x20/0x30 [ 3753.060025] [] nmi_handle.clone.0+0x38/0x50 [ 3753.060025] [] do_nmi+0xae/0x350 [ 3753.060025] [] ? __alloc_skb+0x31/0x110 [ 3753.060025] [] nmi_stack_correct+0x28/0x2d [ 3753.060025] [] ? hrtimer_force_reprogram+0x4b/0xe0 [ 3753.060025] [] ? _raw_spin_lock_irqsave+0x30/0x40 [ 3753.060025] [] __wake_up_sync_key+0x24/0x60 [ 3753.060025] [] sock_def_readable+0x3d/0x70 [ 3753.060025] [] unix_stream_sendmsg+0x217/0x3b0 [ 3753.060025] [] sock_aio_write+0xf8/0x110 [ 3753.060025] [] do_sync_write+0xac/0xf0 [ 3753.060025] [] vfs_write+0x126/0x140 [ 3753.060025] [] ? fget_light+0x77/0xe0 [ 3753.060025] [] sys_write+0x42/0x70 [ 3753.060025] [] sysenter_do_call+0x12/0x22 [ 3753.060025] ---[ end trace 994a369c06c2c20b ]--- [ 3753.110224] ------------[ cut here ]------------ [ 3753.110224] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 3753.110224] Hardware name: ProLiant BL460c G1 [ 3753.110224] Watchdog detected hard LOCKUP on cpu 4 [ 3753.110224] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3753.110224] Pid: 9077, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #761 [ 3753.110224] Call Trace: [ 3753.110224] [] warn_slowpath_common+0x72/0xa0 [ 3753.110224] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.110224] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.110224] [] ? watchdog_timer_fn+0x170/0x170 [ 3753.110224] [] warn_slowpath_fmt+0x33/0x40 [ 3753.110224] [] watchdog_overflow_callback+0x9c/0xc0 [ 3753.110224] [] __perf_event_overflow+0x9a/0x240 [ 3753.110224] [] perf_event_overflow+0x15/0x20 [ 3753.110224] [] intel_pmu_handle_irq+0x129/0x250 [ 3753.110224] [] ? get_parent_ip+0xb/0x40 [ 3753.110224] [] ? get_partial_node.clone.38+0x231/0x290 [ 3753.110224] [] ? check_preempt_curr+0x72/0x90 [ 3753.110224] [] perf_event_nmi_handler+0x20/0x30 [ 3753.110224] [] nmi_handle.clone.0+0x38/0x50 [ 3753.110224] [] do_nmi+0xae/0x350 [ 3753.110224] [] ? get_parent_ip+0xb/0x40 [ 3753.110224] [] nmi_stack_correct+0x28/0x2d [ 3753.110224] [] ? sock_queue_err_skb+0x9b/0xb0 [ 3753.110224] [] ? try_to_wake_up+0x8f/0x220 [ 3753.110224] [] default_wake_function+0x10/0x20 [ 3753.110224] [] autoremove_wake_function+0x20/0x40 [ 3753.110224] [] __wake_up_common+0x48/0x70 [ 3753.110224] [] __wake_up_sync_key+0x43/0x60 [ 3753.110224] [] sock_def_readable+0x3d/0x70 [ 3753.110224] [] unix_stream_sendmsg+0x217/0x3b0 [ 3753.110224] [] sock_aio_write+0xf8/0x110 [ 3753.110224] [] do_sync_write+0xac/0xf0 [ 3753.110224] [] vfs_write+0x126/0x140 [ 3753.110224] [] ? fget_light+0x77/0xe0 [ 3753.110224] [] sys_write+0x42/0x70 [ 3753.110224] [] sysenter_do_call+0x12/0x22 [ 3753.110224] ---[ end trace 994a369c06c2c20c ]--- [ 3753.070006] ------------[ cut here ]------------ [ 3753.070006] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 3753.070006] Hardware name: ProLiant BL460c G1 [ 3753.070006] Watchdog detected hard LOCKUP on cpu 2 [ 3753.070006] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 3753.070006] Pid: 8897, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #761 [ 3753.070006] Call Trace: [ 3753.070006] [] warn_slowpath_common+0x72/0xa0 [ 3753.070006] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.070006] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 3753.070006] [] ? watchdog_timer_fn+0x170/0x170 [ 3753.070006] [] warn_slowpath_fmt+0x33/0x40 [ 3753.070006] [] watchdog_overflow_callback+0x9c/0xc0 [ 3753.070006] [] __perf_event_overflow+0x9a/0x240 [ 3753.070006] [] ? schedule_hrtimeout_range_clock+0x15d/0x180 [ 3753.070006] [] perf_event_overflow+0x15/0x20 [ 3753.070006] [] intel_pmu_handle_irq+0x129/0x250 [ 3753.070006] [] perf_event_nmi_handler+0x20/0x30 [ 3753.070006] [] nmi_handle.clone.0+0x38/0x50 [ 3753.070006] [] do_nmi+0xae/0x350 [ 3753.070006] [] ? sub_preempt_count+0x7b/0xb0 [ 3753.070006] [] ? _raw_spin_unlock+0x15/0x40 [ 3753.070006] [] nmi_stack_correct+0x28/0x2d [ 3753.070006] [] ? try_to_wake_up+0x8d/0x220 [ 3753.070006] [] ? __free_pages+0x1f/0x40 [ 3753.070006] [] default_wake_function+0x10/0x20 [ 3753.070006] [] autoremove_wake_function+0x20/0x40 [ 3753.070006] [] __wake_up_common+0x48/0x70 [ 3753.070006] [] __wake_up_sync_key+0x43/0x60 [ 3753.070006] [] unix_write_space+0x4e/0x90 [ 3753.070006] [] sock_wfree+0x48/0x50 [ 3753.070006] [] unix_destruct_scm+0x80/0xa0 [ 3753.070006] [] skb_release_head_state+0x4d/0xe0 [ 3753.070006] [] __kfree_skb+0x10/0x90 [ 3753.070006] [] consume_skb+0x27/0x90 [ 3753.070006] [] unix_stream_recvmsg+0x1f3/0x590 [ 3753.070006] [] ? wake_up_bit+0x30/0x30 [ 3753.070006] [] sock_aio_read+0x107/0x110 [ 3753.070006] [] do_sync_read+0xac/0xf0 [ 3753.070006] [] vfs_read+0x11f/0x130 [ 3753.070006] [] ? fget_light+0x77/0xe0 [ 3753.070006] [] sys_read+0x42/0x70 [ 3753.070006] [] sysenter_do_call+0x12/0x22 [ 3753.070006] ---[ end trace 994a369c06c2c20d ]--- 2nd one (CONFIG_PREEMPT_NONE=y) ------------------------------- [ 241.920003] ------------[ cut here ]------------ [ 241.920003] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.920003] Hardware name: ProLiant BL460c G1 [ 241.920003] Watchdog detected hard LOCKUP on cpu 0 [ 241.920003] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.920003] Pid: 8977, comm: hackbench Not tainted 3.2.0-05066-g31ae26a-dirty #762 [ 241.920003] Call Trace: [ 241.920003] [] warn_slowpath_common+0x72/0xa0 [ 241.920003] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.920003] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.920003] [] ? watchdog_timer_fn+0x170/0x170 [ 241.920003] [] warn_slowpath_fmt+0x33/0x40 [ 241.920003] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.920003] [] __perf_event_overflow+0x9a/0x240 [ 241.920003] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.920003] [] perf_event_overflow+0x15/0x20 [ 241.920003] [] intel_pmu_handle_irq+0x129/0x250 [ 241.920003] [] ? find_busiest_group+0x150/0xae0 [ 241.920003] [] perf_event_nmi_handler+0x20/0x30 [ 241.920003] [] do_nmi+0xb7/0x350 [ 241.920003] [] nmi_stack_correct+0x28/0x2d [ 241.920003] [] ? _raw_spin_lock+0x24/0x30 [ 241.920003] [] load_balance+0x1b4/0x500 [ 241.920003] [] idle_balance+0xc2/0x140 [ 241.920003] [] __schedule+0x4be/0x650 [ 241.920003] [] ? __slab_free+0x155/0x270 [ 241.920003] [] schedule+0x35/0x50 [ 241.920003] [] schedule_timeout+0x1dd/0x250 [ 241.920003] [] ? unix_destruct_scm+0x80/0xa0 [ 241.920003] [] ? __kfree_skb+0x3d/0x90 [ 241.920003] [] ? __kfree_skb+0x3d/0x90 [ 241.920003] [] ? prepare_to_wait+0x48/0x70 [ 241.920003] [] unix_stream_recvmsg+0x3ed/0x570 [ 241.920003] [] ? wake_up_bit+0x30/0x30 [ 241.920003] [] sock_aio_read+0x107/0x110 [ 241.920003] [] do_sync_read+0xac/0xf0 [ 241.920003] [] ? irq_exit+0x67/0xa0 [ 241.920003] [] ? smp_apic_timer_interrupt+0x59/0x88 [ 241.920003] [] ? sched_clock_cpu+0x109/0x1a0 [ 241.920003] [] vfs_read+0x11f/0x130 [ 241.920003] [] sys_read+0x42/0x70 [ 241.920003] [] sysenter_do_call+0x12/0x22 [ 241.920003] ---[ end trace 066d964ca98ab480 ]--- [ 241.835214] ------------[ cut here ]------------ [ 241.835214] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.835214] Hardware name: ProLiant BL460c G1 [ 241.835214] Watchdog detected hard LOCKUP on cpu 3 [ 241.835214] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.835214] Pid: 8836, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.835214] Call Trace: [ 241.835214] [] warn_slowpath_common+0x72/0xa0 [ 241.835214] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.835214] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.835214] [] ? watchdog_timer_fn+0x170/0x170 [ 241.835214] [] warn_slowpath_fmt+0x33/0x40 [ 241.835214] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.835214] [] __perf_event_overflow+0x9a/0x240 [ 241.835214] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.835214] [] perf_event_overflow+0x15/0x20 [ 241.835214] [] intel_pmu_handle_irq+0x129/0x250 [ 241.835214] [] ? sched_clock_local+0xe2/0x1e0 [ 241.835214] [] ? sched_clock_cpu+0x64/0x1a0 [ 241.835214] [] perf_event_nmi_handler+0x20/0x30 [ 241.835214] [] do_nmi+0xb7/0x350 [ 241.835214] [] nmi_stack_correct+0x28/0x2d [ 241.835214] [] ? try_to_wake_up+0xa2/0x220 [ 241.835214] [] default_wake_function+0x10/0x20 [ 241.835214] [] autoremove_wake_function+0x20/0x40 [ 241.835214] [] __wake_up_common+0x48/0x70 [ 241.835214] [] __wake_up_sync_key+0x43/0x60 [ 241.835214] [] sock_def_readable+0x38/0x60 [ 241.835214] [] unix_stream_sendmsg+0x215/0x3a0 [ 241.835214] [] sock_aio_write+0xf8/0x110 [ 241.835214] [] do_sync_write+0xac/0xf0 [ 241.835214] [] vfs_write+0x126/0x140 [ 241.835214] [] sys_write+0x42/0x70 [ 241.835214] [] sysenter_do_call+0x12/0x22 [ 241.835214] ---[ end trace 066d964ca98ab481 ]--- [ 241.344106] ------------[ cut here ]------------ [ 241.344106] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.344106] Hardware name: ProLiant BL460c G1 [ 241.344106] Watchdog detected hard LOCKUP on cpu 1 [ 241.344106] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.344106] Pid: 8622, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.344106] Call Trace: [ 241.344106] [] warn_slowpath_common+0x72/0xa0 [ 241.344106] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.344106] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.344106] [] ? watchdog_timer_fn+0x170/0x170 [ 241.344106] [] warn_slowpath_fmt+0x33/0x40 [ 241.344106] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.344106] [] __perf_event_overflow+0x9a/0x240 [ 241.344106] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.344106] [] perf_event_overflow+0x15/0x20 [ 241.344106] [] intel_pmu_handle_irq+0x129/0x250 [ 241.344106] [] ? unix_seqpacket_recvmsg+0x50/0x50 [ 241.344106] [] perf_event_nmi_handler+0x20/0x30 [ 241.344106] [] do_nmi+0xb7/0x350 [ 241.344106] [] nmi_stack_correct+0x28/0x2d [ 241.344106] [] ? hidinput_connect+0xa8b/0x2720 [ 241.344106] [] ? _raw_spin_lock_irqsave+0x22/0x30 [ 241.344106] [] __wake_up_sync_key+0x24/0x60 [ 241.344106] [] ? _copy_from_user+0x37/0x60 [ 241.344106] [] sock_def_readable+0x38/0x60 [ 241.344106] [] unix_stream_sendmsg+0x215/0x3a0 [ 241.344106] [] sock_aio_write+0xf8/0x110 [ 241.344106] [] do_sync_write+0xac/0xf0 [ 241.344106] [] vfs_write+0x126/0x140 [ 241.344106] [] sys_write+0x42/0x70 [ 241.344106] [] sysenter_do_call+0x12/0x22 [ 241.344106] ---[ end trace 066d964ca98ab482 ]--- [ 241.419616] ------------[ cut here ]------------ [ 241.419616] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.419616] Hardware name: ProLiant BL460c G1 [ 241.419616] Watchdog detected hard LOCKUP on cpu 4 [ 241.419616] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.419616] Pid: 8637, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.419616] Call Trace: [ 241.419616] [] warn_slowpath_common+0x72/0xa0 [ 241.419616] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.419616] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.419616] [] ? watchdog_timer_fn+0x170/0x170 [ 241.419616] [] warn_slowpath_fmt+0x33/0x40 [ 241.419616] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.419616] [] __perf_event_overflow+0x9a/0x240 [ 241.419616] [] ? poll_freewait+0xb0/0xb0 [ 241.419616] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.419616] [] perf_event_overflow+0x15/0x20 [ 241.419616] [] intel_pmu_handle_irq+0x129/0x250 [ 241.419616] [] perf_event_nmi_handler+0x20/0x30 [ 241.419616] [] do_nmi+0xb7/0x350 [ 241.419616] [] nmi_stack_correct+0x28/0x2d [ 241.419616] [] ? try_to_wake_up+0xa2/0x220 [ 241.419616] [] default_wake_function+0x10/0x20 [ 241.419616] [] autoremove_wake_function+0x20/0x40 [ 241.419616] [] __wake_up_common+0x48/0x70 [ 241.419616] [] __wake_up_sync_key+0x43/0x60 [ 241.419616] [] sock_def_readable+0x38/0x60 [ 241.419616] [] unix_stream_sendmsg+0x215/0x3a0 [ 241.419616] [] sock_aio_write+0xf8/0x110 [ 241.419616] [] do_sync_write+0xac/0xf0 [ 241.419616] [] vfs_write+0x126/0x140 [ 241.419616] [] sys_write+0x42/0x70 [ 241.419616] [] sysenter_do_call+0x12/0x22 [ 241.419616] ---[ end trace 066d964ca98ab483 ]--- [ 241.897579] ------------[ cut here ]------------ [ 241.897579] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.897579] Hardware name: ProLiant BL460c G1 [ 241.897579] Watchdog detected hard LOCKUP on cpu 7 [ 241.897579] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.897579] Pid: 8814, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.897579] Call Trace: [ 241.897579] [] warn_slowpath_common+0x72/0xa0 [ 241.897579] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.897579] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.897579] [] ? watchdog_timer_fn+0x170/0x170 [ 241.897579] [] warn_slowpath_fmt+0x33/0x40 [ 241.897579] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.897579] [] __perf_event_overflow+0x9a/0x240 [ 241.897579] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.897579] [] perf_event_overflow+0x15/0x20 [ 241.897579] [] intel_pmu_handle_irq+0x129/0x250 [ 241.897579] [] ? intel_pmu_handle_irq+0x16b/0x250 [ 241.897579] [] ? find_busiest_group+0x150/0xae0 [ 241.897579] [] perf_event_nmi_handler+0x20/0x30 [ 241.897579] [] do_nmi+0xb7/0x350 [ 241.897579] [] nmi_stack_correct+0x28/0x2d [ 241.897579] [] ? _raw_spin_lock+0x24/0x30 [ 241.897579] [] load_balance+0x358/0x500 [ 241.897579] [] idle_balance+0xc2/0x140 [ 241.897579] [] __schedule+0x4be/0x650 [ 241.897579] [] ? __slab_free+0x155/0x270 [ 241.897579] [] ? __kmalloc_node_track_caller+0xf6/0x1b0 [ 241.897579] [] schedule+0x35/0x50 [ 241.897579] [] schedule_timeout+0x1dd/0x250 [ 241.897579] [] ? unix_destruct_scm+0x80/0xa0 [ 241.897579] [] ? __kfree_skb+0x3d/0x90 [ 241.897579] [] ? __kfree_skb+0x3d/0x90 [ 241.897579] [] ? prepare_to_wait+0x48/0x70 [ 241.897579] [] unix_stream_recvmsg+0x3ed/0x570 [ 241.897579] [] ? wake_up_bit+0x30/0x30 [ 241.897579] [] sock_aio_read+0x107/0x110 [ 241.897579] [] do_sync_read+0xac/0xf0 [ 241.897579] [] ? call_rcu_sched+0x12/0x20 [ 241.897579] [] vfs_read+0x11f/0x130 [ 241.897579] [] sys_read+0x42/0x70 [ 241.897579] [] sysenter_do_call+0x12/0x22 [ 241.897579] ---[ end trace 066d964ca98ab484 ]--- [ 241.736351] ------------[ cut here ]------------ [ 241.736351] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.736351] Hardware name: ProLiant BL460c G1 [ 241.736351] Watchdog detected hard LOCKUP on cpu 2 [ 241.736351] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.736351] Pid: 8725, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.736351] Call Trace: [ 241.736351] [] warn_slowpath_common+0x72/0xa0 [ 241.736351] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.736351] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.736351] [] ? watchdog_timer_fn+0x170/0x170 [ 241.736351] [] warn_slowpath_fmt+0x33/0x40 [ 241.736351] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.736351] [] __perf_event_overflow+0x9a/0x240 [ 241.736351] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.736351] [] perf_event_overflow+0x15/0x20 [ 241.736351] [] intel_pmu_handle_irq+0x129/0x250 [ 241.736351] [] ? find_busiest_group+0x150/0xae0 [ 241.736351] [] perf_event_nmi_handler+0x20/0x30 [ 241.736351] [] do_nmi+0xb7/0x350 [ 241.736351] [] nmi_stack_correct+0x28/0x2d [ 241.736351] [] ? _raw_spin_lock+0x24/0x30 [ 241.736351] [] load_balance+0x358/0x500 [ 241.736351] [] idle_balance+0xc2/0x140 [ 241.736351] [] __schedule+0x4be/0x650 [ 241.736351] [] ? try_to_wake_up+0x93/0x220 [ 241.736351] [] ? __slab_free+0x155/0x270 [ 241.736351] [] schedule+0x35/0x50 [ 241.736351] [] schedule_timeout+0x1dd/0x250 [ 241.736351] [] ? unix_destruct_scm+0x80/0xa0 [ 241.736351] [] ? kmem_cache_free+0xe8/0x100 [ 241.736351] [] ? __kfree_skb+0x3d/0x90 [ 241.736351] [] ? __kfree_skb+0x3d/0x90 [ 241.736351] [] ? __kfree_skb+0x3d/0x90 [ 241.736351] [] ? prepare_to_wait+0x48/0x70 [ 241.736351] [] unix_stream_recvmsg+0x3ed/0x570 [ 241.736351] [] ? wake_up_bit+0x30/0x30 [ 241.736351] [] sock_aio_read+0x107/0x110 [ 241.736351] [] do_sync_read+0xac/0xf0 [ 241.736351] [] ? rb_insert_color+0xcc/0x110 [ 241.736351] [] ? irq_exit+0x67/0xa0 [ 241.736351] [] ? smp_apic_timer_interrupt+0x59/0x88 [ 241.736351] [] ? rw_verify_area+0x4a/0x100 [ 241.736351] [] vfs_read+0x11f/0x130 [ 241.736351] [] sys_read+0x42/0x70 [ 241.736351] [] sysenter_do_call+0x12/0x22 [ 241.736351] ---[ end trace 066d964ca98ab485 ]--- [ 241.344282] ------------[ cut here ]------------ [ 241.344282] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.344282] Hardware name: ProLiant BL460c G1 [ 241.344282] Watchdog detected hard LOCKUP on cpu 5 [ 241.344282] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.344282] Pid: 8600, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.344282] Call Trace: [ 241.344282] [] warn_slowpath_common+0x72/0xa0 [ 241.344282] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.344282] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.344282] [] ? watchdog_timer_fn+0x170/0x170 [ 241.344282] [] warn_slowpath_fmt+0x33/0x40 [ 241.344282] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.344282] [] __perf_event_overflow+0x9a/0x240 [ 241.344282] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.344282] [] perf_event_overflow+0x15/0x20 [ 241.344282] [] intel_pmu_handle_irq+0x129/0x250 [ 241.344282] [] ? intel_pmu_handle_irq+0x16b/0x250 [ 241.344282] [] ? find_busiest_group+0x150/0xae0 [ 241.344282] [] perf_event_nmi_handler+0x20/0x30 [ 241.344282] [] do_nmi+0xb7/0x350 [ 241.344282] [] nmi_stack_correct+0x28/0x2d [ 241.344282] [] ? _raw_spin_lock+0x24/0x30 [ 241.344282] [] load_balance+0x358/0x500 [ 241.344282] [] idle_balance+0xc2/0x140 [ 241.344282] [] __schedule+0x4be/0x650 [ 241.344282] [] ? find_busiest_group+0x150/0xae0 [ 241.344282] [] ? __slab_free+0x155/0x270 [ 241.344282] [] ? __kmalloc_node_track_caller+0xf6/0x1b0 [ 241.344282] [] schedule+0x35/0x50 [ 241.344282] [] schedule_timeout+0x1dd/0x250 [ 241.344282] [] ? unix_destruct_scm+0x80/0xa0 [ 241.344282] [] ? kmem_cache_free+0xe8/0x100 [ 241.344282] [] ? __kfree_skb+0x3d/0x90 [ 241.344282] [] ? __kfree_skb+0x3d/0x90 [ 241.344282] [] ? __kfree_skb+0x3d/0x90 [ 241.344282] [] ? prepare_to_wait+0x48/0x70 [ 241.344282] [] unix_stream_recvmsg+0x3ed/0x570 [ 241.344282] [] ? wake_up_bit+0x30/0x30 [ 241.344282] [] sock_aio_read+0x107/0x110 [ 241.344282] [] do_sync_read+0xac/0xf0 [ 241.344282] [] ? do_page_fault+0x16b/0x3d0 [ 241.344282] [] vfs_read+0x11f/0x130 [ 241.344282] [] sys_read+0x42/0x70 [ 241.344282] [] sysenter_do_call+0x12/0x22 [ 241.344282] ---[ end trace 066d964ca98ab486 ]--- [ 241.750004] ------------[ cut here ]------------ [ 241.750004] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9c/0xc0() [ 241.750004] Hardware name: ProLiant BL460c G1 [ 241.750004] Watchdog detected hard LOCKUP on cpu 6 [ 241.750004] Modules linked in: netconsole xt_hashlimit ipmi_devintf ipmi_si ipmi_msghandler af_packet tg3 bonding [ 241.750004] Pid: 8745, comm: hackbench Tainted: G W 3.2.0-05066-g31ae26a-dirty #762 [ 241.750004] Call Trace: [ 241.750004] [] warn_slowpath_common+0x72/0xa0 [ 241.750004] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.750004] [] ? watchdog_overflow_callback+0x9c/0xc0 [ 241.750004] [] ? watchdog_timer_fn+0x170/0x170 [ 241.750004] [] warn_slowpath_fmt+0x33/0x40 [ 241.750004] [] watchdog_overflow_callback+0x9c/0xc0 [ 241.750004] [] __perf_event_overflow+0x9a/0x240 [ 241.750004] [] ? x86_perf_event_set_period+0x118/0x210 [ 241.750004] [] perf_event_overflow+0x15/0x20 [ 241.750004] [] intel_pmu_handle_irq+0x129/0x250 [ 241.750004] [] ? sched_clock_local+0xe2/0x1e0 [ 241.750004] [] ? sched_clock_cpu+0x109/0x1a0 [ 241.750004] [] perf_event_nmi_handler+0x20/0x30 [ 241.750004] [] do_nmi+0xb7/0x350 [ 241.750004] [] nmi_stack_correct+0x28/0x2d [ 241.750004] [] ? try_to_wake_up+0xa2/0x220 [ 241.750004] [] default_wake_function+0x10/0x20 [ 241.750004] [] autoremove_wake_function+0x20/0x40 [ 241.750004] [] __wake_up_common+0x48/0x70 [ 241.750004] [] __wake_up_sync_key+0x43/0x60 [ 241.750004] [] sock_def_readable+0x38/0x60 [ 241.750004] [] unix_stream_sendmsg+0x215/0x3a0 [ 241.750004] [] sock_aio_write+0xf8/0x110 [ 241.750004] [] do_sync_write+0xac/0xf0 [ 241.750004] [] ? rb_erase+0x16a/0x290 [ 241.750004] [] ? irq_exit+0x67/0xa0 [ 241.750004] [] ? smp_apic_timer_interrupt+0x59/0x88 [ 241.750004] [] vfs_write+0x126/0x140 [ 241.750004] [] sys_write+0x42/0x70 [ 241.750004] [] sysenter_do_call+0x12/0x22 [ 241.750004] ---[ end trace 066d964ca98ab487 ]---