From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932946Ab2GYM0T (ORCPT ); Wed, 25 Jul 2012 08:26:19 -0400 Received: from e23smtp08.au.ibm.com ([202.81.31.141]:47041 "EHLO e23smtp08.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932089Ab2GYM0R (ORCPT ); Wed, 25 Jul 2012 08:26:17 -0400 Message-ID: <500FE5C9.4040407@linux.vnet.ibm.com> Date: Wed, 25 Jul 2012 17:55:45 +0530 From: "Srivatsa S. Bhat" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120605 Thunderbird/13.0 MIME-Version: 1.0 To: Thomas Gleixner CC: LKML , Ingo Molnar , Peter Zijlstra , Rusty Russell , "Paul E. McKenney" , Namhyung Kim , Tejun Heo , Nikunj A Dadhania Subject: Re: [Patch 0/7] Per cpu thread hotplug infrastructure - V3 References: <20120716103749.122800930@linutronix.de> In-Reply-To: <20120716103749.122800930@linutronix.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit x-cbid: 12072512-5140-0000-0000-000001D0471B Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/16/2012 04:12 PM, Thomas Gleixner wrote: > The following series implements the infrastructure for parking and > unparking kernel threads to avoid the full teardown and fork on cpu > hotplug operations along with management infrastructure for hotplug > and users. > > Changes vs. V2: > > Use callbacks for all functionality. Thanks to Rusty for pointing > that out. It makes the use sites nice and simple and keeps all the > code which would be duplicated otherwise on the core. > Hi Thomas, I am observing stalls detected by RCU and the subsequent NMI backtraces while running CPU hotplug stress tests on this patchset. Will check whether I hit this without your patches too... (But I think the baseline is a little bit old.. 3.5-rc1..) ---------------------------- Booting Node 1 Processor 7 APIC 0x16 NMI watchdog: enabled, takes one hw-pmu counter. lockdep: fixing up alternatives. Booting Node 0 Processor 8 APIC 0x1 NMI watchdog: enabled, takes one hw-pmu counter. INFO: task stress_cpu_hotp:6903 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. stress_cpu_hotp D 0000000000000000 0 6903 6850 0x00000000 ffff8808cda09ad8 0000000000000046 0000000000000001 ffff8808cda08010 ffff8808cda08000 ffff8808cda08000 ffff8808cda08000 ffff8808cda09fd8 ffff8808cda08000 ffff8808cda09fd8 ffff8808d04f45c0 ffff8808d2d8a200 Call Trace: [] ? cpufreq_notify_transition+0x174/0x2f0 [] ? acpi_cpufreq_cpu_exit+0xb0/0xb0 [acpi_cpufreq] [] ? mark_held_locks+0x80/0x120 [] ? __mutex_unlock_slowpath+0xf7/0x190 [] schedule+0x29/0x70 [] schedule_timeout+0x2d5/0x3f0 [] ? mark_held_locks+0x80/0x120 [] ? _raw_spin_unlock_irq+0x30/0x50 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] wait_for_common+0x115/0x190 [] ? try_to_wake_up+0x2f0/0x2f0 [] ? queue_delayed_work_on+0x132/0x140 [] wait_for_completion+0x1d/0x20 [] hpet_cpuhp_notify+0x119/0x160 [] ? init_one_hpet_msi_clockevent+0x1c0/0x1c0 [] ? __queue_work+0x500/0x500 [] notifier_call_chain+0x72/0x130 [] __raw_notifier_call_chain+0xe/0x10 [] _cpu_up+0x15b/0x164 [] cpu_up+0xbc/0x11f [] store_online+0x9d/0xe0 [] dev_attr_store+0x20/0x30 [] sysfs_write_file+0xd9/0x150 [] vfs_write+0xcb/0x130 [] sys_write+0x55/0x90 [] system_call_fastpath+0x16/0x1b 5 locks held by stress_cpu_hotp/6903: #0: (&buffer->mutex){+.+.+.}, at: [] sysfs_write_file+0x49/0x150 #1: (s_active#111){.+.+.+}, at: [] sysfs_write_file+0xc2/0x150 #2: (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [] cpu_hotplug_driver_lock+0x17/0x20 #3: (cpu_add_remove_lock){+.+.+.}, at: [] cpu_up+0x78/0x11f #4: (cpu_hotplug.lock){+.+.+.}, at: [] cpu_hotplug_begin+0x2f/0x60 lockdep: fixing up alternatives. Booting Node 0 Processor 9 APIC 0x3 NMI watchdog: enabled, takes one hw-pmu counter. lockdep: fixing up alternatives. Booting Node 0 Processor 10 APIC 0x5 CPU 2 is now offline CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 3 is now offline CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 4 is now offline CPU 5 MCA banks CMCI:6 CMCI:8 CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 5 is now offline CPU 6 MCA banks CMCI:6 CMCI:8 CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 6 is now offline CPU 7 MCA banks CMCI:6 CMCI:8 CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 7 is now offline CPU 12 MCA banks CMCI:6 CMCI:8 CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 8 is now offline ------------[ cut here ]------------ WARNING: at kernel/rcutree.c:1545 rcu_do_batch+0x35d/0x370() Hardware name: IBM System x -[7870C4Q]- Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 11788, comm: migration/9 Not tainted 3.5.0-rc1-threadhotplugv3-paulfix #1 Call Trace: [] ? rcu_do_batch+0x35d/0x370 [] ? rcu_do_batch+0x35d/0x370 [] warn_slowpath_common+0x7f/0xc0 [] warn_slowpath_null+0x1a/0x20 [] rcu_do_batch+0x35d/0x370 [] ? trace_hardirqs_on_caller+0x70/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] __rcu_process_callbacks+0x193/0x1f0 [] rcu_process_callbacks+0x8d/0x260 [] __do_softirq+0x151/0x3e0 [] call_softirq+0x1c/0x30 [] do_softirq+0x9d/0xd0 [] irq_exit+0xe5/0x100 [] smp_apic_timer_interrupt+0x6e/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? stop_machine_cpu_stop+0xe2/0x140 [] ? stop_one_cpu_nowait+0x50/0x50 [] cpu_stopper_thread+0xf3/0x1e0 [] ? _raw_spin_unlock_irqrestore+0x44/0x90 [] ? res_counter_init+0x60/0x60 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] ? res_counter_init+0x60/0x60 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 ---[ end trace e18e7d52508d4205 ]--- CPU 9 is now offline CPU 10 is now offline Booting Node 1 Processor 15 APIC 0x17 NMI watchdog: enabled, takes one hw-pmu counter. CPU 1 is now offline CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 2 is now offline CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 3 is now offline CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5 NOHZ: local_softirq_pending 100 CPU 4 is now offline CPU 5 MCA banks CMCI:6 CMCI:8 CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5 Broke affinity for irq 81 CPU 5 is now offline CPU 6 MCA banks CMCI:6 CMCI:8 CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 6 is now offline CPU 7 MCA banks CMCI:6 CMCI:8 CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5 NOHZ: local_softirq_pending 200 CPU 7 is now offline CPU 12 MCA banks CMCI:6 CMCI:8 CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5 CPU 8 is now offline CPU 9 is now offline CPU 10 is now offline CPU 11 is now offline CPU 12 is now offline CPU 13 MCA banks CMCI:6 CMCI:8 CPU 13 is now offline CPU 14 MCA banks CMCI:6 CMCI:8 CPU 14 is now offline CPU 15 MCA banks CMCI:6 CMCI:8 CPU 15 is now offline Booting Node 0 Processor 10 APIC 0x5 NMI watchdog: enabled, takes one hw-pmu counter. lockdep: fixing up alternatives. Booting Node 0 Processor 11 APIC 0x7 NMI watchdog: enabled, takes one hw-pmu counter. INFO: rcu_sched self-detected stall on CPU { 3} (t=91070 jiffies) INFO: rcu_sched self-detected stall on CPU { 4} (t=91070 jiffies) sending NMI to all CPUs: NMI backtrace for cpu 0 CPU 0 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas INFO: rcu_sched self-detected stall on CPU Pid: 0, comm: swapper/0 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]- { /68Y8033 5 RIP: 0010:[] [] default_send_IPI_mask_sequence_phys+0xa1/0x130 RSP: 0018:ffff8808ffc03d98 EFLAGS: 00000002 RAX: 00000000000010ef RBX: 0000000000000006 RCX: 0000000000000010 } (t=91070 jiffies) RDX: 0000000000000fc0 RSI: 0000000000000006 RDI: 0000000000000000 RBP: ffff8808ffc03dd8 R08: 0000000000000010 R09: ffffffff81da9d60 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014 R13: 00000000000000ef R14: 000000000000dd40 R15: 00000000000000ef FS: 0000000000000000(0000) GS:ffff8808ffc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/0 (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a14420) Stack: 0000000000000000 ffffffff81da9d60 0000000000000092 000000000000ea20 ffffffff81da9d60 000001732e239870 ffffffff81a1ef00 7fffffffffffffff ffff8808ffc03de8 ffffffff8103e80e ffff8808ffc03df8 ffffffff8103a43b Call Trace: [] physflat_send_IPI_mask+0xe/0x10 [] lapic_timer_broadcast+0x1b/0x20 [] tick_do_broadcast+0x66/0xa0 [] tick_handle_oneshot_broadcast+0xf4/0x130 [] timer_interrupt+0x15/0x20 [] handle_irq_event_percpu+0xcc/0x3e0 [] handle_irq_event+0x48/0x70 [] handle_edge_irq+0x65/0x130 [] handle_irq+0x5e/0x150 [] ? atomic_notifier_call_chain+0x16/0x20 [] do_IRQ+0x60/0xe0 [] common_interrupt+0x6f/0x6f [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] ? schedule+0x29/0x70 [] rest_init+0xff/0x180 [] ? csum_partial_copy_generic+0x170/0x170 [] start_kernel+0x3be/0x48c [] ? kernel_init+0x290/0x290 [] ? memblock_reserve+0x2e/0x4e [] x86_64_start_reservations+0xa5/0xc2 [] x86_64_start_kernel+0x10f/0x12a [] ? early_idt_handlers+0x120/0x120 Code: a0 e3 ae 81 41 83 fd 02 45 0f b7 24 06 74 4b 8b 04 25 00 a3 5f ff f6 c4 10 74 1b 0f 1f 80 00 00 00 00 f3 90 8b 04 25 00 a3 5f ff c4 10 75 f2 8b 0d 5c ea ab 00 44 89 e0 c1 e0 18 89 04 25 10 NMI backtrace for cpu 1 CPU 1 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/1 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] native_read_tsc+0xb/0x20 RSP: 0018:ffff8808d2d8dc70 EFLAGS: 00000002 RAX: 00000000000004d1 RBX: 0000000000000001 RCX: 00000000f9c4c38a RDX: 00000000000004d1 RSI: 01a8000000000000 RDI: 0000000000000001 RBP: ffff8808d2d8dc70 R08: 00000000f9c4c356 R09: ffff8808d2d8aab8 R10: ffff8808d2d8aab8 R11: 0000000000000000 R12: 00000000000011dc R13: 00000000aeda8ed8 R14: ffff8808d2d8c000 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff8808ffc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/1 (pid: 0, threadinfo ffff8808d2d8c000, task ffff8808d2d8a200) Stack: ffff8808d2d8dcb8 ffffffff812aff41 ffff8808d2d8c000 00000001f9c4c32e ffffffff81a34580 00000000000011dc 00000000aeda8ed8 0000000000000001 ffff8808d2d8de64 ffff8808d2d8dcc8 ffffffff812affef ffff8808d2d8dcf8 Call Trace: [] delay_tsc+0x41/0xc0 [] __delay+0xf/0x20 [] do_raw_spin_lock+0xab/0x150 [] _raw_spin_lock_irqsave+0x61/0x70 [] ? tick_broadcast_oneshot_control+0x83/0x130 [] tick_broadcast_oneshot_control+0x83/0x130 [] tick_notify+0xf8/0x200 [] notifier_call_chain+0x72/0x130 [] raw_notifier_call_chain+0x16/0x20 [] clockevents_notify+0x3d/0x150 [] intel_idle+0xe2/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 9f 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 48 89 d0 <48> c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 NMI backtrace for cpu 2 CPU 2 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/2 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] delay_tsc+0x3c/0xc0 RSP: 0018:ffff8808d2db1d40 EFLAGS: 00000006 RAX: 0000000000005f7d RBX: 0000000000000001 RCX: 000000005f7d5f7c RDX: ffff8808d2db1fd8 RSI: 0018000000000000 RDI: 0000000000000001 RBP: ffff8808d2db1d78 R08: 00000000f9c93f00 R09: ffff8808d2daead0 R10: ffff8808d2daead0 R11: 0000000000000001 R12: 000000000000117f R13: 00000000aeda8ed8 R14: ffff8808d2db0000 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffff8808ffc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/2 (pid: 0, threadinfo ffff8808d2db0000, task ffff8808d2dae240) Stack: ffff8808d2db0000 00000002f9c93ecc ffffffff81a344c0 000000000000117f 00000000aeda8ed8 0000000000000001 ffff8808ffc98970 ffff8808d2db1d88 ffffffff812affef ffff8808d2db1db8 ffffffff812b769b ffffffff81a344c0 Call Trace: [] __delay+0xf/0x20 [] do_raw_spin_lock+0xab/0x150 [] _raw_spin_lock_irqsave+0x61/0x70 [] ? clockevents_notify+0x28/0x150 [] clockevents_notify+0x28/0x150 [] intel_idle+0xe2/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 66 66 66 66 90 65 48 8b 14 25 c8 c7 00 00 4c 8d b2 28 e0 ff ff 89 fb 41 83 46 1c 01 65 44 8b 3c 25 08 dd 00 00 66 66 90 0f ae e8 2f fd d6 ff 66 90 89 45 d0 44 89 7d d4 4c 89 75 c8 eb 1d 48 NMI backtrace for cpu 3 CPU 3 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/3 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] io_serial_in+0x18/0x20 RSP: 0018:ffff8808ffcc3be8 EFLAGS: 00000002 RAX: ffff8808d2db7f00 RBX: ffffffff826876b0 RCX: 0000000000000000 RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff826876b0 RBP: ffff8808ffcc3be8 R08: 00000000f9cdb3aa R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000f32 R12: 000000000000270b R13: 0000000000000020 R14: 0000000000000043 R15: ffffffff826876b0 FS: 0000000000000000(0000) GS:ffff8808ffcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/3 (pid: 0, threadinfo ffff8808d2db6000, task ffff8808d2db4280) Stack: ffff8808ffcc3c18 ffffffff8137c96b 000000000000ffff ffffffff826876b0 0000000000000020 ffffffff81d1f740 ffff8808ffcc3c38 ffffffff8137c9f6 0000000000000006 ffffffff8137c9d0 ffff8808ffcc3c78 ffffffff81377fbd Call Trace: [] wait_for_xmitr+0x3b/0xa0 [] serial8250_console_putchar+0x26/0x40 [] ? wait_for_xmitr+0xa0/0xa0 [] uart_console_write+0x3d/0x70 [] serial8250_console_write+0xc3/0x1a0 [] call_console_drivers+0xa7/0x1b0 [] console_unlock+0x10d/0x220 [] vprintk_emit+0x1a6/0x610 [] ? __lock_acquire+0x301/0x520 [] printk+0x4f/0x51 [] print_cpu_stall+0x74/0x130 [] __rcu_pending+0xee/0x210 [] rcu_check_callbacks+0x108/0x310 [] update_process_times+0x46/0x90 [] tick_sched_timer+0x66/0xc0 [] ? tick_nohz_handler+0x100/0x100 [] __run_hrtimer+0x12e/0x330 [] hrtimer_interrupt+0xcf/0x200 [] smp_apic_timer_interrupt+0x69/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 00 00 00 d3 e6 48 63 f6 48 03 77 40 8b 06 c9 c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 0f b6 8f 91 00 00 00 89 f2 d3 e2 03 57 38 ec <0f> b6 c0 c9 c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 0f b6 8f 91 NMI backtrace for cpu 4 CPU 4 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/4 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] delay_tsc+0x5e/0xc0 RSP: 0018:ffff88117fc03d10 EFLAGS: 00000002 RAX: ffff8808d2dba000 RBX: 00000000000479ad RCX: 00000000f9d24aaa RDX: 00000000f9d24aaa RSI: 0000000000000005 RDI: 00000000000479ad RBP: ffff88117fc03d48 R08: 00000000f9d24aaa R09: ffffffff81af8a60 R10: 0000000000000001 R11: 0000000000000f34 R12: 0000000000001000 R13: 0000000000000002 R14: ffff8808d2dba000 R15: 0000000000000004 FS: 0000000000000000(0000) GS:ffff88117fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f9f8b3ba000 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/4 (pid: 0, threadinfo ffff8808d2dba000, task ffff8808d2db82c0) Stack: ffff8808d2dba000 00000004f9d24a08 0000000000000000 0000000000001000 0000000000000002 000000000000dd40 0000000000000002 ffff88117fc03d58 ffffffff812b0028 ffff88117fc03d78 ffffffff8103b212 0000000000000005 Call Trace: [] __const_udelay+0x28/0x30 [] native_safe_apic_wait_icr_idle+0x22/0x60 [] default_send_IPI_mask_sequence_phys+0xdd/0x130 [] physflat_send_IPI_all+0x19/0x20 [] arch_trigger_all_cpu_backtrace+0x6c/0xb0 [] print_cpu_stall+0x79/0x130 [] __rcu_pending+0xee/0x210 [] rcu_check_callbacks+0x108/0x310 [] update_process_times+0x46/0x90 [] tick_sched_timer+0x66/0xc0 [] ? tick_nohz_handler+0x100/0x100 [] __run_hrtimer+0x12e/0x330 [] hrtimer_interrupt+0xcf/0x200 [] smp_apic_timer_interrupt+0x69/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: dd 00 00 66 66 90 0f ae e8 e8 2f fd d6 ff 66 90 89 45 d0 44 89 7d d4 4c 89 75 c8 eb 1d 48 8b 45 c8 83 68 1c 01 f3 90 83 40 1c 01 <65> 44 8b 3c 25 08 dd 00 00 44 3b 7d d4 75 36 66 66 90 0f ae e8 NMI backtrace for cpu 5 CPU 5 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/5 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] run_timer_softirq+0xb7/0x2c0 RSP: 0000:ffff88117fc43e30 EFLAGS: 00000002 RAX: 0000000100139276 RBX: 0000000000000075 RCX: ffff881152e08790 RDX: 0000000100139275 RSI: ffff88117fc43df0 RDI: ffff881152e08000 RBP: ffff88117fc43eb0 R08: ffff8808d0893d70 R09: ffff8808d2dbcb90 R10: ffff8808d2dbcb90 R11: 0000000000000001 R12: ffff881152e08000 R13: 0000000000000141 R14: ffff88117fc43e70 R15: ffffffff810784b0 FS: 0000000000000000(0000) GS:ffff88117fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/5 (pid: 0, threadinfo ffff8808d2dbe000, task ffff8808d2dbc300) Stack: 0000000000000002 ffff8808d2dbe000 ffff8808d2dbe000 ffff88117fc50ac0 ffff881152e09c50 ffff881152e09850 ffff881152e09450 ffff881152e09050 ffff88117fc43e70 ffff88117fc43e70 ffff88117fc43ea0 0000000000000282 Call Trace: [] __do_softirq+0x151/0x3e0 [] call_softirq+0x1c/0x30 [] do_softirq+0x9d/0xd0 [] irq_exit+0xe5/0x100 [] smp_apic_timer_interrupt+0x6e/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 00 00 00 00 48 8b 05 39 76 a9 00 49 8b 54 24 40 48 39 d0 0f 88 6b 01 00 00 89 d3 81 e3 ff 00 00 00 0f 84 35 01 00 00 48 8d 42 01 <49> 89 44 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 4a 8b 54 20 10 NMI backtrace for cpu 6 CPU 6 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/6 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] run_timer_softirq+0xd9/0x2c0 RSP: 0000:ffff88117fc83e30 EFLAGS: 00000006 RAX: 0000000000000550 RBX: 0000000000000051 RCX: ffff881152e58560 RDX: ffff881152e58560 RSI: ffff88117fc83df0 RDI: ffff881152e58000 RBP: ffff88117fc83eb0 R08: ffffffffff640027 R09: ffff8808d2dc0bd0 R10: ffff8808d2dc0bd0 R11: 0000000000000001 R12: ffff881152e58000 R13: 0000000000000141 R14: ffff88117fc83e70 R15: ffffffff810689f0 FS: 0000000000000000(0000) GS:ffff88117fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/6 (pid: 0, threadinfo ffff8808d2dc2000, task ffff8808d2dc0340) Stack: 0000000000000002 ffff8808d2dc2000 ffff8808d2dc2000 ffff8808d28d4740 ffff881152e59c50 ffff881152e59850 ffff881152e59450 ffff881152e59050 ffff881152e58560 ffff88117fc83e70 ffff88117fc83ea0 0000000000000282 Call Trace: [] __do_softirq+0x151/0x3e0 [] call_softirq+0x1c/0x30 [] do_softirq+0x9d/0xd0 [] irq_exit+0xe5/0x100 [] smp_apic_timer_interrupt+0x6e/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 84 35 01 00 00 48 8d 42 01 49 89 44 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 4a 8b 54 20 10 49 8d 4c 04 10 48 89 55 c0 4c 89 72 08 <4a> 8b 54 20 18 48 89 55 c8 4c 89 32 4a 89 4c 20 10 4a 89 4c 20 NMI backtrace for cpu 7 CPU 7 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 6687, comm: pickup Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] read_hpet+0x18/0x20 RSP: 0018:ffff88117fcc3e98 EFLAGS: 00000086 INFO: task stress_cpu_hotp:6903 blocked for more than 120 seconds. RAX: 00000000516a74c9 RBX: ffffffff81a1efc0 RCX: 7fffffffffffffff "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. RDX: 0000000000000000 RSI: 000001732e339800 RDI: ffffffff81a1efc0 RBP: ffff88117fcc3e98 R08: fffffffffe04007f R09: ffff8808cdb43238 R10: ffff8808cdb43238 R11: 0000000000000000 R12: 00000000238c1d76 stress_cpu_hotp D 0000000000000000 0 6903 6850 0x00000000 R13: 00000000245c0de7 R14: 0000000000000639 R15: 000000000001dcbc FS: 0000000000000000(0000) GS:ffff88117fcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f63164d8dd8 CR3: 00000008c9745000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 ffff8808cda09ad8 0000000000000046 0000000000000001 ffff8808cda08010 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process pickup (pid: 6687, threadinfo ffff8808cc59a000, task ffff8808cdb42980) Stack: ffff8808cda08000 ffff8808cda08000 ffff8808cda08000 ffff88117fcc3ed8 ffff8808cda09fd8 ffffffff810b446c ffff88117fcce818 ffff8808cda08000 000001732e339800 ffff8808cda09fd8 ffff8808d04f45c0 ffff88117fccdd80 ffff8808d2dae240 Call Trace: 0000000000000000 000001732e339800 000001732e339800 [] ? cpufreq_notify_transition+0x174/0x2f0 ffff88117fcc3f08 ffffffff810ba353 000001732e339800 [] ? acpi_cpufreq_cpu_exit+0xb0/0xb0 [acpi_cpufreq] ffff88117fcce8c0 Call Trace: [] ? mark_held_locks+0x80/0x120 [] ? __mutex_unlock_slowpath+0xf7/0x190 [] ktime_get+0x6c/0xe0 [] schedule+0x29/0x70 [] clockevents_program_event+0x43/0x100 [] schedule_timeout+0x2d5/0x3f0 [] tick_program_event+0x24/0x30 [] ? mark_held_locks+0x80/0x120 [] hrtimer_interrupt+0x116/0x200 [] ? _raw_spin_unlock_irq+0x30/0x50 [] smp_apic_timer_interrupt+0x69/0x99 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] apic_timer_interrupt+0x6f/0x80 [] wait_for_common+0x115/0x190 [] ? kmem_cache_free+0x183/0x280 [] ? try_to_wake_up+0x2f0/0x2f0 [] unlink_anon_vmas+0xae/0x180 [] ? queue_delayed_work_on+0x132/0x140 [] free_pgtables+0xa9/0x110 [] wait_for_completion+0x1d/0x20 [] exit_mmap+0x96/0x140 [] hpet_cpuhp_notify+0x119/0x160 [] mmput+0x87/0x110 [] exit_mm+0x10a/0x140 [] ? init_one_hpet_msi_clockevent+0x1c0/0x1c0 [] ? _raw_spin_unlock_irq+0x30/0x50 [] ? __queue_work+0x500/0x500 [] ? tty_audit_exit+0x28/0xa0 [] notifier_call_chain+0x72/0x130 [] do_exit+0x168/0x4b0 [] __raw_notifier_call_chain+0xe/0x10 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] _cpu_up+0x15b/0x164 [] do_group_exit+0x4c/0xc0 [] cpu_up+0xbc/0x11f [] sys_exit_group+0x17/0x20 [] store_online+0x9d/0xe0 [] system_call_fastpath+0x16/0x1b [] dev_attr_store+0x20/0x30 Code: [] sysfs_write_file+0xd9/0x150 c1 80 c9 [] vfs_write+0xcb/0x130 0f 4c c2 [] sys_write+0x55/0x90 c3 66 66 [] system_call_fastpath+0x16/0x1b 5 locks held by stress_cpu_hotp/6903: 66 2e 0f 1f #0: 84 ( 00 &buffer->mutex 00 ){+.+.+.} 00 , at: [] sysfs_write_file+0x49/0x150 00 00 #1: 55 ( 48 s_active 89 #111 e5 ){.+.+.+} 66 , at: [] sysfs_write_file+0xc2/0x150 66 66 #2: 66 ( 90 x86_cpu_hotplug_driver_mutex 48 ){+.+.+.} 8b , at: [] cpu_hotplug_driver_lock+0x17/0x20 05 a8 #3: 36 ( cc cpu_add_remove_lock 00 ){+.+.+.} 48 , at: [] cpu_up+0x78/0x11f 05 f0 #4: 00 ( 00 cpu_hotplug.lock 00 ){+.+.+.} 8b , at: [] cpu_hotplug_begin+0x2f/0x60 00 <89> c0 c9 c3 0f 1f 40 00 55 48 89 e5 66 66 66 66 90 8b 35 91 36 NMI backtrace for cpu 8 CPU 8 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/8 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] run_timer_softirq+0xe5/0x2c0 RSP: 0018:ffff8808ffd03e30 EFLAGS: 00000002 RAX: 0000000000000ce0 RBX: 00000000000000ca RCX: ffff8808d2f98cf0 RDX: ffff8808ffd03e70 RSI: ffff8808ffd03df0 RDI: ffff8808d2f98000 RBP: ffff8808ffd03eb0 R08: ffffffffff640027 R09: ffff8808d2dcac50 R10: ffff8808d2dcac50 R11: 0000000000000001 R12: ffff8808d2f98000 R13: 0000000000000141 R14: ffff8808ffd03e70 R15: ffffffff810784b0 FS: 0000000000000000(0000) GS:ffff8808ffd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/8 (pid: 0, threadinfo ffff8808d2dcc000, task ffff8808d2dca3c0) Stack: 0000000000000002 ffff8808d2dcc000 ffff8808d2dcc000 ffff8808ffd109a0 ffff8808d2f99c50 ffff8808d2f99850 ffff8808d2f99450 ffff8808d2f99050 ffff8808ffd03e70 ffff8808ffd03e70 ffff8808ffd03ea0 0000000000000282 Call Trace: [] __do_softirq+0x151/0x3e0 [] call_softirq+0x1c/0x30 [] do_softirq+0x9d/0xd0 [] irq_exit+0xe5/0x100 [] smp_apic_timer_interrupt+0x6e/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 4a 8b 54 20 10 49 8d 4c 04 10 48 89 55 c0 4c 89 72 08 4a 8b 54 20 18 48 89 55 c8 4c 89 32 <4a> 89 4c 20 10 4a 89 4c 20 18 48 8b 5d c0 4c 39 f3 74 98 48 8b NMI backtrace for cpu 9 CPU 9 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 5, comm: kworker/u:0 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] process_one_work+0x312/0x480 RSP: 0018:ffff8808d2d3dd40 EFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff8808cfb34030 RCX: 000000002dfe2dfe RDX: 0000000000000000 RSI: 000000008d0b56c0 RDI: ffffffff81da2d00 RBP: ffff8808d2d3ddd0 R08: fffffffffe340073 R09: ffff8808d2d3a9d0 R10: ffff8808d2d3a9d0 R11: 0000000000000001 R12: ffff8808cfb34028 R13: ffff881152ca8980 R14: ffffffff81da2d00 R15: ffff8808d0b56c00 FS: 0000000000000000(0000) GS:ffff8808ffd40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f28bade13e0 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/u:0 (pid: 5, threadinfo ffff8808d2d3c000, task ffff8808d2d3a140) Stack: ffffffff8107a4ff 0000000000000046 0000000000000004 ffffffff8107b2d8 ffffffff00000000 0000000000000000 ffffffffa0b23b00 0ffff8808d0b56c0 ffffffffa0b26f98 ffffffff820155d0 0000000000000000 ffffffffa0b25353 Call Trace: [] ? process_one_work+0x13f/0x480 [] ? worker_thread+0x68/0x3b0 [] ? edac_pci_reset_delay_period+0x60/0x60 [edac_core] [] worker_thread+0x186/0x3b0 [] ? manage_workers+0x120/0x120 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 48 89 02 74 04 48 89 50 08 49 c7 45 00 00 00 00 00 49 c7 45 08 00 00 00 00 48 c1 6d a8 04 49 c7 45 10 00 00 00 00 31 d2 8b 75 a8 <49> c7 45 18 00 00 00 00 4c 89 ff 83 e6 0f e8 2b fc ff ff 48 83 NMI backtrace for cpu 10 CPU 10 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 16471, comm: kworker/10:1 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] __might_sleep+0x19/0x240 RSP: 0018:ffff8808c9657cc0 EFLAGS: 00000286 RAX: 000000010013bf80 RBX: ffff8808d253d900 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 000000000000109d RDI: ffffffff817c61e4 RBP: ffff8808c9657ce0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000003 R11: 0000000000000001 R12: ffff8808d253d840 R13: ffff8808d25309c0 R14: 0000000000000001 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8808ffd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fee87eccb60 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/10:1 (pid: 16471, threadinfo ffff8808c9656000, task ffff8808cf8e4380) Stack: ffff8808d253d840 ffff8808d253d900 ffff8808d253d840 ffff8808d25309c0 ffff8808c9657d30 ffffffff8118d7c4 ffff8808ffd90ac0 00000000c9657d80 0000000000000000 ffff8808ffd90ac8 ffff8808ffd90ac0 ffff8808cf888140 Call Trace: [] cache_reap+0xf4/0x270 [] process_one_work+0x201/0x480 [] ? process_one_work+0x13f/0x480 [] ? worker_thread+0x68/0x3b0 [] ? drain_freelist+0xb0/0xb0 [] worker_thread+0x186/0x3b0 [] ? manage_workers+0x120/0x120 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 55 55 48 83 c4 08 5b c9 c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 66 66 66 66 90 <49> 89 fd 41 89 f4 89 d3 e8 2a d6 fe ff 85 c0 74 0d 80 3d d8 e6 NMI backtrace for cpu 11 CPU 11 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 0, comm: swapper/11 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] run_timer_softirq+0xc7/0x2c0 RSP: 0018:ffff8808ffdc3e30 EFLAGS: 00000002 RAX: 0000000000000e70 RBX: 00000000000000e3 RCX: ffff8808d2898e70 RDX: 0000000100131fe3 RSI: ffff8808ffdc3df0 RDI: ffff8808d2898000 RBP: ffff8808ffdc3eb0 R08: ffffffffff640027 R09: ffff8808d2dd8d10 R10: ffff8808d2dd8d10 R11: 0000000000000001 R12: ffff8808d2898000 R13: 0000000000000141 R14: ffff8808ffdc3e70 R15: ffffffff810784b0 FS: 0000000000000000(0000) GS:ffff8808ffdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/11 (pid: 0, threadinfo ffff8808d2dda000, task ffff8808d2dd8480) Stack: 0000000000000002 ffff8808d2dda000 ffff8808d2dda000 ffff8808ffdd09a0 ffff8808d2899c50 ffff8808d2899850 ffff8808d2899450 ffff8808d2899050 ffff8808ffdc3e70 ffff8808ffdc3e70 ffff8808ffdc3ea0 0000000000000282 Call Trace: [] __do_softirq+0x151/0x3e0 [] call_softirq+0x1c/0x30 [] do_softirq+0x9d/0xd0 [] irq_exit+0xe5/0x100 [] smp_apic_timer_interrupt+0x6e/0x99 [] apic_timer_interrupt+0x6f/0x80 [] ? intel_idle+0xc8/0x170 [] ? intel_idle+0xc4/0x170 [] cpuidle_enter+0x17/0x20 [] cpuidle_idle_call+0x143/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xd1/0xd4 Code: 48 39 d0 0f 88 6b 01 00 00 89 d3 81 e3 ff 00 00 00 0f 84 35 01 00 00 48 8d 42 01 49 89 44 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 <4a> 8b 54 20 10 49 8d 4c 04 10 48 89 55 c0 4c 89 72 08 4a 8b 54 lockdep: fixing up alternatives. Booting Node 1 Processor 12 APIC 0x11 sending NMI to all CPUs: NMI backtrace for cpu 9 CPU 9 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 16456, comm: migration/9 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] stop_machine_cpu_stop+0x85/0x140 RSP: 0018:ffff8808d2f93d20 EFLAGS: 00000293 RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808d2da0400 RDX: ffff8808d2da0400 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8 RBP: ffff8808d2f93d50 R08: ffff8808ffd4ebc0 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8 R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01 FS: 0000000000000000(0000) GS:ffff8808ffd40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process migration/9 (pid: 16456, threadinfo ffff8808d2f92000, task ffff8808d2da0400) Stack: 00000000d2f93d30 ffff8808d2ddfe28 ffff8808d2da0400 ffff8808ffd4eb80 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808d2f93e50 ffffffff810e5e53 ffff8808d2f92000 ffff8808d2f93fd8 ffff8808d2f92000 ffff8808ffd4ebc0 Call Trace: [] ? stop_one_cpu_nowait+0x50/0x50 [] cpu_stopper_thread+0xf3/0x1e0 [] ? _raw_spin_unlock_irqrestore+0x44/0x90 [] ? res_counter_init+0x60/0x60 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] ? res_counter_init+0x60/0x60 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 <89> d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 NMI backtrace for cpu 10 CPU 10 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 16466, comm: migration/10 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] stop_machine_cpu_stop+0x89/0x140 RSP: 0018:ffff8808cfa63d20 EFLAGS: 00000293 RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808cbcc81c0 RDX: ffff8808cbcc81c0 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8 RBP: ffff8808cfa63d50 R08: ffff8808ffd8ebc0 R09: 0000000000000000 R10: 0000000000000000 R11: 000000000000000 R12: ffff8808d2ddfdf8 R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01 FS: 0000000000000000(0000) GS:ffff8808ffd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process migration/10 (pid: 16466, threadinfo ffff8808cfa62000, task ffff8808cbcc81c0) Stack: 00000000cfa63d30 ffff8808d2ddfe28 ffff8808cbcc81c0 ffff8808ffd8eb80 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808cfa63e50 ffffffff810e5e53 ffff8808cfa62000 ffff8808cfa63fd8 ffff8808cfa62000 ffff8808ffd8ebc0 Call Trace: [] ? stop_one_cpu_nowait+0x50/0x50 [] cpu_stopper_thread+0xf3/0x1e0 [] ? _raw_spin_unlock_irqrestore+0x44/0x90 [] ? res_counter_init+0x60/0x60 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] ? res_counter_init+0x60/0x60 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 NMI backtrace for cpu 11 CPU 11 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 16475, comm: migration/11 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] stop_machine_cpu_stop+0x80/0x140 RSP: 0018:ffff8808ce103d20 EFLAGS: 00000246 RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808ce0a4640 RDX: ffff8808ce0a4640 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8 RBP: ffff8808ce103d50 R08: ffff8808ffdcebc0 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8 R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01 FS: 0000000000000000(0000) GS:ffff8808ffdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process migration/11 (pid: 16475, threadinfo ffff8808ce102000, task ffff8808ce0a4640) Stack: 00000000ce103d30 ffff8808d2ddfe28 ffff8808ce0a4640 ffff8808ffdceb80 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808ce103e50 ffffffff810e5e53 ffff8808ce102000 ffff8808ce103fd8 ffff8808ce102000 ffff8808ffdcebc0 Call Trace: [] ? stop_one_cpu_nowait+0x50/0x50 [] cpu_stopper_thread+0xf3/0x1e0 [] ? _raw_spin_unlock_irqrestore+0x44/0x90 [] ? res_counter_init+0x60/0x60 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] ? res_counter_init+0x60/0x60 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 44 00 00 83 fb 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 <83> fb 04 74 43 89 d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 NMI backtrace for cpu 2 CPU 2 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 16393, comm: migration/2 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] stop_machine_cpu_stop+0x89/0x140 RSP: 0018:ffff8808cdaf1d20 EFLAGS: 00000293 RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808cb53a400 RDX: ffff8808cb53a400 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8 RBP: ffff8808cdaf1d50 R08: ffff8808ffc8ebc0 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8 R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01 FS: 0000000000000000(0000) GS:ffff8808ffc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f97183ca000 CR3: 00000008c965d000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process migration/2 (pid: 16393, threadinfo ffff8808cdaf0000, task ffff8808cb53a400) Stack: 00000000cdaf1d30 ffff8808d2ddfe28 ffff8808cb53a400 ffff8808ffc8eb80 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808cdaf1e50 ffffffff810e5e53 ffff8808cdaf0000 ffff8808cdaf1fd8 ffff8808cdaf0000 ffff8808ffc8ebc0 Call Trace: [] ? stop_one_cpu_nowait+0x50/0x50 [] cpu_stopper_thread+0xf3/0x1e0 [] ? _raw_spin_unlock_irqrestore+0x44/0x90 [] ? res_counter_init+0x60/0x60 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] ? res_counter_init+0x60/0x60 [] kthread+0xee/0x100 [] kernel_thread_helper+0x4/0x10 [] ? retint_restore_args+0x13/0x13 [] ? __init_kthread_worker+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 NMI backtrace for cpu 0 CPU 0 Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas Pid: 6, comm: migration/0 Tainted: G W 3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] stop_machine_cpu_stop+0x89/0x140 RSP: 0018:ffff8808d2d83d20 EFLAGS: 00000293 RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808d2d80180 RDX: ffff8808d2d80180 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8 RBP: ffff8808d2d83d50 R08: ffff8808ffc0ebc0 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8 R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01 FS: 0000000000000000(0000) GS:ffff8808ffc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process migration/0 (pid: 6, threadinfo ffff8808d2d82000, task ffff8808d2d80180) Stack: 00000000d2d83d30 ffff8808d2ddfe28 ffff8808d2d80180 ffff8808ffc0eb80 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808d2d83e50 ffffffff810e5e53 ffff8808d2d82000 ffff8808d2d83fd8 ffff8808d2d82000 ffff8808ffc0ebc0 Call Trace: [] ? stop_one_cpu_nowait+0x50/0x50 [] cpu_stopper_thread+0xf3/0x1e0 [] ? _raw_spin_unlock_irqrestore+0x44/0x90 [] ? res_counter_init+0x60/0x60 [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [] ? trace_hardirqs_on+0xd/0x10 [] ? res_counter_init+0x60/0x60 [