From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Srivatsa S. Bhat" Subject: Re: [PATCH v5 7/8] mm: Only IPI CPUs to drain local pages if they exist Date: Fri, 06 Jan 2012 11:36:11 +0530 Message-ID: <4F068F53.50402@linux.vnet.ibm.com> References: <1325499859-2262-1-git-send-email-gilad@benyossef.com> <1325499859-2262-8-git-send-email-gilad@benyossef.com> <4F033EC9.4050909@gmail.com> <20120105142017.GA27881@csn.ul.ie> <20120105144011.GU11810@n2100.arm.linux.org.uk> <20120105161739.GD27881@csn.ul.ie> <20120105163529.GA11810@n2100.arm.linux.org.uk> <20120105183504.GF2393@linux.vnet.ibm.com> <20120105222116.GF27881@csn.ul.ie> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: "Paul E. McKenney" , Russell King - ARM Linux , KOSAKI Motohiro , Gilad Ben-Yossef , linux-kernel@vger.kernel.org, Chris Metcalf , Peter Zijlstra , Frederic Weisbecker , linux-mm@kvack.org, Pekka Enberg , Matt Mackall , Sasha Levin , Rik van Riel , Andi Kleen , Andrew Morton , Alexander Viro , Greg KH , linux-fsdevel@vger.kernel.org, Avi Kivity To: Mel Gorman Return-path: Received: from e23smtp03.au.ibm.com ([202.81.31.145]:35718 "EHLO e23smtp03.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751127Ab2AFGGa (ORCPT ); Fri, 6 Jan 2012 01:06:30 -0500 Received: from /spool/local by e23smtp03.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 6 Jan 2012 06:00:06 +1000 In-Reply-To: <20120105222116.GF27881@csn.ul.ie> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On 01/06/2012 03:51 AM, Mel Gorman wrote: > (Adding Greg to cc to see if he recalls seeing issues with sysfs dentry > suffering from recursive locking recently) > > On Thu, Jan 05, 2012 at 10:35:04AM -0800, Paul E. McKenney wrote: >> On Thu, Jan 05, 2012 at 04:35:29PM +0000, Russell King - ARM Linux wrote: >>> On Thu, Jan 05, 2012 at 04:17:39PM +0000, Mel Gorman wrote: >>>> Link please? >>> >>> Forwarded, as its still in my mailbox. >>> >>>> I'm including a patch below under development that is >>>> intended to only cope with the page allocator case under heavy memory >>>> pressure. Currently it does not pass testing because eventually RCU >>>> gets stalled with the following trace >>>> >>>> [ 1817.176001] [] arch_trigger_all_cpu_backtrace+0x87/0xa0 >>>> [ 1817.176001] [] __rcu_pending+0x149/0x260 >>>> [ 1817.176001] [] rcu_check_callbacks+0x5f/0x110 >>>> [ 1817.176001] [] update_process_times+0x3f/0x80 >>>> [ 1817.176001] [] tick_sched_timer+0x5b/0xc0 >>>> [ 1817.176001] [] __run_hrtimer+0xbe/0x1a0 >>>> [ 1817.176001] [] hrtimer_interrupt+0xc1/0x1e0 >>>> [ 1817.176001] [] smp_apic_timer_interrupt+0x63/0xa0 >>>> [ 1817.176001] [] apic_timer_interrupt+0x13/0x20 >>>> [ 1817.176001] [] vfsmount_lock_local_lock+0x25/0x30 >>>> [ 1817.176001] [] path_init+0x2d5/0x370 >>>> [ 1817.176001] [] path_lookupat+0x2d/0x620 >>>> [ 1817.176001] [] do_path_lookup+0x2f/0xd0 >>>> [ 1817.176001] [] user_path_at_empty+0x9f/0xd0 >>>> [ 1817.176001] [] vfs_fstatat+0x4b/0x90 >>>> [ 1817.176001] [] sys_newlstat+0x1f/0x50 >>>> [ 1817.176001] [] system_call_fastpath+0x16/0x1b >>>> >>>> It might be a separate bug, don't know for sure. >> > > I rebased the patch on top of 3.2 and tested again with a bunch of > debugging options set (PROVE_RCU, PROVE_LOCKING etc). Same results. CPU > hotplug is a lot more reliable and less likely to hang but eventually > gets into trouble. > Hi everyone, I was running some CPU hotplug stress tests recently and found it to be problematic too. Mel, I have some logs from those tests which appear very relevant to the "IPI to offline CPU" issue that has been discussed in this thread. Kernel: 3.2-rc7 Here is the log: (Unfortunately I couldn't capture the log intact, due to some annoying serial console issues, but I hope this log is good enough to analyze.) [ 907.825267] Booting Node 1 Processor 15 APIC 0x17 [ 907.830117] smpboot cpu 15: start_ip = 97000 [ 906.104006] Calibrating delay loop (skipped) already calibrated this CPU [ 907.860875] NMI watchdog enabled, takes one hw-pmu counter. [ 907.898899] Broke affinity for irq 81 [ 907.904539] CPU 1 is now offline [ 907.912891] CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 907.929462] CPU 2 is now offline [ 907.939573] CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 907.969514] CPU 3 is now offline [ 907.978644] CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 908.021903] Broke affinity for irq 74 [ 908.024021] ------------[ cut here ]------------ [ 908.024021] WARNING: at kernel/smp.c:258 generic_smp_call_function_single_interrupt+0x109/0x120() [ 908.024021] Hardware name: IBM System x -[7870C4Q]- [ 908.024021] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 908.024021] Pid: 22076, comm: migration/4 Not tainted 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 [ 908.024021] Call Trace: [ 908.024021] [] ? generic_smp_call_function_single_interrupt+0x109/0x120 [ 908.024021] [] warn_slowpath_common+0x7a/0xb0 [ 908.024021] [] warn_slowpath_null+0x15/0x20 [ 908.024021] [] generic_smp_call_function_single_interrupt+0x109/0x120 [ 908.024021] [] smp_call_function_single_interrupt+0x22/0x40 [ 908.024021] [] call_function_single_interrupt+0x73/0x80 [ 908.024021] [] ? stop_machine_cpu_stop+0xda/0x130 [ 908.024021] [] ? stop_one_cpu_nowait+0x50/0x50 [ 908.024021] [] cpu_stopper_thread+0xba/0x180 [ 908.024021] [] ? _raw_spin_unlock_irqrestore+0x3f/0x70 [ 908.024021] [] ? res_counter_init+0x50/0x50 [ 908.024021] [] ? trace_hardirqs_on_caller+0x12d/0x1b0 [ 908.024021] [] ? trace_hardirqs_on+0xd/0x10 [ 908.024021] [] ? res_counter_init+0x50/0x50 [ 908.024021] [] kthread+0x96/0xa0 [ 908.024021] [] kernel_thread_helper+0x4/0x10 [ 908.024021] [] ? retint_restore_args+0x13/0x13 [ 908.024021] [] ? __init_kthread_worker+0x70/0x70 [ 908.024021] [] ? gs_change+0x13/0x13 [ 908.024021] ---[ end trace f4c7a25be63a672a ]--- [ 908.328208] CPU 4 is now offline [ 908.332730] CPU 5 MCA banks CMCI:6 CMCI:8 [ 908.337074] CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 908.349270] CPU 5 is now offline [ 908.353888] CPU 6 MCA banks CMCI:6 CMCI:8 [ 908.376131] CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 908.391939] CPU 6 is now offline [ 908.413193] CPU 7 MCA banks CMCI:6 CMCI:8 [ 908.443245] CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 908.475871] CPU 7 is now offline [ 908.481601] CPU 12 MCA banks CMCI:6 CMCI:8 [ 908.485923] CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 908.519889] CPU 8 is now offline [ 908.565926] CPU 9 is now offline [ 908.602874] CPU 10 is now offline [ 908.634696] CPU 11 is now offline [ 908.674735] CPU 12 is now offline [ 908.680343] CPU 13 MCA banks CMCI:6 CMCI:8 [ 908.721887] CPU 13 is now offline [ 908.728086] CPU 14 MCA banks CMCI:6 CMCI:8 [ 908.789105] CPU 14 is now offline [ 908.794969] CPU 15 MCA banks CMCI:6 CMCI:8 [ 908.881878] CPU 15 is now offline [ 908.885301] lockdep: fixing up alternatives. [ 908.889663] SMP alternatives: switching to UP code [ 909.140900] lockdep: fixing up alternatives. [ 909.145281] SMP alternatives: switching to SMP code [ 909.153536] Booting Node 0 Processor 1 APIC 0x2 [ 909.158157] smpboot cpu 1: start_ip = 97000 [ 907.900022] Calibrating delay loop (skipped) already calibrated this CPU [ 909.181323] NMI watchdog enabled, takes one hw-pmu counter. [ 909.275696] lockdep: fixing up alternatives. [ 909.280106] Booting Node 0 Processor 2 APIC 0x4 [ 909.280107] smpboot cpu 2: start_ip = 97000 [ 907.928015] Calibrating delay loop (skipped) already calibrated this CPU [ 909.308538] NMI watchdog enabled, takes one hw-pmu counter. [ 909.376170] lockdep: fixing up alternatives. [ 909.380589] Booting Node 0 Processor 3 APIC 0x6 [ 1319.109486] Booting Node 1 Processor 14 APIC 0x15 [ 1319.114320] smpboot cpu 14: start_ip = 97000 [ 1318.456153] Calibrating delay loop (skipped) already calibrated this CPU [ 1319.139762] NMI watchdog enabled, takes one hw-pmu counter. [ 1319.150412] lockdep: fixing up alternatives. [ 1319.155062] Booting Node 1 Processor 15 APIC 0x17 [ 1319.160165] smpboot cpu 15: start_ip = 97000 [ 1318.472003] Calibrating delay loop (skipped) already calibrated this CPU [ 1319.188592] NMI watchdog enabled, takes one hw-pmu counter. [ 1319.216529] CPU 1 is now offline [ 1319.224915] CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.240750] CPU 2 is now offline [ 1319.256419] CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.269161] CPU 3 is now offline [ 1319.280258] CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.293433] CPU 4 is now offline [ 1319.298109] CPU 5 MCA banks CMCI:6 CMCI:8 [ 1319.312516] CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.325377] CPU 5 is now offline [ 1319.331679] CPU 6 MCA banks CMCI:6 CMCI:8 [ 1319.340437] CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.352367] CPU 6 is now offline [ 1319.357553] CPU 7 MCA banks CMCI:6 CMCI:8 [ 1319.372577] CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.385997] CPU 7 is now offline [ 1319.393018] CPU 12 MCA banks CMCI:6 CMCI:8 [ 1319.397604] CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5 [ 1319.409149] CPU 8 is now offline [ 1319.428255] CPU 9 is now offline [ 1319.450764] CPU 10 is now offline [ 1319.474489] CPU 11 is now offline [ 1319.496806] CPU 12 is now offline [ 1319.502966] CPU 13 MCA banks CMCI:6 CMCI:8 [ 1319.511746] CPU 13 is now offline [ 1347.146085] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068] [ 1347.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1347.148005] irq event stamp: 151225746 [ 1347.148005] hardirqs last enabled at (151225745): [] restore_args+0x0/0x30 [ 1347.148005] hardirqs last disabled at (151225746): [] apic_timer_interrupt+0x6e/0x80 [ 1347.148005] softirqs last enabled at (151225744): [] __do_softirq+0x1a1/0x200 [ 1347.148005] softirqs last disabled at (151225739): [] call_softirq+0x1c/0x30 [ 1347.148005] CPU 14 [ 1347.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1347.148005] [ 1347.148005] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1347.148005] RIP: 0010:[] [] flush_tlb_others_ipi+0x108/0x140 [ 1347.148005] RSP: 0000:ffff881147bfdc48 EFLAGS: 00000246 [ 1347.148005] RAX: 0000000000000000 RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1347.148005] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1347.148005] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1347.148005] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdbb8 [ 1347.148005] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1347.148005] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1347.148005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1347.148005] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1347.148005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1347.148005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1347.148005] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1347.148005] Stack: [ 1347.148005] ffff881147bfdcb8 ffff881146a20d80 00007fff56156db8 ffff881146a20de0 [ 1347.148005] ffff88114739f818 ffff881147776ab0 ffff881147bfdc88 ffffffff81033e29 [ 1347.148005] ffff881147bfdcb8 ffffffff81033f2a ffff881146a20df8 0000000000000001 [ 1347.148005] Call Trace: [ 1347.148005] [] native_flush_tlb_others+0x9/0x10 [ 1347.148005] [] flush_tlb_page+0x5a/0xa0 [ 1347.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1347.148005] [] do_wp_page+0x469/0x7e0 [ 1347.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1347.148005] [] handle_mm_fault+0x178/0x2e0 [ 1347.148005] [] do_page_fault+0x1e5/0x490 [ 1347.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1347.148005] [] page_fault+0x25/0x30 [ 1347.148005] Code: 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 8b 35 00 4f 9b 00 <4c> 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 2e 0f 1f 84 00 00 [ 1347.148005] Call Trace: [ 1347.148005] [] native_flush_tlb_others+0x9/0x10 [ 1347.148005] [] flush_tlb_page+0x5a/0xa0 [ 1347.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1347.148005] [] do_wp_page+0x469/0x7e0 [ 1347.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1347.148005] [] handle_mm_fault+0x178/0x2e0 [ 1347.148005] [] do_page_fault+0x1e5/0x490 [ 1347.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1347.148005] [] page_fault+0x25/0x30 [ 1384.508005] INFO: rcu_sched detected stall on CPU 14 (t=16250 jiffies) [ 1384.508007] sending NMI to all CPUs: [ 1384.516014] INFO: rcu_sched detected stalls on CPUs/tasks: { 14} (detected by 15, t=16252 jiffies) [ 1384.527400] NMI backtrace for cpu 0 [ 1384.528012] CPU 0 [ 1384.528012] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1384.528012] [ 1384.528012] Pid: 24575, comm: cc1 Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1384.528012] RIP: 0033:[<00002abf6f4f7c48>] [<00002abf6f4f7c48>] 0x2abf6f4f7c47 [ 1384.528012] RSP: 002b:00007fffa24587e8 EFLAGS: 00000206 [ 1384.528012] RAX: 000000000000002b RBX: 00007fffa24587f0 RCX: 0000000000000004 [ 1384.528012] RDX: 0000000000000000 RSI: 00002abf7029b714 RDI: 00007fffa24587f0 [ 1384.528012] RBP: 00007fffa2458860 R08: fffffffffffffffc R09: 00007fffa245892e [ 1384.528012] R10: 00002abf6ee686c0 R11: 00002abf6f4fa1c6 R12: 0000000000000000 [ 1384.528012] R13: 00002abf7029b714 R14: 0000000000000002 R15: 00007fffa245892f [ 1384.528012] FS: 00002abf6f7d87e0(0000) GS:ffff8808ffc00000(0000) knlGS:0000000000000000 [ 1384.528012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1384.528012] CR2: 00002abf7029e000 CR3: 00000007fb3e4000 CR4: 00000000000006f0 [ 1384.528012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1384.528012] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1384.528012] Process cc1 (pid: 24575, threadinfo ffff8807fb3ea000, task ffff8807fb3ed640) [ 1384.528012] [ 1384.528012] Call Trace: [ 1384.508007] NMI backtrace for cpu 14 [ 1384.508007] CPU 14 [ 1384.508007] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1384.508007] [ 1384.508007] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1384.508007] RIP: 0010:[] [] delay_tsc+0x42/0xa0 [ 1384.508007] RSP: 0000:ffff88117fd83d30 EFLAGS: 00000093 [ 1384.508007] RAX: 00000000000000d4 RBX: 00000000000470af RCX: ffffffff8ac50e81 [ 1384.508007] RDX: 000000008ac50e81 RSI: 000000000000000f RDI: 00000000000470af [ 1384.508007] RBP: ffff88117fd83d68 R08: 0000000000000010 R09: ffffffff819e7660 [ 1384.508007] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000001000 [ 1384.508007] R13: 0000000000000002 R14: 000000000000000e R15: 000000000000000e [ 1384.508007] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1384.508007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1384.508007] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1384.508007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1384.508007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1384.508007] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1384.508007] Stack: [ 1384.508007] ffffffff8ac50dad 0000000e00000046 0000000000000000 0000000000001000 [ 1384.508007] 0000000000000002 000000000000cbc0 0000000000000002 ffff88117fd83d78 [ 1384.508007] ffffffff81258e1f ffff88117fd83d98 ffffffff81021aaa 000000000000000f [ 1384.508007] Call Trace: [ 1384.508007] [ 1384.508007] [] __const_udelay+0x2f/0x40 [ 1384.508007] [] native_safe_apic_wait_icr_idle+0x1a/0x50 [ 1384.508007] [] default_send_IPI_mask_sequence_phys+0xdd/0x130 [ 1384.508007] [] physflat_send_IPI_all+0x14/0x20 [ 1384.508007] [] arch_trigger_all_cpu_backtrace+0x67/0xb0 [ 1384.508007] [] __rcu_pending+0x119/0x280 [ 1384.508007] [] rcu_check_callbacks+0xaa/0x1b0 [ 1384.508007] [] update_process_times+0x41/0x80 [ 1384.508007] [] tick_sched_timer+0x5f/0xc0 [ 1384.508007] [] ? tick_nohz_handler+0x100/0x100 [ 1384.508007] [] __run_hrtimer+0xd1/0x1d0 [ 1384.508007] [] hrtimer_interrupt+0xc7/0x1f0 [ 1384.508007] [] smp_apic_timer_interrupt+0x64/0xa0 [ 1384.508007] [] apic_timer_interrupt+0x73/0x80 [ 1384.508007] [ 1384.508007] [] ? flush_tlb_others_ipi+0x102/0x140 [ 1384.508007] [] ? flush_tlb_others_ipi+0x110/0x140 [ 1384.508007] [] native_flush_tlb_others+0x9/0x10 [ 1384.508007] [] flush_tlb_page+0x5a/0xa0 [ 1384.508007] [] ptep_set_access_flags+0x4d/0x70 [ 1384.508007] [] do_wp_page+0x469/0x7e0 [ 1384.508007] [] handle_pte_fault+0x19d/0x1e0 [ 1384.508007] [] handle_mm_fault+0x178/0x2e0 [ 1384.508007] [] do_page_fault+0x1e5/0x490 [ 1384.508007] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1384.508007] [] page_fault+0x25/0x30 [ 1384.508007] Code: 34 25 b0 cb 00 00 66 66 90 0f ae e8 e8 e8 15 db ff 66 90 48 98 44 89 75 d4 48 89 45 c8 eb 1b 66 2e 0f 1f 84 00 00 00 00 00 f3 90 <65> 44 8b 3c 25 b0 cb 00 00 44 3b 7d d4 75 2b 66 66 90 0f ae e8 [ 1384.508007] Call Trace: [ 1384.508007] [] __const_udelay+0x2f/0x40 [ 1384.508007] [] native_safe_apic_wait_icr_idle+0x1a/0x50 [ 1384.508007] [] default_send_IPI_mask_sequence_phys+0xdd/0x130 [ 1384.508007] [] physflat_send_IPI_all+0x14/0x20 [ 1384.508007] [] arch_trigger_all_cpu_backtrace+0x67/0xb0 [ 1384.508007] [] __rcu_pending+0x119/0x280 [ 1384.508007] [] rcu_check_callbacks+0xaa/0x1b0 [ 1384.508007] [] update_process_times+0x41/0x80 [ 1384.508007] [] tick_sched_timer+0x5f/0xc0 [ 1384.508007] [] ? tick_nohz_handler+0x100/0x100 [ 1384.508007] [] __run_hrtimer+0xd1/0x1d0 [ 1384.508007] [] hrtimer_interrupt+0xc7/0x1f0 [ 1384.508007] [] smp_apic_timer_interrupt+0x64/0xa0 [ 1384.508007] [] apic_timer_interrupt+0x73/0x80 [ 1384.508007] [] ? flush_tlb_others_ipi+0x102/0x140 [ 1384.508007] [] ? flush_tlb_others_ipi+0x110/0x140 [ 1384.508007] [] native_flush_tlb_others+0x9/0x10 [ 1384.508007] [] flush_tlb_page+0x5a/0xa0 [ 1384.508007] [] ptep_set_access_flags+0x4d/0x70 [ 1384.508007] [] do_wp_page+0x469/0x7e0 [ 1384.508007] [] handle_pte_fault+0x19d/0x1e0 [ 1384.508007] [] handle_mm_fault+0x178/0x2e0 [ 1384.508007] [] do_page_fault+0x1e5/0x490 [ 1384.508007] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1384.508007] [] page_fault+0x25/0x30 [ 1385.089722] NMI backtrace for cpu 15 [ 1385.089722] CPU 15 [ 1385.089722] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1385.089722] [ 1385.089722] Pid: 24569, comm: sh Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1385.089722] RIP: 0033:[<00002b292b250c59>] [<00002b292b250c59>] 0x2b292b250c58 [ 1385.089722] RSP: 002b:00007fff6bb94b70 EFLAGS: 00000202 [ 1385.089722] RAX: 00002b292b520580 RBX: 00000000006b0907 RCX: 00007fff6bb94c20 [ 1385.089722] RDX: 00007fff6bb94bd0 RSI: 00000000006b0907 RDI: 00007fff6bb94bd0 [ 1385.089722] RBP: 000000000000001b R08: 00000000006b0907 R09: 0000000000000001 [ 1385.089722] R10: 0000000000000000 R11: 00007fff6bb94c20 R12: 0000000000000000 [ 1385.089722] R13: 00000000006b0907 R14: 00007fff6bb94bd0 R15: 00002b292b76eba0 [ 1385.089722] FS: 00002b292b76eba0(0000) GS:ffff88117fdc0000(0000) knlGS:0000000000000000 [ 1385.089722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1385.089722] CR2: 00000000006b1110 CR3: 00000010a023d000 CR4: 00000000000006e0 [ 1385.089722] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1385.089722] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1385.089722] Process sh (pid: 24569, threadinfo ffff8810a0224000, task ffff8810a0220d40) [ 1385.089722] [ 1385.089722] Call Trace: [ 1411.146085] BUG: soft lockup - CPU#14 stuck for 23s! [udevd:1068] [ 1411.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1411.148005] irq event stamp: 151367944 [ 1411.148005] hardirqs last enabled at (151367943): [] restore_args+0x0/0x30 [ 1411.148005] hardirqs last disabled at (151367944): [] apic_timer_interrupt+0x6e/0x80 [ 1411.148005] softirqs last enabled at (151367942): [] __do_softirq+0x1a1/0x200 [ 1411.148005] softirqs last disabled at (151367937): [] call_softirq+0x1c/0x30 [ 1411.148005] CPU 14 [ 1411.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1411.148005] [ 1411.148005] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1411.148005] RIP: 0010:[] [] flush_tlb_others_ipi+0x102/0x140 [ 1411.148005] RSP: 0000:ffff881147bfdc48 EFLAGS: 00000246 [ 1411.148005] RAX: 0000000000000000 RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1411.148005] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1411.148005] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1411.148005] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdbb8 [ 1411.148005] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1411.148005] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1411.148005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1411.148005] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1411.148005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1411.148005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1411.148005] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1411.148005] Stack: [ 1411.148005] ffff881147bfdcb8 ffff881146a20d80 00007fff56156db8 ffff881146a20de0 [ 1411.148005] ffff88114739f818 ffff881147776ab0 ffff881147bfdc88 ffffffff81033e29 [ 1411.148005] ffff881147bfdcb8 ffffffff81033f2a ffff881146a20df8 0000000000000001 [ 1411.148005] Call Trace: [ 1411.148005] [] native_flush_tlb_others+0x9/0x10 [ 1411.148005] [] flush_tlb_page+0x5a/0xa0 [ 1411.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1411.148005] [] do_wp_page+0x469/0x7e0 [ 1411.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1411.148005] [] handle_mm_fault+0x178/0x2e0 [ 1411.148005] [] do_page_fault+0x1e5/0x490 [ 1411.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1411.148005] [] page_fault+0x25/0x30 [ 1411.148005] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 <8b> 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 [ 1411.148005] Call Trace: [ 1411.148005] [] native_flush_tlb_others+0x9/0x10 [ 1411.148005] [] flush_tlb_page+0x5a/0xa0 [ 1411.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1411.148005] [] do_wp_page+0x469/0x7e0 [ 1411.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1411.148005] [] handle_mm_fault+0x178/0x2e0 [ 1411.148005] [] do_page_fault+0x1e5/0x490 [ 1411.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1411.148005] [] page_fault+0x25/0x30 [ 1439.146086] BUG: soft lockup - CPU#14 stuck for 23s! [udevd:1068] [ 1439.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1439.148005] irq event stamp: 151430582 [ 1439.148005] hardirqs last enabled at (151430581): [] restore_args+0x0/0x30 [ 1439.148005] hardirqs last disabled at (151430582): [] apic_timer_interrupt+0x6e/0x80 [ 1439.148005] softirqs last enabled at (151430580): [] __do_softirq+0x1a1/0x200 [ 1439.148005] softirqs last disabled at (151430575): [] call_softirq+0x1c/0x30 [ 1439.148005] CPU 14 [ 1439.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1439.148005] [ 1439.148005] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1439.148005] RIP: 0010:[] [] __bitmap_empty+0x13/0x90 [ 1439.148005] RSP: 0000:ffff881147bfdc38 EFLAGS: 00000246 [ 1439.148005] RAX: 000000000000004f RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1439.148005] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1439.148005] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1439.148005] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdba8 [ 1439.148005] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1439.148005] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1439.148005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1439.148005] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1439.148005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1439.148005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1439.148005] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1439.148005] Stack: [ 1439.148005] ffff881147bfdc78 ffffffff81033df0 ffff881147bfdcb8 ffff881146a20d80 [ 1439.148005] 00007fff56156db8 ffff881146a20de0 ffff88114739f818 ffff881147776ab0 [ 1439.148005] ffff881147bfdc88 ffffffff81033e29 ffff881147bfdcb8 ffffffff81033f2a [ 1439.148005] Call Trace: [ 1439.148005] [] ? flush_tlb_others_ipi+0x110/0x140 [ 1439.148005] [] native_flush_tlb_others+0x9/0x10 [ 1439.148005] [] flush_tlb_page+0x5a/0xa0 [ 1439.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1439.148005] [] do_wp_page+0x469/0x7e0 [ 1439.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1439.148005] [] handle_mm_fault+0x178/0x2e0 [ 1439.148005] [] do_page_fault+0x1e5/0x490 [ 1439.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1439.148005] [] page_fault+0x25/0x30 [ 1439.148005] Code: c7 45 b0 10 00 00 00 48 89 45 c0 e8 38 ff ff ff c9 c3 90 90 90 90 90 90 8d 46 3f 85 f6 41 89 f0 55 44 0f 48 c0 31 d2 41 c1 f8 06 <48> 89 e5 45 85 c0 7e 2a 31 d2 48 83 3f 00 48 89 f9 74 17 eb 60 [ 1439.148005] Call Trace: [ 1439.148005] [] ? flush_tlb_others_ipi+0x110/0x140 [ 1439.148005] [] native_flush_tlb_others+0x9/0x10 [ 1439.148005] [] flush_tlb_page+0x5a/0xa0 [ 1439.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1439.148005] [] do_wp_page+0x469/0x7e0 [ 1439.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1439.148005] [] handle_mm_fault+0x178/0x2e0 [ 1439.148005] [] do_page_fault+0x1e5/0x490 [ 1439.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1439.148005] [] page_fault+0x25/0x30 [ 1467.146085] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068] [ 1467.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1467.148005] irq event stamp: 151493442 [ 1467.148005] hardirqs last enabled at (151493441): [] restore_args+0x0/0x30 [ 1467.148005] hardirqs last disabled at (151493442): [] apic_timer_interrupt+0x6e/0x80 [ 1467.148005] softirqs last enabled at (151493440): [] __do_softirq+0x1a1/0x200 [ 1467.148005] softirqs last disabled at (151493435): [] call_softirq+0x1c/0x30 [ 1467.148005] CPU 14 [ 1467.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1467.148005] [ 1467.148005] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1467.148005] RIP: 0010:[] [] flush_tlb_others_ipi+0x102/0x140 [ 1467.148005] RSP: 0000:ffff881147bfdc48 EFLAGS: 00000246 [ 1467.148005] RAX: 0000000000000000 RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1467.148005] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1467.148005] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1467.148005] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdbb8 [ 1467.148005] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1467.148005] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1467.148005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1467.148005] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1467.148005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1467.148005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1467.148005] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1467.148005] Stack: [ 1467.148005] ffff881147bfdcb8 ffff881146a20d80 00007fff56156db8 ffff881146a20de0 [ 1467.148005] ffff88114739f818 ffff881147776ab0 ffff881147bfdc88 ffffffff81033e29 [ 1467.148005] ffff881147bfdcb8 ffffffff81033f2a ffff881146a20df8 0000000000000001 [ 1467.148005] Call Trace: [ 1467.148005] [] native_flush_tlb_others+0x9/0x10 [ 1467.148005] [] flush_tlb_page+0x5a/0xa0 [ 1467.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1467.148005] [] do_wp_page+0x469/0x7e0 [ 1467.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1467.148005] [] handle_mm_fault+0x178/0x2e0 [ 1467.148005] [] do_page_fault+0x1e5/0x490 [ 1467.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1467.148005] [] page_fault+0x25/0x30 [ 1467.148005] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 <8b> 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 [ 1467.148005] Call Trace: [ 1467.148005] [] native_flush_tlb_others+0x9/0x10 [ 1467.148005] [] flush_tlb_page+0x5a/0xa0 [ 1467.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1467.148005] [] do_wp_page+0x469/0x7e0 [ 1467.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1467.148005] [] handle_mm_fault+0x178/0x2e0 [ 1467.148005] [] do_page_fault+0x1e5/0x490 [ 1467.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1467.148005] [] page_fault+0x25/0x30 [ 1495.146085] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068] [ 1495.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1495.148005] irq event stamp: 151556262 [ 1495.148005] hardirqs last enabled at (151556261): [] restore_args+0x0/0x30 [ 1495.148005] hardirqs last disabled at (151556262): [] apic_timer_interrupt+0x6e/0x80 [ 1495.148005] softirqs last enabled at (151556260): [] __do_softirq+0x1a1/0x200 [ 1495.148005] softirqs last disabled at (151556255): [] call_softirq+0x1c/0x30 [ 1495.148005] CPU 14 [ 1495.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1495.148005] [ 1495.148005] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1495.148005] RIP: 0010:[] [] flush_tlb_others_ipi+0x102/0x140 [ 1495.148005] RSP: 0000:ffff881147bfdc48 EFLAGS: 00000246 [ 1495.148005] RAX: 0000000000000000 RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1495.148005] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1495.148005] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1495.148005] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdbb8 [ 1495.148005] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1495.148005] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1495.148005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1495.148005] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1495.148005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1495.148005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1495.148005] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1495.148005] Stack: [ 1495.148005] ffff881147bfdcb8 ffff881146a20d80 00007fff56156db8 ffff881146a20de0 [ 1495.148005] ffff88114739f818 ffff881147776ab0 ffff881147bfdc88 ffffffff81033e29 [ 1495.148005] ffff881147bfdcb8 ffffffff81033f2a ffff881146a20df8 0000000000000001 [ 1495.148005] Call Trace: [ 1495.148005] [] native_flush_tlb_others+0x9/0x10 [ 1495.148005] [] flush_tlb_page+0x5a/0xa0 [ 1495.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1495.148005] [] do_wp_page+0x469/0x7e0 [ 1495.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1495.148005] [] handle_mm_fault+0x178/0x2e0 [ 1495.148005] [] do_page_fault+0x1e5/0x490 [ 1495.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1495.148005] [] page_fault+0x25/0x30 [ 1495.148005] Code: c3 0f 1f 84 00 00 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 <8b> 35 00 4f 9b 00 4c 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 [ 1495.148005] Call Trace: [ 1495.148005] [] native_flush_tlb_others+0x9/0x10 [ 1495.148005] [] flush_tlb_page+0x5a/0xa0 [ 1495.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1495.148005] [] do_wp_page+0x469/0x7e0 [ 1495.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1495.148005] [] handle_mm_fault+0x178/0x2e0 [ 1495.148005] [] do_page_fault+0x1e5/0x490 [ 1495.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1495.148005] [] page_fault+0x25/0x30 [ 1523.146085] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068] [ 1523.148004] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1523.148004] irq event stamp: 151619118 [ 1523.148004] hardirqs last enabled at (151619117): [] restore_args+0x0/0x30 [ 1523.148004] hardirqs last disabled at (151619118): [] apic_timer_interrupt+0x6e/0x80 [ 1523.148004] softirqs last enabled at (151619116): [] __do_softirq+0x1a1/0x200 [ 1523.148004] softirqs last disabled at (151619111): [] call_softirq+0x1c/0x30 [ 1523.148004] CPU 14 [ 1523.148004] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1523.148004] [ 1523.148004] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1523.148004] RIP: 0010:[] [] __bitmap_empty+0x77/0x90 [ 1523.148004] RSP: 0000:ffff881147bfdc38 EFLAGS: 00000216 [ 1523.148004] RAX: 000000000000ffff RBX: ffff881147bfdbb8 RCX: 0000000000000010 [ 1523.148004] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1523.148004] RBP: ffff881147bfdc38 R08: 0000000000000000 R09: 0000000000000000 [ 1523.148004] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfc000 [ 1523.148004] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 [ 1523.148004] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1523.148004] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1523.148004] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1523.148004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1523.148004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1523.148004] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1523.148004] Stack: [ 1523.148004] ffff881147bfdc78 ffffffff81033df0 ffff881147bfdcb8 ffff881146a20d80 [ 1523.148004] 00007fff56156db8 ffff881146a20de0 ffff88114739f818 ffff881147776ab0 [ 1523.148004] ffff881147bfdc88 ffffffff81033e29 ffff881147bfdcb8 ffffffff81033f2a [ 1523.148004] Call Trace: [ 1523.148004] [] flush_tlb_others_ipi+0x110/0x140 [ 1523.148004] [] native_flush_tlb_others+0x9/0x10 [ 1523.148004] [] flush_tlb_page+0x5a/0xa0 [ 1523.148004] [] ptep_set_access_flags+0x4d/0x70 [ 1523.148004] [] do_wp_page+0x469/0x7e0 [ 1523.148004] [] handle_pte_fault+0x19d/0x1e0 [ 1523.148004] [] handle_mm_fault+0x178/0x2e0 [ 1523.148004] [] do_page_fault+0x1e5/0x490 [ 1523.148004] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1523.148004] [] page_fault+0x25/0x30 [ 1523.148004] Code: 00 00 75 08 c9 c3 66 0f 1f 44 00 00 89 f0 48 63 d2 c1 f8 1f c1 e8 1a 8d 0c 06 83 e1 3f 29 c1 b8 01 00 00 00 48 d3 e0 48 83 e8 01 <48> 85 04 d7 c9 0f 94 c0 0f b6 c0 c3 0f 1f 44 00 00 31 c0 c9 c3 [ 1523.148004] Call Trace: [ 1523.148004] [] flush_tlb_others_ipi+0x110/0x140 [ 1523.148004] [] native_flush_tlb_others+0x9/0x10 [ 1523.148004] [] flush_tlb_page+0x5a/0xa0 [ 1523.148004] [] ptep_set_access_flags+0x4d/0x70 [ 1523.148004] [] do_wp_page+0x469/0x7e0 [ 1523.148004] [] handle_pte_fault+0x19d/0x1e0 [ 1523.148004] [] handle_mm_fault+0x178/0x2e0 [ 1523.148004] [] do_page_fault+0x1e5/0x490 [ 1523.148004] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1523.148004] [] page_fault+0x25/0x30 [ 1551.146087] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068] [ 1551.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1551.148005] irq event stamp: 151681846 [ 1551.148005] hardirqs last enabled at (151681845): [] restore_args+0x0/0x30 [ 1551.148005] hardirqs last disabled at (151681846): [] apic_timer_interrupt+0x6e/0x80 [ 1551.148005] softirqs last enabled at (151681844): [] __do_softirq+0x1a1/0x200 [ 1551.148005] softirqs last disabled at (151681839): [] call_softirq+0x1c/0x30 [ 1551.148005] CPU 14 [ 1551.148005] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1551.148005] [ 1551.148005] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1551.148005] RIP: 0010:[] [] flush_tlb_others_ipi+0x108/0x140 [ 1551.148005] RSP: 0000:ffff881147bfdc48 EFLAGS: 00000246 [ 1551.148005] RAX: 0000000000000000 RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1551.148005] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1551.148005] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1551.148005] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdbb8 [ 1551.148005] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1551.148005] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1551.148005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1551.148005] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1551.148005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1551.148005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1551.148005] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1551.148005] Stack: [ 1551.148005] ffff881147bfdcb8 ffff881146a20d80 00007fff56156db8 ffff881146a20de0 [ 1551.148005] ffff88114739f818 ffff881147776ab0 ffff881147bfdc88 ffffffff81033e29 [ 1551.148005] ffff881147bfdcb8 ffffffff81033f2a ffff881146a20df8 0000000000000001 [ 1551.148005] Call Trace: [ 1551.148005] [] native_flush_tlb_others+0x9/0x10 [ 1551.148005] [] flush_tlb_page+0x5a/0xa0 [ 1551.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1551.148005] [] do_wp_page+0x469/0x7e0 [ 1551.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1551.148005] [] handle_mm_fault+0x178/0x2e0 [ 1551.148005] [] do_page_fault+0x1e5/0x490 [ 1551.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1551.148005] [] page_fault+0x25/0x30 [ 1551.148005] Code: 00 00 00 48 8b 05 91 14 9b 00 41 8d b7 cf 00 00 00 4c 89 e7 ff 90 d0 00 00 00 eb 09 0f 1f 80 00 00 00 00 f3 90 8b 35 00 4f 9b 00 <4c> 89 e7 e8 80 7a 22 00 85 c0 74 ec eb 84 66 2e 0f 1f 84 00 00 [ 1551.148005] Call Trace: [ 1551.148005] [] native_flush_tlb_others+0x9/0x10 [ 1551.148005] [] flush_tlb_page+0x5a/0xa0 [ 1551.148005] [] ptep_set_access_flags+0x4d/0x70 [ 1551.148005] [] do_wp_page+0x469/0x7e0 [ 1551.148005] [] handle_pte_fault+0x19d/0x1e0 [ 1551.148005] [] handle_mm_fault+0x178/0x2e0 [ 1551.148005] [] do_page_fault+0x1e5/0x490 [ 1551.148005] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1551.148005] [] page_fault+0x25/0x30 [ 1579.146085] BUG: soft lockup - CPU#14 stuck for 22s! [udevd:1068] [ 1579.148003] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1579.148003] irq event stamp: 151744636 [ 1579.148003] hardirqs last enabled at (151744635): [] restore_args+0x0/0x30 [ 1579.148003] hardirqs last disabled at (151744636): [] apic_timer_interrupt+0x6e/0x80 [ 1579.148003] softirqs last enabled at (151744634): [] __do_softirq+0x1a1/0x200 [ 1579.148003] softirqs last disabled at (151744629): [] call_softirq+0x1c/0x30 [ 1579.148003] CPU 14 [ 1579.148003] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1579.148003] [ 1579.148003] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1579.148003] RIP: 0010:[] [] __bitmap_empty+0x82/0x90 [ 1579.148003] RSP: 0000:ffff881147bfdc40 EFLAGS: 00000206 [ 1579.148003] RAX: 0000000000000000 RBX: ffffffff81460ab4 RCX: 0000000000000010 [ 1579.148003] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffffff81bbcfc8 [ 1579.148003] RBP: ffff881147bfdc78 R08: 0000000000000000 R09: 0000000000000000 [ 1579.148003] R10: 0000000000000002 R11: ffff8811475a8580 R12: ffff881147bfdbb8 [ 1579.148003] R13: ffff8808ca7e0c80 R14: ffff881147bfc000 R15: 0000000000000000 [ 1579.148003] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1579.148003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1579.148003] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1579.148003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1579.148003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1579.148003] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1579.148003] Stack: [ 1579.148003] ffffffff81033df0 ffff881147bfdcb8 ffff881146a20d80 00007fff56156db8 [ 1579.148003] ffff881146a20de0 ffff88114739f818 ffff881147776ab0 ffff881147bfdc88 [ 1579.148003] ffffffff81033e29 ffff881147bfdcb8 ffffffff81033f2a ffff881146a20df8 [ 1579.148003] Call Trace: [ 1579.148003] [] ? flush_tlb_others_ipi+0x110/0x140 [ 1579.148003] [] native_flush_tlb_others+0x9/0x10 [ 1579.148003] [] flush_tlb_page+0x5a/0xa0 [ 1579.148003] [] ptep_set_access_flags+0x4d/0x70 [ 1579.148003] [] do_wp_page+0x469/0x7e0 [ 1579.148003] [] handle_pte_fault+0x19d/0x1e0 [ 1579.148003] [] handle_mm_fault+0x178/0x2e0 [ 1579.148003] [] do_page_fault+0x1e5/0x490 [ 1579.148003] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1579.148003] [] page_fault+0x25/0x30 [ 1579.148003] Code: 00 89 f0 48 63 d2 c1 f8 1f c1 e8 1a 8d 0c 06 83 e1 3f 29 c1 b8 01 00 00 00 48 d3 e0 48 83 e8 01 48 85 04 d7 c9 0f 94 c0 0f b6 c0 0f 1f 44 00 00 31 c0 c9 c3 0f 1f 40 00 8d 46 3f 85 f6 41 89 [ 1579.148003] Call Trace: [ 1579.148003] [] ? flush_tlb_others_ipi+0x110/0x140 [ 1579.148003] [] native_flush_tlb_others+0x9/0x10 [ 1579.148003] [] flush_tlb_page+0x5a/0xa0 [ 1579.148003] [] ptep_set_access_flags+0x4d/0x70 [ 1579.148003] [] do_wp_page+0x469/0x7e0 [ 1579.148003] [] handle_pte_fault+0x19d/0x1e0 [ 1579.148003] [] handle_mm_fault+0x178/0x2e0 [ 1579.148003] [] do_page_fault+0x1e5/0x490 [ 1579.148003] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1579.148003] [] page_fault+0x25/0x30 [ 1579.636006] INFO: rcu_sched detected stall on CPU 14 (t=65032 jiffies) [ 1579.636006] sending NMI to all CPUs: [ 1579.644010] INFO: rcu_sched detected stalls on CPUs/tasks: { 14} (detected by 0, t=65034 jiffies) [ 1579.655327] NMI backtrace for cpu 0 [ 1579.656012] CPU 0 [ 1579.656012] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1579.656012] [ 1579.656012] Pid: 9736, comm: make Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1579.656012] RIP: 0010:[] [] generic_file_aio_read+0x62/0x280 [ 1579.656012] RSP: 0018:ffff8807ce195b10 EFLAGS: 00000292 [ 1579.656012] RAX: ffff8807ce190800 RBX: ffff8807ce195c38 RCX: ffff8807ce195ab0 [ 1579.656012] RDX: ffff8807ce195ac0 RSI: ffff8807ce195ab0 RDI: ffff8807ce195aa8 [ 1579.656012] RBP: ffff8807ce195b38 R08: 0000000000000040 R09: ffff8807ce1910b8 [ 1579.656012] R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000001 [ 1579.656012] R13: 00000000000001f8 R14: ffff8807ce18e6c0 R15: ffff8807ce195ad8 [ 1579.656012] FS: 00002b9180bbe700(0000) GS:ffff8808ffc00000(0000) knlGS:0000000000000000 [ 1579.656012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1579.656012] CR2: 00002ae261975280 CR3: 0000001146873000 CR4: 00000000000006f0 [ 1579.656012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1579.656012] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1579.656012] Process make (pid: 9736, threadinfo ffff8807ce194000, task ffff8807ce190800) [ 1579.656012] Stack: [ 1579.656012] ffff8807ce195b48 ffff8807ce195c28 ffff8807ce18e6c0 ffff8807ce195ca0 [ 1579.656012] ffff8807ce195f58 ffff8807ce195c58 ffffffff81160f79 ffff8807ce1910b8 [ 1579.656012] ffff8807ce190800 0000000000000000 ffffffff00000001 ffff8807ce18e6c0 [ 1579.656012] Call Trace: [ 1579.656012] [] do_sync_read+0xd9/0x120 [ 1579.656012] [] ? common_file_perm+0x8d/0x110 [ 1579.656012] [] ? security_file_permission+0x93/0xa0 [ 1579.656012] [] vfs_read+0xc8/0x130 [ 1579.656012] [] kernel_read+0x44/0x60 [ 1579.656012] [] load_elf_binary+0x173/0x1060 [ 1579.656012] [] ? __lock_acquire+0x301/0x520 [ 1579.656012] [] ? search_binary_handler+0xfc/0x360 [ 1579.656012] [] ? load_elf_interp+0x5e0/0x5e0 [ 1579.656012] [] ? load_elf_interp+0x5e0/0x5e0 [ 1579.656012] [] search_binary_handler+0x106/0x360 [ 1579.656012] [] ? search_binary_handler+0x4e/0x360 [ 1579.656012] [] do_execve_common+0x27d/0x320 [ 1579.656012] [] do_execve+0x3a/0x40 [ 1579.656012] [] sys_execve+0x49/0x70 [ 1579.656012] [] stub_execve+0x6c/0xc0 [ 1579.656012] Code: c8 4c 8b 77 20 4c 89 e7 48 89 85 58 ff ff ff 48 c7 45 c8 00 00 00 00 e8 8d d8 ff ff 4c 63 e8 4d 85 ed 74 15 48 81 c4 88 00 00 00 <4c> 89 e8 5b 41 5c 41 5d 41 5e 41 5f c9 c3 48 8d bd 70 ff ff ff [ 1579.656012] Call Trace: [ 1579.656012] [] do_sync_read+0xd9/0x120 [ 1579.656012] [] ? common_file_perm+0x8d/0x110 [ 1579.656012] [] ? security_file_permission+0x93/0xa0 [ 1579.656012] [] vfs_read+0xc8/0x130 [ 1579.656012] [] kernel_read+0x44/0x60 [ 1579.656012] [] load_elf_binary+0x173/0x1060 [ 1579.656012] [] ? __lock_acquire+0x301/0x520 [ 1579.656012] [] ? search_binary_handler+0xfc/0x360 [ 1579.656012] [] ? load_elf_interp+0x5e0/0x5e0 [ 1579.656012] [] ? load_elf_interp+0x5e0/0x5e0 [ 1579.656012] [] search_binary_handler+0x106/0x360 [ 1579.656012] [] ? search_binary_handler+0x4e/0x360 [ 1579.656012] [] do_execve_common+0x27d/0x320 [ 1579.656012] [] do_execve+0x3a/0x40 [ 1579.656012] [] sys_execve+0x49/0x70 [ 1579.656012] [] stub_execve+0x6c/0xc0 [ 1579.636006] NMI backtrace for cpu 14 [ 1579.636006] CPU 14 [ 1579.636006] Modules linked in: ipv6 cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf microcode fuse loop dm_mod usbhid i2c_i801 ioatdma i2c_core hid cdc_ether usbnet bnx2 serio_raw mii i7core_edac sg iTCO_wdt dca shpchp iTCO_vendor_support pcspkr mptctl edac_core rtc_cmos tpm_tis tpm tpm_bios button pci_hotplug uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif edd ext3 mbcache jbd fan processor mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal thermal_sys hwmon [ 1579.636006] [ 1579.636006] Pid: 1068, comm: udevd Tainted: G W 3.2.0-rc7-0.0.0.28.36b5ec9-default #1 IBM IBM System x -[7870C4Q]-/68Y8033 [ 1579.636006] RIP: 0010:[] [] delay_tsc+0x51/0xa0 [ 1579.636006] RSP: 0000:ffff88117fd83d30 EFLAGS: 00000046 [ 1579.636006] RAX: 0000000000000042 RBX: 00000000000470af RCX: ffffffffd032bab4 [ 1579.636006] RDX: 00000000d032bab4 RSI: 000000000000000f RDI: 00000000000470af [ 1579.636006] RBP: ffff88117fd83d68 R08: 0000000000000010 R09: ffffffff819e7660 [ 1579.636006] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000001000 [ 1579.636006] R13: 0000000000000002 R14: 000000000000000e R15: 000000000000000e [ 1579.636006] FS: 00007fbaa8d43780(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000 [ 1579.636006] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1579.636006] CR2: 00007fff56156db8 CR3: 00000011473a0000 CR4: 00000000000006e0 [ 1579.636006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1579.636006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1579.636006] Process udevd (pid: 1068, threadinfo ffff881147bfc000, task ffff881146fe96c0) [ 1579.636006] Stack: [ 1579.636006] ffffffffd032ba72 0000000e00000046 0000000000000000 0000000000001000 [ 1579.636006] 0000000000000002 000000000000cbc0 0000000000000002 ffff88117fd83d78 [ 1579.636006] ffffffff81258e1f ffff88117fd83d98 ffffffff81021aaa 000000000000000f [ 1579.636006] Call Trace: [ 1579.636006] [ 1579.636006] [] __const_udelay+0x2f/0x40 [ 1579.636006] [] native_safe_apic_wait_icr_idle+0x1a/0x50 [ 1579.636006] [] default_send_IPI_mask_sequence_phys+0xdd/0x130 [ 1579.636006] [] physflat_send_IPI_all+0x14/0x20 [ 1579.636006] [] arch_trigger_all_cpu_backtrace+0x67/0xb0 [ 1579.636006] [] __rcu_pending+0x119/0x280 [ 1579.636006] [] rcu_check_callbacks+0xaa/0x1b0 [ 1579.636006] [] update_process_times+0x41/0x80 [ 1579.636006] [] tick_sched_timer+0x5f/0xc0 [ 1579.636006] [] ? tick_nohz_handler+0x100/0x100 [ 1579.636006] [] __run_hrtimer+0xd1/0x1d0 [ 1579.636006] [] hrtimer_interrupt+0xc7/0x1f0 [ 1579.636006] [] smp_apic_timer_interrupt+0x64/0xa0 [ 1579.636006] [] apic_timer_interrupt+0x73/0x80 [ 1579.636006] [ 1579.636006] [] ? __bitmap_empty+0x7b/0x90 [ 1579.636006] [] flush_tlb_others_ipi+0x110/0x140 [ 1579.636006] [] native_flush_tlb_others+0x9/0x10 [ 1579.636006] [] flush_tlb_page+0x5a/0xa0 [ 1579.636006] [] ptep_set_access_flags+0x4d/0x70 [ 1579.636006] [] do_wp_page+0x469/0x7e0 [ 1579.636006] [] handle_pte_fault+0x19d/0x1e0 [ 1579.636006] [] handle_mm_fault+0x178/0x2e0 [ 1579.636006] [] do_page_fault+0x1e5/0x490 [ 1579.636006] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 1579.636006] [] page_fault+0x25/0x30 [ 1579.636006] Code: db ff 66 90 48 98 44 89 75 d4 48 89 45 c8 eb 1b 66 2e 0f 1f 84 00 00 00 00 00 f3 90 65 44 8b 3c 25 b0 cb 00 00 44 3b 7d d4 75 2b <66> 66 90 0f ae e8 e8 b4 15 db ff 66 90 48 63 c8 48 89 c8 48 2b [ 1579.636006] Call Trace: [ 1579.636006] [] __const_udelay+0x2f/0x40 [ 1579.636006] [] native_safe_apic_wait_icr_idle+0x1a/0x50 [ 1579.636006] [] default_send_IPI_mask_sequence_phys+0xdd/0x130 [ 1579.636006] [] physflat_send_IPI_all+0x14/0x20 [ 1579.636006] [] arch_trigger_all_cpu_backtrace+0x67/0xb0 [ 1579.636006] [] __rcu_pending+0x119/0x280 [ 1579.636006] [] rcu_check_callbacks+0xaa/0x1b0 [ 1579.636006] [] update_process_times+0x41/0x80 [ 1579.636006] [] tick_sched_timer+0x5f/0xc0 [ 1579.636006] [] ? tick_nohz_handler+0x100/0x100 [ 1579.636006] [] __run_hrtimer+0xd1/0x1d0 [ 1579.636006] [] hrtimer_interrupt+0xc7/0x1f0 [ 1579.636006] [] smp_apic_timer_interrupt+0x64/0xa0 [ 1579.636006] [] apic_timer_interrupt+0x73/0x80 [ 1579.636006] [] ? __bitmap_empty+0x7b/0x90 [ 1579.636006] [] flush_tlb_others_ipi+0x110/0x140 [ 1579.636006] [] native_flush_tlb_others+0x9/0x10 [ 1579.636006] [] flush_tlb_page+0x5a/0xa0 [ 1579.636006] [] ptep_set_access_flags+0x4d/0x70 [ 1579.636006] [