* mm: hangs in collapse_huge_page
@ 2014-04-16 2:06 Sasha Levin
2014-04-24 16:46 ` Sasha Levin
2014-04-30 15:42 ` Kirill A. Shutemov
0 siblings, 2 replies; 8+ messages in thread
From: Sasha Levin @ 2014-04-16 2:06 UTC (permalink / raw)
To: linux-mm; +Cc: LKML, Dave Jones, Andrew Morton
Hi all,
I often see hung task triggering in khugepaged within collapse_huge_page().
I've initially assumed the case may be that the guests are too loaded and
the warning occurs because of load, but after increasing the timeout to
1200 sec I still see the warning.
Here's what I get in the log, up to the point the guest reboots:
[ 2406.651012] INFO: task khugepaged:3562 blocked for more than 1200 seconds.
[ 2406.653331] Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.656549] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2406.662008] khugepaged D ffff88019fa1e4b8 5408 3562 2 0x00000000
[ 2406.663773] ffff8803d6afbb08 0000000000000002 ffffffffb056e740 ffff8803d6b28000
[ 2406.666549] ffff8803d6afbfd8 00000000001d79c0 00000000001d79c0 00000000001d79c0
[ 2406.669558] ffff88019fac8000 ffff8803d6b28000 ffff8803d6afbaf8 ffff8803d6b28000
[ 2406.673316] Call Trace:
[ 2406.676062] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:819 include/linux/spinlock_api_smp.h:168 kernel/locking/spinlock.c:199)
[ 2406.678384] schedule (kernel/sched/core.c:2750)
[ 2406.680418] rwsem_down_write_failed (kernel/locking/rwsem-xadd.c:240)
[ 2406.683053] ? get_parent_ip (kernel/sched/core.c:2472)
[ 2406.684950] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:106)
[ 2406.686998] ? khugepaged_do_scan (arch/x86/include/asm/atomic.h:26 mm/huge_memory.c:1989 mm/huge_memory.c:2591 mm/huge_memory.c:2709)
[ 2406.689314] ? lock_contended (kernel/locking/lockdep.c:3734 kernel/locking/lockdep.c:3812)
[ 2406.691611] ? down_write (kernel/locking/rwsem.c:50 (discriminator 2))
[ 2406.693657] ? collapse_huge_page.isra.30 (arch/x86/include/asm/atomic.h:26 mm/huge_memory.c:1989 mm/huge_memory.c:2378)
[ 2406.696508] collapse_huge_page.isra.30 (arch/x86/include/asm/atomic.h:26 mm/huge_memory.c:1989 mm/huge_memory.c:2378)
[ 2406.699123] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
[ 2406.701570] ? put_lock_stats.isra.12 (arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254)
[ 2406.704058] ? khugepaged_do_scan (include/linux/spinlock.h:343 mm/huge_memory.c:2533 mm/huge_memory.c:2629 mm/huge_memory.c:2709)
[ 2406.706517] ? get_parent_ip (kernel/sched/core.c:2472)
[ 2406.708746] khugepaged_do_scan (mm/huge_memory.c:2633 mm/huge_memory.c:2709)
[ 2406.711252] khugepaged (include/linux/freezer.h:64 mm/huge_memory.c:2722 mm/huge_memory.c:2747)
[ 2406.713333] ? bit_waitqueue (kernel/sched/wait.c:291)
[ 2406.715703] ? khugepaged_do_scan (mm/huge_memory.c:2739)
[ 2406.718031] kthread (kernel/kthread.c:210)
[ 2406.720004] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2406.722590] ret_from_fork (arch/x86/kernel/entry_64.S:555)
[ 2406.724780] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2406.727349] 1 lock held by khugepaged/3562:
[ 2406.728975] #0: (&mm->mmap_sem){++++++}, at: collapse_huge_page.isra.30 (arch/x86/include/asm/atomic.h:26 mm/huge_memory.c:1989 mm/huge_memory.c:2378)
[ 2406.733683] BUG: using __this_cpu_write() in preemptible [00000000] code: khungtaskd/3540
[ 2406.735837] caller is __this_cpu_preempt_check+0x13/0x20
[ 2406.737191] CPU: 21 PID: 3540 Comm: khungtaskd Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.739790] ffffffffb17b99f5 ffff880436b5bd58 ffffffffb05223a7 0000000000000007
[ 2406.741834] 0000000000000015 ffff880436b5bd88 ffffffffadb306d9 00000000000004b0
[ 2406.744275] 00000000003ffecf 00000000000002cf ffff8803d6b28000 ffff880436b5bd98
[ 2406.746446] Call Trace:
[ 2406.747118] dump_stack (lib/dump_stack.c:52)
[ 2406.748927] check_preemption_disabled (arch/x86/include/asm/preempt.h:80 lib/smp_processor_id.c:49)
[ 2406.750736] __this_cpu_preempt_check (lib/smp_processor_id.c:63)
[ 2406.752708] touch_softlockup_watchdog (kernel/watchdog.c:142)
[ 2406.754405] touch_nmi_watchdog (kernel/watchdog.c:170)
[ 2406.756063] watchdog (kernel/hung_task.c:122 kernel/hung_task.c:180 kernel/hung_task.c:236)
[ 2406.757444] ? watchdog (include/linux/rcupdate.h:800 kernel/hung_task.c:169 kernel/hung_task.c:236)
[ 2406.759472] ? reset_hung_task_detector (kernel/hung_task.c:224)
[ 2406.766326] kthread (kernel/kthread.c:210)
[ 2406.769513] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2406.773602] ret_from_fork (arch/x86/kernel/entry_64.S:555)
[ 2406.776132] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2406.778998] sending NMI to all CPUs:
[ 2406.781135] NMI backtrace for cpu 21
[ 2406.783323] CPU: 21 PID: 3540 Comm: khungtaskd Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.789592] task: ffff880436a88000 ti: ffff880436b5a000 task.ti: ffff880436b5a000
[ 2406.790928] RIP: native_write_msr_safe (arch/x86/include/asm/msr.h:95)
[ 2406.790928] RSP: 0018:ffff880436b5bd28 EFLAGS: 00000082
[ 2406.790928] RAX: 0000000000000400 RBX: 0000000000000015 RCX: 0000000000000830
[ 2406.790928] RDX: 0000000000000015 RSI: 0000000000000400 RDI: 0000000000000830
[ 2406.790928] RBP: ffff880436b5bd28 R08: ffffffffb30bc580 R09: 0000000000000000
[ 2406.790928] R10: 0000000000000001 R11: 3a73555043206c6c R12: ffffffffb30bc580
[ 2406.790928] R13: 0000000000000015 R14: 0000000000080000 R15: 000000000000b022
[ 2406.790928] FS: 0000000000000000(0000) GS:ffff880467000000(0000) knlGS:0000000000000000
[ 2406.790928] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.790928] CR2: 00007f07ed23d099 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.790928] Stack:
[ 2406.790928] ffff880436b5bd88 ffffffffad0b3ac2 ffff880436b5bd98 0000000000000282
[ 2406.790928] 0000000000000002 0000000200000008 ffff880436b5bda8 0000000000002710
[ 2406.790928] 00000000003ffecf 00000000000002cf ffff8803d6b28000 ffff8803d6b28000
[ 2406.790928] Call Trace:
[ 2406.790928] __x2apic_send_IPI_mask (arch/x86/include/asm/paravirt.h:133 arch/x86/include/asm/apic.h:169 arch/x86/include/asm/x2apic.h:26 arch/x86/kernel/apic/x2apic_phys.c:52)
[ 2406.790928] x2apic_send_IPI_all (arch/x86/kernel/apic/x2apic_phys.c:77)
[ 2406.790928] arch_trigger_all_cpu_backtrace (include/linux/bitmap.h:265 include/linux/cpumask.h:443 arch/x86/kernel/apic/hw_nmi.c:54)
[ 2406.790928] watchdog (kernel/hung_task.c:124 kernel/hung_task.c:180 kernel/hung_task.c:236)
[ 2406.790928] ? watchdog (include/linux/rcupdate.h:800 kernel/hung_task.c:169 kernel/hung_task.c:236)
[ 2406.790928] ? reset_hung_task_detector (kernel/hung_task.c:224)
[ 2406.790928] kthread (kernel/kthread.c:210)
[ 2406.790928] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2406.790928] ret_from_fork (arch/x86/kernel/entry_64.S:555)
[ 2406.790928] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2406.790928] Code: 00 55 89 f9 48 89 e5 0f 32 31 ff 89 c0 48 c1 e2 20 89 3e 48 09 c2 5d 48 89 d0 c3 66 0f 1f 44 00 00 55 89 f0 89 f9 48 89 e5 0f 30 <31> c0 5d c3 66 90 55 89 f9 48 89 e5 0f 33 89 c0 48 c1 e2 20 5d
[ 2406.732546] NMI backtrace for cpu 2
[ 2406.790928] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 129.351 msecs
[ 2406.732546] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.732546] task: ffff8801ee8f8000 ti: ffff88002ad6e000 task.ti: ffff88002ad6e000
[ 2406.732546] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.732546] RSP: 0018:ffff88002ad6fe28 EFLAGS: 00000282
[ 2406.732546] RAX: ffff8801ee8f8000 RBX: ffff88002ad6ffd8 RCX: 0000000000000000
[ 2406.732546] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.732546] RBP: ffff88002ad6fe28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.732546] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.732546] R13: ffff88002ad6e000 R14: ffffffffb30bc580 R15: ffff88002ad6ffd8
[ 2406.732546] FS: 0000000000000000(0000) GS:ffff880095000000(0000) knlGS:0000000000000000
[ 2406.732546] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.732546] CR2: 00007f026359f170 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.732546] Stack:
[ 2406.732546] ffff88002ad6fe48 ffffffffad07c04d ffff88002ad6ffd8 0000000000000000
[ 2406.732546] ffff88002ad6fe58 ffffffffad07cd4f ffff88002ad6fed8 ffffffffad1b73f5
[ 2406.732546] 0000000000000000 b14dfbbab381bb01 ffff88002ad6ffd8 000000000000f000
[ 2406.732546] Call Trace:
[ 2406.732546] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.732546] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.732546] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.732546] cpu_startup_entry (??:?)
[ 2406.732546] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.732546] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.726090] NMI backtrace for cpu 11
[ 2407.051480] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 269.944 msecs
[ 2406.726090] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.726090] task: ffff88002ad88000 ti: ffff88002ad84000 task.ti: ffff88002ad84000
[ 2406.726090] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.726090] RSP: 0018:ffff88002ad85e28 EFLAGS: 00000282
[ 2406.726090] RAX: ffff88002ad88000 RBX: ffff88002ad85fd8 RCX: 0000000000000000
[ 2406.726090] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.726090] RBP: ffff88002ad85e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.726090] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.726090] R13: ffff88002ad84000 R14: ffffffffb30bc580 R15: ffff88002ad85fd8
[ 2406.726090] FS: 0000000000000000(0000) GS:ffff880285000000(0000) knlGS:0000000000000000
[ 2406.726090] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.726090] CR2: 00007f5b4d8d397a CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.726090] Stack:
[ 2406.726090] ffff88002ad85e48 ffffffffad07c04d ffff88002ad85fd8 0000000000000000
[ 2406.726090] ffff88002ad85e58 ffffffffad07cd4f ffff88002ad85ed8 ffffffffad1b73f5
[ 2406.726090] 0000000000000000 e0335f8592de6dea ffff88002ad85fd8 000000000000f000
[ 2406.726090] Call Trace:
[ 2406.726090] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.726090] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.726090] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.726090] cpu_startup_entry (??:?)
[ 2406.726090] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.726090] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.770400] NMI backtrace for cpu 4
[ 2407.227620] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 446.006 msecs
[ 2406.770400] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.770400] task: ffff880436910000 ti: ffff88002ad72000 task.ti: ffff88002ad72000
[ 2406.770400] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.770400] RSP: 0018:ffff88002ad73e28 EFLAGS: 00000282
[ 2406.770400] RAX: ffff880436910000 RBX: ffff88002ad73fd8 RCX: 0000000000000000
[ 2406.770400] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.770400] RBP: ffff88002ad73e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.770400] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.770400] R13: ffff88002ad72000 R14: ffffffffb30bc580 R15: ffff88002ad73fd8
[ 2406.770400] FS: 0000000000000000(0000) GS:ffff880127000000(0000) knlGS:0000000000000000
[ 2406.770400] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.770400] CR2: 00007f0e063095e0 CR3: 000000005e3c5000 CR4: 00000000000006a0
[ 2406.770400] Stack:
[ 2406.770400] ffff88002ad73e48 ffffffffad07c04d ffff88002ad73fd8 0000000000000000
[ 2406.770400] ffff88002ad73e58 ffffffffad07cd4f ffff88002ad73ed8 ffffffffad1b73f5
[ 2406.770400] 0000000000000000 78cea51f19eb6d9a ffff88002ad73fd8 000000000000f000
[ 2406.770400] Call Trace:
[ 2406.770400] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.770400] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.770400] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.770400] cpu_startup_entry (??:?)
[ 2406.770400] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.770400] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.770587] NMI backtrace for cpu 8
[ 2407.392169] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 610.715 msecs
[ 2406.770587] CPU: 8 PID: 0 Comm: swapper/8 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.770587] task: ffff880316910000 ti: ffff88002ad7e000 task.ti: ffff88002ad7e000
[ 2406.770587] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.770587] RSP: 0018:ffff88002ad7fe28 EFLAGS: 00000282
[ 2406.770587] RAX: ffff880316910000 RBX: ffff88002ad7ffd8 RCX: 0000000000000000
[ 2406.770587] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.770587] RBP: ffff88002ad7fe28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.770587] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.770587] R13: ffff88002ad7e000 R14: ffffffffb30bc580 R15: ffff88002ad7ffd8
[ 2406.770587] FS: 0000000000000000(0000) GS:ffff8801ef000000(0000) knlGS:0000000000000000
[ 2406.770587] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.770587] CR2: 00007fd82dbefb80 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.770587] Stack:
[ 2406.770587] ffff88002ad7fe48 ffffffffad07c04d ffff88002ad7ffd8 0000000000000000
[ 2406.770587] ffff88002ad7fe58 ffffffffad07cd4f ffff88002ad7fed8 ffffffffad1b73f5
[ 2406.770587] 0000000000000000 c37c456b12092c72 ffff88002ad7ffd8 000000000000f000
[ 2406.770587] Call Trace:
[ 2406.770587] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.770587] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.770587] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.770587] cpu_startup_entry (??:?)
[ 2406.770587] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.770587] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.776572] NMI backtrace for cpu 9
[ 2407.455680] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 674.457 msecs
[ 2406.776572] CPU: 9 PID: 0 Comm: swapper/9 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.776572] task: ffff880436918000 ti: ffff88002ad80000 task.ti: ffff88002ad80000
[ 2406.776572] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.776572] RSP: 0018:ffff88002ad81e28 EFLAGS: 00000282
[ 2406.776572] RAX: ffff880436918000 RBX: ffff88002ad81fd8 RCX: 0000000000000000
[ 2406.776572] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.776572] RBP: ffff88002ad81e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.776572] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.776572] R13: ffff88002ad80000 R14: ffffffffb30bc580 R15: ffff88002ad81fd8
[ 2406.776572] FS: 0000000000000000(0000) GS:ffff880220e00000(0000) knlGS:0000000000000000
[ 2406.776572] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.776572] CR2: 00007fd9fe85e099 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.776572] Stack:
[ 2406.776572] ffff88002ad81e48 ffffffffad07c04d ffff88002ad81fd8 0000000000000000
[ 2406.776572] ffff88002ad81e58 ffffffffad07cd4f ffff88002ad81ed8 ffffffffad1b73f5
[ 2406.776572] 0000000000000000 982768e623c64a41 ffff88002ad81fd8 000000000000f000
[ 2406.776572] Call Trace:
[ 2406.776572] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.776572] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.776572] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.776572] cpu_startup_entry (??:?)
[ 2406.776572] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.776572] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.773931] NMI backtrace for cpu 16
[ 2407.522960] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 741.709 msecs
[ 2406.773931] CPU: 16 PID: 0 Comm: swapper/16 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.773931] task: ffff880316920000 ti: ffff88002ad9a000 task.ti: ffff88002ad9a000
[ 2406.773931] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.773931] RSP: 0018:ffff88002ad9be28 EFLAGS: 00000282
[ 2406.773931] RAX: ffff880316920000 RBX: ffff88002ad9bfd8 RCX: 0000000000000000
[ 2406.773931] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.773931] RBP: ffff88002ad9be28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.773931] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.773931] R13: ffff88002ad9a000 R14: ffffffffb30bc580 R15: ffff88002ad9bfd8
[ 2406.773931] FS: 0000000000000000(0000) GS:ffff880377000000(0000) knlGS:0000000000000000
[ 2406.773931] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.773931] CR2: 00007faa2c8d59de CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.773931] Stack:
[ 2406.773931] ffff88002ad9be48 ffffffffad07c04d ffff88002ad9bfd8 0000000000000000
[ 2406.773931] ffff88002ad9be58 ffffffffad07cd4f ffff88002ad9bed8 ffffffffad1b73f5
[ 2406.773931] 0000000000000000 fc58c7b9f54f958d ffff88002ad9bfd8 000000000000f000
[ 2406.773931] Call Trace:
[ 2406.773931] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.773931] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.773931] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.773931] cpu_startup_entry (??:?)
[ 2406.773931] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.773931] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.791002] NMI backtrace for cpu 7
[ 2407.623400] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 842.127 msecs
[ 2406.791002] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.791002] task: ffff8801ee900000 ti: ffff88002ad7c000 task.ti: ffff88002ad7c000
[ 2406.791002] RIP: native_write_msr_safe (arch/x86/include/asm/msr.h:95)
[ 2406.791002] RSP: 0018:ffff88002ad7dcf8 EFLAGS: 00000046
[ 2406.791002] RAX: 000000007fffffff RBX: ffff8801bd00f040 RCX: 0000000000000838
[ 2406.791002] RDX: 0000000000000000 RSI: 000000007fffffff RDI: 0000000000000838
[ 2406.791002] RBP: ffff88002ad7dcf8 R08: 0000000000800000 R09: ffffffffffffffff
[ 2406.791002] R10: 00032cdb7c0be5d4 R11: 0000000000000000 R12: 00000007ff111e51
[ 2406.791002] R13: 0000000000000001 R14: 0000000000000001 R15: ffff8801bd1d0a40
[ 2406.791002] FS: 0000000000000000(0000) GS:ffff8801bd000000(0000) knlGS:0000000000000000
[ 2406.791002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.791002] CR2: 00007f0c5de1e099 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.791002] Stack:
[ 2406.791002] ffff88002ad7dd08 ffffffffad0b3831 ffff88002ad7dd18 ffffffffad0ac09d
[ 2406.791002] ffff88002ad7dd48 ffffffffad1f6bca 0000000000000000 00000238653e3600
[ 2406.791002] 0000000000000001 0000000000000000 ffff88002ad7dd68 ffffffffad1f9600
[ 2406.791002] Call Trace:
[ 2406.791002] native_apic_msr_write (arch/x86/include/asm/paravirt.h:133 arch/x86/include/asm/apic.h:136)
[ 2406.791002] lapic_next_event (arch/x86/kernel/apic/apic.c:483)
[ 2406.791002] clockevents_program_event (kernel/time/clockevents.c:270)
[ 2406.791002] tick_program_event (kernel/time/tick-oneshot.c:32)
[ 2406.791002] hrtimer_force_reprogram (kernel/hrtimer.c:574)
[ 2406.791002] __remove_hrtimer (kernel/hrtimer.c:915)
[ 2406.791002] hrtimer_try_to_cancel (kernel/hrtimer.c:952 kernel/hrtimer.c:1078)
[ 2406.791002] hrtimer_cancel (kernel/hrtimer.c:1100)
[ 2406.791002] tick_nohz_restart (kernel/time/tick-sched.c:841)
[ 2406.791002] tick_nohz_idle_exit (kernel/time/tick-sched.c:879 include/linux/jump_label.h:105 include/linux/context_tracking_state.h:27 include/linux/vtime.h:22 kernel/time/tick-sched.c:887 kernel/time/tick-sched.c:929)
[ 2406.791002] cpu_idle_loop (kernel/sched/idle.c:241)
[ 2406.791002] cpu_startup_entry (??:?)
[ 2406.791002] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.791002] Code: 00 55 89 f9 48 89 e5 0f 32 31 ff 89 c0 48 c1 e2 20 89 3e 48 09 c2 5d 48 89 d0 c3 66 0f 1f 44 00 00 55 89 f0 89 f9 48 89 e5 0f 30 <31> c0 5d c3 66 90 55 89 f9 48 89 e5 0f 33 89 c0 48 c1 e2 20 5d
[ 2406.780287] NMI backtrace for cpu 10
[ 2406.791002] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1087.712 msecs
[ 2406.780287] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.780287] task: ffff880556938000 ti: ffff88002ad82000 task.ti: ffff88002ad82000
[ 2406.780287] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.780287] RSP: 0018:ffff88002ad83e28 EFLAGS: 00000282
[ 2406.780287] RAX: ffff880556938000 RBX: ffff88002ad83fd8 RCX: 0000000000000000
[ 2406.780287] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.780287] RBP: ffff88002ad83e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.780287] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.780287] R13: ffff88002ad82000 R14: ffffffffb30bc580 R15: ffff88002ad83fd8
[ 2406.780287] FS: 0000000000000000(0000) GS:ffff880253000000(0000) knlGS:0000000000000000
[ 2406.780287] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.780287] CR2: 00007fb641e0d070 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.780287] Stack:
[ 2406.780287] ffff88002ad83e48 ffffffffad07c04d ffff88002ad83fd8 0000000000000000
[ 2406.780287] ffff88002ad83e58 ffffffffad07cd4f ffff88002ad83ed8 ffffffffad1b73f5
[ 2406.780287] 0000000000000000 9f3d8b5b532a750a ffff88002ad83fd8 000000000000f000
[ 2406.780287] Call Trace:
[ 2406.780287] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.780287] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.780287] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.780287] cpu_startup_entry (??:?)
[ 2406.780287] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.780287] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.790114] NMI backtrace for cpu 1
[ 2408.073317] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1291.403 msecs
[ 2406.790114] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.790114] task: ffff8800948f0000 ti: ffff88002ad6c000 task.ti: ffff88002ad6c000
[ 2406.790114] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.790114] RSP: 0018:ffff88002ad6de28 EFLAGS: 00000282
[ 2406.790114] RAX: ffff8800948f0000 RBX: ffff88002ad6dfd8 RCX: 0000000000000000
[ 2406.790114] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.790114] RBP: ffff88002ad6de28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.790114] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.790114] R13: ffff88002ad6c000 R14: ffffffffb30bc580 R15: ffff88002ad6dfd8
[ 2406.790114] FS: 0000000000000000(0000) GS:ffff880063000000(0000) knlGS:0000000000000000
[ 2406.790114] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.790114] CR2: 00007faa2c8d59de CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.790114] Stack:
[ 2406.790114] ffff88002ad6de48 ffffffffad07c04d ffff88002ad6dfd8 0000000000000000
[ 2406.790114] ffff88002ad6de58 ffffffffad07cd4f ffff88002ad6ded8 ffffffffad1b73f5
[ 2406.790114] 0000000000000000 070355202ebd0a66 ffff88002ad6dfd8 000000000000f000
[ 2406.790114] Call Trace:
[ 2406.790114] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.790114] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.790114] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.790114] cpu_startup_entry (??:?)
[ 2406.790114] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.790114] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.773449] NMI backtrace for cpu 12
[ 2408.114944] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1333.636 msecs
[ 2406.773449] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.773449] task: ffff88002ad8b000 ti: ffff88002ad86000 task.ti: ffff88002ad86000
[ 2406.773449] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.773449] RSP: 0018:ffff88002ad87e28 EFLAGS: 00000282
[ 2406.773449] RAX: ffff88002ad8b000 RBX: ffff88002ad87fd8 RCX: 0000000000000000
[ 2406.773449] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.773449] RBP: ffff88002ad87e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.773449] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.773449] R13: ffff88002ad86000 R14: ffffffffb30bc580 R15: ffff88002ad87fd8
[ 2406.773449] FS: 0000000000000000(0000) GS:ffff8802b7000000(0000) knlGS:0000000000000000
[ 2406.773449] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.773449] CR2: 00007f5fae0ef170 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.773449] Stack:
[ 2406.773449] ffff88002ad87e48 ffffffffad07c04d ffff88002ad87fd8 0000000000000000
[ 2406.773449] ffff88002ad87e58 ffffffffad07cd4f ffff88002ad87ed8 ffffffffad1b73f5
[ 2406.773449] 0000000000000000 c6410aa475d0509c ffff88002ad87fd8 000000000000f000
[ 2406.773449] Call Trace:
[ 2406.773449] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.773449] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.773449] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.773449] cpu_startup_entry (??:?)
[ 2406.773449] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.773449] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.783731] NMI backtrace for cpu 20
[ 2406.783731] CPU: 20 PID: 0 Comm: swapper/20 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.783731] task: ffff8802b6918000 ti: ffff88002adaa000 task.ti: ffff88002adaa000
[ 2406.783731] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.783731] RSP: 0018:ffff88002adabe28 EFLAGS: 00000282
[ 2406.783731] RAX: ffff8802b6918000 RBX: ffff88002adabfd8 RCX: 0000000000000000
[ 2406.783731] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.783731] RBP: ffff88002adabe28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.783731] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.783731] R13: ffff88002adaa000 R14: ffffffffb30bc580 R15: ffff88002adabfd8
[ 2406.783731] FS: 0000000000000000(0000) GS:ffff880437000000(0000) knlGS:0000000000000000
[ 2406.783731] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.783731] CR2: 00007f9fbf31f170 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.783731] Stack:
[ 2406.783731] ffff88002adabe48 ffffffffad07c04d ffff88002adabfd8 0000000000000000
[ 2406.783731] ffff88002adabe58 ffffffffad07cd4f ffff88002adabed8 ffffffffad1b73f5
[ 2406.783731] 0000000000000000 ec4da5f4ddcd59c7 ffff88002adabfd8 000000000000f000
[ 2406.783731] Call Trace:
[ 2406.783731] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.783731] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.783731] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.783731] cpu_startup_entry (??:?)
[ 2406.783731] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.783731] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.780604] NMI backtrace for cpu 17
[ 2406.780604] CPU: 17 PID: 0 Comm: swapper/17 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.780604] task: ffff880496920000 ti: ffff88002ad9c000 task.ti: ffff88002ad9c000
[ 2406.780604] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.780604] RSP: 0018:ffff88002ad9de28 EFLAGS: 00000282
[ 2406.780604] RAX: ffff880496920000 RBX: ffff88002ad9dfd8 RCX: 0000000000000000
[ 2406.780604] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.780604] RBP: ffff88002ad9de28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.780604] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.780604] R13: ffff88002ad9c000 R14: ffffffffb30bc580 R15: ffff88002ad9dfd8
[ 2406.780604] FS: 0000000000000000(0000) GS:ffff8803a7000000(0000) knlGS:0000000000000000
[ 2406.780604] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.780604] CR2: 00007f446d8cd170 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.780604] Stack:
[ 2406.780604] ffff88002ad9de48 ffffffffad07c04d ffff88002ad9dfd8 0000000000000000
[ 2406.780604] ffff88002ad9de58 ffffffffad07cd4f ffff88002ad9ded8 ffffffffad1b73f5
[ 2406.780604] 0000000000000000 970676ae9739a05a ffff88002ad9dfd8 000000000000f000
[ 2406.780604] Call Trace:
[ 2406.780604] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.780604] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.780604] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.780604] cpu_startup_entry (??:?)
[ 2406.780604] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.780604] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.773106] NMI backtrace for cpu 18
[ 2406.773106] CPU: 18 PID: 0 Comm: swapper/18 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.773106] task: ffff880616928000 ti: ffff88002ad9e000 task.ti: ffff88002ad9e000
[ 2406.773106] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.773106] RSP: 0018:ffff88002ad9fe28 EFLAGS: 00000282
[ 2406.773106] RAX: ffff880616928000 RBX: ffff88002ad9ffd8 RCX: 0000000000000000
[ 2406.773106] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.773106] RBP: ffff88002ad9fe28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.773106] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.773106] R13: ffff88002ad9e000 R14: ffffffffb30bc580 R15: ffff88002ad9ffd8
[ 2406.773106] FS: 0000000000000000(0000) GS:ffff8803d7000000(0000) knlGS:0000000000000000
[ 2406.773106] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.773106] CR2: 00007f5b4d3ff099 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.773106] Stack:
[ 2406.773106] ffff88002ad9fe48 ffffffffad07c04d ffff88002ad9ffd8 0000000000000000
[ 2406.773106] ffff88002ad9fe58 ffffffffad07cd4f ffff88002ad9fed8 ffffffffad1b73f5
[ 2406.773106] 0000000000000000 ccdf3c6031d54cd1 ffff88002ad9ffd8 000000000000f000
[ 2406.773106] Call Trace:
[ 2406.773106] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.773106] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.773106] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.773106] cpu_startup_entry (??:?)
[ 2406.773106] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.773106] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.770609] NMI backtrace for cpu 13
[ 2406.770609] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.770609] task: ffff8803d6928000 ti: ffff88002ad90000 task.ti: ffff88002ad90000
[ 2406.770609] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.770609] RSP: 0018:ffff88002ad91e28 EFLAGS: 00000282
[ 2406.770609] RAX: ffff8803d6928000 RBX: ffff88002ad91fd8 RCX: 0000000000000000
[ 2406.770609] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.770609] RBP: ffff88002ad91e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.770609] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.770609] R13: ffff88002ad90000 R14: ffffffffb30bc580 R15: ffff88002ad91fd8
[ 2406.770609] FS: 0000000000000000(0000) GS:ffff8802e7000000(0000) knlGS:0000000000000000
[ 2406.770609] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.770609] CR2: 00007f10cbecf070 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.770609] Stack:
[ 2406.770609] ffff88002ad91e48 ffffffffad07c04d ffff88002ad91fd8 0000000000000000
[ 2406.770609] ffff88002ad91e58 ffffffffad07cd4f ffff88002ad91ed8 ffffffffad1b73f5
[ 2406.770609] 0000000000000000 618f5836413e64b6 ffff88002ad91fd8 000000000000f000
[ 2406.770609] Call Trace:
[ 2406.770609] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.770609] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.770609] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.770609] cpu_startup_entry (??:?)
[ 2406.770609] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.770609] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.785234] NMI backtrace for cpu 19
[ 2406.785234] CPU: 19 PID: 0 Comm: swapper/19 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.785234] task: ffff88018a910000 ti: ffff88002ada8000 task.ti: ffff88002ada8000
[ 2406.785234] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.785234] RSP: 0018:ffff88002ada9e28 EFLAGS: 00000282
[ 2406.785234] RAX: ffff88018a910000 RBX: ffff88002ada9fd8 RCX: 0000000000000000
[ 2406.785234] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.785234] RBP: ffff88002ada9e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.785234] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.785234] R13: ffff88002ada8000 R14: ffffffffb30bc580 R15: ffff88002ada9fd8
[ 2406.785234] FS: 0000000000000000(0000) GS:ffff880407000000(0000) knlGS:0000000000000000
[ 2406.785234] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.785234] CR2: 00007f7bc2f1d099 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.785234] Stack:
[ 2406.785234] ffff88002ada9e48 ffffffffad07c04d ffff88002ada9fd8 0000000000000000
[ 2406.785234] ffff88002ada9e58 ffffffffad07cd4f ffff88002ada9ed8 ffffffffad1b73f5
[ 2406.785234] 0000000000000000 43f465af05da1bde ffff88002ada9fd8 000000000000f000
[ 2406.785234] Call Trace:
[ 2406.785234] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.785234] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.785234] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.785234] cpu_startup_entry (??:?)
[ 2406.785234] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.785234] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.722636] NMI backtrace for cpu 5
[ 2406.722636] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.722636] task: ffff880556930000 ti: ffff88002ad74000 task.ti: ffff88002ad74000
[ 2406.722636] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.722636] RSP: 0018:ffff88002ad75e28 EFLAGS: 00000282
[ 2406.722636] RAX: ffff880556930000 RBX: ffff88002ad75fd8 RCX: 0000000000000000
[ 2406.722636] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.722636] RBP: ffff88002ad75e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.722636] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.722636] R13: ffff88002ad74000 R14: ffffffffb30bc580 R15: ffff88002ad75fd8
[ 2406.722636] FS: 0000000000000000(0000) GS:ffff880158e00000(0000) knlGS:0000000000000000
[ 2406.722636] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.722636] CR2: 00007f9ba4e9f170 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.722636] Stack:
[ 2406.722636] ffff88002ad75e48 ffffffffad07c04d ffff88002ad75fd8 0000000000000000
[ 2406.722636] ffff88002ad75e58 ffffffffad07cd4f ffff88002ad75ed8 ffffffffad1b73f5
[ 2406.722636] 0000000000000000 9a6095db8c2e6f9a ffff88002ad75fd8 000000000000f000
[ 2406.722636] Call Trace:
[ 2406.722636] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.722636] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.722636] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.722636] cpu_startup_entry (??:?)
[ 2406.722636] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.722636] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.780256] NMI backtrace for cpu 0
[ 2406.780256] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.780256] task: ffffffffb1e344c0 ti: ffffffffb1e00000 task.ti: ffffffffb1e00000
[ 2406.780256] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.780256] RSP: 0000:ffffffffb1e01e38 EFLAGS: 00000282
[ 2406.780256] RAX: ffffffffb1e344c0 RBX: ffffffffb1e01fd8 RCX: 0000000000000000
[ 2406.780256] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.780256] RBP: ffffffffb1e01e38 R08: 0000000000000000 R09: 0000000000000000
[ 2406.780256] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.780256] R13: ffffffffb1e00000 R14: ffffffffb30bc580 R15: ffffffffb1e01fd8
[ 2406.780256] FS: 0000000000000000(0000) GS:ffff88002be00000(0000) knlGS:0000000000000000
[ 2406.780256] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.780256] CR2: 00007f0e067fe821 CR3: 0000000031e2d000 CR4: 00000000000006b0
[ 2406.780256] Stack:
[ 2406.780256] ffffffffb1e01e58 ffffffffad07c04d ffffffffb1e01fd8 0000000000000000
[ 2406.780256] ffffffffb1e01e68 ffffffffad07cd4f ffffffffb1e01ee8 ffffffffad1b73f5
[ 2406.780256] 0000000000000000 d81b28d1e484efc7 ffffffffb1e01fd8 0000000000000002
[ 2406.780256] Call Trace:
[ 2406.780256] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.780256] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.780256] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.780256] cpu_startup_entry (??:?)
[ 2406.780256] rest_init (init/main.c:398)
[ 2406.780256] ? rest_init (init/main.c:373)
[ 2406.780256] start_kernel (init/main.c:653)
[ 2406.780256] ? repair_env_string (init/main.c:260)
[ 2406.780256] ? early_idt_handlers (arch/x86/kernel/head_64.S:340)
[ 2406.780256] x86_64_start_reservations (arch/x86/kernel/head64.c:194)
[ 2406.780256] x86_64_start_kernel (arch/x86/kernel/head64.c:183)
[ 2406.780256] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2406.790091] NMI backtrace for cpu 3
[ 2406.790091] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2406.790091] task: ffff880316908000 ti: ffff88002ad70000 task.ti: ffff88002ad70000
[ 2406.790091] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2406.790091] RSP: 0018:ffff88002ad71e28 EFLAGS: 00000282
[ 2406.790091] RAX: ffff880316908000 RBX: ffff88002ad71fd8 RCX: 0000000000000000
[ 2406.790091] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2406.790091] RBP: ffff88002ad71e28 R08: 0000000000000000 R09: 0000000000000000
[ 2406.790091] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2406.790091] R13: ffff88002ad70000 R14: ffffffffb30bc580 R15: ffff88002ad71fd8
[ 2406.790091] FS: 0000000000000000(0000) GS:ffff8800c7000000(0000) knlGS:0000000000000000
[ 2406.790091] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2406.790091] CR2: 00007fbb037b0334 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2406.790091] Stack:
[ 2406.790091] ffff88002ad71e48 ffffffffad07c04d ffff88002ad71fd8 0000000000000000
[ 2406.790091] ffff88002ad71e58 ffffffffad07cd4f ffff88002ad71ed8 ffffffffad1b73f5
[ 2406.790091] 0000000000000000 a42a8a608a0df9ca ffff88002ad71fd8 000000000000f000
[ 2406.790091] Call Trace:
[ 2406.790091] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2406.790091] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2406.790091] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2406.790091] cpu_startup_entry (??:?)
[ 2406.790091] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2406.790091] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2408.693358] NMI backtrace for cpu 15
[ 2408.693358] CPU: 15 PID: 0 Comm: swapper/15 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2408.722811] task: ffff88018a908000 ti: ffff88002ad98000 task.ti: ffff88002ad98000
[ 2408.722811] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2408.722811] RSP: 0018:ffff88002ad99e28 EFLAGS: 00000282
[ 2408.722811] RAX: ffff88018a908000 RBX: ffff88002ad99fd8 RCX: 0000000000000000
[ 2408.722811] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2408.722811] RBP: ffff88002ad99e28 R08: 0000000000000000 R09: 0000000000000000
[ 2408.722811] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2408.722811] R13: ffff88002ad98000 R14: ffffffffb30bc580 R15: ffff88002ad99fd8
[ 2408.722811] FS: 0000000000000000(0000) GS:ffff880347000000(0000) knlGS:0000000000000000
[ 2408.722811] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2408.722811] CR2: 00007f93c7ead070 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2408.722811] Stack:
[ 2408.722811] ffff88002ad99e48 ffffffffad07c04d ffff88002ad99fd8 0000000000000000
[ 2408.722811] ffff88002ad99e58 ffffffffad07cd4f ffff88002ad99ed8 ffffffffad1b73f5
[ 2408.722811] 0000000000000000 d415927b269676da ffff88002ad99fd8 000000000000f000
[ 2408.722811] Call Trace:
[ 2408.722811] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2408.722811] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2408.722811] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2408.722811] cpu_startup_entry (??:?)
[ 2408.722811] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2408.722811] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2408.722619] NMI backtrace for cpu 14
[ 2408.722619] CPU: 14 PID: 0 Comm: swapper/14 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2408.722619] task: ffff8805b6908000 ti: ffff88002ad92000 task.ti: ffff88002ad92000
[ 2408.722619] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2408.722619] RSP: 0018:ffff88002ad93e28 EFLAGS: 00000282
[ 2408.722619] RAX: ffff8805b6908000 RBX: ffff88002ad93fd8 RCX: 0000000000000000
[ 2408.722619] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2408.722619] RBP: ffff88002ad93e28 R08: 0000000000000000 R09: 0000000000000000
[ 2408.722619] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2408.722619] R13: ffff88002ad92000 R14: ffffffffb30bc580 R15: ffff88002ad93fd8
[ 2408.722619] FS: 0000000000000000(0000) GS:ffff880317000000(0000) knlGS:0000000000000000
[ 2408.722619] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2408.722619] CR2: 00007f30be8c9730 CR3: 0000000157401000 CR4: 00000000000006a0
[ 2408.722619] Stack:
[ 2408.722619] ffff88002ad93e48 ffffffffad07c04d ffff88002ad93fd8 0000000000000000
[ 2408.722619] ffff88002ad93e58 ffffffffad07cd4f ffff88002ad93ed8 ffffffffad1b73f5
[ 2408.722619] 0000000000000000 89f157ff9e713d03 ffff88002ad93fd8 000000000000f000
[ 2408.837217] Call Trace:
[ 2408.837217] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2408.837217] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2408.837217] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2408.837217] cpu_startup_entry (??:?)
[ 2408.837217] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2408.837217] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2408.691691] NMI backtrace for cpu 6
[ 2408.691691] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2408.691691] task: ffff8800948f8000 ti: ffff88002ad76000 task.ti: ffff88002ad76000
[ 2408.691691] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2408.691691] RSP: 0018:ffff88002ad77e28 EFLAGS: 00000282
[ 2408.691691] RAX: ffff8800948f8000 RBX: ffff88002ad77fd8 RCX: 0000000000000000
[ 2408.691691] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2408.691691] RBP: ffff88002ad77e28 R08: 0000000000000000 R09: 0000000000000000
[ 2408.691691] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2408.691691] R13: ffff88002ad76000 R14: ffffffffb30bc580 R15: ffff88002ad77fd8
[ 2408.691691] FS: 0000000000000000(0000) GS:ffff88018b000000(0000) knlGS:0000000000000000
[ 2408.691691] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2408.691691] CR2: 00007f1f306df170 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2408.691691] Stack:
[ 2408.691691] ffff88002ad77e48 ffffffffad07c04d ffff88002ad77fd8 0000000000000000
[ 2408.691691] ffff88002ad77e58 ffffffffad07cd4f ffff88002ad77ed8 ffffffffad1b73f5
[ 2408.691691] 0000000000000000 154a1c73da38ea65 ffff88002ad77fd8 000000000000f000
[ 2408.691691] Call Trace:
[ 2408.691691] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2408.691691] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2408.691691] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2408.691691] cpu_startup_entry (??:?)
[ 2408.691691] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2408.691691] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2408.696506] NMI backtrace for cpu 23
[ 2408.696506] CPU: 23 PID: 0 Comm: swapper/23 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2408.696506] task: ffff880616930000 ti: ffff88002adb0000 task.ti: ffff88002adb0000
[ 2408.696506] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2408.696506] RSP: 0018:ffff88002adb1e28 EFLAGS: 00000282
[ 2408.696506] RAX: ffff880616930000 RBX: ffff88002adb1fd8 RCX: 0000000000000000
[ 2408.696506] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2408.696506] RBP: ffff88002adb1e28 R08: 0000000000000000 R09: 0000000000000000
[ 2408.696506] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2408.696506] R13: ffff88002adb0000 R14: ffffffffb30bc580 R15: ffff88002adb1fd8
[ 2408.696506] FS: 0000000000000000(0000) GS:ffff8804c7000000(0000) knlGS:0000000000000000
[ 2408.696506] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2408.696506] CR2: 00007faa2c8d59de CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2408.696506] Stack:
[ 2408.696506] ffff88002adb1e48 ffffffffad07c04d ffff88002adb1fd8 0000000000000000
[ 2408.696506] ffff88002adb1e58 ffffffffad07cd4f ffff88002adb1ed8 ffffffffad1b73f5
[ 2408.696506] 0000000000000000 5f651cf4a238a6ff ffff88002adb1fd8 000000000000f000
[ 2408.696506] Call Trace:
[ 2408.696506] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2408.696506] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2408.696506] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2408.696506] cpu_startup_entry (??:?)
[ 2408.696506] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2408.696506] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2408.837579] NMI backtrace for cpu 22
[ 2408.837579] CPU: 22 PID: 0 Comm: swapper/22 Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2408.837579] task: ffff8804f6938000 ti: ffff88002adae000 task.ti: ffff88002adae000
[ 2408.837579] RIP: native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 2408.837579] RSP: 0018:ffff88002adafe28 EFLAGS: 00000282
[ 2408.837579] RAX: ffff8804f6938000 RBX: ffff88002adaffd8 RCX: 0000000000000000
[ 2408.837579] RDX: 0000000000000000 RSI: ffffffffb1820b10 RDI: ffffffffb17b99f5
[ 2408.837579] RBP: ffff88002adafe28 R08: 0000000000000000 R09: 0000000000000000
[ 2408.837579] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 2408.837579] R13: ffff88002adae000 R14: ffffffffb30bc580 R15: ffff88002adaffd8
[ 2408.837579] FS: 0000000000000000(0000) GS:ffff880497000000(0000) knlGS:0000000000000000
[ 2408.837579] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2408.837579] CR2: 00007fffb3da2ff8 CR3: 0000000031e2d000 CR4: 00000000000006a0
[ 2408.837579] Stack:
[ 2408.837579] ffff88002adafe48 ffffffffad07c04d ffff88002adaffd8 0000000000000000
[ 2408.837579] ffff88002adafe58 ffffffffad07cd4f ffff88002adafed8 ffffffffad1b73f5
[ 2408.837579] 0000000000000000 21d4616f708a6f6a ffff88002adaffd8 000000000000f000
[ 2408.837579] Call Trace:
[ 2408.837579] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 2408.837579] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 2408.837579] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 2408.837579] cpu_startup_entry (??:?)
[ 2408.837579] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 2408.837579] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
[ 2409.043054] Kernel panic - not syncing: hung_task: blocked tasks
[ 2409.046306] CPU: 21 PID: 3540 Comm: khungtaskd Not tainted 3.15.0-rc1-next-20140415-sasha-00020-gaa90d09 #398
[ 2409.050074] 00000000003ffecf ffff880436b5bd38 ffffffffb05223a7 0000000000000001
[ 2409.050074] ffffffffb16caa40 ffff880436b5bdb8 ffffffffb0513f93 ffff880436b5bd58
[ 2409.050074] 0000000000000008 ffff880436b5bdc8 ffff880436b5bd68 0000000000001f15
[ 2409.050074] Call Trace:
[ 2409.050074] dump_stack (lib/dump_stack.c:52)
[ 2409.050074] panic (kernel/panic.c:117)
[ 2409.050074] watchdog (kernel/hung_task.c:124 kernel/hung_task.c:180 kernel/hung_task.c:236)
[ 2409.050074] ? watchdog (include/linux/rcupdate.h:800 kernel/hung_task.c:169 kernel/hung_task.c:236)
[ 2409.050074] ? reset_hung_task_detector (kernel/hung_task.c:224)
[ 2409.050074] kthread (kernel/kthread.c:210)
[ 2409.050074] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2409.050074] ret_from_fork (arch/x86/kernel/entry_64.S:555)
[ 2409.050074] ? kthread_create_on_node (kernel/kthread.c:176)
[ 2409.050074] Dumping ftrace buffer:
[ 2409.050074] (ftrace buffer empty)
[ 2409.050074] Kernel Offset: 0x2c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Thanks,
Sasha
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2014-04-16 2:06 mm: hangs in collapse_huge_page Sasha Levin
@ 2014-04-24 16:46 ` Sasha Levin
2014-04-30 15:42 ` Kirill A. Shutemov
1 sibling, 0 replies; 8+ messages in thread
From: Sasha Levin @ 2014-04-24 16:46 UTC (permalink / raw)
To: linux-mm; +Cc: LKML, Dave Jones, Andrew Morton
On 04/15/2014 10:06 PM, Sasha Levin wrote:
> Hi all,
>
> I often see hung task triggering in khugepaged within collapse_huge_page().
>
> I've initially assumed the case may be that the guests are too loaded and
> the warning occurs because of load, but after increasing the timeout to
> 1200 sec I still see the warning.
Ping? It happens quite often right now.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2014-04-16 2:06 mm: hangs in collapse_huge_page Sasha Levin
2014-04-24 16:46 ` Sasha Levin
@ 2014-04-30 15:42 ` Kirill A. Shutemov
2014-05-01 14:38 ` Hillf Danton
` (2 more replies)
1 sibling, 3 replies; 8+ messages in thread
From: Kirill A. Shutemov @ 2014-04-30 15:42 UTC (permalink / raw)
To: Sasha Levin, Andrea Arcangeli; +Cc: linux-mm, LKML, Dave Jones, Andrew Morton
On Tue, Apr 15, 2014 at 10:06:56PM -0400, Sasha Levin wrote:
> Hi all,
>
> I often see hung task triggering in khugepaged within collapse_huge_page().
>
> I've initially assumed the case may be that the guests are too loaded and
> the warning occurs because of load, but after increasing the timeout to
> 1200 sec I still see the warning.
I suspect it's race (although I didn't track down exact scenario) with
__khugepaged_exit().
Comment in __khugepaged_exit() says that khugepaged_test_exit() always
called under mmap_sem:
2045 void __khugepaged_exit(struct mm_struct *mm)
...
2063 } else if (mm_slot) {
2064 /*
2065 * This is required to serialize against
2066 * khugepaged_test_exit() (which is guaranteed to run
2067 * under mmap sem read mode). Stop here (after we
2068 * return all pagetables will be destroyed) until
2069 * khugepaged has finished working on the pagetables
2070 * under the mmap_sem.
2071 */
2072 down_write(&mm->mmap_sem);
2073 up_write(&mm->mmap_sem);
2074 }
2075 }
But this is not true. At least khugepaged_scan_mm_slot() calls it without
the sem:
2566 static unsigned int khugepaged_scan_mm_slot(unsigned int pages,
2567 struct page **hpage)
...
2046 {
2047 struct mm_slot *mm_slot;
2048 int free = 0;
2049
2050 spin_lock(&khugepaged_mm_lock);
2051 mm_slot = get_mm_slot(mm);
2052 if (mm_slot && khugepaged_scan.mm_slot != mm_slot) {
2053 hash_del(&mm_slot->hash);
2054 list_del(&mm_slot->mm_node);
2055 free = 1;
2056 }
2057 spin_unlock(&khugepaged_mm_lock);
2058
2059 if (free) {
2060 clear_bit(MMF_VM_HUGEPAGE, &mm->flags);
2061 free_mm_slot(mm_slot);
2062 mmdrop(mm);
Not sure yet if it's a real problem or not. Andrea, could you comment on
this?
Sasha, please try patch below.
Not-Yet-Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index b4b1feba6472..1c6ace5207b9 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1986,6 +1986,8 @@ static void insert_to_mm_slots_hash(struct mm_struct *mm,
static inline int khugepaged_test_exit(struct mm_struct *mm)
{
+ VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem) &&
+ !spin_is_locked(&khugepaged_mm_lock));
return atomic_read(&mm->mm_users) == 0;
}
@@ -2062,14 +2064,16 @@ void __khugepaged_exit(struct mm_struct *mm)
mmdrop(mm);
} else if (mm_slot) {
/*
- * This is required to serialize against
- * khugepaged_test_exit() (which is guaranteed to run
- * under mmap sem read mode). Stop here (after we
- * return all pagetables will be destroyed) until
- * khugepaged has finished working on the pagetables
+ * This is required to serialize against khugepaged_test_exit()
+ * (which is guaranteed to run under mmap sem read mode or
+ * khugepaged_mm_lock).
+ * Stop here (after we return all pagetables will be destroyed)
+ * until khugepaged has finished working on the pagetables
* under the mmap_sem.
*/
down_write(&mm->mmap_sem);
+ spin_lock(&khugepaged_mm_lock);
+ spin_unlock(&khugepaged_mm_lock);
up_write(&mm->mmap_sem);
}
}
--
Kirill A. Shutemov
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2014-04-30 15:42 ` Kirill A. Shutemov
@ 2014-05-01 14:38 ` Hillf Danton
2014-05-11 0:34 ` Sasha Levin
2015-04-30 22:17 ` Sasha Levin
2 siblings, 0 replies; 8+ messages in thread
From: Hillf Danton @ 2014-05-01 14:38 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Sasha Levin, Andrea Arcangeli, linux-mm, LKML, Dave Jones, Andrew Morton
hi all
On Wed, Apr 30, 2014 at 11:42 PM, Kirill A. Shutemov
<kirill@shutemov.name> wrote:
> On Tue, Apr 15, 2014 at 10:06:56PM -0400, Sasha Levin wrote:
>> Hi all,
>>
>> I often see hung task triggering in khugepaged within collapse_huge_page().
>>
>> I've initially assumed the case may be that the guests are too loaded and
>> the warning occurs because of load, but after increasing the timeout to
>> 1200 sec I still see the warning.
>
> I suspect it's race (although I didn't track down exact scenario) with
> __khugepaged_exit().
>
> Comment in __khugepaged_exit() says that khugepaged_test_exit() always
> called under mmap_sem:
>
> 2045 void __khugepaged_exit(struct mm_struct *mm)
> ...
> 2063 } else if (mm_slot) {
> 2064 /*
> 2065 * This is required to serialize against
> 2066 * khugepaged_test_exit() (which is guaranteed to run
> 2067 * under mmap sem read mode). Stop here (after we
> 2068 * return all pagetables will be destroyed) until
> 2069 * khugepaged has finished working on the pagetables
> 2070 * under the mmap_sem.
> 2071 */
> 2072 down_write(&mm->mmap_sem);
> 2073 up_write(&mm->mmap_sem);
> 2074 }
> 2075 }
>
> But this is not true. At least khugepaged_scan_mm_slot() calls it without
> the sem:
>
> 2566 static unsigned int khugepaged_scan_mm_slot(unsigned int pages,
> 2567 struct page **hpage)
> ...
> 2046 {
> 2047 struct mm_slot *mm_slot;
> 2048 int free = 0;
> 2049
> 2050 spin_lock(&khugepaged_mm_lock);
> 2051 mm_slot = get_mm_slot(mm);
> 2052 if (mm_slot && khugepaged_scan.mm_slot != mm_slot) {
> 2053 hash_del(&mm_slot->hash);
> 2054 list_del(&mm_slot->mm_node);
> 2055 free = 1;
> 2056 }
> 2057 spin_unlock(&khugepaged_mm_lock);
> 2058
> 2059 if (free) {
> 2060 clear_bit(MMF_VM_HUGEPAGE, &mm->flags);
> 2061 free_mm_slot(mm_slot);
> 2062 mmdrop(mm);
>
> Not sure yet if it's a real problem or not. Andrea, could you comment on
> this?
>
> Sasha, please try patch below.
>
This box is quite,
CPU: 0 PID: 0 Comm: swapper/0
CPU: 1 PID: 0 Comm: swapper/1
CPU: 2 PID: 0 Comm: swapper/2
CPU: 3 PID: 0 Comm: swapper/3
CPU: 4 PID: 0 Comm: swapper/4
CPU: 5 PID: 0 Comm: swapper/5
CPU: 6 PID: 0 Comm: swapper/6
CPU: 7 PID: 0 Comm: swapper/7
CPU: 8 PID: 0 Comm: swapper/8
CPU: 9 PID: 0 Comm: swapper/9
CPU: 10 PID: 0 Comm: swapper/10
CPU: 11 PID: 0 Comm: swapper/11
CPU: 12 PID: 0 Comm: swapper/12
CPU: 13 PID: 0 Comm: swapper/13
CPU: 14 PID: 0 Comm: swapper/14
CPU: 15 PID: 0 Comm: swapper/15
CPU: 16 PID: 0 Comm: swapper/16
CPU: 17 PID: 0 Comm: swapper/17
CPU: 18 PID: 0 Comm: swapper/18
CPU: 19 PID: 0 Comm: swapper/19
CPU: 20 PID: 0 Comm: swapper/20
CPU: 21 PID: 3540 Comm: khungtaskd
CPU: 22 PID: 0 Comm: swapper/22
CPU: 23 PID: 0 Comm: swapper/23
and lets make more noise.
Hillf
---
--- a/mm/huge_memory.c Thu May 1 22:20:20 2014
+++ b/mm/huge_memory.c Thu May 1 22:24:06 2014
@@ -2732,7 +2732,8 @@ static void khugepaged_wait_work(void)
}
if (khugepaged_enabled())
- wait_event_freezable(khugepaged_wait, khugepaged_wait_event());
+ wait_event_freezable_timeout(khugepaged_wait, khugepaged_wait_event(),
+ msecs_to_jiffies(2000));
}
static int khugepaged(void *none)
--
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2014-04-30 15:42 ` Kirill A. Shutemov
2014-05-01 14:38 ` Hillf Danton
@ 2014-05-11 0:34 ` Sasha Levin
2014-05-14 21:29 ` Kirill A. Shutemov
2015-04-30 22:17 ` Sasha Levin
2 siblings, 1 reply; 8+ messages in thread
From: Sasha Levin @ 2014-05-11 0:34 UTC (permalink / raw)
To: Kirill A. Shutemov, Andrea Arcangeli
Cc: linux-mm, LKML, Dave Jones, Andrew Morton
On 04/30/2014 11:42 AM, Kirill A. Shutemov wrote:
> On Tue, Apr 15, 2014 at 10:06:56PM -0400, Sasha Levin wrote:
>> > Hi all,
>> >
>> > I often see hung task triggering in khugepaged within collapse_huge_page().
>> >
>> > I've initially assumed the case may be that the guests are too loaded and
>> > the warning occurs because of load, but after increasing the timeout to
>> > 1200 sec I still see the warning.
> I suspect it's race (although I didn't track down exact scenario) with
> __khugepaged_exit().
>
> Comment in __khugepaged_exit() says that khugepaged_test_exit() always
> called under mmap_sem:
>
> 2045 void __khugepaged_exit(struct mm_struct *mm)
> ...
> 2063 } else if (mm_slot) {
> 2064 /*
> 2065 * This is required to serialize against
> 2066 * khugepaged_test_exit() (which is guaranteed to run
> 2067 * under mmap sem read mode). Stop here (after we
> 2068 * return all pagetables will be destroyed) until
> 2069 * khugepaged has finished working on the pagetables
> 2070 * under the mmap_sem.
> 2071 */
> 2072 down_write(&mm->mmap_sem);
> 2073 up_write(&mm->mmap_sem);
> 2074 }
> 2075 }
>
> But this is not true. At least khugepaged_scan_mm_slot() calls it without
> the sem:
>
> 2566 static unsigned int khugepaged_scan_mm_slot(unsigned int pages,
> 2567 struct page **hpage)
> ...
> 2046 {
> 2047 struct mm_slot *mm_slot;
> 2048 int free = 0;
> 2049
> 2050 spin_lock(&khugepaged_mm_lock);
> 2051 mm_slot = get_mm_slot(mm);
> 2052 if (mm_slot && khugepaged_scan.mm_slot != mm_slot) {
> 2053 hash_del(&mm_slot->hash);
> 2054 list_del(&mm_slot->mm_node);
> 2055 free = 1;
> 2056 }
> 2057 spin_unlock(&khugepaged_mm_lock);
> 2058
> 2059 if (free) {
> 2060 clear_bit(MMF_VM_HUGEPAGE, &mm->flags);
> 2061 free_mm_slot(mm_slot);
> 2062 mmdrop(mm);
>
> Not sure yet if it's a real problem or not. Andrea, could you comment on
> this?
>
> Sasha, please try patch below.
That patch really solved the problem for me, I didn't see a single hang
up until today. So I suspect that while that patch is good there is another
(smaller) case which may case a hang.
[ 6006.253399] INFO: task khugepaged:3814 blocked for more than 1200 seconds.
[ 6006.254711] Tainted: G W 3.15.0-rc4-next-20140508-sasha-00020-gec9304b-dirty #452
[ 6006.257591] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6006.260710] khugepaged D ffff8805bb7a64b8 4968 3814 2 0x00000000
[ 6006.263261] ffff880291e61b48 0000000000000002 ffffffff955890d0 ffff8802927f0000
[ 6006.264986] ffff880291e61fd8 00000000001d7840 00000000001d7840 00000000001d7840
[ 6006.265923] ffff8805b040b000 ffff8802927f0000 ffff880291e61b38 ffff8802927f0000
[ 6006.267193] Call Trace:
[ 6006.267629] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:819 include/linux/spinlock_api_smp.h:168 kernel/locking/spinlock.c:199)
[ 6006.268673] schedule (kernel/sched/core.c:2765)
[ 6006.269674] rwsem_down_write_failed (kernel/locking/rwsem-xadd.c:289)
[ 6006.270998] ? get_parent_ip (kernel/sched/core.c:2485)
[ 6006.271984] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:106)
[ 6006.273240] ? khugepaged_scan_mm_slot (mm/huge_memory.c:1991 mm/huge_memory.c:2598)
[ 6006.274327] ? lock_contended (kernel/locking/lockdep.c:3734 kernel/locking/lockdep.c:3812)
[ 6006.275299] ? down_write (kernel/locking/rwsem.c:50 (discriminator 2))
[ 6006.276126] ? collapse_huge_page.isra.31 (mm/huge_memory.c:1991 mm/huge_memory.c:2385)
[ 6006.277281] collapse_huge_page.isra.31 (mm/huge_memory.c:1991 mm/huge_memory.c:2385)
[ 6006.278328] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
[ 6006.279534] ? put_lock_stats.isra.12 (arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254)
[ 6006.281314] ? khugepaged_scan_mm_slot (include/linux/spinlock.h:343 mm/huge_memory.c:2540 mm/huge_memory.c:2636)
[ 6006.282140] ? get_parent_ip (kernel/sched/core.c:2485)
[ 6006.283039] khugepaged_scan_mm_slot (mm/huge_memory.c:2640)
[ 6006.284074] khugepaged (include/linux/spinlock.h:343 mm/huge_memory.c:2720 mm/huge_memory.c:2753)
[ 6006.284986] ? bit_waitqueue (kernel/sched/wait.c:291)
[ 6006.285864] ? khugepaged_scan_mm_slot (mm/huge_memory.c:2746)
[ 6006.286897] kthread (kernel/kthread.c:210)
[ 6006.287705] ? kthread_create_on_node (kernel/kthread.c:176)
[ 6006.288796] ret_from_fork (arch/x86/kernel/entry_64.S:553)
[ 6006.289668] ? kthread_create_on_node (kernel/kthread.c:176)
[ 6006.291708] 1 lock held by khugepaged/3814:
[ 6006.292368] #0: (&mm->mmap_sem){++++++}, at: collapse_huge_page.isra.31 (mm/huge_memory.c:1991 mm/huge_memory.c:2385)
Thanks,
Sasha
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2014-05-11 0:34 ` Sasha Levin
@ 2014-05-14 21:29 ` Kirill A. Shutemov
0 siblings, 0 replies; 8+ messages in thread
From: Kirill A. Shutemov @ 2014-05-14 21:29 UTC (permalink / raw)
To: Sasha Levin; +Cc: Andrea Arcangeli, linux-mm, LKML, Dave Jones, Andrew Morton
On Sat, May 10, 2014 at 08:34:40PM -0400, Sasha Levin wrote:
> On 04/30/2014 11:42 AM, Kirill A. Shutemov wrote:
> > Sasha, please try patch below.
>
> That patch really solved the problem for me, I didn't see a single hang
> up until today. So I suspect that while that patch is good there is another
> (smaller) case which may case a hang.
Looks like my patch didn't fix the problem but hide it. Most like due some
shift in timings because of additional lock/unlock.
I'll try to look more for a real issue.
>
> [ 6006.253399] INFO: task khugepaged:3814 blocked for more than 1200 seconds.
> [ 6006.254711] Tainted: G W 3.15.0-rc4-next-20140508-sasha-00020-gec9304b-dirty #452
> [ 6006.257591] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 6006.260710] khugepaged D ffff8805bb7a64b8 4968 3814 2 0x00000000
> [ 6006.263261] ffff880291e61b48 0000000000000002 ffffffff955890d0 ffff8802927f0000
> [ 6006.264986] ffff880291e61fd8 00000000001d7840 00000000001d7840 00000000001d7840
> [ 6006.265923] ffff8805b040b000 ffff8802927f0000 ffff880291e61b38 ffff8802927f0000
> [ 6006.267193] Call Trace:
> [ 6006.267629] ? _raw_spin_unlock_irq (arch/x86/include/asm/paravirt.h:819 include/linux/spinlock_api_smp.h:168 kernel/locking/spinlock.c:199)
> [ 6006.268673] schedule (kernel/sched/core.c:2765)
> [ 6006.269674] rwsem_down_write_failed (kernel/locking/rwsem-xadd.c:289)
> [ 6006.270998] ? get_parent_ip (kernel/sched/core.c:2485)
> [ 6006.271984] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:106)
> [ 6006.273240] ? khugepaged_scan_mm_slot (mm/huge_memory.c:1991 mm/huge_memory.c:2598)
> [ 6006.274327] ? lock_contended (kernel/locking/lockdep.c:3734 kernel/locking/lockdep.c:3812)
> [ 6006.275299] ? down_write (kernel/locking/rwsem.c:50 (discriminator 2))
> [ 6006.276126] ? collapse_huge_page.isra.31 (mm/huge_memory.c:1991 mm/huge_memory.c:2385)
> [ 6006.277281] collapse_huge_page.isra.31 (mm/huge_memory.c:1991 mm/huge_memory.c:2385)
> [ 6006.278328] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
> [ 6006.279534] ? put_lock_stats.isra.12 (arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254)
> [ 6006.281314] ? khugepaged_scan_mm_slot (include/linux/spinlock.h:343 mm/huge_memory.c:2540 mm/huge_memory.c:2636)
> [ 6006.282140] ? get_parent_ip (kernel/sched/core.c:2485)
> [ 6006.283039] khugepaged_scan_mm_slot (mm/huge_memory.c:2640)
> [ 6006.284074] khugepaged (include/linux/spinlock.h:343 mm/huge_memory.c:2720 mm/huge_memory.c:2753)
> [ 6006.284986] ? bit_waitqueue (kernel/sched/wait.c:291)
> [ 6006.285864] ? khugepaged_scan_mm_slot (mm/huge_memory.c:2746)
> [ 6006.286897] kthread (kernel/kthread.c:210)
> [ 6006.287705] ? kthread_create_on_node (kernel/kthread.c:176)
> [ 6006.288796] ret_from_fork (arch/x86/kernel/entry_64.S:553)
> [ 6006.289668] ? kthread_create_on_node (kernel/kthread.c:176)
> [ 6006.291708] 1 lock held by khugepaged/3814:
> [ 6006.292368] #0: (&mm->mmap_sem){++++++}, at: collapse_huge_page.isra.31 (mm/huge_memory.c:1991 mm/huge_memory.c:2385)
>
>
> Thanks,
> Sasha
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2014-04-30 15:42 ` Kirill A. Shutemov
2014-05-01 14:38 ` Hillf Danton
2014-05-11 0:34 ` Sasha Levin
@ 2015-04-30 22:17 ` Sasha Levin
2015-04-30 22:24 ` Kirill A. Shutemov
2 siblings, 1 reply; 8+ messages in thread
From: Sasha Levin @ 2015-04-30 22:17 UTC (permalink / raw)
To: Kirill A. Shutemov, Andrea Arcangeli
Cc: linux-mm, LKML, Dave Jones, Andrew Morton
On 04/30/2014 11:42 AM, Kirill A. Shutemov wrote:
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index b4b1feba6472..1c6ace5207b9 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -1986,6 +1986,8 @@ static void insert_to_mm_slots_hash(struct mm_struct *mm,
>
> static inline int khugepaged_test_exit(struct mm_struct *mm)
> {
> + VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem) &&
> + !spin_is_locked(&khugepaged_mm_lock));
> return atomic_read(&mm->mm_users) == 0;
> }
I've managed to hit this during testing:
[ 8048.304275] kernel BUG at mm/huge_memory.c:2060!
[ 8048.305878] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
[ 8048.307479] Modules linked in: quota_v2 quota_tree xfs libcrc32c x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ast kvm ttm drm_kms_helper crct10dif_pclmul crc32_pclmul drm ghash_clmulni_intel aesni_
intel aes_x86_64 lrw glue_helper ablk_helper cryptd joydev i2c_algo_bit sb_edac syscopyarea sysfillrect edac_core sysimgblt lpc_ich ipmi_si ipmi_msghandler ioatdma shpchp mac_hid btrfs xor mlx4_en vxlan raid6_pq
hid_generic ixgbe mlx4_core usbhid hid dca megaraid_sas ahci ptp libahci pps_core mdio
[ 8048.314422] CPU: 31 PID: 13065 Comm: thp01 Not tainted 4.1.0-rc1-next-20150430+ #8
[ 8048.316215] Hardware name: Oracle Corporation OVCA X3-2 /ASSY,MOTHERBOARD,1U , BIOS 17021300 06/19/2012
[ 8048.318070] task: ffff8837ba9b3b40 ti: ffff8837bfcf8000 task.ti: ffff8837bfcf8000
[ 8048.319941] RIP: __khugepaged_enter (mm/huge_memory.c:2059 mm/huge_memory.c:2075)
[ 8048.321856] RSP: 0018:ffff8837bfcff8a0 EFLAGS: 00010246
[ 8048.323752] RAX: 000000000000d800 RBX: ffff8837b8314b00 RCX: 0000000000000000
[ 8048.325665] RDX: 00000000000000d8 RSI: 00000000000000fc RDI: ffff8837b8314ba8
[ 8048.327570] RBP: ffff8837bfcff8e0 R08: ffff8837df1e5040 R09: ffffed06f4b701b8
[ 8048.329486] R10: 000000002a82d01f R11: 1ffff106f82c0f77 R12: ffff8837a5b80d98
[ 8048.331414] R13: ffff8837c6c58b80 R14: ffff8837c6c58bd0 R15: 0000000000000000
[ 8048.333357] FS: 00007f238e593740(0000) GS:ffff8837df1c0000(0000) knlGS:0000000000000000
[ 8048.335329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8048.337304] CR2: 00007f8a8d2c4740 CR3: 00000037c7c10000 CR4: 00000000000407e0
[ 8048.339369] Stack:
[ 8048.341343] 0000000000000000 00000007fffffffe ffff883700000001 ffff8837b8314b00
[ 8048.343382] 00007fffffc00000 ffff8837c6c58b80 ffff8837c6c58bd0 0000000000000000
[ 8048.345421] ffff8837bfcff910 ffffffff815cfa60 ffff8837bfcff910 ffffffff81249cd3
[ 8048.347473] Call Trace:
[ 8048.349502] khugepaged_enter_vma_merge (include/linux/khugepaged.h:46 mm/huge_memory.c:2115)
[ 8048.351584] ? up_write (kernel/locking/rwsem.h:9 kernel/locking/rwsem.c:93)
[ 8048.353654] expand_downwards (mm/mmap.c:2278)
[ 8048.355719] ? __mem_cgroup_count_vm_event (mm/memcontrol.c:1156)
[ 8048.357791] handle_mm_fault (mm/memory.c:2673 mm/memory.c:3250 mm/memory.c:3371 mm/memory.c:3400)
[ 8048.359886] ? follow_page_pte (mm/gup.c:48)
[ 8048.361952] ? __pmd_alloc (mm/memory.c:3382)
[ 8048.364020] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:154 kernel/locking/spinlock.c:183)
[ 8048.366083] ? follow_page_pte (mm/gup.c:125)
[ 8048.368139] ? follow_page_mask (mm/gup.c:209)
[ 8048.370181] __get_user_pages (mm/gup.c:285 mm/gup.c:477)
[ 8048.372214] ? follow_page_mask (mm/gup.c:420)
[ 8048.374242] ? might_fault (./arch/x86/include/asm/current.h:14 mm/memory.c:3762)
[ 8048.376269] get_user_pages (mm/gup.c:818)
[ 8048.378295] copy_strings.isra.20 (fs/exec.c:197 fs/exec.c:510)
[ 8048.380392] ? count.isra.18.constprop.36 (fs/exec.c:454)
[ 8048.382439] ? copy_strings_kernel (fs/exec.c:556)
[ 8048.384464] do_execveat_common.isra.32 (fs/exec.c:1577)
[ 8048.386469] ? do_execveat_common.isra.32 (include/linux/spinlock.h:312 fs/exec.c:1263 fs/exec.c:1518)
[ 8048.388448] ? prepare_bprm_creds (fs/exec.c:1475)
[ 8048.390395] ? kmem_cache_alloc (include/trace/events/kmem.h:53 mm/slub.c:2524)
[ 8048.392309] ? getname_flags (fs/namei.c:135)
[ 8048.394187] ? up_read (./arch/x86/include/asm/rwsem.h:156 kernel/locking/rwsem.c:81)
[ 8048.396027] ? getname_flags (fs/namei.c:146)
[ 8048.397869] SyS_execve (fs/exec.c:1701)
[ 8048.399715] stub_execve (arch/x86/kernel/entry_64.S:510)
[ 8048.401482] ? system_call_fastpath (arch/x86/kernel/entry_64.S:261)
[ 8048.403207] Code: 1f 84 00 00 00 00 00 b8 f4 ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f b7 05 a9 fb db 01 0f b6 d4 31 d0 a8 fe 0f 85 3e fe ff ff <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 48 89 df e8 18 88 f6 ff 0f
All code
========
0: 1f (bad)
1: 84 00 test %al,(%rax)
3: 00 00 add %al,(%rax)
5: 00 00 add %al,(%rax)
7: b8 f4 ff ff ff mov $0xfffffff4,%eax
c: c3 retq
d: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
14: 00 00 00
17: 0f b7 05 a9 fb db 01 movzwl 0x1dbfba9(%rip),%eax # 0x1dbfbc7
1e: 0f b6 d4 movzbl %ah,%edx
21: 31 d0 xor %edx,%eax
23: a8 fe test $0xfe,%al
25: 0f 85 3e fe ff ff jne 0xfffffffffffffe69
2b:* 0f 0b ud2 <-- trapping instruction
2d: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
34: 00 00 00
37: 48 89 df mov %rbx,%rdi
3a: e8 18 88 f6 ff callq 0xfffffffffff68857
3f:
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
9: 00 00 00
c: 48 89 df mov %rbx,%rdi
f: e8 18 88 f6 ff callq 0xfffffffffff6882c
14:
[ 8048.406837] RIP __khugepaged_enter (mm/huge_memory.c:2059 mm/huge_memory.c:2075)
[ 8048.408525] RSP <ffff8837bfcff8a0>
Thanks,
Sasha
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: mm: hangs in collapse_huge_page
2015-04-30 22:17 ` Sasha Levin
@ 2015-04-30 22:24 ` Kirill A. Shutemov
0 siblings, 0 replies; 8+ messages in thread
From: Kirill A. Shutemov @ 2015-04-30 22:24 UTC (permalink / raw)
To: Sasha Levin; +Cc: Andrea Arcangeli, linux-mm, LKML, Dave Jones, Andrew Morton
On Thu, Apr 30, 2015 at 06:17:34PM -0400, Sasha Levin wrote:
> On 04/30/2014 11:42 AM, Kirill A. Shutemov wrote:
> > diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> > index b4b1feba6472..1c6ace5207b9 100644
> > --- a/mm/huge_memory.c
> > +++ b/mm/huge_memory.c
> > @@ -1986,6 +1986,8 @@ static void insert_to_mm_slots_hash(struct mm_struct *mm,
> >
> > static inline int khugepaged_test_exit(struct mm_struct *mm)
> > {
> > + VM_BUG_ON(!rwsem_is_locked(&mm->mmap_sem) &&
> > + !spin_is_locked(&khugepaged_mm_lock));
> > return atomic_read(&mm->mm_users) == 0;
> > }
>
> I've managed to hit this during testing:
>
> [ 8048.304275] kernel BUG at mm/huge_memory.c:2060!
> [ 8048.305878] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
> [ 8048.307479] Modules linked in: quota_v2 quota_tree xfs libcrc32c x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ast kvm ttm drm_kms_helper crct10dif_pclmul crc32_pclmul drm ghash_clmulni_intel aesni_
> intel aes_x86_64 lrw glue_helper ablk_helper cryptd joydev i2c_algo_bit sb_edac syscopyarea sysfillrect edac_core sysimgblt lpc_ich ipmi_si ipmi_msghandler ioatdma shpchp mac_hid btrfs xor mlx4_en vxlan raid6_pq
> hid_generic ixgbe mlx4_core usbhid hid dca megaraid_sas ahci ptp libahci pps_core mdio
> [ 8048.314422] CPU: 31 PID: 13065 Comm: thp01 Not tainted 4.1.0-rc1-next-20150430+ #8
> [ 8048.316215] Hardware name: Oracle Corporation OVCA X3-2 /ASSY,MOTHERBOARD,1U , BIOS 17021300 06/19/2012
> [ 8048.318070] task: ffff8837ba9b3b40 ti: ffff8837bfcf8000 task.ti: ffff8837bfcf8000
> [ 8048.319941] RIP: __khugepaged_enter (mm/huge_memory.c:2059 mm/huge_memory.c:2075)
> [ 8048.321856] RSP: 0018:ffff8837bfcff8a0 EFLAGS: 00010246
> [ 8048.323752] RAX: 000000000000d800 RBX: ffff8837b8314b00 RCX: 0000000000000000
> [ 8048.325665] RDX: 00000000000000d8 RSI: 00000000000000fc RDI: ffff8837b8314ba8
> [ 8048.327570] RBP: ffff8837bfcff8e0 R08: ffff8837df1e5040 R09: ffffed06f4b701b8
> [ 8048.329486] R10: 000000002a82d01f R11: 1ffff106f82c0f77 R12: ffff8837a5b80d98
> [ 8048.331414] R13: ffff8837c6c58b80 R14: ffff8837c6c58bd0 R15: 0000000000000000
> [ 8048.333357] FS: 00007f238e593740(0000) GS:ffff8837df1c0000(0000) knlGS:0000000000000000
> [ 8048.335329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 8048.337304] CR2: 00007f8a8d2c4740 CR3: 00000037c7c10000 CR4: 00000000000407e0
> [ 8048.339369] Stack:
> [ 8048.341343] 0000000000000000 00000007fffffffe ffff883700000001 ffff8837b8314b00
> [ 8048.343382] 00007fffffc00000 ffff8837c6c58b80 ffff8837c6c58bd0 0000000000000000
> [ 8048.345421] ffff8837bfcff910 ffffffff815cfa60 ffff8837bfcff910 ffffffff81249cd3
> [ 8048.347473] Call Trace:
> [ 8048.349502] khugepaged_enter_vma_merge (include/linux/khugepaged.h:46 mm/huge_memory.c:2115)
> [ 8048.351584] ? up_write (kernel/locking/rwsem.h:9 kernel/locking/rwsem.c:93)
> [ 8048.353654] expand_downwards (mm/mmap.c:2278)
> [ 8048.355719] ? __mem_cgroup_count_vm_event (mm/memcontrol.c:1156)
> [ 8048.357791] handle_mm_fault (mm/memory.c:2673 mm/memory.c:3250 mm/memory.c:3371 mm/memory.c:3400)
> [ 8048.359886] ? follow_page_pte (mm/gup.c:48)
> [ 8048.361952] ? __pmd_alloc (mm/memory.c:3382)
> [ 8048.364020] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:154 kernel/locking/spinlock.c:183)
> [ 8048.366083] ? follow_page_pte (mm/gup.c:125)
> [ 8048.368139] ? follow_page_mask (mm/gup.c:209)
> [ 8048.370181] __get_user_pages (mm/gup.c:285 mm/gup.c:477)
> [ 8048.372214] ? follow_page_mask (mm/gup.c:420)
> [ 8048.374242] ? might_fault (./arch/x86/include/asm/current.h:14 mm/memory.c:3762)
> [ 8048.376269] get_user_pages (mm/gup.c:818)
We call get_user_pages() here without ->mmap_sem taken. It violates
get_user_pages() interface but should not cause a problem because we don't
have concurency for the mm yet -- it's exec path.
Not sure if we should correct it.
Hm. __bprm_mm_init() in the same exec path takes ->mmap_sem.
Any comments?
> [ 8048.378295] copy_strings.isra.20 (fs/exec.c:197 fs/exec.c:510)
> [ 8048.380392] ? count.isra.18.constprop.36 (fs/exec.c:454)
> [ 8048.382439] ? copy_strings_kernel (fs/exec.c:556)
> [ 8048.384464] do_execveat_common.isra.32 (fs/exec.c:1577)
> [ 8048.386469] ? do_execveat_common.isra.32 (include/linux/spinlock.h:312 fs/exec.c:1263 fs/exec.c:1518)
> [ 8048.388448] ? prepare_bprm_creds (fs/exec.c:1475)
> [ 8048.390395] ? kmem_cache_alloc (include/trace/events/kmem.h:53 mm/slub.c:2524)
> [ 8048.392309] ? getname_flags (fs/namei.c:135)
> [ 8048.394187] ? up_read (./arch/x86/include/asm/rwsem.h:156 kernel/locking/rwsem.c:81)
> [ 8048.396027] ? getname_flags (fs/namei.c:146)
> [ 8048.397869] SyS_execve (fs/exec.c:1701)
> [ 8048.399715] stub_execve (arch/x86/kernel/entry_64.S:510)
> [ 8048.401482] ? system_call_fastpath (arch/x86/kernel/entry_64.S:261)
> [ 8048.403207] Code: 1f 84 00 00 00 00 00 b8 f4 ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f b7 05 a9 fb db 01 0f b6 d4 31 d0 a8 fe 0f 85 3e fe ff ff <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 48 89 df e8 18 88 f6 ff 0f
> All code
> ========
> 0: 1f (bad)
> 1: 84 00 test %al,(%rax)
> 3: 00 00 add %al,(%rax)
> 5: 00 00 add %al,(%rax)
> 7: b8 f4 ff ff ff mov $0xfffffff4,%eax
> c: c3 retq
> d: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
> 14: 00 00 00
> 17: 0f b7 05 a9 fb db 01 movzwl 0x1dbfba9(%rip),%eax # 0x1dbfbc7
> 1e: 0f b6 d4 movzbl %ah,%edx
> 21: 31 d0 xor %edx,%eax
> 23: a8 fe test $0xfe,%al
> 25: 0f 85 3e fe ff ff jne 0xfffffffffffffe69
> 2b:* 0f 0b ud2 <-- trapping instruction
> 2d: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
> 34: 00 00 00
> 37: 48 89 df mov %rbx,%rdi
> 3a: e8 18 88 f6 ff callq 0xfffffffffff68857
> 3f:
>
> Code starting with the faulting instruction
> ===========================================
> 0: 0f 0b ud2
> 2: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
> 9: 00 00 00
> c: 48 89 df mov %rbx,%rdi
> f: e8 18 88 f6 ff callq 0xfffffffffff6882c
> 14:
> [ 8048.406837] RIP __khugepaged_enter (mm/huge_memory.c:2059 mm/huge_memory.c:2075)
> [ 8048.408525] RSP <ffff8837bfcff8a0>
>
>
> Thanks,
> Sasha
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-04-30 22:24 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-04-16 2:06 mm: hangs in collapse_huge_page Sasha Levin
2014-04-24 16:46 ` Sasha Levin
2014-04-30 15:42 ` Kirill A. Shutemov
2014-05-01 14:38 ` Hillf Danton
2014-05-11 0:34 ` Sasha Levin
2014-05-14 21:29 ` Kirill A. Shutemov
2015-04-30 22:17 ` Sasha Levin
2015-04-30 22:24 ` Kirill A. Shutemov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).